An end-to-end noisy speech separation method based on channel attention mechanism and transformer

Abstract

The application discloses an end-to-end noisy speech separation method based on a channel attention mechanism and a transformer, comprising: a time sequence perception context channel attention layer is constructed, and features are effectively filtered and screened from the channel dimension of the features; meanwhile, due to the existence of time sequence perception and context perception characteristics, the rationality of channel weight distribution under noise is further improved; secondly, in order to enhance the global expression and modeling ability of the speech potential features output by the encoder, a bidirectional LSTM-Transformer encoder layer is proposed, wherein a feedforward layer with a bidirectional LSTM further enriches the global context information in the multi-head attention features, and the effectiveness of the encoder in feature coding is improved. The application realizes the improvement of the performance of the speech separation system under complex noise, and shows improvement in various separation test indicators. In addition, the reduction of model complexity and the improvement of effectiveness brought by reasonable feature screening make the application suitable for most application scenarios related to human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the field of speech separation technology, specifically relating to an end-to-end noisy speech separation method based on channel attention mechanism and Transformer. Background Technology

[0002] Speech separation is a classic problem stemming from the "cocktail party problem," where discerning and understanding each speaker's message becomes challenging amidst overlapping speech and complex background noise. Speech separation technology is frequently applied in various everyday scenarios (such as conference transcripts and multimedia processing), and most of these scenarios involve some degree of background noise. The development of speech technology aims not only to improve the quality of speaker speech extracted by speech separation systems but also, and perhaps more importantly, to significantly and effectively reduce background noise interference, thereby enhancing the robustness of speech separation systems to background noise. This is of paramount importance in many real-world applications.

[0003] The goal of speech separation is not only to isolate the speech of a single speaker from a noisy mixed signal, but also to identify and segment individual speaker speech fragments for use as input for other important downstream tasks. In this process, the speech separation system, as an upstream task, can serve as a preprocessing module for intelligent devices, ultimately adapting to the specific needs of the user's application scenario for corresponding downstream tasks (such as automatic speech recognition and speech emotion analysis). Deep learning technology plays an increasingly important role in this process. Applications of deep learning technology cover image processing, natural language processing, signal processing, and other related fields. It represents a deeper learning and adaptation of human brain thinking, and in some areas, its performance can even rival human intelligence. Especially with the rapid development and advancement of artificial intelligence technology and high-performance hardware devices (such as CPUs and GPUs), some deep learning-based speech separation technologies are no longer limited by inherent hardware and software characteristics, and many speech separation algorithms have been deployed in practical applications. Deep learning technology can effectively identify the speech patterns of speakers in noisy mixed signals (including those between the speaker and background noise, and between different speakers), significantly improving and enhancing the understanding of human and intelligent machine speech.

[0004] Many current deep learning-based speech separation techniques primarily involve analysis from the time domain or a joint time-frequency domain. While the latter can fully utilize the complementarity of time-frequency information in speech signals, many algorithms involve high computational and parameter counts, which poses a constraint for resource-constrained scenarios. Furthermore, the short-time Fourier transform technique used to transform time-domain speech signals to the time-frequency domain suffers from the limitation of having the same receptive field (window size) for different frequency components in the speech signal, which to some extent restricts the performance improvement of time-frequency domain-based speech separation systems. Currently, mainstream speech separation models can be viewed as an encoder + separation module + decoder structure (this structure originates from the Conv-TasNet model). The encoder obtains the latent feature representation in the noisy input speech signal. The separator then further learns and captures the speaker's speech patterns and reduces background noise interference from the feature space to obtain a mask for the estimated signal. Finally, the decoder maps the speaker's speech obtained by the separation module from the latent feature space to the real speech signal space.

[0005] The end-to-end architecture based on Conv-TasNet has not only found numerous applications in speech separation but has also profoundly impacted model architectures for other speech tasks (such as speech enhancement) and other fields. In the Conv-TasNet network, the separation module captures and represents the dynamic temporal changes in the speech waveform by stacking multiple one-dimensional convolutional layers, thereby modeling the speaker's speech patterns. However, the multi-layered convolutional operations result in redundancy in the feature mask information learned by the separation module, which to some extent reduces the effectiveness of the speech separation system. Therefore, it is necessary to reasonably filter and select the features output by the separation module. Furthermore, the Conv-TasNet encoder and separation module lack full utilization of the global contextual dependencies of speech, because in some cases, background noise based on contextual information may potentially improve the performance of noisy speech separation systems. Both local and global characteristics of speech are crucial for speech separation systems to separate the speech of different speakers and reduce background noise interference. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings and deficiencies of existing technologies by proposing an end-to-end noisy speech separation method based on channel attention and Transformer. This method proposes performing channel attention from the channel dimension of features, applying it to both the encoder and the separator. Furthermore, the method introduces additional temporal awareness and contextual information before the channel attention makes its decisions. Temporal awareness helps enhance the temporal characteristics of the features, while the presence of contextual information effectively improves the rationality of channel weight allocation in the channel attention mechanism, enabling effective filtering and selection of speech features. Secondly, this invention proposes a bidirectional LSTM-Transformer coding layer to enhance the global representation and modeling capabilities of the encoder's output features, further improving the effectiveness of feature encoding by the encoder.

[0007] The technical solution adopted by this invention to solve its technical problem is: an end-to-end noisy speech separation method based on the attention mechanism and Transformer, the method comprising the following steps:

[0008] Step 1: Input noisy mixed signal S mix It is composed of clean speech signals S1 and S2 from two speakers and ambient background noise n superimposed. First, the input mixed signal is preprocessed through a convolutional layer 0 and four cascaded, parameter-shared convolutional layers 1. Then, a temporally aware contextual channel attention layer and a bidirectional LSTM-Transformer encoder layer are used to perform feature filtering and impart global contextual information to the preprocessed result, so as to obtain a latent feature representation F of the speech signal that is effective and has a certain contextual modeling capability. Encoder ;

[0009] Step 2: The three cascaded steps of the separator basic module, consisting of multiple cascaded 1D convolutional layers and temporally aware contextual channel attention layers, progressively process the feature F. Encoder The process generates a speech separation mask, and finally combines the mask with the latent feature representation F of the mixed signal output by the encoder. Encoder Element-wise multiplication is performed to separate the latent spatial features F corresponding to each speaker's speech and background noise in the mixed signal. Separator ;

[0010] Step 3: Characteristic representation F of the separator output Separator Finally, the decoder, consisting of transposed convolutional layers corresponding to the four cascaded convolutional layers 1 and one convolutional layer 0 at the encoder's front end, completes the mapping from the latent feature space to the real temporal speech signal space and the background noise signal space. The decoder ultimately outputs three estimated temporal signals, corresponding to the speech signals of the two speakers respectively. and background noise signal

[0011] Step 4: Constrain and optimize the performance of the speech separation system using the overall loss function of speech separation;

[0012] Step 5: Evaluate the performance of the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer.

[0013] Furthermore, the specific steps of step 1 are as follows:

[0014] Step 1-1: The temporal noisy mixed speech signal is passed through convolution layer 0, and then through a 1D convolution with a kernel size of L / 2 = 16 and a stride of S = L / 2, to obtain approximately stationary speech segments s that overlap with each other. mix-k The number of output channels is C0 = 512, and PReLU is used to obtain the non-linear expressive power of the features;

[0015] Step 1-2: Segment the speech signal s mix-k Furthermore, by using four cascaded, parameter-shared convolutional layers, the model training parameters are reduced while the expressive power of speech features is further improved.

[0016] Steps 1-3: The speech nonlinear feature map output after passing through one convolutional layer 0 and four convolutional layers 1 is passed through a temporally aware contextual channel attention layer to filter out redundant features in its output channel dimension and improve the effectiveness of separating useful features.

[0017] Steps 1-4: After performing attention-based feature filtering on the channel dimension, the output enhances the speech separation system's ability to globally model speech signals. Finally, through a bidirectional LSTM-Transformer encoder layer, an effective latent feature representation F of the speech signal with certain global contextual information is obtained. Encoder .

[0018] Furthermore, in steps 1-2, the specific method for implementing four cascaded, parameter-sharing convolutional layers includes the following steps:

[0019] Step 1-2-1: A single convolutional layer consists of a 1D convolution with dilated convolution and a non-linear activation function PReLU, which improves the non-linear expressive power of speech feature mapping. The number of input channels, number of output channels, kernel size, kernel stride, dilated convolution dilation rate, and number of padding elements in the 1D convolution are 512, 512, 3, 1, 1, and 1, respectively.

[0020] Step 1-2-2: Concatenate the four convolutional layers and share parameters among the four layers.

[0021] Furthermore, in steps 1-3, the specific method for implementing the time-aware contextual channel attention layer includes the following steps:

[0022] Step 1-3-1: Add time-aware characteristics to the output of step 1-2;

[0023] Step 1-3-2: Introduce contextual relationships into the output of Step 1-3-1 in order to provide reasonable channel weight allocation for the subsequent channel attention module;

[0024] Step 1-3-3: Perform feature weighting and filtering on the channel dimension for speech features with temporal-aware context.

[0025] Furthermore, in step 1-3-1, the specific method for implementing speech feature representation with time-aware characteristics includes the following steps:

[0026] Step 1-3-1-1: Average pool the last dimension of the input features to 1, retaining only the time dimension, thus fully learning the dynamic changes of the input signal in the time domain. This operation can be defined as:

[0027]

[0028] Where N is the number of dimensions in the last dimension of the input feature, x t and y t Let f represent the features at the t-th time point, respectively. gap Represents global average pooling;

[0029] Step 1-3-1-2: Perform channel transformation on the result that retains only the time dimension through a 1×1 convolution;

[0030] Step 1-3-1-3: ReLU activation to provide non-linear expressive power;

[0031] Steps 1-3-1-4: The 1×1 convolution learns the temporal characteristics of the speech again and calculates the weights through the Sigmoid function as a temporal awareness of the input features;

[0032] Step 1-3-1-5: Multiply the learned time-aware weights of the input features element-wise with the input features to add global time-aware characteristics to the time-domain signal.

[0033] Furthermore, in step 1-3-2, the specific method for implementing time-aware contextual speech features includes the following steps:

[0034] Step 1-3-2-1: Increase the number of channels by performing a 1×1 convolution on the two-dimensional output of step 1-3-1-5;

[0035] Step 1-3-2-2: Merge the last two dimensions of the output from the previous step;

[0036] Step 1-3-2-3: A 1×1 convolution is used to compress the channel dimension of the two-dimensional output result of Step 1-3-2-2 and to calculate the context features;

[0037] Steps 1-3-2-4: Apply the Softmax function to calculate the mask of the context features;

[0038] Step 1-3-2-5: Perform element-wise multiplication of the context features and the context mask to obtain speech features with temporal awareness context.

[0039] Furthermore, in step 1-3-3, the specific method for assigning and filtering speech features with temporally aware context at the channel dimension includes the following steps:

[0040] Step 1-3-3-1: A 1×1 convolution transforms the number of channels of the output result of step 1-3-2-5;

[0041] Step 1-3-3-2: Layer normalization enhances the stability of speech features and model training;

[0042] Step 1-3-3-3: Apply 1×1 convolution again to perform a channel number transformation;

[0043] Step 1-3-3-4: The Sigmoid function estimates the attention score (i.e., the weight assigned to each channel) of the output of Step 1-3-1-5. Since the acquisition of the attention score involves global contextual information of the speech, this makes the allocation of channel weights more reasonable and effective.

[0044] Step 1-3-3-5: Multiply the output of Step 1-3-1-5 with its corresponding channel attention score in the channel dimension to achieve effective filtering of temporal-aware contextual speech features in the channel dimension.

[0045] Furthermore, the specific method for implementing the bidirectional LSTM-Transformer encoder layer in steps 1-4 includes the following steps:

[0046] Step 1-4-1: Enhance the long-distance temporal dependencies of speech in the output of Step 1-3-3-5 through the multi-head attention mechanism of the Transformer encoder layer;

[0047] Step 1-4-2: Add the temporally aware contextual speech features output in Step 1-3-3-5 to the result in Step 1-4-1 and apply layer normalization. This smooths the output and helps ensure the stability of model training and prevent gradient explosion and gradient vanishing.

[0048] Step 1-4-3: Use a feedforward layer with a globally aware bidirectional LSTM to enhance the global modeling and expressive power of speech features;

[0049] Step 1-4-4: Add the outputs of Step 1-4-2 and Step 1-4-3 and apply layer normalization. Here, the final speech latent feature representation F of the encoder is obtained. Encoder This is for the separator to learn further.

[0050] Furthermore, in step 1-4-1, the specific method for implementing the multi-head attention mechanism includes the following steps:

[0051] Step 1-4-1-1: Define the number of heads h in the multi-head attention mechanism. Different heads are responsible for focusing on different features and information angles of the input features and capturing the global dependencies in the speech temporal features, thereby improving the model's ability to understand and model the contextual relationships of speech signals.

[0052] Step 1-4-1-2: Each head internally calculates K (Key), Q (Query), and V (Value) for the input features. Different heads have different weight matrices for calculating K, Q, and V, and their parameters are learnable.

[0053] Step 1-4-1-3: Softmax calculates the attention score for each head, and multiplies the attention score by V to obtain the feature representation of the input features with respect to attention;

[0054] Step 1-4-1-4: Concatenate the outputs of the multi-head system and multiply them with a learnable weight matrix of one parameter to form a multi-head attention feature representation of the input features.

[0055] Furthermore, in step 1-4-3, the specific method for implementing the feedforward layer with a globally aware bidirectional LSTM includes the following steps:

[0056] Step 1-4-3-1: Copy the output of Step 1-4-2, and reverse one of the copies according to the hidden time dimension, that is, in reverse time order;

[0057] Step 1-4-3-2: Feed the forward and reverse order speech features of the hidden time dimension into the LSTM network respectively, and concatenate the two outputs to further enhance the global contextual representation and modeling ability of the multi-channel speech attention features output in Step 1-4-1;

[0058] Step 1-4-3-3: ReLU activation and a linear layer further enhance the non-linear expressive power of the features.

[0059] Furthermore, in step 2, the specific steps for implementing the separator include:

[0060] Step 2-1: Apply layer normalization to the output of Step 1 to improve the stability of model training, and then use 1×1 convolution to transform the number of channels;

[0061] Step 2-2: Construct the basic module of the separator by cascaded 1D convolutional layers and the temporal-aware contextual channel attention layer constructed in Step 1-3;

[0062] Step 2-3: Cascade the basic separator modules obtained in Step 2-2 three times;

[0063] Steps 2-4: Perform feature aggregation and nonlinear mapping on the relevant outputs of the three cascaded basic separator modules to obtain the speech separation mask;

[0064] Steps 2-5: Combine the speech separation mask with the encoder output feature F Encoder Multiplying them yields the latent feature estimates F corresponding to the speech signals of the two speakers and the background noise signal. Separator .

[0065] Furthermore, in step 2-2, the specific method for implementing the basic module of the separator includes the following steps:

[0066] Step 2-2-1: Construct a 1D convolutional layer;

[0067] Step 2-2-2: Adjust the 1D convolutional layer constructed in step 2-2-1 according to the dilation factor 2. m (where m = 0, 1, 2, 3, 4, 5, 6, 7) are cascaded in sequence, that is, 1-dimensional convolutional layers are stacked eight times;

[0068] Step 2-2-3: Next, the output of Step 2-2-2 is fed into the temporal-aware contextual channel attention layer constructed in Step 1-3. Reasonable feature selection is performed in the channel dimension and certain contextual information is given to the features, thereby completing the construction of the basic module of the separator.

[0069] Furthermore, in step 2-2-1, the specific method for implementing the 1D convolutional layer includes the following steps:

[0070] Step 2-2-1-1: Perform channel transformation on the input using a 1×1 convolution to accommodate computation in subsequent layers;

[0071] Step 2-2-1-2: PReLU activation and layer normalization are used to enhance the nonlinearity of features and the stability of model training, respectively;

[0072] Step 2-2-1-3: Feed the output of step 2-2-1-2 into a depthwise convolution to reduce the amount of computation and improve the efficiency of the convolution kernel parameters.

[0073] Step 2-2-1-4: PReLU activation and layer normalization;

[0074] Step 2-2-1-5: Apply two parallel 1×1 convolutions to the output of Step 2-2-1-4 to perform a transformation with different numbers of channels. One of the convolution outputs is added element-wise with the output of Step 2-2-1-1 to finally obtain the two outputs of the 1D convolutional layer.

[0075] Furthermore, in steps 2-4, the specific method for performing feature aggregation and nonlinear mapping on the relevant outputs of the three cascaded basic separator modules to obtain the final speech separation mask includes the following steps:

[0076] Step 2-4-1: Add the outputs of the 1D convolutional layers and the temporal-aware contextual channel attention layers in the three cascaded separator basic modules to obtain the overall feature aggregation representation F of the separator cascaded modules. sum ;

[0077] Step 2-4-2: PReLU aggregates features F sum Perform nonlinear mapping;

[0078] Step 2-4-3: Adjust the number of channels in the 1×1 convolution to accommodate subsequent activation layers;

[0079] Step 2-4-4: Sigmoid calculates the mixture feature F at the encoder output. Encoder The mask corresponding to the speech of the two speakers and the background noise.

[0080] Furthermore, step 3 specifically includes the following steps:

[0081] Step 3-1: The separation feature F output in step 2... Separator After four cascaded transposed convolutional layers, the kernel size, stride, and dilation factor correspond to the four cascaded convolutional layers at the front end of the encoder, and the parameters are not shared. The feature representation of the time-domain waveform of the real speech signal and background noise signal is learned step by step.

[0082] Step 3-2: Finally, after passing through the transposed convolutional layer corresponding to the frontmost convolutional layer 0 of the encoder, the two speaker speech time-domain signals and the background noise time-domain signal are obtained.

[0083] Furthermore, step 4 specifically includes the following steps:

[0084] Step 4-1: Define the overall loss function for speech separation.

[0085] Loss total =Loss SI-SNR +αLoss MR ,

[0086] Loss SI-SNR =-SI-SNR represents the scale-invariant signal-to-noise ratio loss function (SI-SNR). The input is the separated signal and the corresponding target signal. It is used to reduce the difference between the speech estimation and background noise estimation of the two speakers and the actual speaker speech and background noise. Loss MR The corresponding multi-resolution (MR) loss aims to improve the quality of speech separation and more effectively remove noise from the speaker's speech by considering both time-domain and frequency-domain information. α is an adjustable hyperparameter used to adjust the contribution and influence of the multi-resolution loss on the performance of the speech separation system.

[0087] Step 4-2: Apply a permutation-invariant training strategy to reduce the dependence of the model's separation performance on the speaker order, enhance the model's generalization ability, and gradually reduce the overall loss of the speech separation algorithm through gradient descent.

[0088] Furthermore, in step 5, the specific method for performance evaluation of the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer includes the following steps:

[0089] Step 5-1: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with existing temporal speech separation models to verify its overall performance;

[0090] Step 5-2: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on Transformer to verify the effectiveness of the time-aware context channel attention layer;

[0091] Step 5-3: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with another end-to-end noisy speech separation method based on channel attention mechanism to verify the effectiveness of the bidirectional LSTM-Transformer encoder layer.

[0092] Beneficial effects:

[0093] 1. In order to filter out features useful for separation, this invention proposes to perform channel attention from the channel dimension of the features, which is applied to the encoder and the separator respectively. Furthermore, the method of this invention introduces additional temporal awareness and contextual information before the channel attention completes the decision. Temporal awareness helps to enhance the speech temporal characteristics of the features, and the presence of contextual information effectively improves the rationality of channel weight allocation in the channel attention mechanism, thereby achieving effective filtering and selection of speech features.

[0094] 2. In order to improve the global representation and modeling ability of the latent speech features obtained by the encoder in the speech separation system, this invention constructs a Transformer encoder layer based on the Transformer encoder layer and a feedforward layer with bidirectional LSTM. While improving the effectiveness of encoder feature encoding, it also enhances the system's global modeling ability of speech temporal sequence, thereby further enhancing the performance of the system in performing speech separation under noisy conditions.

[0095] 3. This invention also estimates the noise signal, and the estimated noise signal indirectly reflects the performance of the speech separation system, especially in removing noise interference from the speaker's speech. Furthermore, the estimated noise helps the model learn the differences and connections between the speaker's speech and noise from the mixed feature representation, improving the model's robustness to noise, as noise can sometimes contribute to the speech separation effect. Attached Figure Description

[0096] Figure 1 This is a schematic diagram of the overall speech separation process of the present invention.

[0097] Figure 2 This is a framework diagram of the temporal awareness context channel attention layer of the present invention.

[0098] Figure 3 This is a diagram of the Transformer encoder layer framework upon which this invention is based.

[0099] Figure 4 This is a diagram of the feedforward layer framework with bidirectional LSTM proposed in this invention. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be noted that, to avoid obscuring the invention with unnecessary details, only structures and / or processing steps closely related to the invention are shown in the drawings, while other details not directly related to the invention are omitted.

[0101] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0102] like Figures 1 to 4 As shown, this invention proposes an end-to-end noisy speech separation method based on channel attention mechanism and Transformer. The temporal-aware contextual channel attention layer further enhances the effectiveness of output features in the encoder and separator by performing reasonable feature selection and filtering along the channel dimension. Furthermore, the constructed bidirectional LSTM-Transformer further improves the global representation and modeling capability of the encoder output features and enriches the contextual information in the latent speech features. These two designs effectively improve the utilization rate of useful features and global contextual information in the speech separation system, improving the performance of the noisy speech separation system. Specifically, the method includes the following steps:

[0103] Step 1: Input noisy mixed signal S mix It is composed of the clean speech signals S1 and S2 from two speakers and the ambient background noise n superimposed. First, the input noisy mixed signal is preprocessed by a convolutional layer 0 and four cascaded convolutional layers 1 with shared parameters. Then, a temporally aware contextual channel attention layer and a bidirectional LSTM-Transformer encoder layer are used to filter features from the preprocessed result and impart global contextual information to obtain a latent feature representation F of the speech signal that is effective and has a certain ability to model context. Encoder ;

[0104] Step 2: The three cascaded steps of the separator basic module, consisting of multiple cascaded 1D convolutional layers and temporally aware contextual channel attention layers, progressively process the feature F. Encoder The speech separation mask is generated, and finally the mask is combined with the latent feature representation F of the mixed signal output by the encoder. Encoder Element-wise multiplication is performed to separate the latent spatial features F corresponding to each speaker's speech and background noise in the mixed signal. Separator ;

[0105] Step 3: Characteristic representation F of the separator output Separator Finally, the decoder, consisting of transposed convolutional layers corresponding to the four cascaded convolutional layers 1 and one convolutional layer 0 at the encoder's front end, completes the mapping from the latent feature space to the real temporal speech signal space and the background noise signal space. The decoder ultimately outputs three estimated temporal signals, corresponding to the speech signals of the two speakers respectively. and background noise signal

[0106] Step 4: Constrain and optimize the performance of the speech separation system using the overall loss function of speech separation;

[0107] Step 5: Evaluate the performance of the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer.

[0108] This invention uses two noisy speech datasets: the WHAM! dataset and the Libri2Mix dataset. For the WHAM! dataset: First, speech recordings from two speakers were randomly selected from the WSJ0 corpus. Then, the two speakers' voices were mixed using a signal-to-noise ratio randomly selected from -5dB to 5dB, thus constructing a clean speech training set consisting of 49 male speakers and 51 female speakers, with each mixed speech recording containing two speakers, totaling 30 hours. Additionally, a 10-hour validation set and a 5-hour test set were obtained from speeches by 16 other speakers from WSJ0 (different from the training set). Finally, after downsampling at 8kHz, a training set of 20,000 voice recordings, a validation set of 5,000 voice recordings, and a test set of 3,000 voice recordings were obtained, with a precision of 16 bits. The audio noise data in WHAM! comes from a large amount of sampling data from different urban environments in the San Francisco Bay Area at the end of 2018. These sampling areas mainly include restaurants, coffee shops, bars, and parks. For the Libri2Mix dataset: it is a noisy mixed speech dataset generated by mixing the clean speech set of LibriSpeech and the noisy set of WHAM, with each speech segment containing only two random speakers. The number of speech segments used for model training, validation, and testing are 13,900, 3,000, and 3,000, respectively. For both the WHAM! and Libri2Mix datasets, the noisy mixed speech signal is first passed sequentially through the first convolutional layer 0 and four cascaded convolutional layers 1 of the encoder to obtain preliminary feature representations. Subsequent layers of the encoder then perform in-depth and refined processing of these features.

[0109] The subsequent layers of step 1 in this invention specifically include:

[0110] Step 1-1: The temporal noisy mixed speech signal is passed through convolution layer 0, and then through a 1D convolution with a kernel size of L / 2 = 16 and a stride of S = L / 2, to obtain approximately stationary speech segments s that overlap with each other. mix-k The number of output channels is C0 = 512, and PReLU is used to obtain the non-linear expressive power of the features;

[0111] Step 1-2: Segment the speech signal s mix-kFurthermore, by using four cascaded, parameter-shared convolutional layers, the model training parameters are reduced while the expressive power of speech features is further improved.

[0112] Steps 1-3: The speech nonlinear feature map output after passing through one convolutional layer 0 and four convolutional layers 1 is passed through a temporally aware contextual channel attention layer to filter out redundant features in its output channel dimension and improve the effectiveness of separating useful features.

[0113] Steps 1-4: The output of the attention-based feature filtering at the channel dimension enhances the speech separation system's ability to model the global speech signal. This step is mainly achieved through a bidirectional LSTM-Transformer encoder layer, ultimately yielding an effective latent feature representation F of the speech signal with certain global contextual information. Encoder .

[0114] In steps 1-2 of the present invention, the specific method for implementing four cascaded, parameter-sharing convolutional layers includes the following steps:

[0115] Step 1-2-1: A single convolutional layer consists of a 1D convolution with dilated convolution and a non-linear activation function PReLU, which improves the non-linear expressive power of speech feature mapping. The number of input channels, number of output channels, kernel size, kernel stride, dilated convolution dilation rate, and number of padding elements in the 1D convolution are 512, 512, 3, 1, 1, and 1, respectively.

[0116] Step 1-2-2: Concatenate the four convolutional layers and share parameters among the four layers.

[0117] In steps 1-3 of the present invention, the specific method for implementing the time-aware contextual channel attention layer includes the following steps:

[0118] Step 1-3-1: Add time-aware characteristics to the output of Step 1-2 above;

[0119] Step 1-3-2: Introduce contextual relationships into the output of Step 1-3-1 above, so as to provide reasonable channel weight allocation for the subsequent channel attention module;

[0120] Step 1-3-3: Perform feature weighting and filtering on the channel dimension for speech features with temporal-aware context.

[0121] In step 1-3-1 of the present invention, the specific method for realizing speech feature representation with time-aware characteristics includes the following steps: Step 1-3-1-1: The last dimension of the input feature is averaged and pooled to 1, retaining only the time dimension, thereby fully learning the dynamic changes of the input signal in the time domain. This operation can be defined as:

[0122]

[0123] Where N is the number of dimensions in the last dimension of the input feature, x t and y t Let f represent the features at the t-th time point, respectively. gap Represents global average pooling;

[0124] Step 1-3-1-2: Perform channel transformation on the result that retains only the time dimension through a 1×1 convolution;

[0125] Step 1-3-1-3: ReLU activation to provide non-linear expressive power;

[0126] Steps 1-3-1-4: The 1×1 convolution learns the temporal characteristics of the speech again and calculates the weights through the Sigmoid function as a temporal awareness of the input features;

[0127] Step 1-3-1-5: Multiply the learned time-aware weights of the input features element-wise with the input features to add global time-aware characteristics to the time-domain signal.

[0128] In steps 1-3-2 of the present invention, the specific method for implementing time-aware contextual speech features includes the following steps:

[0129] Step 1-3-2-1: Increase the number of channels by performing a 1×1 convolution on the two-dimensional output of step 1-3-1-5 above;

[0130] Step 1-3-2-2: Merge the last two dimensions of the output from the previous step;

[0131] Step 1-3-2-3: A 1×1 convolution is used to compress the channel dimension of the two-dimensional output result from Step 1-3-2-2 above, and the context features are calculated.

[0132] Steps 1-3-2-4: Apply the Softmax function to calculate the mask of the context features;

[0133] Step 1-3-2-5: Perform element-wise multiplication of the context features and the context mask to obtain speech features with temporal awareness context.

[0134] In step 1-3-3 of the present invention, the specific method for performing feature weighting and filtering of speech features with temporal-aware context in the channel dimension includes the following steps: Step 1-3-3-1: 1×1 convolution transforms the number of channels of the output result of step 1-3-2-5 above;

[0135] Step 1-3-3-2: Layer normalization enhances the stability of speech features and model training;

[0136] Step 1-3-3-3: Apply 1×1 convolution again to perform a channel number transformation;

[0137] Step 1-3-3-4: The Sigmoid function estimates the attention score (i.e. the weight assigned to each channel) of the output of Step 1-3-1-5 above. Since the acquisition of the attention score involves global contextual information of the speech, this makes the allocation of channel weights more reasonable and effective.

[0138] Step 1-3-3-5: Multiply the output of Step 1-3-1-5 above with its corresponding channel attention score in the channel dimension to achieve effective filtering of temporal-aware contextual speech features in the channel dimension.

[0139] In steps 1-4 of the present invention, the specific method for implementing the bidirectional LSTM-Transformer encoder layer includes the following steps:

[0140] Step 1-4-1: Enhance the long-distance temporal dependencies of speech in the output of Step 1-3-3-5 by using the multi-head attention mechanism of the Transformer encoder layer;

[0141] Step 1-4-2: Add the temporally aware contextual speech features output from Step 1-3-3-5 to the result of Step 1-4-1 and apply layer normalization. This smooths the output and helps ensure the stability of model training and prevent gradient explosion and gradient vanishing.

[0142] Step 1-4-3: Use a feedforward layer with a globally aware bidirectional LSTM to enhance the global modeling and expressive power of speech features;

[0143] Step 1-4-4: Add the outputs of Step 1-4-2 and Step 1-4-3 above and apply layer normalization. Here, the speech latent feature representation of the encoder in the present invention is obtained for the separator to learn further.

[0144] In step 1-4-1 of the present invention, the specific method for implementing the multi-head attention mechanism includes the following steps: Step 1-4-1-1: Define the number of heads h in the multi-head attention mechanism. Different heads are responsible for focusing on different features and information angles of the input features, and capturing the global dependencies in the speech temporal features to improve the model's ability to understand and model the contextual relationships of the speech signal.

[0145] Step 1-4-1-2: Each head internally calculates K (Key), Q (Query), and V (Value) for the input features. Different heads have different weight matrices for calculating K, Q, and V, and their parameters are learnable.

[0146] Step 1-4-1-3: Softmax calculates the attention score for each head, and multiplies the attention score by V to obtain the feature representation of the input features with respect to attention;

[0147] Step 1-4-1-4: Concatenate the outputs of the multi-head system and multiply them with a learnable weight matrix to form a multi-head attention feature representation of the input features.

[0148] In step 1-4-3 of the present invention, the specific method for implementing the feedforward layer with global awareness bidirectional LSTM includes the following steps: Step 1-4-3-1: Copy the output of step 1-4-2 above, and reverse one of the copies according to the hidden time dimension, that is, the time is reversed;

[0149] Step 1-4-3-2: Feed the forward and reverse order speech features of the hidden time dimension into the LSTM network respectively, and concatenate the two outputs to further enhance the global contextual representation and modeling ability of the multi-channel speech attention features output in Step 1-4-1;

[0150] Step 1-4-3-3: ReLU activation and a linear layer further enhance the non-linear expressive power of the features.

[0151] Step 2 focuses on a detailed explanation of the basic modules of the separator, feature aggregation, and nonlinear mapping to obtain the separation mask:

[0152] Step 2-1: Apply layer normalization to the output of Step 1 to improve the stability of model training, and then use 1×1 convolution to transform the number of channels;

[0153] Step 2-2: Construct the basic module of the separator by cascaded 1D convolutional layers and the temporal-aware contextual channel attention layer constructed in Step 1-3;

[0154] Step 2-3: Cascade the basic separator modules obtained in Step 2-2 three times;

[0155] Steps 2-4: Perform feature aggregation and nonlinear mapping on the relevant outputs of the three cascaded basic separator modules to obtain the speech separation mask;

[0156] Steps 2-5: Combine the speech separation mask with the encoder output feature F Encoder Multiplying them yields the latent feature estimates F corresponding to the speech signals of the two speakers and the background noise signal. Separator .

[0157] In step 2-2 of the present invention, the specific method for implementing the basic module of the separator includes the following steps: Step 2-2-1: Construct a 1D convolutional layer;

[0158] Step 2-2-2: The 1D convolutional layer constructed in step 2-2-1 is then expanded according to an inflation factor of 2. m (where m = 0, 1, 2, 3, 4, 5, 6, 7) are cascaded in sequence, that is, 1-dimensional convolutional layers are stacked eight times;

[0159] Step 2-2-3: Next, the output of Step 2-2-2 is fed into the temporal-aware contextual channel attention layer constructed in Step 1-3. Reasonable feature selection is performed in the channel dimension and certain contextual information is given to the features, thereby completing the construction of the basic module of the separator.

[0160] In step 2-2-1 of the present invention, the specific method for implementing a 1D convolutional layer includes the following steps: Step 2-2-1-1: Perform channel transformation on the input of the 1×1 convolution to adapt to the calculation of subsequent layers;

[0161] Step 2-2-1-2: PReLU activation and layer normalization enhance the nonlinearity of features and the stability of model training, respectively;

[0162] Step 2-2-1-3: Feed the output of step 2-2-1-2 above into a depthwise convolution to reduce the amount of computation and improve the efficiency of the convolution kernel parameters.

[0163] Step 2-2-1-4: PReLU activation and layer normalization;

[0164] Step 2-2-1-5: Apply two parallel 1×1 convolutions to the output of Step 2-2-1-4 above to perform a transformation with different numbers of channels. One of the convolution outputs is added element-wise with the output of Step 2-2-1-1 to finally obtain the two outputs of the 1D convolutional layer.

[0165] In steps 2-4 of the present invention, the specific method for performing feature aggregation and nonlinear mapping on the relevant outputs of the three cascaded basic separator modules to obtain the final speech separation mask includes the following steps: Step 2-4-1: Add the outputs of the 1D convolutional layer and the temporal-aware context channel attention layer in the three cascaded basic separator modules to obtain the feature aggregation representation of the entire cascaded separator module;

[0166] Step 2-4-2: PReLU performs a non-linear mapping on the aggregated features;

[0167] Step 2-4-3: Adjust the number of channels in the 1×1 convolution to accommodate subsequent activation layers;

[0168] Step 2-4-4: Sigmoid calculates the masks corresponding to the two speakers' speech and the background noise in the mixed features.

[0169] Step 3 includes the following specific steps:

[0170] Step 3-1: The output of Step 2 above is passed through four cascaded transposed convolutional layers. The kernel size, stride and dilation factor of the convolutional layers correspond to the four cascaded convolutional layers at the front end of the encoder, and the parameters are not shared. The characteristic representation of the time-domain waveform of the real speech signal and the background noise signal is learned step by step.

[0171] Step 3-2: Finally, after transpose convolution corresponding to the frontmost convolution layer 0 of the encoder, the two speaker speech time-domain signals and the background noise time-domain signal are separated.

[0172] Step 4 includes the following specific steps:

[0173] Step 4-1: Define the overall loss function for speech separation.

[0174] Loss total =Loss SI-SNR +αLoss MR ,

[0175] Loss SI-SNR =-SI-SNR represents the scale-invariant signal-to-noise ratio loss function (SI-SNR). The input is the separated signal and the corresponding target signal. It is used to reduce the difference between the speech estimation and background noise estimation of the two speakers and the actual speaker speech and background noise. Loss MR The corresponding multi-resolution (MR) loss aims to improve the quality of speech separation and more effectively remove noise from the speaker's speech by considering both time-domain and frequency-domain information. α is an adjustable hyperparameter used to adjust the contribution and influence of the multi-resolution loss on the performance of the speech separation system.

[0176] Step 4-2: Apply a permutation-invariant training strategy to reduce the dependence of the model's separation performance on the speaker order, enhance the model's generalization ability, and gradually reduce the overall loss of the speech separation algorithm through gradient descent.

[0177] In step 5 of the present invention, the specific method for evaluating the performance of the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer includes the following steps: Step 5-1: Compare the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer with existing time-domain speech separation models to verify the overall performance of the present invention;

[0178] Step 5-2: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on Transformer to verify the effectiveness of the time-aware context channel attention layer;

[0179] Step 5-3: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with another end-to-end noisy speech separation method based on channel attention mechanism to verify the effectiveness of the bidirectional LSTM-Transformer encoder layer.

[0180] This invention is based on the PyTorch deep learning framework. The deep learning acceleration hardware is a single NVIDIA Tesla P100 SXM2 graphics card, and the CPU RTF test uses an Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz. The original mixed speech is divided into 4-second segments with a 2-second overlap between adjacent segments (this ensures that adjacent segments retain some temporal and contextual characteristics of the original complete speech). The number of training epochs is set to 200, with an initial learning rate of 0.001. If the model's performance on the validation set does not decline after two consecutive epochs, the learning rate for the next iteration is halved, with a minimum learning rate of 1e-8. The Adam optimizer is used to reasonably optimize the method of updating learnable parameters using gradient descent.

[0181] The experimental evaluation metrics in the embodiments of the present invention mainly include perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and scale-invariant signal-to-noise ratio improvement (SI-SNRi).

[0182] First, this invention focuses on comparing the impact of using different channel attention modules in the basic encoder and separator modules on speech separation performance. These modules are embedded into the encoder, separator, and their combination in the overall speech separation process, and their impact on speech separation quality is evaluated using three metrics: PESQ, STOI, and SI-SNRi. The three types of channel attention mainly include: basic channel attention, context-aware channel attention, and time-series-aware context channel attention module. The experimental results are shown in Table 1:

[0183] Table 1 compares the performance of adding different channel attention mechanism modules at three locations in the overall separation framework.

[0184]

[0185] As shown in Table 1, with the improvement of the channel attention mechanism from basic channel attention to temporal-aware contextual channel attention, the scores of all metrics showed significant improvements. The increase in PESQ score indicates that the model's ability to simulate human auditory perception has been enhanced. The improvement in STOI score indicates that the model performs better in maintaining the intelligibility of speech transmission, while the increase in SI-SNRi score reflects the model's progress in improving the target speech-to-noise ratio. At the encoder level, the temporal-aware and contextual-aware channel attention module exhibits superior performance compared to the other two modules. This result suggests that the integration of temporal information is crucial for capturing the dynamic characteristics of speech signals, enabling the encoder to process speech information more accurately. At the separator level, the same module also showed the best performance, indicating that the contextual and temporal awareness characteristics have a significant advantage in distinguishing target speech from background noise during speech signal separation and recovery.

[0186] Furthermore, the overall performance of the model peaked when the channel attention mechanism module was applied to both the encoder and the separator. This phenomenon indicates that the global optimization of the channel attention mechanism is crucial throughout the entire speech separation process. The synergistic effect of the temporal-aware and context-aware channel attention module in the encoding and separation stages significantly improves the processing quality of the speech signal, which is reflected in the improvements across various evaluation metrics.

[0187] Finally, this invention also focuses on analyzing the impact of the proposed bidirectional LSTM-Transformer encoder layer on separation performance. Through waveform comparison from experiments, the importance of the LSTM-enhanced Transformer encoder layer is demonstrated. In the model without LSTM enhancement, the estimated speech signal is mixed with noise components, which manifest as irregular fluctuations in the waveform. This not only affects the clarity of the speech but also reduces the accuracy of the speech back-end processing. In contrast, the signal waveform processed by introducing the bidirectional LSTM-Transformer encoder layer appears smooth and clean. This smoothness indicates that the model effectively removes noise during processing, preserving the main features of the speech.

[0188] In summary, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer proposed in this embodiment can significantly and effectively improve the performance of noisy speech separation systems from two main aspects: the selection of channel attention mode and its position in the separation framework, and the bidirectional LSTM-Transformer encoder layer. Experimental results verify the effectiveness and rationality of the temporal-aware contextual channel attention layer and the bidirectional LSTM-Transformer encoder layer in terms of feature selection and global contextual modeling ability to enhance features, respectively. Furthermore, compared with existing deep learning-based speech separation methods, this embodiment achieves improvements in multiple measurement indicators, and to a certain extent enhances the adaptability and robustness of the separation algorithm to various background noises, making it of great learning and reference value in practical applications.

[0189] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An end-to-end noisy speech separation method based on channel attention mechanism and Transformer, characterized in that, The method includes the following steps: Step 1: Input noisy mixed signal S mix It consists of the clean speech signals of the two speakers. S 1. S 2 and ambient background noise n The algorithm is constructed by stacking layers. First, the input noisy mixed signal is preprocessed through a convolutional layer 0 and four cascaded convolutional layers 1 with shared parameters. Then, a temporally aware contextual channel attention layer and a bidirectional LSTM-Transformer encoder layer are used to filter features from the preprocessed result and impart global contextual information, thereby obtaining a latent feature representation of the speech signal that is effective and has a certain ability to model context. F Encoder ; Step 2: The three cascaded steps of the separator basic module, consisting of multiple cascaded 1D convolutional layers and temporally aware contextual channel attention layers, progressively process features. F Encoder Generate speech separation mask Mask Finally, the mask Mask Latent feature representation of the mixed signal with encoder output F Encoder Element-wise multiplication is performed to separate the latent spatial features corresponding to each speaker's speech and background noise in the mixed signal. F Separator ; Step 3: Characteristic representation of the separator output F Separator Finally, the decoder, consisting of transposed convolutional layers corresponding to the four cascaded convolutional layers 1 and one convolutional layer 0 at the encoder's front end, completes the mapping from the latent feature space to the real temporal speech signal space and the background noise signal space. The decoder ultimately outputs three estimated temporal signals, corresponding to the speech signals of the two speakers respectively. , and background noise signal ; Step 4: Constrain and optimize the performance of the speech separation system using the overall loss function of speech separation; Step 5: Evaluate the performance of the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer; The specific steps of step 1 include: Step 1-1: The temporal noisy mixed speech signal is passed through convolution layer 0, and then through a convolution kernel with a size of... L / 2 = 16, stride S = L A 1D convolution with a bit width of 2 yields stationary speech segments that overlap. s mix-k The number of output channels is C 0 = 512, and PreLU is used to obtain the non-linear representation of the features; Step 1-2: Segment the speech signal s mix-k By using four cascaded, parameter-shared convolutional layers, the model training parameters are reduced while the expressive power of speech features is improved. Steps 1-3: The speech nonlinear feature map output after passing through one convolutional layer 0 and four convolutional layers 1 is passed through a temporally aware contextual channel attention layer to filter out redundant features in its output channel dimension and improve the effectiveness of separating useful features. Steps 1-4: After performing attention-based feature filtering on the channel dimension, the output enhances the speech separation system's ability to globally model speech signals. Finally, through a bidirectional LSTM-Transformer encoder layer, an effective latent feature representation of the speech signal with global contextual information is obtained. F Encoder ; In steps 1-3, the specific method for implementing the temporally aware contextual channel attention layer includes the following steps: Step 1-3-1: Add time-aware characteristics to the output of step 1-2; Step 1-3-2: Introduce contextual relationships into the output of Step 1-3-1 in order to provide reasonable channel weight allocation for the subsequent channel attention module; Step 1-3-3: Perform feature weighting and filtering on the channel dimension for speech features with temporal-aware context.

2. The end-to-end noisy speech separation method based on channel attention mechanism and Transformer according to claim 1, characterized in that, In step 1-3-1, the specific method for implementing speech feature representation with time-aware characteristics includes the following steps: Step 1-3-1-1: Average pool the last dimension of the input features to 1, retaining only the time dimension, thus fully learning the dynamic changes of the input signal in the time domain. This operation is defined as: ， in N It is the number of dimensions in the last dimension of the input feature. x t and y t They represent the first t Features at a specific point in time, f gap Represents global average pooling; Step 1-3-1-2: Perform channel transformation on the result that retains only the time dimension through a 1×1 convolution; Step 1-3-1-3: ReLU activation to provide non-linear expressive power; Steps 1-3-1-4: The 1×1 convolution learns the temporal characteristics of the speech again and calculates the weights through the Sigmoid function as a temporal awareness of the input features; Step 1-3-1-5: Multiply the learned time-aware weights of the input features element-wise with the input features to add global time-aware characteristics to the time-domain signal; In step 1-3-2, the specific method for implementing time-aware contextual speech features includes the following steps: Step 1-3-2-1: Increase the number of channels by performing a 1×1 convolution on the two-dimensional output of step 1-3-1-5; Step 1-3-2-2: Merge the last two dimensions of the output from the previous step; Step 1-3-2-3: A 1×1 convolution is used to compress the channel dimension of the two-dimensional output result of Step 1-3-2-2 and to calculate the context features; Steps 1-3-2-4: Apply the Softmax function to calculate the mask of the context features; Step 1-3-2-5: Perform element-wise multiplication between the context features and the context mask to obtain speech features with temporal awareness context; In step 1-3-3, the specific method for assigning and filtering speech features with temporal-aware context at the channel dimension includes the following steps: Step 1-3-3-1: A 1×1 convolution transforms the number of channels of the output result of step 1-3-2-5; Step 1-3-3-2: Layer normalization enhances the stability of speech features and model training; Step 1-3-3-3: Apply 1×1 convolution again to perform a channel number transformation; Step 1-3-3-4: The Sigmoid function estimates the attention score in the channel dimension of the output of Step 1-3-1-5, that is, the weight assigned to each channel; Step 1-3-3-5: Multiply the output of Step 1-3-1-5 with its corresponding channel attention score in the channel dimension to achieve effective filtering of temporal-aware contextual speech features in the channel dimension.

3. The end-to-end noisy speech separation method based on channel attention mechanism and Transformer according to claim 2, characterized in that, In steps 1-4, the specific method for implementing the bidirectional LSTM-Transformer encoder layer includes the following steps: Step 1-4-1: Enhance the long-distance temporal dependencies of speech in the output of Step 1-3-3-5 through the multi-head attention mechanism of the Transformer encoder layer; Step 1-4-2: Add the temporally aware contextual speech features output in Step 1-3-3-5 to the result in Step 1-4-1 and apply layer normalization. This smooths the output and helps ensure the stability of model training and prevent gradient explosion and gradient vanishing. Step 1-4-3: Use a feedforward layer with a globally aware bidirectional LSTM to enhance the global modeling and expressive power of speech features; Step 1-4-4: Add the outputs of Step 1-4-2 and Step 1-4-3 and apply layer normalization to obtain the final speech latent feature representation output by the encoder. F Encoder , for the separator to learn; In step 1-4-1, the specific method for implementing the multi-head attention mechanism includes the following steps: Step 1-4-1-1: Define the number of heads in the multi-head attention mechanism h Different heads are responsible for focusing on different features and information angles of the input features, and capturing global dependencies in speech temporal features, thereby improving the model's ability to understand and model the contextual relationships of speech signals. Step 1-4-1-2: Each head internally calculates the features related to the input. K ( Key ), Q ( Query )and V ( Value Different heads have different calculations. K , Q and V The weight matrix has parameters that can be learned; Step 1-4-1-3: Softmax calculates the attention score for each head, and multiplies the attention score by V to obtain the feature representation of the input features with respect to attention; Step 1-4-1-4: Concatenate the outputs of the multi-head system and multiply them with the weight matrix learned by one of the parameters to form a multi-head attention feature representation of the input features. In step 1-4-3, the specific method for implementing the feedforward layer with a globally aware bidirectional LSTM includes the following steps: Step 1-4-3-1: Copy the output of Step 1-4-2, and reverse one of the copies according to the hidden time dimension, that is, in reverse time order; Step 1-4-3-2: Feed the forward and reverse speech features of the hidden time dimension into the LSTM network respectively, and concatenate the two outputs to enhance the global contextual representation and modeling ability of the multi-channel speech attention features output in Step 1-4-1; Step 1-4-3-3: ReLU activation and a linear layer enhance the non-linear expressive power of features.

4. The end-to-end noisy speech separation method based on channel attention mechanism and Transformer according to claim 1, characterized in that, In step 2, the specific steps for implementing the separator include: Step 2-1: Apply layer normalization to the output of Step 1 to improve the stability of model training, and then use 1×1 convolution to transform the number of channels; Step 2-2: Construct the basic module of the separator by multiple cascaded 1D convolutional layers and the temporal-aware contextual channel attention layer constructed in Step 1-3; Step 2-3: Cascade the basic separator modules obtained in Step 2-2 three times; Steps 2-4: Perform feature aggregation and nonlinear mapping on the relevant outputs of the three cascaded basic separator modules to obtain the speech separation mask. Mask ; Steps 2-5: Separate the speech mask Mask With encoder output features F Encoder Multiplying them yields latent feature estimates corresponding to the speech signals of the two speakers and the background noise signal. F Separator .

5. The end-to-end noisy speech separation method based on channel attention mechanism and Transformer according to claim 4, characterized in that, In step 2-2, the specific method for implementing the basic module of the separator includes the following steps: Step 2-2-1: Construct a 1D convolutional layer; Step 2-2-2: Adjust the 1D convolutional layer constructed in step 2-2-1 according to the dilation factor 2. m ,in m =0, 1, 2, 3, 4, 5, 6, 7, cascaded in sequence, that is, stacking 1-dimensional convolutional layers eight times; Step 2-2-3: Next, the output of Step 2-2-2 is fed into the temporal-aware contextual channel attention layer constructed in Step 1-3. Reasonable feature selection is performed in the channel dimension and certain contextual information is given to the features to complete the construction of the basic module of the separator. In step 2-2-1, the specific method for implementing a 1D convolutional layer includes the following steps: Step 2-2-1-1: Perform channel transformation on the input using a 1×1 convolution to accommodate computation in subsequent layers; Step 2-2-1-2: PReLU activation and layer normalization are used to enhance the nonlinearity of features and the stability of model training, respectively; Step 2-2-1-3: Feed the output of step 2-2-1-2 into a depthwise convolution to reduce the amount of computation and improve the efficiency of the convolution kernel parameters. Step 2-2-1-4: PReLU activation and layer normalization; Step 2-2-1-5: Apply two parallel 1×1 convolutions to the output of Step 2-2-1-4 to perform a transformation with different numbers of channels. One of the convolution outputs is added element-wise with the output of Step 2-2-1-1 to finally obtain the two outputs of the 1D convolutional layer.

6. The end-to-end noisy speech separation method based on channel attention mechanism and Transformer according to claim 4, characterized in that, In steps 2-4, the specific method for performing feature aggregation and nonlinear mapping on the relevant outputs of the three cascaded basic separator modules to obtain the final speech separation mask includes the following steps: Step 2-4-1: Add the outputs of the 1D convolutional layers and the temporal-aware contextual channel attention layers in the three cascaded separator basic modules to obtain the overall feature aggregation representation of the separator cascaded modules. F sum ; Step 2-4-2: PReLU for aggregated features F sum Perform nonlinear mapping; Step 2-4-3: Adjust the number of channels in the 1×1 convolution to accommodate subsequent activation layers; Step 2-4-4: Sigmoid calculates the blended features at the encoder output. F Encoder Masks corresponding to the speech of two speakers and background noise Mask .

7. The end-to-end noisy speech separation method based on channel attention mechanism and Transformer according to claim 1, characterized in that, The specific steps of step 3 include: Step 3-1: Analyze the separation features output in Step 2. F Separator After four cascaded transposed convolutional layers, the kernel size, stride, and dilation factor correspond to the four cascaded convolutional layers at the front end of the encoder, and the parameters are not shared. The learning process conforms to the feature representation of the time-domain waveform of the real speech signal and background noise signal. Step 3-2: Finally, after transpose convolution corresponding to the frontmost convolution layer 0 of the encoder, the two speaker speech time-domain signals and the background noise time-domain signal are separated.

8. The end-to-end noisy speech separation method based on channel attention mechanism and Transformer according to claim 1, characterized in that, The specific steps of step 4 include: Step 4-1: Define the overall loss function for speech separation. Loss total ＝ Loss SI-SNR + αLoss MR ， in Loss SI-SNR =- SI-SNR This represents a scale-invariant signal-to-noise ratio loss function. The inputs are the separated signal and the corresponding target signal. It is used to reduce the difference between the speech estimation and background noise estimation of the two speakers and the actual speaker speech and background noise. Loss MR To address multi-resolution loss, this method improves the quality of speech separation and more effectively removes noise from speaker speech by considering both time and frequency domain information. α These are hyperparameters that are manually adjusted to adjust the contribution and impact of multi-resolution loss on the performance of the speech separation system. Step 4-2: Apply a permutation-invariant training strategy to reduce the dependence of the model's separation performance on the speaker order, enhance the model's generalization ability, and reduce the overall loss of the speech separation algorithm through gradient descent.

9. The end-to-end noisy speech separation method based on channel attention mechanism and Transformer according to claim 1, characterized in that, Step 5, specifically the method for performance evaluation of the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer, includes the following steps: Step 5-1: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with existing temporal speech separation models to verify its overall performance; Step 5-2: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on Transformer to verify the effectiveness of the time-aware context channel attention layer; Step 5-3: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with another end-to-end noisy speech separation method based on channel attention mechanism to verify the effectiveness of the bidirectional LSTM-Transformer encoder layer.

Images