Target speech extraction method based on block feature fusion and double-path transformer

Abstract

A target speech extraction method based on block feature fusion and double-path Transformer, first, the method of block feature fusion is used, so that each block can obtain complete target speaker reference speech features, and the utilization rate of features is improved; then, the double-path Transformer is used for global and local modeling of the signal, and the target speaker speech information is efficiently extracted; finally, the target speaker reference speech feature extraction network is pre-trained, the training efficiency is improved, and each component in the whole network is optimized through multi-task learning, so that the extraction effect is best. The method realizes target speaker speech extraction, compared with the existing extraction method, the model parameter quantity is lower, the extraction speed is faster, and higher extraction purity is realized.

Description

Technical Field

[0001] The present invention belongs to the field of audio processing, and in particular relates to a target speech extraction method based on block feature fusion and dual-path Transformer. Background Art

[0002] With the rapid development of artificial intelligence, voice interaction has become a crucial method of human-computer interaction, effectively improving the user experience. However, most existing voice interaction technologies are based on a single speaker, and performance deteriorates dramatically in scenarios with multiple speakers simultaneously. Traditionally, voice separation techniques have been used to isolate multiple speakers. However, these methods require pre-determining or estimating the number of speakers in a mixed speech stream, which is often unpredictable in real-world environments. As the number of speakers in an environment increases, the computing power required for voice separation also increases. Therefore, a voice processing technology is needed to isolate the target speaker's voice from the complex background.

[0003] The human brain can focus auditory attention on specific sounds by blocking out background noise. Target speech extraction is a technique that mimics human auditory attention. Based on a neural network, the system is provided with a reference speech from a speaker not seen during training. Based on this reference speech, the system directs attention to the target speaker, simulating human attentional self-attention, thereby extracting the target speaker's speech from a mixed speech. Previous research has commonly used frequency domain extraction methods. These methods extract frequency domain information from the reference and mixed audio sources as input, generate a mask that only allows the target speaker's voice to pass through, apply the mask to the mixed audio, and finally convert the target speech into a time domain signal for output. Zmolikova K et al. performed a short-time Fourier transform (STFT) on the input speech and used it as input for their proposed SpeakerBeam model. They then used a bidirectional long short-term memory (LSTM) network to extract the target speech, which can be used to extract long and complex mixed speech. Wang Q et al. divided the system into two parts: reference speech feature extraction and a frequency domain mask generation network. They combined a convolutional neural network (CNN) with a LSTM network to improve the extraction performance of frequency domain methods. However, frequency-domain methods are subject to windowing effects and phase estimation issues during signal reconstruction. Therefore, some researchers have begun studying target speaker extraction in the time domain. Chenglin Xu et al. proposed a multi-scale time-domain speaker extraction network, SpEX, which uses a time-domain convolutional network (TCN) module to model and extract mixed audio, achieving better results than frequency-domain extraction and validating the feasibility of their approach. They further improved upon reference audio information extraction and proposed SpEx+, which uses time-domain feature information from reference audio instead of spectrograms to achieve complete time-domain speaker extraction.

[0004] In summary, directly extracting the target speaker's voice from time-domain signals has become the mainstream approach. However, existing methods make limited use of the characteristics of the reference speech, and the resulting reconstructed extraction results are generally not pure and still contain some noise. Addressing these issues and further improving the performance of target speaker voice extraction are of great significance to the practical application of artificial intelligence. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention provides a target speech extraction method based on block feature fusion and dual-path Transformer. In order to solve the problem of target speaker speech extraction, the present invention proposes a block feature fusion method to improve the utilization rate of the target speaker reference features. At the same time, the present invention also proposes the introduction of a dual-path Transformer network for target speaker extraction to further improve the purity of target speech reconstruction. Finally, the present invention achieves better target speaker speech extraction by pre-training the target speaker reference speech feature extraction network and using multi-task learning to optimize the overall network.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A target speech extraction method based on block feature fusion and dual-path Transformer includes the following steps:

[0008] Step 1: Perform convolution encoding on the mixed audio to convert the speech information into feature representation. Use a trainable convolution kernel to encode the speech. Optimize the feature representation through training to achieve the best effect. Input the mixed speech signal y and output the mixed speech feature matrix Y.

[0009] Step 2: Convolutionally encode the target speaker's reference speech and extract its feature sequence. Multi-scale convolution can obtain features of different resolutions and simplify feature expression by reducing feature dimensions. Input the target speaker's reference speech. Output target speaker speech features

[0010] Step 3: Divide the mixed audio features and reference speech features into blocks and perform block-level feature fusion to improve the utilization of reference features and enable the extraction process to more accurately identify the speaker's voice; input the mixed speech feature matrix Y and the target speaker's speech features Output feature fusion matrix E;

[0011] Step 4: Use a dual-path Transformer to process the fused features. By calculating multi-head attention on the fused features, local and global modeling is performed to achieve accurate recognition and extraction of the target speaker. Finally, a mask matrix is ​​generated that only allows the target speaker's voice information to pass through. The feature fusion matrix E is input and the target speaker mask Mask is output.

[0012] Step 5: Modulate the mixed audio features with the extracted mask, filter out the speech features that are not related to the speaker, decode the modulated response features, restore the time domain signal from the features, obtain the target speech, input the mixed speech coding feature matrix Y and the target speaker mask Mask, and output the extracted target speaker speech.

[0013] Step 6: Pre-train the reference feature extraction network to speed up the training convergence. Use multi-task learning to jointly optimize the network as a whole and adjust the network to the best state. Input the target speaker's speech feature Y and the target speaker's speech extraction result. Output pre-trained reference feature extraction network and multi-task learning loss.

[0014] Furthermore, in step 1, the mixed voice signal Input a one-dimensional convolution layer with N convolution kernels. The size of each convolution kernel is L, and its stride is L / 2. The encoding of the mixed speech sequence can be expressed as:

[0015]

[0016] Where k represents the frame sequence number, k∈{0,1,2,...,K}, K=2(TL) / L+1 is the total number of frames under the step size L / 2, y represents the frame sequence, L is the convolution kernel size, U represents the convolution kernel matrix, N is the number of convolution kernels, i and j represent the matrix element numbers, i = 0, 1, ..., N-1, j = 0, 1, ..., K-1, ReLU represents the activation function f(x) = max(x, 0);

[0017] Perform encoding calculation to obtain the feature matrix containing mixed audio information

[0018] Furthermore, the process of step 2 is as follows:

[0019] Step 2.1: Perform time-domain multi-scale convolution coding on the reference speech to convert the time-domain information into feature representation. Reference speech feature extraction is based on the speech coding module of the previous process. The reference speech s is passed through three parallel convolution encoders. The convolution kernel size of each convolution layer is L1, L2, and L3 respectively. The convolution kernels of different scales encode the reference speech at different resolutions. Multi-scale speech feature S' = [S1, S2, S3];

[0020] Step 2.2: Use the residual and pooling modules to process the multi-scale speech features, reduce the feature dimension to extract the feature sequence, and the reference speech feature S' will pass through N R A stacked residual module to obtain appropriate speaker feature representation;

[0021] The residual module consists of two layers of CNN with a convolution kernel size of 1*1, a one-dimensional maximum pooling layer and a residual connection. The batch normalization BN layer and the parameter ReLU (PReLU) activation function normalize and nonlinearly transform the output of each CNN layer; the kernel size of the 1-D maximum pooling layer is 1*3, which is used to remove silence and compress the time series to 1 / 3 of the original; finally, point convolution is used to adjust the number of channels and average pooling is performed to obtain the reference feature coefficient Where D is the number of channels, that is, the predetermined feature dimension.

[0022] The process of step three is:

[0023] Step 3.1: Perform 50% overlapping cuts on each channel of the mixed speech feature and reorganize the blocks into a new feature matrix. Taking the method of segmentation in the feature sequence dimension, the long feature sequence of length K is segmented into several small segments of length F, where each small segment overlaps with the previous segment by 50%, and finally can be divided into M small segments. The chunking operation is defined as Chunk(-), and the chunking result can be expressed as

[0024] Step 3.2: Fuse the mixed speech features after segmentation with the reference features of the target speaker, and add complete reference feature information to each segment. In order to solve the problems of low reference feature utilization, high hardware resource occupancy and slow training speed, a block feature fusion method is proposed. In the production In the process, the feature dimension is set to the same length as the block, that is, D = F, Will By exchanging the dimensions of The P(-) operation is a dimension swap, which copies the second and third dimensions. The size is the same as E S Consistent, that is The C(-) operation is a copy operation, and pytorch's broadcast mechanism can implement this operation more efficiently; finally, the feature matrix Y of the same size is obtained c and Then use the addition method to achieve feature fusion, that is After copying and adding, the final feature matrix used for extraction is obtained. The dotted box represents a small block of a feature channel, which has obtained all the speech features of the target speaker and improved the utilization rate of the features.

[0025] The process of step 4 is as follows:

[0026] Step 4.1: Use the dual-path Transformer to process the fused features, identify and extract the target speaker's speech information. In order to effectively learn the order information of the sequence in the dual-path structure and promote the convergence of the model, the long short-term memory network is introduced. Its calculation formula is as follows:

[0027]

[0028] Among them, Z X Represents the input data of the Transformer module, Z M represents the intermediate variable after multi-head attention calculation, and O represents the output after the Transformer module operation;

[0029] The dual-path Transformer structure is a local and global Transformer. The two modules have the same structure and are used for local and global information modeling respectively. The modeling of different parts is achieved by dimension exchange. Both the local and global modules use a residual structure, that is, the input E is divided into two paths, and then added again after processing. The local and global processing is repeated B times in total to obtain the speech extraction output E o ;

[0030] Step 4.2: Restore the feature blocks processed by the dual-path Transformer to generate the extraction mask Mask, the output of the dual-path Transformer Y after block division with mixed speech feature matrix Y c The dimensions are consistent. In order to perform mask modulation with Y, the inverse operation of the block is required, which is defined as M(-). Mask is a mask matrix that only allows the target speaker to pass through, and its dimension is consistent with the mixed speech feature matrix Y.

[0031] The process of step five is:

[0032] Step 5.1: Mask modulation is performed on the mixed speech feature matrix. Based on the provided target speaker reference features, a mask matrix Mask is trained to allow only the target speaker's speech to pass through the mixed speech coding feature matrix Y. The voices of non-target speakers (interference speakers, noise) are filtered out as much as possible. This process is called mask modulation. The result S is the modulation response of the target speaker, which is calculated as follows:

[0033]

[0034] Where Y represents the hybrid speech coding feature matrix, Mask represents the target speaker mask matrix, S represents the modulation response of the target speaker, represents element-wise multiplicative modulation;

[0035] Step 5.2: Decode the modulated response to obtain the extracted speech. Target speech decoding converts speech features from a two-dimensional feature matrix into one-dimensional speech information through an inverse convolution operation. This process is also called target speech decoding. The target speech decoder is composed of transposed convolution. Transposed convolution does not initially have fixed parameters, but is composed of learnable parameters. After training, the optimal decoding parameters are obtained to optimize the extraction task. The calculation formula is as follows:

[0036]

[0037] Where S represents the modulation response of the target speaker, represents the extracted time-domain speech signal of the target speaker, and Tconv represents the transposed convolution operation.

[0038] The process of step six is:

[0039] Step 6.1: Pre-train the reference feature extraction network to enable it to distinguish speakers based on their voices. The reference feature extraction network extracts the speech features of the target speaker from the reference audio. This speech feature can be used to identify different speakers and confirm the target speaker from multiple voices. The extracted speech features are passed through a linear layer and a normalized exponential layer to obtain the probability of speaker prediction. The cross-entropy loss is used as the loss function to train and optimize the network. The calculation formula is as follows:

[0040]

[0041] Where i represents the speaker number, i∈{1,2,…,N s}, N s is the total number of speakers in the training dataset, p i represents the true category label of the i-th speaker, represents the predicted speaker probability;

[0042] Step 6.2: Use multi-task learning to train multiple modules of the network to optimize extraction performance. The target speaker extraction network is composed of multiple modules, each of which has its own function. To complete the speech extraction task, all modules need to be combined. Using multi-task learning training methods to train each module of the network simultaneously can help the network reach its optimal state. The pre-trained reference speech feature extraction module is also optimized to adapt to the speech extraction task. At the same time, due to the pre-training of this module, the multi-task learning training process of the entire network will be accelerated.

[0043] The evaluation of speech extraction effect is to compare the similarity between the extracted speech and the original speech before mixing. The closer the similarity, the better the extraction effect. SI-SDR is an indicator used to calculate the similarity between two signals. s is the original signal before mixing, that is, the correct answer. is the reconstructed signal after the network. There is a certain distance between these two vectors, so Decompose into perpendicular to s and parallel to s The calculation formula of SI-SDR is:

[0044]

[0045] Calculate the extracted time domain speech The SI-SDR of the original speech s is used as an indicator of the network extraction effect, with the goal of minimizing the signal reconstruction error, namely SI-SDR loss:

[0046] ML-loss=αSI-SDR-loss+(1-α)CE-loss (7)

[0047] Among them, α represents the weight of multi-task learning loss, SI-SDR-loss represents the SI-SDR loss value, and CE-loss represents the cross entropy loss value.

[0048] The technical concept of the present invention is: in scenarios where multiple people are speaking, the human-computer voice interaction function is difficult to apply. The target speaker's voice extraction can extract the target voice from the mixed voice of multiple people by obtaining the target's reference voice information. Compared with traditional voice separation, it is more targeted to multi-person voice scenarios. Existing extraction methods are mostly based on frequency domain information, and there are problems such as window effect and phase estimation during signal reconstruction. The method of the present invention first encodes the speech in the time domain, and adopts a block feature fusion method to improve the utilization rate of the target reference feature information. Then, a dual-path Transformer is used to process the fused information, calculate multi-head attention, perform local and global modeling of the voice information, identify the target speaker from the mixed voice, generate an extraction mask, and reconstruct the target voice through mask modulation and decoding. Finally, pre-training and multi-task learning are used to train and optimize the entire network.

[0049] The beneficial effects of the present invention are mainly manifested in: reducing the number of network parameters, improving the extraction operation speed, and achieving better extraction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a system structure diagram of the target speech extraction method based on block feature fusion and dual-path Transformer.

[0051] Figure 2 It is the flow chart of hybrid speech coding.

[0052] Figure 3 This is a diagram of the convolutional coding calculation principle.

[0053] Figure 4 This is a reference speech feature extraction flowchart.

[0054] Figure 5 It is the structure diagram of the residual and pooling module.

[0055] Figure 6 It is a feature matrix block flow chart.

[0056] Figure 7 It is a flowchart of block feature fusion.

[0057] Figure 8 This is the Transformer structure diagram.

[0058] Figure 9 It is a dual-path Transformer structure diagram.

[0059] Figure 10 This is a reference feature extraction network pre-training flowchart.

[0060] Figure 11 This is the SI-SDR calculation principle diagram.

[0061] Figure 12 These are some experimental results of the target speaker’s speech extraction experiment. DETAILED DESCRIPTION

[0062] The present invention will be further described below with reference to the accompanying drawings.

[0063] Reference Figures 1 to 12 , a target speech extraction method based on block feature fusion and dual-path Transformer, including the following steps:

[0064] Step 1: Convolutionally encode the mixed audio to convert the speech information into a feature representation. Use a trainable convolution kernel to encode the speech, and optimize the feature representation through training to achieve the best results. Input the mixed speech signal y and output the mixed speech feature matrix Y. The implementation method is as follows:

[0065] Mixed speech signal Enter a one-dimensional convolutional layer, such as Figure 2 As shown in the figure, the convolution layer has N convolution kernels, the size of each convolution kernel is L, and its step size is L / 2. The encoding of the mixed speech sequence can be expressed as:

[0066]

[0067] Where k represents the frame sequence number, k∈{0,1,2,...,K}, K=2(TL) / L+1 is the total number of frames under the step size L / 2, y represents the frame sequence, L is the convolution kernel size, U represents the convolution kernel matrix, N is the number of convolution kernels, i and j represent the matrix element numbers, i = 0, 1, ..., N-1, j = 0, 1, ..., K-1, and ReLU represents the activation function f(x) = max(x, 0).

[0068] Encoding calculation as Figure 3 As shown, the final feature matrix containing mixed audio information is obtained

[0069]

[0070] Step 2: Convolutionally encode the target speaker's reference speech and extract its feature sequence. Multi-scale convolution can obtain features of different resolutions and simplify feature expression by reducing feature dimensions. Input the target speaker's reference speech Output target speaker speech features The process of this step is as follows Figure 4 , the process is:

[0071] Step 2.1: Perform time-domain multi-scale convolution coding on the reference speech to convert the time-domain information into feature representation. Reference speech feature extraction is based on the speech coding module of the previous process. The reference speech s is passed through three parallel convolution encoders. The convolution kernel size of each convolution layer is L1, L2, and L3 respectively. The convolution kernels of different scales encode the reference speech at different resolutions. Multi-scale speech features S' = [S1, S2, S3].

[0072] Step 2.2: Use the residual and pooling modules to process the multi-scale speech features and reduce the feature dimension to extract the feature sequence. The reference speech feature S' will be processed by N R To obtain a suitable speaker feature representation, a stacked residual module is constructed. The residual module structure is as follows: Figure 5 shown.

[0073] The residual module consists of two CNN layers with a convolution kernel size of 1*1, a 1D max pooling layer, and a residual connection. The batch normalization (BN) layer and the parametric ReLU (PReLU) activation function normalize and nonlinearly transform the output of each CNN layer. The 1-D max pooling layer has a kernel size of 1*3, which is used to remove silence and compress the time series to 1 / 3 of its original size. Finally, point convolution is used to adjust the number of channels and average pooling is performed to obtain the reference feature coefficients. Where D is the number of channels, that is, the predetermined feature dimension.

[0074] Step 3: Divide the mixed audio features and reference speech features into blocks and perform block-level feature fusion to improve the utilization of reference features and make the extraction process more accurate in identifying the speaker's voice. Output feature fusion matrix E, the process is:

[0075] Step 3.1: Perform 50% overlap cutting on each channel of the mixed speech feature and reorganize the blocks into a new feature matrix. Figure 6 As shown, for the mixed speech feature matrix Taking the method of segmentation in the feature sequence dimension, the long feature sequence of length K is segmented into several small segments of length F, where each small segment overlaps with the previous segment by 50%, and finally can be divided into M small segments. The chunking operation is defined as Chunk(-), and the chunking result can be expressed as

[0076] Step 3.2: Fuse the mixed speech features after segmentation with the reference features of the target speaker, and add complete reference feature information for each segment. In order to solve the problems of low reference feature utilization, high hardware resource occupancy and slow training speed, the present invention proposes a segment feature fusion method, such as Figure 7 As shown. For the reference speech features In the production In the process, the feature dimension is set to the same length as the block, that is, D = F, Will By exchanging the dimensions of The P(-) operation is a dimension swap, which copies the second and third dimensions. The size is the same as E S Consistent, that is The C(-) operation is a copy operation, and the broadcast mechanism of pytorch can implement this operation more efficiently. Finally, the feature matrix Y of the same size is obtained. c and Then use the addition method to achieve feature fusion, that is The specific process is shown in the figure. After replication and addition, the final feature matrix used for extraction is obtained. The dotted box represents a small block of a feature channel. All speech features of the target speaker have been obtained, which improves feature utilization.

[0077] Step 4: Use a dual-path Transformer to process the fused features. By calculating multi-head attention on the fused features, local and global modeling is performed to achieve accurate recognition and extraction of the target speaker. Finally, a mask matrix is ​​generated that only allows the target speaker's voice information to pass through. Input the feature fusion matrix E and output the target speaker mask Mask. The process is:

[0078] Step 4.1: Use the dual-path Transformer to process the fused features, identify and extract the target speaker’s speech information. In order to effectively learn the order information of the sequence in the dual-path structure and promote the convergence of the model, the method of the present invention introduces a long short-term memory network, the structure of which is as follows: Figure 8 As shown, the calculation formula is as follows:

[0079]

[0080] Among them, Z X Represents the input data of the Transformer module, Z M Represents the intermediate variable after multi-head attention calculation, and O represents the output after the Transformer module operation.

[0081] The dual-path Transformer structure is as follows Figure 9 As shown in Figure 1, the main structure is the local and global Transformer. These two modules have the same structure and are used for local and global information modeling respectively. They achieve modeling of different parts by exchanging dimensions. Both the local and global modules use a residual structure, that is, the input E is divided into two paths, which are then added together after processing. The local and global processing is repeated B times in total to obtain the speech extraction output E. o .

[0082] Step 4.2: Restore the feature blocks processed by the dual-path Transformer to generate the extraction mask. Output of the dual-path Transformer Y after block division with mixed speech feature matrix Y c In order to perform mask modulation with Y, the inverse operation of the block is required, which is defined as M(-). Mask is a mask matrix that only allows the target speaker to pass through, and its dimension is consistent with the mixed speech feature matrix Y.

[0083] Step 5: Modulate the mixed audio features with the extracted mask to filter out speech features unrelated to the speaker. Decode the modulated response features, restore the time domain signal from the features, and obtain the target speech. Input the mixed speech coding feature matrix Y and the target speaker mask Mask, and output the extracted target speaker speech. The process is:

[0084] Step 5.1: Mask modulation is performed on the mixed speech feature matrix. Based on the provided target speaker reference features, a mask matrix Mask is trained to allow only the target speaker's speech to pass through the mixed speech coding feature matrix Y. The voices of non-target speakers (interference speakers, noise) are filtered out as much as possible. This process is called mask modulation. The result S is the modulation response of the target speaker, which is calculated as follows:

[0085]

[0086] Where Y represents the hybrid speech coding feature matrix, Mask represents the target speaker mask matrix, S represents the modulation response of the target speaker, Represents element-wise multiplicative modulation.

[0087] Step 5.2: Decode the modulated response to extract the speech. Target speech decoding involves converting speech features from a two-dimensional feature matrix into one-dimensional speech information through an inverse convolution operation. This process is also called target speech decoding. The target speech decoder is composed of transposed convolutions. Transposed convolutions initially have no fixed parameters but are composed of learnable parameters. After training, the optimal decoding parameters are obtained, thereby optimizing the extraction task. The calculation formula is as follows:

[0088]

[0089] Where S represents the modulation response of the target speaker, represents the extracted time-domain speech signal of the target speaker, and Tconv represents the transposed convolution operation.

[0090] Step 6: Pre-train the reference feature extraction network to speed up the training convergence. Use multi-task learning to jointly optimize the entire network and adjust the network to the best state. Input the target speaker's speech feature Y and the target speaker's speech extraction result Output pre-trained reference feature extraction network and multi-task learning loss. The process is:

[0091] Step 6.1: Pre-train the reference feature extraction network so that it can distinguish the speaker's identity based on the sound. The function of the reference feature extraction network is to extract the target speaker's speech features from the reference audio. This speech feature can be used to identify different speakers and confirm the target speaker from multiple voices. The pre-training process of the feature extraction network is as follows: Figure 10 As shown in the figure, the extracted speech features are passed through a linear layer and a normalized exponential layer to obtain the probability of speaker prediction. The cross-entropy loss is used as the loss function to train and optimize the network. The calculation formula is as follows:

[0092]

[0093] Where i represents the speaker number, i∈{1,2,…,N s}, N s is the total number of speakers in the training dataset, p i represents the true category label of the i-th speaker, represents the predicted speaker probability.

[0094] Step 6.2: Use multi-task learning to train multiple modules of the network and optimize extraction performance. The target speaker extraction network is composed of multiple modules, each with its own function. These modules must be combined to complete the speech extraction task. Using multi-task learning to train each module simultaneously can help the network reach optimal performance. The pre-trained reference speech feature extraction module is also optimized simultaneously to suit the speech extraction task. Furthermore, pre-training this module accelerates the multi-task learning training process for the entire network.

[0095] The evaluation of speech extraction effect is generally to compare the similarity between the extracted speech and the original speech before mixing. The closer the similarity, the better the extraction effect. SI-SDR is an indicator used to calculate the similarity between two signals, such as Figure 11 As shown, s is the original signal before mixing, that is, the correct answer, is the reconstructed signal after the network. There is a certain distance between these two vectors, so Decompose into perpendicular to s and parallel to s The calculation formula of SI-SDR is:

[0096]

[0097] Calculate the extracted time domain speech The SI-SDR of the original speech s is used as an indicator of the network extraction effect, with the goal of minimizing the signal reconstruction error, namely SI-SDR loss:

[0098] ML-loss=αSI-SDR-loss+(1-α)CE-loss (7)

[0099] Among them, α represents the weight of multi-task learning loss, SI-SDR-loss represents the SI-SDR loss value, and CE-loss represents the cross entropy loss value.

[0100] To verify the performance of the proposed method, a comparative experiment was conducted. The dataset used in the experiment is WSJ0-2mix-extr, a two-person mixed speech dataset. The speech signals are resampled to an 8kHz sampling rate based on the WSJ0 speech database. This dataset is divided into three subsets: a training set (20,000 sentences), a validation set (5,000 sentences), and a test set (3,000 sentences). Each sentence data set contains three sentences: the mixed speech, the original speech, and the reference speech.

[0101] The loss weight α for multi-task learning was set to 0.5, the encoder convolution kernel length L = 4, the encoder stride = L / 2 = 2, the number of encoder convolution kernels N = 64, the number of multi-head attention heads H = 4, and the number of dual-path transformer cycles B = 6. Training was performed for 100 epochs, with a batch size of 4 and an initial learning rate of 0.125. A warm-up learning strategy was used, and Adam was used as the optimizer.

[0102] Experiment 1: Target speaker speech extraction experiment. The test set has 3000 samples. Some of the experimental results are as follows Figure 12 As shown, the left side is the time domain diagram of the audio (horizontal axis time, vertical axis amplitude), and the right side is the spectrogram of the audio (horizontal axis time, vertical axis frequency). From top to bottom, they are the mixed speech y, the original audio of the target speaker s, and the extracted result and reference audio The time domain graph of the mixed speech is the superposition of the two speech sounds, and the spectrogram is relatively messy. The reference speech and the original speech have no connection except for the speaker. Compared with the mixed audio, the extraction results Only the target speaker's voice is retained in the mixed speech. Both the time-domain graph and the spectrogram reveal that interfering speech is reduced, resulting in a high degree of extraction purity. Furthermore, compared with the original audio, the extracted results reconstruct nearly every utterance, eliminating any excess noise. The spectrograms are virtually identical, demonstrating excellent extraction performance.

[0103] Experiment 2: Performance Comparison. Using the same test set of 3,000 samples, we tested existing methods, using average SI-SDR as the performance metric. The results, shown in Table 1, show that our method not only reduces the number of model parameters but also achieves a 1.43dB improvement in extraction performance compared to previous time-domain methods, achieving the best performance.

[0104] Model Methods Time-frequency domain processing Parameter quantity (M) SI-SDR (dB) Unprocessed - - 2.5 SpeakerBeam Frequency domain 19.30 9.22 VoiceFilter Frequency domain 18.90 12.4 SpEx Time Domain 10.80 14.6 SpEx+ Time Domain 11.10 18.2 Method of the present invention Time Domain 4.39 19.63

[0105] Table 1

[0106] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A target speech extraction method based on block feature fusion and dual-path Transformer, characterized in that: The method comprises the following steps: Step 1: Perform convolution encoding on the mixed audio to convert the speech information into feature representation. Use a trainable convolution kernel to encode the speech. Optimize the feature representation through training to achieve the best effect. Input the mixed speech signal y and output the mixed speech feature matrix Y. Step 2: Convolutionally encode the target speaker's reference speech and extract its feature sequence. Multi-scale convolution can obtain features of different resolutions and simplify feature expression by reducing feature dimensions. Input the target speaker's reference speech s and output the target speaker's speech features. Step 3: Divide the mixed audio features and reference speech features into blocks and perform block-level feature fusion to improve the utilization of reference features and enable the extraction process to more accurately identify the speaker's voice; input the mixed speech feature matrix Y and the target speaker's speech features Output feature fusion matrix E; Step 4: Use a dual-path Transformer to process the fused features. By calculating multi-head attention on the fused features, local and global modeling is performed to achieve accurate recognition and extraction of the target speaker. Finally, a mask matrix is ​​generated that only allows the target speaker's voice information to pass through. The feature fusion matrix E is input and the target speaker mask Mask is output. Step 5: Modulate the mixed audio features with the extracted mask, filter out the speech features that are not related to the speaker, decode the modulated response features, restore the time domain signal from the features, obtain the target speech, input the mixed speech coding feature matrix Y and the target speaker mask Mask, and output the extracted target speaker speech. Step 6: Pre-train the reference feature extraction network to speed up the training convergence. Use multi-task learning to jointly optimize the network as a whole and adjust the network to the best state. Input the target speaker's speech feature Y and the target speaker's speech extraction result. Output pre-trained reference feature extraction network and multi-task learning loss.

2. The target speech extraction method based on block feature fusion and dual-path Transformer according to claim 1, characterized in that: In the step 1, the mixed speech signal Input a one-dimensional convolution layer with N convolution kernels. The size of each convolution kernel is L, and its stride is L / 2. The encoding of the mixed speech sequence can be expressed as: Where k represents the frame sequence number, k∈{0,1,2,...,K}, K=2(TL) / L+1 is the total number of frames under the step size L / 2, y represents the frame sequence, L is the convolution kernel size, U represents the convolution kernel matrix, N is the number of convolution kernels, i and j represent the matrix element numbers, i = 0, 1, ..., N-1, j = 0, 1, ..., K-1, ReLU represents the activation function f(x) = max(x, 0); Perform encoding calculation to obtain the feature matrix containing mixed audio information 3. The target speech extraction method based on block feature fusion and dual-path Transformer according to claim 1 or 2, characterized in that: The process of step 2 is: Step 2.1: Perform time-domain multi-scale convolution coding on the reference speech to convert the time-domain information into feature representation. The reference speech feature extraction is based on the speech coding module in the previous process. The reference speech s is passed through three parallel convolution encoders. The convolution kernel sizes of each convolution layer are L1, L2, and L3 respectively. Convolution kernels of different scales encode the reference speech at different resolutions. Multi-scale speech feature S' = [S1, S2, S3]; Step 2.2: Use the residual and pooling modules to process the multi-scale speech features, reduce the feature dimension to extract the feature sequence, and the reference speech feature S' will pass through N R A stacked residual module to obtain appropriate speaker feature representation; The residual module consists of two layers of CNN with a convolution kernel size of 1*1, a one-dimensional maximum pooling layer and a residual connection. The batch normalization BN layer and the parameter ReLU activation function normalize and nonlinearly transform the output of each CNN layer; the kernel size of the 1-D maximum pooling layer is 1*3, which is used to remove silence and compress the time series to 1 / 3 of the original; finally, point convolution is used to adjust the number of channels and average pooling is performed to obtain the reference feature coefficient Where D is the number of channels, that is, the predetermined feature dimension.

4. The target speech extraction method based on block feature fusion and dual-path Transformer according to claim 1 or 2, characterized in that: The process of step three is: Step 3.1: Perform 50% overlapping cuts on each channel of the mixed speech feature and reorganize the blocks into a new feature matrix. Taking the method of segmentation in the feature sequence dimension, the long feature sequence of length K is segmented into several small segments of length F, where each small segment overlaps with the previous segment by 50%, and finally can be divided into M small segments. The chunking operation is defined as Chunk(-), and the chunking result can be expressed as Step 3.2: Fuse the mixed speech features after segmentation with the reference features of the target speaker, and add complete reference feature information to each segment. In order to solve the problems of low reference feature utilization, high hardware resource occupancy and slow training speed, a block feature fusion method is proposed. In the production In the process, the feature dimension is set to the same length as the block, that is, D = F, Will By exchanging the dimensions of The P(-) operation is a dimension swap, which copies the second and third dimensions. The size is the same as E S Consistent, that is The C(-) operation is a copy operation, and pytorch's broadcast mechanism can implement this operation more efficiently; finally, the feature matrix Y of the same size is obtained c and Then use the addition method to achieve feature fusion, that is After copying and adding, the final feature matrix used for extraction is obtained. The dotted box represents a small block of a feature channel, which has obtained all the speech features of the target speaker and improved the utilization rate of the features.

5. The target speech extraction method based on block feature fusion and dual-path Transformer according to claim 1 or 2, characterized in that: The process of step 4 is as follows: Step 4.1: Use the dual-path Transformer to process the fused features, identify and extract the target speaker's speech information. In order to effectively learn the order information of the sequence in the dual-path structure and promote the convergence of the model, the long short-term memory network is introduced. Its calculation formula is as follows: Among them, Z X Represents the input data of the Transformer module, Z M represents the intermediate variable after multi-head attention calculation, and O represents the output after the Transformer module operation; The dual-path Transformer structure is a local and global Transformer. The two modules have the same structure and are used for local and global information modeling respectively. The modeling of different parts is achieved by dimension exchange. Both the local and global modules use a residual structure, that is, the input E is divided into two paths, and then added again after processing. The local and global processing is repeated B times in total to obtain the speech extraction output E o ; Step 4.2: Restore the feature blocks processed by the dual-path Transformer to generate the extraction mask Mask, the output of the dual-path Transformer Y after block division with mixed speech feature matrix Y c The dimensions are consistent. In order to perform mask modulation with Y, the inverse operation of the block is required, which is defined as M(-). Mask is a mask matrix that only allows the target speaker to pass through, and its dimension is consistent with the mixed speech feature matrix Y.

6. The target speech extraction method based on block feature fusion and dual-path Transformer according to claim 1 or 2, characterized in that: The process of step five is: Step 5.1: Perform mask modulation on the mixed speech feature matrix. For the mixed speech coding feature matrix Y, based on the provided target speaker reference features, a mask matrix Mask is trained to allow only the target speaker's speech to pass through. The voices of non-target speakers are filtered out as much as possible. This process is called mask modulation. The result S is the modulation response of the target speaker, which is calculated as follows: Where Y represents the hybrid speech coding feature matrix, Mask represents the target speaker mask matrix, S represents the modulation response of the target speaker, represents element-wise multiplicative modulation; Step 5.2: Decode the modulated response to obtain the extracted speech. Target speech decoding converts speech features from a two-dimensional feature matrix into one-dimensional speech information through an inverse convolution operation. This process is also called target speech decoding. The target speech decoder is composed of transposed convolution. The transposed convolution does not initially have fixed parameters, but is composed of learnable parameters. After training, the optimal decoding parameters are obtained to optimize the extraction task. The calculation formula is as follows: Where S represents the modulation response of the target speaker, represents the extracted time-domain speech signal of the target speaker, and Tconv represents the transposed convolution operation.

7. The target speech extraction method based on block feature fusion and dual-path Transformer according to claim 1 or 2, characterized in that: The process of step six is: Step 6.1: Pre-train the reference feature extraction network so that it can distinguish speakers based on their voices. Pass the extracted speech features through a linear layer and a normalized exponential layer to obtain the probability of speaker prediction. Use the cross-entropy loss as the loss function to train and optimize the network. The calculation formula is as follows: Where i represents the speaker number, i∈{1,2,…,N s }, N s is the total number of speakers in the training dataset, p i represents the true category label of the i-th speaker, represents the predicted speaker probability; Step 6.2: Use multi-task learning to train multiple modules of the network and optimize extraction performance. The target speaker extraction network is composed of multiple modules, each of which has its own function. To complete the speech extraction task, all modules need to be combined. Using multi-task learning training methods to train each module of the network simultaneously can make the network reach the optimal state. The pre-trained reference speech feature extraction module will also be optimized to adapt to the speech extraction task. At the same time, due to the pre-training of this module, the multi-task learning training process of the entire network will be accelerated. The evaluation of speech extraction effect is to compare the similarity between the extracted speech and the original speech before mixing. The closer the similarity, the better the extraction effect. SI-SDR is an indicator used to calculate the similarity between two signals. s is the original signal before mixing, that is, the correct answer. is the reconstructed signal after the network. There is a certain distance between these two vectors, so Decompose into perpendicular to s and parallel to s The calculation formula of SI-SDR is: Calculate the extracted time domain speech The SI-SDR of the original speech s is used as an indicator of the network extraction effect, with the goal of minimizing the signal reconstruction error, namely SI-SDRloss: ML-loss=αSI-SDR-loss+(1-α)CE-loss (7) Among them, α represents the weight of multi-task learning loss, SI-SDR-loss represents the SI-SDR loss value, and CE-loss represents the cross entropy loss value.

Images