Multi-resolution UNet dereverberation network

Abstract

The present invention provides a multi-resolution UNet dereverberation network, which achieves information complementarity between different resolution blocks by slicing the speech spectrum along the time dimension in different resolution blocks and adding information interaction between different resolution blocks, which helps to improve the speech dereverberation performance. Due to the use of multi-resolution features, the present invention interacts the information in the lower-level resolution blocks with the information between the upper-level resolution blocks, thereby gradually improving the dereverberation performance and solving the performance limitation problem brought about by the use of single-resolution features in the baseline method. Experimental verification was carried out in a strong reverberation environment of actual recording. The experimental results show that the proposed dereverberation network still has good dereverberation performance, solving the problem of poor performance of the original method in a strong reverberation environment.

Description

Technical Field

[0001] The present invention relates to the field of speech processing, and in particular to a dereverberation network. By slicing the speech spectrum along the time dimension into blocks of different resolutions and introducing information interaction between the blocks of different resolutions, information complementarity between the blocks of different resolutions is achieved, which helps to improve the speech dereverberation performance. Background Art

[0002] In real life, people spend most of their time indoors. In relatively enclosed indoor environments, when using mobile phones, making calls, or holding video conferences, the sound from the source not only travels directly through the air to the microphone (often referred to as direct sound), but also travels multiple times through surfaces like walls, ceilings, and floors before reaching the microphone. The signal received by the microphone is a combination of the original sound signal from the source and the delay and attenuation caused by its reflections from these surfaces. This phenomenon is known as reverberation.

[0003] For people with normal hearing, reverberation can severely impact speech quality and intelligibility, a problem exacerbated by longer reverberation times. For people with hearing loss, the effects are even more severe. Furthermore, reverberation can severely impact many practical applications, such as Automatic Speech Recognition (ASR), widely used in smart speakers and in-car voice control, hearing aids for the hearing impaired, and sound source localization systems (SLS) picked up by microphones.

[0004] In recent years, with the rapid development of deep learning technology, a large number of deep learning-based dereverberation methods have been proposed, which have greatly improved the performance of dereverberation. In 2014, Han et al. first proposed the use of deep neural networks (DNN) for speech dereverberation. After observing that the reverberation time (T 60 ) After selecting appropriate frame length and frame shift to improve the performance of dereverberation, Wu et al. 60 Incorporate it into feature selection and DNN training; Williamson et al. performed denoising and dereverberation in the complex domain for the first time in supervised learning, and jointly enhanced the amplitude spectrum and phase spectrum of speech by estimating the complex ideal ratio mask (CIRM) in the real domain and imaginary domain; Zhao et al. observed that spectral mapping is better than the time-frequency masking method in the speech dereverberation task, while the time-frequency masking method is better than the spectral mapping method in the denoising task, and constructed a two-stage algorithm.

[0005] Since 2015, UNet has achieved significant breakthroughs in medical image segmentation, ushering in the era of deep learning. In 2018, Ernst et al. introduced the UNet network to the task of speech dereverberation. In UNet, the reverberant speech spectrum is first downsampled with a step size of 2 until the bottleneck feature is reached. The bottleneck feature is then upsampled to restore the original spectral shape. Furthermore, to reduce the loss of important information caused by downsampling, skip connections are added between the downsampling layer and the corresponding upsampling layer.

[0006] However, this method feeds the entire speech spectrum into the UNet network, which lacks local information and may lead to suboptimal performance. Summary of the Invention

[0007] In order to overcome the deficiencies of the prior art, the present invention provides a multi-resolution UNet dereverberation network. The multi-resolution UNet (MR-UNet) dereverberation network proposed in the present invention can effectively remove the adverse effects of reverberation. Although the UNet-based speech dereverberation network reduces the information loss caused by downsampling through the jump connection between the downsampling layer and the upsampling layer, it still lacks local information and may lead to suboptimal performance. To address this problem, the present invention proposes a multi-resolution UNet dereverberation network, which achieves information complementarity between different resolution blocks by slicing the speech spectrum along the time dimension in different resolution blocks and adding information interaction between different resolution blocks, which helps to improve the speech dereverberation performance.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A multi-resolution UNet dereverberation network includes three resolution blocks. The input features of each resolution block are time-frequency features of a specific resolution. The dereverberation network adopts an encoder-decoder structure. The three resolution blocks are called B1, B2 and B3 respectively. The resolution of the time-frequency features input by each resolution block is different. In resolution block B3, the input features are evenly divided into four parts along the time dimension. Each part is encoded by a convolution block (Conv Block, CB) and an encoder and then spliced ​​in pairs along the time dimension. The spliced ​​features are sent to the decoder for decoding. The output of the decoder is passed through a mask module (Mask The dereverberation result of B3 is obtained after the input features of resolution block B2 are divided into two parts of equal size along the time dimension. Each part is spliced ​​along the time dimension after passing through the convolution block and the encoder, and then sent to the decoder for decoding. After passing through the mask module, the dereverberation result of B2 is obtained. No segmentation operation is performed in resolution block B1. After directly passing through the encoder and decoder output, the dereverberation result of B1 is obtained through two-dimensional convolution. Through direct addition of information and channel splicing, the useful information output by the current resolution block is passed to the next resolution block, thereby gradually improving the dereverberation performance of the MR-UNet network.

[0010] In the convolution block, CB is regarded as a residual network composed of multiple two-dimensional convolutional layers that extracts features in each resolution block; the amplitude spectrum of the reverberant speech is represented by X, and the output of CB is represented by Z, then:

[0011] Z=X+W CB X (1)

[0012] Where W CB Represents the weight matrix of CB. The result of passing the input feature X through the two-dimensional convolution layer and the parametric rectified linear unit (PReLU) and then through the output of the two-dimensional convolution layer is called X1. X1 is obtained by the global pooling layer to obtain X2. The formula is:

[0013]

[0014] Where H and W represent the dimensions of the spectrum, X1(i,j) represents the spectrum value at the time-frequency point (i,j); H GP It is a global pooling function that helps to represent all information of the entire spectrum;

[0015] In order to make full use of the information after global pooling, the gate function and the Sigmoid function are used:

[0016] X3=f(W U δ(W D X2)) (3)

[0017] Where f(·) represents the Sigmoid activation function, δ(·) represents the ReLU (Rectified Linear Unit) activation function, and W D represents the first 2D convolutional layer after global pooling, with downsampling rate r, W U represents the second two-dimensional convolutional layer after global pooling, with an upsampling rate of r. Finally, we get:

[0018] Z=X+X1X3 (4)

[0019] Where Z is the output result of the convolution block.

[0020] In the masking module, the unprocessed amplitude spectrum of the current resolution block is added to the result of a 1×1 convolution with the intermediate product to obtain the processed amplitude spectrum of the current stage. The processed amplitude spectrum of the current stage is masked by a 1×1 convolution and a sigmoid activation function with the result of a two-dimensional convolution with the intermediate product. The masked result is then channel-connected with the unprocessed amplitude spectrum of the next resolution block after a residual connection. The benefit of the masking module is that it suppresses features with less information in the current resolution block and only allows useful features to propagate to the next resolution block. The "unprocessed amplitude spectrum of the current resolution block" and the "unprocessed amplitude spectrum of the next resolution block" refer to the unprocessed amplitude spectrum of the reverberated speech of two adjacent resolution blocks. The "intermediate product" refers to the output of the decoder in the current resolution block, and the "processed amplitude spectrum" refers to the estimated amplitude spectrum of the dereverberated speech.

[0021] In each resolution block, the U-Net network is used to stack the convolution blocks to form a multi-convolution block (MCB). The MCB does not contain downsampling operations, so there is no loss of spectral information. Each MCB contains 8 CBs and a two-dimensional convolution layer. The input and output of the MCB are connected through a residual connection, and 3 MCBs are used instead of UNet.

[0022] The loss function used by the multi-resolution UNet dereverberation network is:

[0023]

[0024] Among them L MR-UNet Represents the loss function of MR-UNet, R represents the resolution of a resolution block, represents the de-reverberated speech estimated by the resolution block, Y represents the amplitude spectrum of the direct sound speech, L R Expressed as:

[0025]

[0026] where ε is set to 10 -3 .

[0027] The beneficial effect of the present invention lies in the fact that, by using multi-resolution features, the information in the lower-resolution blocks interacts with the information between the upper-resolution blocks, thereby progressively improving the dereverberation performance and overcoming the performance limitations of the baseline method using single-resolution features. The proposed dereverberation network was experimentally verified in an actual recorded strong reverberation environment. The experimental results show that the proposed dereverberation network still has good dereverberation performance, solving the problem of the poor performance of the original method in strong reverberation environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural block diagram of MR-UNet of the present invention.

[0029] Figure 2 It is a structural block diagram of the convolution block of the present invention.

[0030] Figure 3 It is a structural block diagram of the mask module of the present invention.

[0031] Figure 4 It is a structural block diagram of information fusion between blocks of different resolutions in the present invention.

[0032] Figure 5 It is a structural block diagram of the multi-convolution block of the present invention.

[0033] Figure 6 The spectrogram of the sample speech after using each dereverberation algorithm, where Figure 6 (a) is the spectrum of reverberant speech (T60=756ms), Figure 6 (b) is the direct sound speech spectrum diagram, Figure 6 (c) is the WPE (1ch) dedeverberation speech spectrum. Figure 6 (d) is the LSTM dedeverberation speech spectrum. Figure 6 (e) is the Late-LSTM dedeverberation speech spectrum. Figure 6 (f) is the UNet dedeverberation speech spectrum, Figure 6 (g) is the SkipConvNet dedeverberation speech spectrum. Figure 6 (h) is the MR-UNet dedeverberation speech spectrum. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and examples.

[0035] Figure 1The block diagram of the proposed MR-UNet dereverberation network is shown. The proposed MR-UNet is divided into three resolution blocks. The input features of each resolution block are time-frequency features of a specific resolution. The dereverberation network uses an encoder-decoder neural network, namely a UNet. The output is the dereverberation output for the current resolution block. Specifically, the three resolution blocks are referred to as B1, B2, and B3, from top to bottom. The resolution of the time-frequency features input to each resolution block is different. In B3, the input features are evenly divided into four parts along the time dimension. Each part is encoded by a convolution block (CB) and an encoder, and then concatenated along the time dimension. The concatenated features are then fed into the decoder for decoding. The decoder output passes through a mask block (MB) to produce the dereverberation result for B3. Similarly, the input features of B2 are divided into two equal-sized parts along the time dimension. Each part is concatenated along the time dimension after passing through a convolution block and an encoder, and then fed into the decoder and mask block to produce the dereverberation result for B2. In order to preserve the original reverberation information, no segmentation operation is performed in B1. The dereverberation result of B1 is obtained by directly passing the encoder and decoder output through two-dimensional convolution. In addition, in order to achieve information interaction between blocks of different resolutions and improve the dereverberation performance of the MR-UNet network, Figure 1 The dot-dash line represents direct addition of information, and the dash line represents channel concatenation. The purpose of the two information interactions is to pass useful information output by the current resolution block to the next resolution block, thereby gradually improving the dereverberation performance of the MR-UNet network.

[0036] The structure of the three submodules in MR-UNet, Figure 2 The structural diagram of the convolutional block in MR-UNet is shown. Figure 1 In , CB is regarded as a residual network composed of multiple two-dimensional convolutional layers that extract features in each resolution block; the amplitude spectrum of the reverberant speech is represented by X, and the output of CB is represented by Z, then:

[0037] Z=X+W CB X (1)

[0038] Where W CB Represents the weight matrix of CB. Figure 2 In the example, the input feature X passes through the two-dimensional convolution layer and the parametric rectified linear unit (PReLU), and then the output of the two-dimensional convolution layer is called X1. X1 is obtained by the global pooling layer to obtain X2. The formula is:

[0039]

[0040] Where H and W represent the dimensions of the spectrum, X1(i,j) represents the spectrum value at the time-frequency point (i,j); H GP It is a global pooling function that helps to represent all information of the entire spectrum;

[0041] In order to make full use of the information after global pooling, the gate function and the Sigmoid function are used:

[0042] X3=f(W U δ(W D X2)) (3)

[0043] Where f(·) represents the Sigmoid activation function, δ(·) represents the ReLU (Rectified Linear Unit) activation function, and W D represents the first 2D convolutional layer after global pooling, with downsampling rate r, W U represents the second two-dimensional convolutional layer after global pooling, with an upsampling rate of r. Finally, the output Z is:

[0044] Z=X+X1X3 (4)

[0045] Figure 3 Shown in Figure 1 The specific structure diagram of the mask module in , where "unprocessed amplitude spectrum of the current resolution block" and "unprocessed amplitude spectrum of the next resolution block" refer to the unprocessed amplitude spectrum of the reverberant speech of the two adjacent resolution blocks. "Intermediate product" refers to the output of the decoder in the current resolution block, and "processed amplitude spectrum" refers to the estimated amplitude spectrum of the dereverberated speech. The unprocessed amplitude spectrum of the current resolution block will be added to the result of 1×1 convolution with the intermediate product to obtain the processed amplitude spectrum of the current stage. The processed amplitude spectrum of the current stage obtained will be masked by 1×1 convolution and Sigmoid activation function with the result of two-dimensional convolution with the intermediate product. The processed result will be channel-wise connected with the unprocessed amplitude spectrum of the next resolution block after residual connection. The advantage of the mask module is that it suppresses features with less information in the current resolution block and only allows useful features to propagate to the next resolution block.

[0046] Figure 1 The dashed lines in represent information fusion between blocks of different resolutions. The outputs of the encoder and decoder of the current resolution block are added to the features of the next resolution block after 1×1 convolution, as shown in Figure 4 The information fusion between different resolution blocks can reduce the information loss caused by repeated upsampling and downsampling. At the same time, the information of the current resolution block will supplement the information of the next resolution block, making it more information-rich.

[0047] At each resolution block, the network that actually performs dereverberation is the U-Net network. However, downsampling in the UNet network will lose some spectral information. To solve this problem, some convolution blocks are stacked to form a multi-convolution block (MCB). MCB does not contain downsampling operations, so there will be no loss of spectral information. The structural block diagram of MCB is shown below. Figure 5 As shown in the figure, each MCB contains 8 CBs and a 2D convolutional layer. The input and output of the MCB are connected through residual connections. Use 3 MCBs to replace UNet;

[0048] The loss function of the proposed MR-UNet is:

[0049]

[0050] Among them L MR-UNet Represents the loss function of MR-UNet, R represents the resolution of a resolution block, represents the de-reverberated speech estimated by the resolution block, Y represents the amplitude spectrum of the direct sound speech, L R Expressed as:

[0051]

[0052] where ε is set to 10 in all experiments. -3 .

[0053] The present invention evaluates the dereverberation results of the MR-UNet method under simulated reverberation datasets and real-recorded reverberation datasets.

[0054] A series of experiments were conducted on a simulated reverberation dataset to evaluate the proposed MR-UNet network. The scoring results are shown in Table 1. Bold numbers indicate the best performance. For all metrics, higher scores indicate better speech quality. Clearly, MR-UNet outperforms all baseline systems in all metrics, demonstrating superior performance.

[0055] Table 1 Average scores of STOI, PESQ, and fwSegSNR for the simulated reverberation dataset

[0056]

[0057] exist Figure 6 In this paper, a sample is randomly selected from the simulation data set to intuitively demonstrate the effectiveness of the MR-UNet algorithm for dereverberation proposed in this paper. Figure 6 (a) is the amplitude spectrum of reverberant speech, Figure 6 (b) is the amplitude spectrum of the corresponding clean speech. Figure 6(c), (d), (e), (f), (g), and (h) show the amplitude spectra of the processed speech obtained by dereverberation using WPE (1ch), LSTM, Late-LSTM, UNet, SkipConvNet, and MR-UNet, respectively. Obviously, MR-UNet performs best in suppressing the smearing effect caused by reverberation.

[0058] Figure 6 Spectrogram of samples in the simulated reverberation dataset, where (a) reverberation speech spectrogram (T 60 =756ms), (b) direct sound speech spectrogram, (c) WPE (1ch) dereverberation speech spectrogram, (d) LSTM dereverberation speech spectrogram, (e) Late-LSTM dereverberation speech spectrogram, (f) UNet dereverberation speech spectrogram, (g) SkipConvNet dereverberation speech spectrogram, (h) MR-UNet dereverberation speech spectrogram

[0059] In order to verify the dereverberation effect of the MR-UNet proposed in this paper in a real environment, the STOI, PESQ and fwSegSNR scores of each dereverberation algorithm were recorded under the real reverberation dataset.

[0060] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments. However, the specific embodiments described are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the present invention.

[0061] The dereverberation performance of the proposed MR-UNet network was evaluated using both simulated room impulse responses and real-world room recordings. The simulations were conducted using the Librispeech dataset, a large corpus of approximately 1,000 hours of English speech. The training portion of the corpus was divided into three subsets: 100 hours, 360 hours, and 500 hours. Recordings with higher quality and closer to American accents were selected for the first two subsets. The audio in the dataset was recognized using an acoustic model trained on the Wall Street Journal (WSJ) si-84 model, and the corresponding book text was estimated using a bigram language model. The word error rate (WER) was calculated based on the automatically translated text and reference text. Speakers in the dataset were ranked according to the translation results from the WSJ model, with speakers with lower WERs categorized as "clean" and those with higher WERs as "other." A development set of 20 male and 20 female speakers was randomly selected from the "clean" subset. The test set was constructed in the same manner as the development set. For speakers in the test or development set, approximately 8 minutes of audio is used per speaker. This totals approximately 5 hours and 20 minutes. The remaining spectra in the "clean" section are randomly divided into two training sets, approximately 100 hours and 360 hours respectively. To avoid imbalance in the speaking time of each speaker, the audio of each speaker is limited to approximately 25 minutes. The "other" section is similarly divided into test and development sets and a training set of approximately 500 hours. In the "other" section, the test and development sets are not randomly selected, but rather more challenging data is selected. The WER calculated by the WSJ model is ranked, and the speakers in the test and development sets are randomly selected from the bottom quarter of the ranking. Table 2 provides a brief description of all subsets in the dataset.

[0062] Table 2 Brief description of Librispeech subsets

[0063]

[0064] Based on the Librispeech dataset, we simulated the room impulse response (RIR) using the image source method to generate a small-scale reverberant Librispeech dataset. We randomly selected 7,000 clean speech sentences from Librispeech as the training set, 4,000 as the test set, and 7,000 as the validation set. The room settings and microphone-source positions for each clean speech sentence were randomized. The room length ranged from 3 to 10 meters, the width from 3 to 8 meters, and the height from 2.5 to 6 meters. The microphones and sound sources within the room were randomly selected within a range of greater than 0.3 meters from the walls. The distance between the sound source and the microphones ranged from 0.5 to 10 meters, and the reverberation time ranged from 0.4 to 1.2 seconds. Each sentence in the dataset was sampled at 16,000 Hz and truncated to 4 seconds in length. Table 8-2 shows the experimental settings for the simulated dataset.

[0065] In addition, the proposed MR-UNet was trained and verified on Libri-adhoc40. Libri-adhoc40 uses a speaker to play Librispeech corpus in an actual conference room scenario, and collects the data set generated by the replayed speech through 40 microphones. The conference room can be roughly regarded as a rectangular parallelepiped with a height of 9.8 meters, a width of 10.3 meters, and a height of 4.2 meters. The reverberation in the room is very severe, and the reverberation time T 60 The duration was approximately 0.9 seconds, with almost no additive noise. The distance between the microphone and the sound source ranged from 0.8 to 7.4 meters. The microphone was fixed in position in the room, and the speakers were placed at nine locations.

[0066] Table 3 Experimental settings of simulation datasets

[0067]

[0068] Notably, the speakers at each location were oriented differently, with one location featuring speakers playing from two opposite directions. The audio played was sourced from Librispeech's "train-clean-100," which contains a total of 251 speakers. Each location played audio from approximately 20 to 40 speakers. Three speaker positions in the validation and test sets shared the same positions and orientations as those in the training set. Five new positions were used in the validation and test sets to prevent duplication between the training and test sets, and vice versa. The audio played was sourced from Librispeech's "dev-clean" and "test-clean" files, each containing audio from 40 speakers. The undisturbed speech from Libri-adhoc40 was recorded in an anechoic chamber measuring 11.8 meters long, 4.2 meters wide, and 3.8 meters high. The distance between the microphone and the sound source was 0.4 meters, and the volume was set to the same level as in a conference room. Since the experiments in this paper primarily focus on single-channel dereverberation, the experimental dataset consists of reverberant speech recorded by one of the 40 microphones randomly selected for each sentence. Finally, the recorded data set used in the experiment of the present invention includes 28,540 sentences in the training set, 2,621 sentences in the test set, and 2,704 sentences in the validation set. The sampling rate of all sentences is 16,000 Hz.

[0069] Experimental setup:

[0070] For the speech signal with a sampling rate of 16k used in the experiment, a Hanning window with a frame length of 16 milliseconds and a frame shift of 8 milliseconds was used for windowing. Subsequently, a 256-point Fast Fourier Transform (FFT) was applied to each frame to make it contain 129 frequency points. The maximum-minimum normalization was used to compress its dynamic range to 0-1. The proposed MR-UNet was trained for 100 rounds with a batch size of 4. The optimizer used in the experiment was the Adam optimizer, with the parameter β1 set to 0.9 and the parameter β2 set to 0.999. The initial learning rate of the network was 2×10 -4 , gradually decays to the minimum learning rate 1×10 through the cosine annealing strategy -6 The warm-up process lasted for 5250 steps. Finally, the PESQ, STOI, and fwSegSNR indicators were used to evaluate the dereverberation effect.

[0071] Experimental results:

[0072] (1) Results of simulated reverberation dataset

[0073] The dereverberation results of the MR-UNet method proposed in this invention are evaluated on simulated reverberation datasets and real-recorded reverberation datasets.

[0074] A series of experiments were conducted on a simulated reverberation dataset to evaluate the proposed MR-UNet approach. The STOI, PESQ, and fwSegSNR scores of the proposed MR-UNet dereverberation algorithm were calculated for each sentence. For all metrics, higher scores indicate better speech quality. Table 1 lists the average STOI, PESQ, and fwSegSNR scores for all sentences on the simulated reverberation dataset. Compared to the reverberant speech, except for the LSTM dereverberation algorithm, which only improved on PESQ, all other dereverberation algorithms showed improvements in STOI, PESQ, and fwSegSNR. The proposed MR-UNet surpassed all baseline systems in all three dereverberation metrics. On average, compared to the unprocessed reverberant speech, MR-UNet achieved improvements of 0.184 dB in STOI, 0.61 dB in PESQ, and 4.65 dB in fwSegSNR. Compared to the best-performing U-Net dereverberation algorithm in the baseline system, the proposed algorithm achieved a relative improvement of 4.2% in STOI, 11.2% in PESQ, and 22% in fwSegSNR. Furthermore, it was observed that the U-Net network and its variants generally performed better in dereverberation than the LSTM network and its variants. The single-channel WPE algorithm was not significantly effective in dereverberation.

[0075] exist Figure 6 In Figure 3, a sample is randomly selected from the simulation dataset to intuitively demonstrate the effectiveness of our proposed MR-UNet algorithm for dereverberation. Figure 6 (a) is the amplitude spectrum of reverberant speech, Figure 6 (b) is the amplitude spectrum of the corresponding clean speech. Figure 6(c), (d), (e), (f), (g), and (h) show the amplitude spectra of the processed speech obtained by dereverberation using WPE (1 channel), LSTM, Late-LSTM, UNet, SkipConvNet, and MR-UNet, respectively. Clearly, MR-UNet performs best in suppressing the smearing effect caused by reverberation. Most of the other dereverberation algorithms are able to effectively remove reverberation. Specifically, the UNet algorithm achieves suboptimal performance, but also results in slight spectral leakage. Compared to the MR-UNet algorithm, the SkipConvNet algorithm does not restore its spectral structure very clearly. While the LSTM algorithm relatively clearly restores the spectral structure, it also introduces significant artifacts. The Late-LSTM algorithm performs poorly, possibly due to inappropriate sample selection. The WPE algorithm does not perform well in single-channel dereverberation. In general, although the proposed MR-UNet algorithm is still not satisfactory in the low-frequency part, it still eliminates most of the reverberation effects under the reverberation condition with relatively large reverberation, and has a good dereverberation effect.

[0076] (2) Ablation experiment under simulated reverberation dataset

[0077] To analyze the role of the various components of the proposed MR-UNet, ablation experiments were conducted to examine the number of different resolution blocks, whether information fusion and masking were used between different resolution blocks, UNet subnetworks, and how to combine multiple convolutional block subnetworks for optimal performance. Table 4 shows the average STOI, PESQ, and fwSegSNR scores for different numbers of resolution blocks on a simulated reverberation dataset. Bold numbers indicate the best performance.

[0078] Table 4 Average scores of STOI, PESQ, and fwSegSNR for different numbers of resolution blocks in the simulated reverberation dataset

[0079]

[0080] Table 4 compares the dereverberation performance of different numbers of resolution blocks in MR-UNet. Experimental results show that as the number of resolution blocks increases, the STOI, PESQ, and fwSegSNR scores improve. However, at the same time, the number of network parameters, and therefore the system complexity, increases. The results of using four resolution blocks in MR-UNet compared to using three resolution blocks only slightly improve the STOI and PESQ scores, and even slightly deteriorate the fwSegSNR score, while increasing the number of network parameters by 23%. Ultimately, we balanced dereverberation performance and system complexity and chose to use three resolution blocks in MR-UNet.

[0081] Table 5 Average scores of STOI, PESQ, and fwSegSNR under different network combinations of the simulated reverberation dataset.

[0082]

[0083] Table 5 shows the average STOI, PESQ, and fwSegSNR scores for different network combinations on a simulated reverberation dataset, demonstrating the dereverberation performance of different network combinations. Bold numbers indicate the best performance. For convenience, information fusion between blocks of different resolutions is referred to as "IF," mask blocks as "MB," and multi-convolutional blocks as "MCB." "MR-UNet-IF-MB" indicates no information fusion or mask blocks between blocks of different resolutions, "MR-UNet-IF" indicates no information fusion between blocks of different resolutions, "MR-UNet-MB" indicates no mask blocks, "MR-UNet" uses both information fusion and mask blocks, and "MR-UNet+MCB" replaces the U-Net with MCB. Experimental results show that the dereverberation performance of different network combinations is not significantly different. Specifically, the information fusion between blocks of different resolutions and the information interaction between blocks of different resolutions caused by the mask blocks have limited effects on dereverberation performance. Using U-Net as the dereverberation network for each sub-stage achieves slightly higher fwSegSNR scores than using MCB as the dereverberation network for each sub-stage, but slightly lower STOI and PESQ scores. In summary, the information interaction between blocks of different resolutions is beneficial to the final dereverberation performance, and MCB can be used as an alternative to UNet.

[0084] (3) Results on the recorded reverberation dataset

[0085] To verify the dereverberation performance of the proposed MR-UNet in real-world environments, experiments were conducted using the real-reverberation dataset, Libri-adhoc40. Table 6 lists the STOI, PESQ, and fwSegSNR scores of various dereverberation algorithms on this dataset. The table shows that the proposed MR-UNet achieves a 6.5% relative improvement in STOI over the 40-channel WPE of the next-best baseline system, the SkipCon-vNet, and a 15.7% relative improvement in fwSegSNR over the next-best baseline system, the SkipConvNet. Its PESQ is nearly on par with the best-performing SkipConvNet. This fully demonstrates that even in real-world environments with severe reverberation, our proposed dereverberation algorithm can still achieve excellent dereverberation performance. Compared to the dereverberation baseline system, the dereverberation performance of the U-Net network and its variants continues to outperform that of the LSTM network and its variants across all metrics, consistent with our findings on the simulated reverberation dataset. The dereverberation performance of the single-channel WPE algorithm is poor, but if multi-channel WPE is used, the dereverberation effect is significantly improved.

[0086] Table 6 Average scores of STOI, PESQ, and fwSegSNR for the recorded reverberation dataset.

[0087]

[0088] The speech recognition performance under the recorded reverberation dataset was also tested. The acoustic model of the speech recognition system used is the Conformer trained with 960 hours of annotated speech from the LibriSpeech dataset. The language model is the Transformer trained with 960 hours of annotated speech from LibriSpeech plus 800M words of plain text corpus. The decoding algorithm is the CTC-Attention joint decoding algorithm. The speech recognition model used here uses the word error rate (WER) as the evaluation indicator: given a test set S, the word error rate of the speech recognition model on S is defined as its recognition result. The average edit distance between the corresponding true label sequence y:

[0089]

[0090] in, represents the number of sentences in the test set, |y| represents the length of sequence y, and ED(p,q) represents the edit distance between p and q, which is the minimum number of insertions, substitutions, and deletions required to transform p into q. Lower WER indicates better performance.

[0091] Table 7 lists the WER results obtained by using the MR-UNet proposed in the present invention and various dereverberation baseline systems. The speech recognition system mentioned above uses MR-UNet. It can be seen from Table 6 that the MR-UNet proposed in the present invention achieves the lowest WER when only single-channel data is used. Compared with the best-performing single-channel dereverberation baseline system SkipConvNet, the WER is absolutely improved by 6.46%. Although the WER of MR-UNet is 2.43% higher than that of the WPE (40ch) algorithm, WPE (40ch) uses information from 40 channels, and the spatial information between channels may help to reduce the WER of the speech recognition system. This indirectly reflects the effectiveness of MR-UNet in single-channel speech dereverberation.

[0092] Table 7 WER of different speech dereverberation algorithms under the dataset of recorded reverberation

[0093]

[0094] The above-mentioned specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention and do not constitute a limitation of the present invention. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A multi-resolution UNet dereverberation network, characterized by: The multi-resolution UNet dereverberation network includes three resolution blocks, and the input features of each resolution block are time-frequency features of a specific resolution. The dereverberation network adopts an encoder-decoder structure with three resolution blocks called B1, B2 and B3, respectively. The resolution of the time-frequency features input by each resolution block is different. In resolution block B3, the input features are evenly divided into four parts along the time dimension. After each part is encoded by the convolution block and the encoder, they are spliced ​​along the time dimension in pairs. The spliced ​​features are sent to the decoder for decoding, and the output of the decoder is passed through the mask module to obtain the dereverberation result of B3; similarly, the input features in resolution block B2 are divided into two parts of the same size along the time dimension. After each part is spliced ​​along the time dimension after passing through the convolution block and the encoder, it is sent to the decoder for decoding, and then passed through the mask module to obtain the dereverberation result of B2; No segmentation operation is performed in resolution block B1. After passing through the encoder and decoder output directly, the dereverberation result of B1 is obtained by two-dimensional convolution. Through direct addition of information and channel splicing, the useful information output by the current resolution block is passed to the next resolution block, thereby gradually improving the dereverberation performance of the multi-resolution UNet dereverberation network.

2. The multi-resolution UNet dereverberation network according to claim 1, characterized in that: In the convolution block, the convolution block is regarded as a residual network composed of multiple two-dimensional convolution layers that extracts features in each resolution block; the amplitude spectrum of the reverberant speech is represented by X, and the output of CB is represented by Z, then: Z=X+W CB X (1) Where W CB Represents the weight matrix of the convolution block. The result of passing the input feature X through the two-dimensional convolution layer and the parameterized rectified linear unit and then through the output of the two-dimensional convolution layer is called X1. X1 is obtained by the global pooling layer to obtain X2. The formula is: Where H and W represent the dimensions of the spectrum, X1(i,j) represents the spectrum value at the time-frequency point (i,j); H GP It is a global pooling function that helps to represent all information of the entire spectrum; In order to make full use of the information after global pooling, the gate function and the Sigmoid function are used: X3=f(W U δ(W D X2)) (3) Where f(·) represents the Sigmoid activation function, δ(·) represents the linear rectified unit activation function, and W D represents the first 2D convolutional layer after global pooling, with downsampling rate r, W U represents the second two-dimensional convolutional layer after global pooling, with an upsampling rate of r. Finally, we get: Z=X+X1X3 (4) Where Z is the output result of the convolution block.

3. The multi-resolution UNet dereverberation network according to claim 1, characterized in that: In the masking module, the unprocessed amplitude spectrum of the current resolution block is added to the result of 1×1 convolution of the intermediate product to obtain the processed amplitude spectrum of the current stage. The processed amplitude spectrum of the current stage is masked by 1×1 convolution and Sigmoid activation function and the result of two-dimensional convolution of the intermediate product. The masked result will be channel-connected with the unprocessed amplitude spectrum of the next resolution block after residual connection. The advantage of the mask module is that it suppresses features with less information in the current resolution block and only allows useful features to propagate to the next resolution block. The "unprocessed amplitude spectrum of the current resolution block" and "unprocessed amplitude spectrum of the next resolution block" refer to the unprocessed amplitude spectrum of the reverberant speech of two adjacent resolution blocks, the "intermediate product" refers to the output of the decoder in the current resolution block, and the "processed amplitude spectrum" refers to the estimated amplitude spectrum of the dereverberated speech.

4. The multi-resolution UNet dereverberation network according to claim 1, characterized in that: In each resolution block, the U-Net network is used to stack the convolution blocks to form a multi-convolution block. The multi-convolution block does not contain downsampling operations, so there is no loss of spectral information. Each multi-convolution block contains 8 convolution blocks and a two-dimensional convolution layer. The input and output of the multi-convolution block are connected through residual connections, and 3 multi-convolution blocks are used instead of UNet.

5. The multi-resolution UNet dereverberation network according to claim 1, characterized in that: The loss function used by the multi-resolution UNet dereverberation network is: Among them L MR-UNet Represents the loss function of MR-UNet, R represents the resolution of a resolution block, represents the de-reverberated speech estimated by the resolution block, Y represents the amplitude spectrum of the direct sound speech, L R Expressed as: where ε is set to 10 -3 .

Images