A Speech Enhancement Network Signal-to-Noise Ratio Estimator and Loss Optimization Method

Abstract

The present invention discloses a speech enhancement network signal-to-noise ratio estimator, which includes an encoder and a decoder. A complex-numbered CNN, a complex-numbered LSTM, and a complex-numbered BN layer are arranged between the encoder and the decoder. The encoder includes a complex Conv2D layer, a complex BN layer, and a real-numbered PReLU layer. A 1-D convolution module is arranged after the LSTM layer. The 1-D convolution module is composed of a plurality of one-dimensional convolution layers and a fully connected layer alternately connected in series. The fully connected layer has a sigmoid function. The signal-to-noise ratio estimator adopts a serial combination of two one-dimensional convolution layers and a fully connected layer with a sigmoid function. The input is the splicing of the real part and the imaginary part of the noisy speech signal calculated by the complex LSTM. The output is a frame-level priori signal-to-noise ratio calculated according to a formula, so as to maintain good speech quality.

Description

Technical Field

[0001] The present invention belongs to the technical field related to speech enhancement technology, and particularly relates to a speech enhancement network signal-to-noise ratio estimator and a loss optimization method. Background Art

[0002] In real-world acoustic scenarios, speech signals are subject to varying degrees of noise contamination. Severe levels of contamination can severely degrade speech quality. The goal of speech enhancement is to restore the original, clean speech signal from noisy speech, thereby improving speech intelligibility and perceived quality by suppressing background noise interference. Speech enhancement technology is widely used in modern voice communication systems, such as automatic speech recognition, mobile communications, and conferencing systems.

[0003] Currently, speech enhancement algorithms are primarily categorized as traditional algorithms and deep learning-based algorithms. Traditional speech enhancement algorithms have low computational cost and hardware requirements, achieving good results in removing stationary noise. However, they struggle with non-stationary noise and low signal-to-noise ratios. In recent years, deep learning-based speech enhancement algorithms have achieved very promising results. These methods are generally considered supervised learning tasks, enhancing noisy speech in the time-frequency domain or directly in the time domain.

[0004] However, deep learning-based noise reduction methods inevitably cause speech distortion while reducing noise. When enhancing noisy speech with a high signal-to-noise ratio, the network will cause more severe attenuation, even worse than the original noisy speech. Summary of the Invention

[0005] The object of the present invention is to provide a speech enhancement network signal-to-noise ratio estimator and loss optimization method to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] Compared with the prior art, the present invention provides a speech enhancement network signal-to-noise ratio estimator, comprising an encoder and a decoder, wherein a complex-numbered CNN, a complex-numbered LSTM, and a complex-numbered BN layer are arranged between the encoder and the decoder, and the encoder comprises a complex Conv2D layer, a complex BN layer, and a real-numbered PReLU layer;

[0008] A signal-to-noise ratio estimator is set after the LSTM layer. The signal-to-noise ratio estimator is composed of a series of alternating one-dimensional convolutional layers and fully connected layers. The fully connected layers have a sigmoid function.

[0009] The process of calculating the prior signal-to-noise ratio using the 1-D convolution module is as follows:

[0010]

[0011] Where PSNR represents the prior signal-to-noise ratio, A clean represents the clean speech spectrum, A noise Represents the noise spectrum. Normalizing the PSNR, we can get:

[0012]

[0013] in represents the standard deviation of PSNR, and σ represents the mean of PSNR.

[0014] After that, ERFC (Complementary Error Function) is used to compress the PSNR to between 0 and 1

[0015] PSNR=(erfc(PSNR+1)) / 2.

[0016] 2. Preferably, the Conv2D operation is implemented by a complex-valued filter, and the complex-valued filter W is defined as:

[0017] W=W r +jW i

[0018] Real matrix W r and W i Represents the real and imaginary parts of a complex convolution kernel.

[0019] Define an input complex matrix X = X r +jX i , the complex output of the complex convolution operation is:

[0020]

[0021] Among them, F out Represents the output features of a complex layer.

[0022] 3. According to the above-mentioned method for optimizing network loss for speech enhancement, the method comprises the following steps:

[0023] S1: Use the dynamic mixed speech-noise method to create speech datasets with and without reverberation. The signal-to-noise ratio of these speech is distributed between 0 dB and 20 dB.

[0024] S2: Input the speech produced in S1 into the DCCRN speech enhancement model with signal-to-noise ratio estimator;

[0025] S3: The signal-to-noise ratio estimator calculates the prior signal-to-noise ratio Lable_snr and subtracts it from the SI-SNR to obtain the loss value. The SI-SNR calculation formula is as follows:

[0026]

[0027] s represents the time domain waveform of clean speech, represents the time domain waveform of the estimated speech, <·,·> represents the dot product between two vectors, and ‖·‖2 represents the Euclidean distance;

[0028]

[0029] Loss = (-1) × Loss SI-SNR +2×Loss SNR

[0030] Among them, Loss SNR The coefficient of is set to 2 in order to unify the scales of the two loss functions;

[0031] S4: The priori signal-to-noise ratio calculated by the signal-to-noise ratio estimator is decoded and output by the decoder to obtain enhanced speech.

[0032] A speech enhancement network signal-to-noise ratio estimator and loss optimization method has the following beneficial effects:

[0033] 1. The speech enhancement model of the present invention effectively combines the advantages of DCUNET and CRN. The complex network used between the encoder and decoder can model the correlation between amplitude and phase through complex multiplication, thereby better utilizing the phase information of the speech.

[0034] 2. The present invention designs a signal-to-noise ratio estimator after the two LSTM layers and redesigns the loss function in the speech enhancement algorithm of the signal-to-noise ratio estimator, so that the frame-level prior signal-to-noise ratio can be calculated, maintaining good speech quality and improving the speech attenuation problem caused by the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0036] Figure 1 A schematic diagram of a DCCRN speech enhancement model provided with a speech enhancement network signal-to-noise ratio estimator proposed by the present invention;

[0037] Figure 2 This is a schematic diagram of the signal-to-noise ratio estimator structure proposed in the present invention.

[0038] In the figure: 10, encoder; 11, complex Conv2D layer; 12, complex BN layer; 13, real PReLU layer; 20, decoder; 30, complex CNN; 40, complex LSTM; 50, complex BN layer; 60, 1-D convolution module; 61, one-dimensional convolution layer; 62, fully connected layer. DETAILED DESCRIPTION

[0039] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0040] The present invention provides a speech enhancement network signal-to-noise ratio estimator, which is arranged in Figure 1 The DCCRN speech enhancement model shown.

[0041] The DCCRN speech enhancement model includes an encoder 10 and a decoder 20, with a complex CNN 30, a complex LSTM 40, and a complex BN layer 50 disposed between the encoder 10 and the decoder 20. The encoder 10 includes a complex Conv2D layer 11, a complex BN layer 12, and a real PReLU layer 13. The complex Conv2D layer 11 includes four traditional Conv2D operations to control the flow of complex information in the encoder.

[0042] The complex-valued filter W is defined as:

[0043] W=W r +jW i

[0044] Real matrix W r and W i Represents the real and imaginary parts of a complex convolution kernel.

[0045] Define an input complex matrix X = X r +jX i , the complex output of the complex convolution operation is:

[0046]

[0047] Among them, F out Represents the output features of a complex layer.

[0048] The DCCRN speech enhancement model uses a complex network and can model the correlation between amplitude and phase through complex multiplication, thereby better utilizing the phase information of the speech.

[0049] refer to Figure 2 Two LSTM layers 40 are arranged between the encoder 10 and the decoder 20, and a signal-to-noise ratio estimator 60 is arranged after the LSTM layer 40. The signal-to-noise ratio estimator 60 is composed of a plurality of one-dimensional convolutional layers 61 and a fully connected layer 62 alternately combined in series, and the fully connected layer 62 has a sigmoid function.

[0050] The concatenated signal of the real part and the imaginary part of the noisy speech signal calculated by the LSTM layer 40 is input into the signal-to-noise ratio estimator 60. After processing and calculation by the signal-to-noise ratio estimator 60, a frame-level priori signal-to-noise ratio can be obtained to maintain good speech quality.

[0051] The process of calculating the a priori signal-to-noise ratio by the signal-to-noise ratio estimator 60 is as follows:

[0052]

[0053] Where PSNR represents the prior signal-to-noise ratio, A clean represents the clean speech spectrum, A noise Represents the noise spectrum. Normalizing the PSNR, we can get:

[0054]

[0055] in represents the standard deviation of PSNR, and σ represents the mean of PSNR.

[0056] After that, ERFC (Complementary Error Function) is used to compress the PSNR to between 0 and 1

[0057] PSNR = (erfc(PSNR+1)) / 2

[0058] A method for optimizing speech enhancement network loss comprises the following steps:

[0059] S1: Use the dynamic mixed speech-noise method to create speech datasets with and without reverberation. The signal-to-noise ratio of these speech is distributed between 0 dB and 20 dB.

[0060] S2: Input the speech produced in S1 into the DCCRN speech enhancement model with the signal-to-noise ratio estimator 60;

[0061] S3: The signal-to-noise ratio estimator 60 calculates the a priori signal-to-noise ratio Lable_snr and subtracts it from the SI-SNR to obtain a loss value. The SI-SNR calculation formula is as follows:

[0062]

[0063] Among them, s represents the time domain waveform of clean speech, represents the time domain waveform of the estimated speech, <·,·> represents the dot product between two vectors, and ‖·‖2 represents the Euclidean distance

[0064] The a priori signal-to-noise ratio calculation formula is as follows:

[0065]

[0066] Loss = (-1) × Loss SI-SNR +2×Loss SNR

[0067] Loss SNR The coefficient of is set to 2 in order to unify the scale of the loss function of the two parts;

[0068] S4: The a priori signal-to-noise ratio calculated by the signal-to-noise ratio estimator 60 is decoded and output by the decoder 20 to obtain enhanced speech.

[0069] In order to fully verify the effect of the loss function used by the signal-to-noise ratio estimator 60 on improving speech quality, the present invention uses the dataset provided by the DNS Challenge (INTERSPEECH2020) to train and evaluate the proposed model. The provided corpus contains 500 hours of clean speech clips from 2150 speakers, 180 hours of noise datasets from 150 categories, and room impulse responses from the openSLR26 and openSLR28 datasets. The DNS Challenge also provides a publicly available test dataset, including two types of synthetic clips, the first one with reverberation and the other without reverberation, with 150 clips in each category, and the signal-to-noise ratio is distributed between 0 decibels and 20 decibels.

[0070] To diversify the data, we used a dynamic speech-noise mixing method. We created a 50-hour dataset with and without reverberation, totaling 100 hours. We then mixed clean speech and noise at a random signal-to-noise ratio (SNR) of -5dB to 20dB. To generate reverberant speech, we convolved random samples of speech with the RIR_noises dataset, which simulates room reverberation. Ultimately, we obtained 60,000 6-second speech clips, all sampled at 16kHz.

[0071] In the test model of the present invention, the frame size is set to 400 points, the hop size is set to 256 points, and the FFT length is 512 points. The number of channels of the encoder is {32, 64, 128, 128, 256, 256}, the convolution kernel is set to (5, 2), and the step size is set to (2, 1). In the one-dimensional convolution module, the input channel is set to 4, and the convolution kernel is set to (32, 1). After the last LSTM layer is a dense layer of 1024 units. For the decoder, each convolution layer looks back one frame, a total of 6 frames, for a total of 6x6.25=37.5ms. The model optimizer uses the Adam optimizer.

[0072] This paper uses the PyTorch deep learning framework to train and test the model. The model's input is a 100-second sequence, and the training process is repeated for 100 epochs. The initial learning rate is set to 0.001. If the model fails to improve on the validation set for two consecutive epochs, the learning rate is reduced by one-third. Training terminates if the model fails to improve on the validation set for 10 consecutive epochs.

[0073] As shown in Table 1, Model 1 (Proposed Model 1) corresponds to adding a signal-to-noise ratio estimator after the LSTM layer, and Model 2 (Proposed Model 2) adds the new loss function provided by the present invention based on Model 1.

[0074]

[0075] Table 1 Objective evaluation results of DNS Challenge synthetic development set

[0076] In Table 1, PESQ is an objective assessment of speech quality, STOI is an assessment of speech intelligibility, and SI-SDR is the scale-invariant signal-to-distortion ratio. Higher scores for these metrics indicate better speech quality and a more effective model. Table 1 shows that the proposed method outperforms other methods in all three metrics, both with and without reverberation. Model 2 achieves a 4.62% improvement in PESQ over DCCRN in the unaural state, and 1.53% and 1.77dB improvements over NSnet in STOI and SI-SDR, respectively. In the presence of reverberation, Model 2 achieves an 8.22% improvement in PESQ over DCCRN, a 7.82% improvement in STOI over DTLN, and a 5.927dB improvement in SI-SDR over NSnet.

[0077] Table 2 DNS Challenge blind test set MOS test results

[0078] Model no reverb reverb realrec Ave. Noisy 3.13 2.64 2.83 2.85 NSNet 3.49 2.64 3.00 3.03 DCCRN-E 4.00 2.94 3.37 3.42 Proposed model 1 3.95 2.98 3.42 3.45 Proposed model 2 4.03 3.02 3.57 3.54

[0079] Table 2 shows the test results on the blind test set provided by DNS Challenge. It can be seen that the MOS score of the proposed model under the condition of no reverberation is 15.47% higher than that of NSnet. The MOS scores under the condition of reverberation and real recording are 2.72% and 5.93% higher than those of DCCRN respectively.

[0080] Tables 3-5 show the objective evaluation results of the second test set under undisturbed conditions. As can be seen, the proposed model achieves some improvement in the test results for five typical signal-to-noise ratios. Under undisturbed conditions, the proposed model achieves improvements of 2.05%, 0.47%, and 0.796dB in PESQ, STOI, and SI-SDR, respectively, surpassing the best results of DCCRN-E.

[0081] Table 3 PESQ test results of the undisturbed noisy speech test set

[0082] test SNR 0dB 5dB 10dB 15dB 20dB Ave. Noisy 1.789 2.132 2.47 2.833 3.176 2.48 DCCRN-E 2.704 3.075 3.379 3.628 3.814 3.32 Proposed model 2 2.805 3.137 3.398 3.735 3.867 3.388

[0083] Table 4 STOI test results of the undisturbed noisy speech test set

[0084] test SNR 0dB 5dB 10dB 15dB 20dB Ave. Noisy 77.55 84.78 90.44 94.34 96.76 88.77 DCCRN-E 87.44 92.31 95.31 97.13 98.18 94.07 Proposed model 2 89.07 92.33 95.53 97.43 98.24 94.52

[0085] Table 5 SI-SDR test results of the undisturbed noisy speech test set

[0086] test SNR 0dB 5dB 10dB 15dB 20dB Ave. Noisy 0.005 5.003 10.003 14.899 20.001 9.982 DCCRN-E 11.19 14.905 18.22 21.343 24.039 17.939 Proposed model 2 12.181 15.905 19.208 21.896 24.487 18.735

[0087] In summary, the model proposed in this invention, which incorporates a signal-to-noise ratio estimator, focuses on improving the degree to which neural networks impair speech perception quality. Experimental results and indicator scores indicate that the proposed model outperforms other models, demonstrating the effectiveness of the signal-to-noise ratio estimator 60 using the a priori signal-to-noise ratio calculation method provided by this invention.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A speech enhancement network signal-to-noise ratio estimator, comprising an encoder (10) and a decoder (20), characterized in that: A complex number operation CNN (30), a complex number LSTM layer (40) and a complex number BN layer (50) are provided between the encoder (10) and the decoder (20), and the encoder (10) includes a complex Conv2D layer (11), a complex BN layer (12) and a real number PReLU layer (13); A signal-to-noise ratio estimator (60) is provided after the LSTM layer (40), wherein the signal-to-noise ratio estimator (60) is composed of a plurality of one-dimensional convolutional layers (61) and a fully connected layer (62) alternately connected in series, wherein the fully connected layer (62) has a sigmoid function; The process of calculating the priori signal-to-noise ratio by the signal-to-noise ratio estimator (60) is as follows: ； Where PSNR represents the prior signal-to-noise ratio, A clean represents the clean speech spectrum, A noise Represents the noise spectrum; normalizing the PSNR, we can get: ； in represents the standard deviation of PSNR, represents the mean of PSRN; After that, ERFC (Complementary Error Function) is used to compress the PSNR to between 0 and 1 。 2. The speech enhancement network signal-to-noise ratio estimator according to claim 1, wherein: The Conv2D operation is implemented by a complex-valued filter, where W is defined as: ； Real matrix W r and W i Represents the real and imaginary parts of a complex convolution kernel; Define an input complex matrix X=X r +jX i , the complex output of the complex convolution operation is: ； Among them, F out Represents the output features of a complex layer.

3. The loss optimization method of the speech enhancement network signal-to-noise ratio estimator according to any one of claims 1 or 2, comprising the following steps: S1: Use the dynamic mixed speech-noise method to create speech datasets with and without reverberation. The signal-to-noise ratio of these speech is distributed between 0 dB and 20 dB. S2: Input the speech produced in S1 into the DCCRN speech enhancement model with the signal-to-noise ratio estimator (60); S3: The signal-to-noise ratio estimator (60) calculates the prior signal-to-noise ratio Lable_snr and subtracts it from the SI-SNR to obtain the loss value. The SI-SNR calculation formula is as follows: ； represents the time domain waveform of clean speech, represents the time domain waveform of the estimated speech, represents the dot product between two vectors, represents the Euclidean distance; The calculation formula of the prior signal-to-noise ratio Lable_SNR in the loss function is as follows: ； Among them, Loss SNR The coefficient of is set to 2 in order to unify the scales of the two loss functions; S4: The priori signal-to-noise ratio calculated by the signal-to-noise ratio estimator (60) is decoded and output by the decoder (20) to obtain enhanced speech.

Images