A speech enhancement method for low signal-to-noise ratio audio signals

Abstract

The application provides a speech enhancement method for a low signal-to-noise ratio audio signal, comprising the following steps: a first stage training: S1, inputting a spectrum of the audio signal into a codec network to obtain a first stage enhanced speech; S4, calculating an acoustic parameter loss; S5, calculating a fine-grained spectrum feature loss; S6, combining the two losses and a frequency domain loss to construct a first stage loss for training, and obtaining a first stage enhancer; a second stage training: S7, obtaining features by an encoder and calculating a contrastive learning loss based on the S1 output, pure speech and residual noise; S9, constructing a second stage loss based on the first stage loss and the contrastive learning loss for second stage training, and obtaining a second stage enhancer; an inference stage: S10, inputting a noisy audio into the two-stage enhancer connected in sequence to obtain a target speech. The application considers the influence of acoustic parameter features and fine-grained spectrum features on the enhanced speech signal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of marine communication, in particular, especially relates to a speech enhancement method for low signal-to-noise ratio audio signals. BACKGROUND

[0002] Marine communication audio signals are one of the most critical data recorded in ship voyage recorders and are the main communication method during ship operation, which are widely used for voice and data transmission between ships and shores. However, during marine communication of ships, marine communication audio signals are often affected by marine environment and complexity of ship navigation, resulting in various noises mixed in the voice. Wind and wave sound in the marine environment, mechanical noise of the ship engine, and electromagnetic interference of other electronic devices all interfere with the VHF voice signal. Under the combined action of these factors, the signal-to-noise ratio of marine communication audio signals is usually low, which affects the intelligibility and clarity of the voice. Therefore, enhancing marine communication audio signals is directly related to the efficiency of ship-shore cooperation and maritime supervision, as well as the timely transmission of search and rescue information.

[0003] Speech enhancement has been developed for a long time. Traditional speech enhancement methods, such as minimum mean square error filtering and spectral subtraction filtering, improved signal subspace combined with Wiener filtering, and Kalman filtering algorithm, have certain effects, but their performance is not satisfactory when facing invisible noise or low signal-to-noise ratio. The fundamental reason is that traditional methods rely too much on prior assumptions. If the noise assumption is underestimated, noise residues will occur, and spectral peaks will appear, forming artificial noise. Overestimating the noise will destroy the original speech information and cause speech distortion. Deep learning methods based on data-driven include time-domain waveform mapping, spectral mapping, and mask generation methods. Time-domain waveform mapping method refers to inputting audio waveform into the model, and the model directly outputs pure speech waveform; spectral mapping method transforms time-domain input to frequency domain, and uses spectral mapping method to process enhancement task; the output of the mask generation method is a mask value, and the enhanced speech is obtained based on the mask and the spectrum. Deep learning methods do not need prior assumptions, and have high robustness to complex noise, and have gradually become the mainstream method. However, the existing deep learning methods have deficiencies in modeling the acoustic features and fine-grained spectrum of speech, and also have problems of excessive suppression and noise residue due to poor decoupling ability. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a speech enhancement method for low signal-to-noise ratio audio signals to solve the technical problem of noise residue in existing speech enhancement methods.

[0005] The technical means adopted by the present application are as follows:

[0006] A speech enhancement method of a low signal-to-noise ratio audio signal, comprising the following steps:

[0007] First stage training:

[0008] S1, input the short-time Fourier transform spectrum of the audio signal into the complex encoder network, extract the real and imaginary part features, and obtain the output of the encoder network, i.e., the intermediate feature;

[0009] S2, input the intermediate feature into the Conformer module, and the Conformer module models the timing information using the attention network to obtain the output of the Conformer module;

[0010] S3, splice the intermediate feature and the output of the Conformer module, input into the complex decoder network to obtain the frequency domain output, and perform inverse Fourier transform on the frequency domain output to obtain the first stage time domain enhanced speech;

[0011] S4, input the pure speech in the audio signal and the first stage time domain enhanced speech into the acoustic parameter extractor to obtain the embedding of the pure speech and the first stage time domain enhanced speech, calculate the root mean square error of the embedding of the pure speech and the first stage time domain enhanced speech, and construct an acoustic parameter loss function;

[0012] S5, input the pure speech in the audio signal and the first stage time domain enhanced speech into the fine-grained spectral feature extractor, extract the penultimate layer output of the fine-grained spectral feature extractor network as the fine-grained spectral feature, align the fine-grained spectral feature using the root mean square error loss, and construct a fine-grained feature loss function;

[0013] S6, based on the acoustic parameter loss function, the fine-grained feature loss function and the spectral loss, construct the loss function of the first stage, use the loss function of the first stage for first stage training, and output the trained first stage enhancer;

[0014] The training process is as follows: calculate the gradient of the first stage loss function on the network parameters, subtract the product of the gradient and the learning rate from the network parameters, repeatedly update the network parameters to reduce the loss, and terminate the training when the updated network parameters have no performance improvement on the validation set for 20 consecutive iterations;

[0015] Second stage training:

[0016] S7, obtain features by encoding the first stage time domain enhanced speech, the pure speech and the residual noise respectively;

[0017] S8, construct a triple based on the features obtained in S7, input the triple into a contrast learning method to obtain a contrast learning loss function;

[0018] S9, based on the fine-grained feature loss function, the contrastive learning loss function and L log-PCM constructing a loss function of the second stage, using the loss function of the second stage for second stage training, and outputting the trained second stage enhancer;

[0019] Speech enhancement:

[0020] S10, inputting the audio to be enhanced into the trained first stage enhancer, outputting the first stage enhanced audio, inputting the first stage enhanced audio into the trained second stage enhancer, and outputting the enhanced target audio.

[0021] Further, S1 specifically includes the following steps:

[0022] performing short-time Fourier transform on the audio signal to obtain real and imaginary parts;

[0023] extracting acoustic parameter features and fine-grained spectral features in the enhanced speech using the feature consistency module; inputting the obtained real and imaginary parts into a complex encoder network; the complex encoder network includes a complex convolution layer, a complex batch normalization and a PReLU activation function; the complex convolution layer can simultaneously convolve the real and imaginary parts, and the real and imaginary parts are simultaneously processed, and the specific representation calculation process is as follows:

[0024]

[0025] wherein, A k is the input of the complex two-dimensional convolution, W r and W i are the weights of the real part and the weights of the imaginary part of the complex convolution kernel, and are the real part and the imaginary part of the input signal;

[0026] A k After batch normalization processing and PReLU activation function of the complex coding layer, the corresponding encoder network output F k is obtained, and the formula is as follows:

[0027] F k =PReLU(Batchnorm2d(A k )).

[0028] Further, in S2, the formula for the Conformer module to model the timing information using the attention network is as follows:

[0029]

[0030] wherein, H is the output of the Conformer module, is the real part, is the imaginary part.

[0031] Further, in S3, the frequency domain output is represented as:

[0032] O = Decoder(concatenate(F k , H)).

[0033] Further, in S4, the acoustic parameter loss function and in S5, the fine-grained feature loss function are as follows:

[0034]

[0035] where T is the acoustic parameter embedding corresponding to the audio signal in the training set, is the acoustic parameter embedding after the acoustic parameter extractor, F is the fine-grained spectral embedding corresponding to the audio signal in the training set, is the fine-grained spectral embedding after the acoustic parameter extractor.

[0036] Further, in S7, the formula for obtaining the feature through the encoder is as follows:

[0037]

[0038] where S is the output of the first stage, X is the pure signal, w, w + and w - represent the first stage output feature, pure audio feature and residual noise feature extracted after the encoder, respectively.

[0039] Further, the loss function of the first stage is:

[0040] L PFL-stage1 = L log-PCM + aL TAP + bL FM

[0041] The loss function of the second stage is:

[0042] L PFL-stage2 = L log-PCM + aL TAP + bL FM + lL CL

[0043] where a, b, l are hyperparameters;

[0044] L log-PCM is:

[0045]

[0046] where S(t,f) and are the coefficients of the STFT of s and respectively, T is the number of time frames, F is the number of frequency points; r and i represent the real and imaginary parts of a complex number respectively; L SM is the average absolute error loss between the clean short-time Fourier transform coefficients and the estimated short-time Fourier transform coefficients;

[0047] The contrast loss function is as follows:

[0048]

[0049] where N is the number of samples, tau is the temperature function, and the triplets are [w,w + ,w - ], in which [w,w + ] is a positive sample pair and [w,w - ] is a negative sample pair.

[0050] The application further provides a storage medium, which comprises a stored program, wherein the program performs any of the above voice enhancement methods for a low signal-to-noise ratio audio signal when running.

[0051] The application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor performs any of the above voice enhancement methods for a low signal-to-noise ratio audio signal by running the computer program.

[0052] Compared with the prior art, the application has the following advantages:

[0053] The application considers the influence of acoustic parameter features and fine-grained spectral features of a signal on the construction of an enhanced voice signal. The real part and the imaginary part of a voice signal are simultaneously modeled by a complex encoder-decoder network, the time sequence features of the voice signal are modeled by a Conformer module, and the voice is reconstructed by a complex decoder network.

[0054] The feature consistency module proposed in the method can model 25 acoustic features such as fundamental frequency (F0), jitter, center frequency and bandwidth of the first three formants through the acoustic parameter modeling part, and can capture the tiny spectral details in the audio signal through the fine-grained spectral feature module, so that better voice enhancement effect is achieved.

[0055] The contrast learning module proposed by the method improves the decoupling ability of the model, thereby realizing the removal of residual noise and the correction of excessive suppression. In addition, the loss function proposed by the method combines the phase-constrained frequency domain loss, acoustic parameter loss, fine-grained spectral feature loss and contrast learning loss to guide the network to update the parameters, and obtain better network parameters and enhancement effect. The experimental results show that the enhancement effect of the speech signal of the present application is obviously higher than that of other methods, which fully proves the effectiveness of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0057] Figure 1 The method flowchart of the present application.

[0058] Figure 2 The enhanced audio time domain waveform diagram of the present application before enhancement.

[0059] Figure 3 The enhanced audio spectrum diagram of the present application before enhancement.

[0060] Figure 4 The enhanced audio time domain waveform diagram of the present application after enhancement.

[0061] Figure 5 The enhanced audio spectrum diagram of the present application after enhancement. DETAILED DESCRIPTION

[0062] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0063] It is to be understood that the terms "first", "second", and the like, used in the description and the claims of the application, as well as the above-described drawings, are used to distinguish between similar objects and are not necessarily used to describe a particular sequential or chronological order. It is to be understood that the use of data herein is not limited to any particular order, and that the embodiments of the application described herein can be practiced in any order other than the one illustrated or described herein. Furthermore, the terms "comprise", "comprising", and "comprises", and any variation thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a list of steps or units can not necessarily be limited to those steps or units which are clearly recited, but can include other not expressly recited steps or units.

[0064] As shown in Figure 1 , the application provides a speech enhancement method for a low signal-to-noise ratio audio signal, comprising the following steps:

[0065] First, the audio signal is subjected to a short-time Fourier transform to obtain real and imaginary parts.

[0066] The application adopts a segmented training, and in the first stage, the feature consistency module is used to align the acoustic parameter features and fine-grained spectral features in the enhanced speech. The obtained real and imaginary parts are input into a complex encoder network to enable the model to better reconstruct the real and imaginary parts of the speech. The complex encoder network is composed of a complex convolution layer, a complex batch normalization, and a PReLU activation function. The complex convolution layer capable of simultaneously performing convolution on the real and imaginary parts processes the real and imaginary parts of the speech at the same time, more effectively capturing and processing the complex features of the speech signal. If A k is the output of the complex two-dimensional convolution, the specific representation calculation process is as follows:

[0067]

[0068] where W r and W i represent the weights of the real part and the imaginary part of the complex convolution kernel, respectively, and are the real and imaginary parts of the input signal, respectively. Then, A k is subjected to batch normalization processing and PReLU activation function of the complex encoding layer to obtain the corresponding encoder network output F k . The formula is as follows:

[0069] F k = PReLU(Batchnorm2d(A k )) (2)

[0070] The intermediate features extracted by the complex encoder are sent to the Conformer module, which uses its attention mechanism to model the temporal information, and its output can be represented as:

[0071]

[0072] Then, the output F of the encoder is concatenated with the output H of the Conformer and input into the decoder, and the frequency domain output O of the decoder can be represented as: k

[0073] O = Decoder(concatenate(F k ,H)) (4)

[0074] The frequency domain output O is inverse Fourier transformed to obtain the first stage time domain enhanced speech S.

[0075] Then, the clean speech and the enhanced speech are input into the acoustic parameter extractor to obtain the embeddings of the clean speech and the enhanced speech, respectively, and the root mean square error is calculated to reduce the distance between the embeddings to ensure the consistency of the acoustic parameters between the enhanced speech and the clean speech. At the same time, the clean speech and the enhanced speech are input into the fine-grained spectral feature extractor, and the output of the second-to-last layer of the network is extracted as the fine-grained spectral feature. The fine-grained spectral features are aligned using the root mean square error loss, and the acoustic parameter loss function and the fine-grained spectral feature loss function can be represented as:

[0076]

[0077] where T is the acoustic parameter embedding corresponding to the audio signal in the training set, is the acoustic parameter embedding obtained by the acoustic parameter extractor, F is the fine-grained spectral embedding corresponding to the audio signal in the training set, and F is the fine-grained spectral embedding obtained by the fine-grained spectral feature extractor.

[0078] The fine-grained spectral feature extractor network is sequentially connected by 16 convolutional layers and 3 fully connected layers.

[0079] The second stage introduces contrastive learning, and the output of the first stage, the clean audio and the residual noise are also obtained by the Encoder with shared weights to obtain the features, which can be represented as:

[0080]

[0081] where R is the residual noise, S is the output of the first stage, X is the clean signal, w, w + and w - represent the first stage output feature, clean audio feature and residual noise feature extracted by the encoder, respectively. ​

[0082] A triple [w, w + , w - ] is constructed for each sample, where [w, w + ] is the positive sample pair and [w, w - ] is the negative sample pair. Contrastive learning usually maximizes the consistency between positive samples and minimizes the consistency between negative samples in the learned representation space using a contrastive loss function, thereby enhancing the decoupling ability of the model, as shown in the following formula:

[0083]

[0084] During model training, the loss function used is proposed in the present application, L PFL , which uses L log-PCM that can limit the phase on the main body, and adds a feature consistency module that can model the acoustic parameters and fine-grained spectral features of the audio, and finally adds a contrastive learning loss function to improve the decoupling ability of the model.

[0085] Therefore, the loss function of the first stage is defined as:

[0086] L PFL-stage1 =L log-PCM +αL TAP +βL FM (8)

[0087] The loss function of the second stage is defined as:

[0088] L PFL-stage2 =L log-PCM +αL TAP +βL FM +λL CL (9)

[0089] Where α, β, λ are hyperparameters, which are set to 0.1, 0.6 and 1.8 respectively here.

[0090] L log-PCM can be defined as:

[0091]

[0092] Where S(t,f) and are the coefficients of the STFT of s and , T is the number of time frames, and F is the frequency interval. Subscripts r and i represent the real and imaginary parts of a complex number, respectively. L SM is the average absolute error loss between the pure short-time Fourier transform coefficient and the estimated short-time Fourier transform coefficient.

[0093] The network structure used in the two-stage training is a complex encoder-Conformer-complex decoder structure, wherein the encoder and the decoder each comprise six complex convolution layers, and the corresponding channel numbers of each layer are set to (32, 64, 128, 256, 256, 256). The complex encoder and the complex decoder are connected by two layers of Conformer modules, and an eight-head attention network is used. The training processes of the first stage and the second stage are the same, and the training process is as follows: the gradient of the loss function to the network parameters is calculated, the network parameters are updated by subtracting the product of the gradient and the learning rate, the network parameters are repeatedly updated to reduce the loss, and when the updated network parameters do not have performance improvement on the validation set for 20 consecutive iterations, the training is terminated.

[0094] All audio data is 16kHz. The window function uses a Hamming window, and the Fourier transform point number is 512. The learning rate in the experiment is set to 0.001, and the Adam optimizer is used. The epoch is set to 30, and the batch size used is 1. The server version used for training the model is Ubuntu 18.04.6, the method is implemented using python 3.8, and the framework uses pytorch. If the training result weight does not have performance improvement on the validation set for 20 consecutive iterations, the training is terminated.

[0095] Figure 2 The enhanced pre-audio time domain waveform diagram of the application is shown. It can be found by observation that the pre-enhanced speech contains a large amount of noise, which is caused by wind and wave noise, electromagnetic interference noise and mechanical noise, so that the signal-to-noise ratio of the speech is very low. Figure 3 The enhanced pre-audio spectrum diagram of the application is shown. The speech spectrum in the pre-enhanced audio spectrum diagram is covered by the noise spectrum, and the harmonic structure cannot be captured. Figure 4 The enhanced post-audio time domain waveform diagram of the application is shown. The noise in the enhanced post-audio is suppressed, the speech content of the speaker is retained in the speech waveform diagram, and the signal-to-noise ratio is improved. Figure 5 The enhanced post-audio spectrum diagram of the application is shown. The harmonic structure in the enhanced post-audio spectrum diagram is clear.

[0096] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A method for speech enhancement of low signal-to-noise ratio audio signals, characterized in that, Includes the following steps: Phase One Training: S1. Input the short-time Fourier transform spectrum of the audio signal into the complex encoder network, extract the real and imaginary part features, and obtain the output of the encoder network, i.e., the intermediate features. S2. Input the intermediate features into the Conformer module. The Conformer module uses the attention network to model the temporal information and obtains the output of the Conformer module. S3. The intermediate features and the output of the Conformer module are concatenated and then input into the complex decoder network to obtain the frequency domain output. The frequency domain output is then subjected to inverse Fourier transform to obtain the first stage time-domain enhanced speech. S4. Input the clean speech and the first-stage time-domain enhanced speech in the audio signal into the acoustic parameter extractor to obtain the embeddings of the clean speech and the first-stage time-domain enhanced speech respectively. Calculate the root mean square error of the embeddings of the clean speech and the first-stage time-domain enhanced speech, and calculate the acoustic parameter loss function. S5. Input the clean speech and the first-stage time-domain enhanced speech in the audio signal into the fine-grained spectral feature extractor, extract the output of the penultimate layer of the fine-grained spectral feature extractor network as fine-grained spectral features, align the fine-grained spectral features using the root mean square error loss, and calculate the fine-grained feature loss function. S6. Construct the first-stage loss function based on the acoustic parameter loss function, fine-grained feature loss function, and spectral loss function. Use the first-stage loss function to perform the first-stage training and output the trained first-stage enhancer. The training process is as follows: Calculate the gradient of the loss function with respect to the network parameters in the first stage, subtract the product of the gradient and the learning rate from the network parameters, and repeatedly update the network parameters to reduce the loss until the updated network parameters do not improve for 20 consecutive iterations on the validation set, then terminate the training. Second phase of training: S7. The features of the first-stage temporal enhanced speech, clean speech and residual noise are obtained by the encoder respectively; S8. Construct triples based on the features obtained in S7, and input the triples into the contrastive learning method to obtain the contrastive learning loss function; S9, based on acoustic parameter loss function, fine-grained feature loss function, contrastive learning loss function and L log-PCM Construct a loss function for the second stage, use the loss function for the second stage of training, and output the trained second-stage augmenter. Voice enhancement: S10. Input the audio to be enhanced into the first-stage enhancer after training, and output the enhanced audio after the first stage. Input the enhanced audio after the first stage into the second-stage enhancer after training, and output the enhanced target audio.

2. The speech enhancement method for low signal-to-noise ratio audio signals according to claim 1, characterized in that, S1 specifically includes the following steps: Perform a short-time Fourier transform on the audio signal to obtain the real and imaginary components; The feature consistency module is used to extract acoustic parameter features and fine-grained spectral features from enhanced speech; The obtained real and imaginary components are input into a complex encoder network; the complex encoder network includes complex convolutional layers, complex batch normalization, and the PReLU activation function; the real and imaginary components are processed simultaneously by complex convolutional layers that can convolve both the real and imaginary components, and the specific calculation process is as follows: Among them, A k W is the output of a complex two-dimensional convolution. r and W i These are the weights of the real part and the imaginary part of the complex convolution kernel, respectively. and These represent the real and imaginary parts of the input signal, respectively. A k After batch normalization and PReLU activation function of the complex coding layer, the corresponding encoder network output F is obtained. k The formula is as follows: F k =PReLU(Batchnorm2d(A k ))。 3. The speech enhancement method for low signal-to-noise ratio audio signals according to claim 1, characterized in that, In S2, the Conformer module uses the following formula to model temporal information using an attention network: Where H is the output of the Conformer module. For the real part, This is the imaginary part.

4. The speech enhancement method for low signal-to-noise ratio audio signals according to claim 1, characterized in that, In S3, the frequency domain output is represented as: O=Decoder(concatenate(F k ,H))。 5. The speech enhancement method for low signal-to-noise ratio audio signals according to claim 1, characterized in that, The acoustic parameter loss function in S4 and the fine-grained feature loss function in S5 are as follows: Where T is the acoustic parameter embedding corresponding to the audio signals in the training set. It is the acoustic parameter embedding after the post-acoustic parameter extractor, where F is the fine-grained spectral embedding corresponding to the audio signals in the training set. It is a fine-grained spectral embedding after the post-acoustic parameter extractor.

6. The speech enhancement method for low signal-to-noise ratio audio signals according to claim 1, characterized in that, The formula for obtaining features through the encoder in S7 is as follows: Where S is the output of the first stage, X is the clean signal, and w, w + and w - These represent the first-stage output features, clean audio features, and residual noise features extracted after the encoder, respectively.

7. The speech enhancement method for low signal-to-noise ratio audio signals according to claim 5, characterized in that, The loss function for the first stage is: L PFL-stage1 =L log-PCM +αL TAP +βL FM The loss function for the second stage is: L PFL-stage2 =L log-PCM +αL TAP +βL FM +λL CL Where α, β, and λ are hyperparameters; L log-PCM for: Where S(t,f) and They are s and The coefficients of the STFT, where T is the number of time frames, F is the number of frequency points; r and i represent the imaginary parts of the real sum of complex numbers, respectively; L SM It is the average absolute error loss between the pure short-time Fourier transform coefficients and the estimated short-time Fourier coefficients; The contrastive learning loss function is as follows: Where N is the sample size, τ is the temperature function, and the triplet is [w, w + ,w - ], where [w,w + As a positive sample pair, [w,w] - [This is used as a negative sample pair.] 8. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it performs the speech enhancement method for low signal-to-noise ratio audio signals as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the speech enhancement method for low signal-to-noise ratio audio signals as described in any one of claims 1 to 7 through the computer program.

Images