Speech signal joint noise reduction compression method and system

Abstract

The present invention provides a method and system for joint noise reduction and compression of speech signals, comprising: step 1: training an encoder, quantizer, and decoder based on a multi-scale spectral reconstruction loss to obtain an optimal noise reduction encoder and vector quantizer; step 2: fixing the encoder and quantizer, using the encoder, quantizer, and decoder as generators, introducing a waveform-based discriminator and a spectrum-based discriminator, and training a decoder under optimal perceptual quality constraints based on the generation loss; and step 3: compressing the speech signal through multiple downsampling on the encoder, and performing noise reduction and compression on the speech signal on the encoder. The optimal training framework proposed in the present invention has a reasonable theoretical basis and is a theoretically optimal structure that can achieve the maximum possible noise reduction on the encoder side and increase the effective information in the encoded bitstream.

Description

Technical Field

[0001] The present invention relates to the technical fields of speech enhancement technology and speech encoder technology, and in particular to a speech signal joint noise reduction and compression method and system. Background Art

[0002] In the communications field, the primary function of a speech codec is to encode speech into a low-rate bitstream for transmission. The receiving end then reconstructs the original signal from the received bitstream. This requires encoding the audio with as few bits as possible and decoding it back to produce the original audio with minimal distortion and a more natural sound. To address audio with real-world noise, an additional speech enhancement module is often added before the encoder to remove the noise.

[0003] Patent document CN102945674A (application number: CN201210505189.0) discloses a method for noise reduction processing of speech signals using a digital noise reduction algorithm. The noisy signal is first processed using a subspace method to obtain a relatively pure processed signal, and then the signal is input into an adaptive filter to perform an LMS (least mean square algorithm) to eliminate speech noise. The present invention combines the subspace noise reduction algorithm with the adaptive LMS algorithm to filter noise.

[0004] Traditional speech codecs, such as OPUS and EVS, use discontinuous transmission (DTX) and filters to mitigate the effects of noise, resulting in very limited noise reduction capabilities. While the end-to-end Soundstream codec provides the conditions for combined noise reduction and compression, this framework is not the optimal noise reduction and compression architecture. It is necessary to add an additional noise reduction encoder on the encoder side to ensure that the encoded bitstream contains more effective information. Furthermore, the audio reconstructed by the speech codec must maintain both naturalness and similarity to the original audio, but these two aspects are often not achieved simultaneously. Compression bitrate, distortion, and perception are mutually constrained. Spectral distortion is often used to measure the similarity between speech sounds, but spectral distortion cannot accurately reflect the naturalness of the audio. Currently, most research uses generative adversarial methods to align the distribution of reconstructed and original audio, improving audio naturalness. Weighting parameters are used to adjust the balance between distortion and naturalness. Summary of the Invention

[0005] In view of the defects in the prior art, the present invention aims to provide a method and system for joint noise reduction and compression of speech signals.

[0006] The speech signal joint noise reduction and compression method provided by the present invention includes:

[0007] Step 1: Train the encoder, quantizer, and decoder based on the multi-scale spectral reconstruction loss to obtain the optimal denoising encoder and vector quantizer;

[0008] Step 2: Fix the encoder and quantizer, use the encoder, quantizer, and decoder as generators, introduce waveform-based discriminators and spectrum-based discriminators, and train the decoder under the optimal perceptual quality constraint based on the generation loss;

[0009] Step 3: The speech signal is compressed by multiple downsampling on the encoder, and the speech signal is denoised and compressed on the encoder.

[0010] Preferably, the encoder includes a one-dimensional convolutional layer and four encoding modules, each encoding module includes three residual units and a downsampling layer; the residual unit extracts audio features while stably retaining effective information and avoiding gradient disappearance; the downsampling layer performs downsampling to compress the signal;

[0011] The decoder includes a one-dimensional convolution layer and four decoding modules, each of which includes a transposed convolution for upsampling and three residual units; the upsampling transposed convolution performs upsampling to restore the audio dimension from the compressed signal;

[0012] The decoder and the encoder use the same parameters;

[0013] It uses 8 cascaded quantizers with a codebook size of 1024, supporting compression at a target bit rate of 3kbps-18kbps.

[0014] The waveform-based discriminators input the original waveform, the waveform downsampled by two times, and the waveform downsampled by four times, respectively, and extract waveform features at different scales. Each discriminator includes one ordinary convolution and four grouped convolutions. The ordinary convolution initially extracts waveform features. The grouped convolution further extracts features using a grouping approach with less computation and parameters. Each grouped convolution has a group size of 4, a downsampling factor of 4, and a channel multiplier of 4, outputting a maximum of 1024 channels. Finally, two convolutional layers gradually compress the features, reducing the number of channels to the specified number, and producing the final output.

[0015] The spectrum-based discriminator first performs STFT transformation on the waveform and then performs two-dimensional convolution on the transformed time-frequency spectrum to extract features. The discriminator consists of a series of residual units and a two-dimensional convolution layer, and finally obtains a one-dimensional signal in the downsampled time domain.

[0016] Preferably, the generation loss includes generation adversarial loss, feature matching loss and multi-scale spectrum reconstruction loss, and the expression is:

[0017]

[0018] Among them, X is the original audio, For the reconstructed audio, Xd Audio reconstructed for the first stage; L G represents the generator loss; λ adv represents the weight of generating adversarial loss; feat represents the feature matching loss weight; λ dis represents the distortion loss weight; To generate adversarial loss; is the feature matching loss; is the multi-scale spectral reconstruction loss;

[0019] The expression of multi-scale spectrum reconstruction loss is:

[0020]

[0021] in, is the Mel spectrum of the tth frame of the original audio; Reconstruct the Mel spectrum of the tth frame of audio; s is the window length; α s is the weight parameter;

[0022] The expression of feature matching loss is:

[0023]

[0024] Among them, K is the number of discriminators; L is the number of identification layers; is the t-th output of the l-th layer of the k-th discriminator of the original audio; E x [] represents the expectation of x; T k, Represents the total number of output logits of the fully connected layer of the k-th discriminator layer l along the time dimension;

[0025] The expression of generative adversarial loss is:

[0026]

[0027] Among them, T k Represents the total number of logits along the time dimension of the output of the k-th discriminator; Indicates that the audio will be reconstructed The t-th output after inputting the k-th discriminator.

[0028] Preferably, the expression of the discriminator loss is:

[0029]

[0030] Preferably, in the speech compression task, the encoder encodes the source sequence into a sequence with a target bit rate R, and the decoder decodes the sequence to obtain the reconstructed original sequence. Given the target bit rate, the constraint relationship between distortion and perception is expressed as:

[0031]

[0032]

[0033] Among them, R (I) () represents the minimum achievable bit rate under given distortion and perception indicators; D represents distortion; P represents perception; Indicates that under the condition that X is known The conditional distribution of represents expectation; Δ() represents the formula for calculating distortion; p x represents the prior distribution of X; d() represents the divergence measure function of the two distributions.

[0034] The speech signal joint noise reduction and compression system provided by the present invention includes:

[0035] Module M1: Train the encoder, quantizer, and decoder based on multi-scale spectral reconstruction loss to obtain the optimal denoising encoder and vector quantizer;

[0036] Module M2: The encoder and quantizer are fixed, and the encoder, quantizer, and decoder are used as generators. A waveform-based discriminator and a spectrum-based discriminator are introduced, and the decoder under the optimal perceptual quality constraint is trained based on the generation loss.

[0037] Module M3: compresses the speech signal through multiple downsampling on the encoder, and performs noise reduction and compression on the speech signal on the encoder.

[0038] Preferably, the encoder includes a one-dimensional convolutional layer and four encoding modules, each encoding module includes three residual units and a downsampling layer; the residual unit extracts audio features while stably retaining effective information and avoiding gradient disappearance; the downsampling layer performs downsampling to compress the signal;

[0039] The decoder includes a one-dimensional convolution layer and four decoding modules, each of which includes a transposed convolution for upsampling and three residual units; the upsampling transposed convolution performs upsampling to restore the audio dimension from the compressed signal;

[0040] The decoder and the encoder use the same parameters;

[0041] It uses 8 cascaded quantizers with a codebook size of 1024, supporting compression at a target bit rate of 3kbps-18kbps.

[0042] The waveform-based discriminators input the original waveform, the waveform downsampled by two times, and the waveform downsampled by four times, respectively, and extract waveform features at different scales. Each discriminator includes one ordinary convolution and four grouped convolutions. The ordinary convolution initially extracts waveform features. The grouped convolution further extracts features using a grouping approach with less computation and parameters. Each grouped convolution has a group size of 4, a downsampling factor of 4, and a channel multiplier of 4, outputting a maximum of 1024 channels. Finally, two convolutional layers gradually compress the features, reducing the number of channels to the specified number, and producing the final output.

[0043] The spectrum-based discriminator first performs STFT transformation on the waveform and then performs two-dimensional convolution on the transformed time-frequency spectrum to extract features. The discriminator consists of a series of residual units and a two-dimensional convolution layer, and finally obtains a one-dimensional signal in the downsampled time domain.

[0044] Preferably, the generation loss includes generation adversarial loss, feature matching loss and multi-scale spectrum reconstruction loss, and the expression is:

[0045]

[0046] Among them, X is the original audio, For the reconstructed audio, X d Audio reconstructed for the first stage; L G represents the generator loss; λ adv represents the weight of generating adversarial loss; feat represents the feature matching loss weight; λ dis represents the distortion loss weight; To generate adversarial loss; is the feature matching loss; is the multi-scale spectral reconstruction loss;

[0047] The expression of multi-scale spectrum reconstruction loss is:

[0048]

[0049] in, is the Mel spectrum of the tth frame of the original audio; Reconstruct the Mel spectrum of the tth frame of audio; s is the window length; α s is the weight parameter;

[0050] The expression of feature matching loss is:

[0051]

[0052] Among them, K is the number of discriminators; L is the number of identification layers; is the t-th output of the l-th layer of the k-th discriminator of the original audio; Ex [] represents the expectation of x; T k, Represents the total number of output logits of the fully connected layer of the k-th discriminator layer l along the time dimension;

[0053] The expression of generative adversarial loss is:

[0054]

[0055] Among them, T k Represents the total number of logits along the time dimension of the output of the k-th discriminator; Indicates that the audio will be reconstructed The t-th output after inputting the k-th discriminator.

[0056] Preferably, the expression of the discriminator loss is:

[0057]

[0058] Preferably, in the speech compression task, the encoder encodes the source sequence into a sequence with a target bit rate R, and the decoder decodes the sequence to obtain the reconstructed original sequence. Given the target bit rate, the constraint relationship between distortion and perception is expressed as:

[0059]

[0060]

[0061] Among them, R (I) () represents the minimum achievable bit rate under given distortion and perception indicators; D represents distortion; P represents perception; Indicates that under the condition that X is known The conditional distribution of represents expectation; Δ() represents the formula for calculating distortion; p x represents the prior distribution of X; d() represents the divergence measure function of the two distributions.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) The codec proposed in the present invention can achieve noise reduction of real environmental noise during the encoding and decoding process without adding an additional speech enhancement module. Compared with the most advanced traditional codecs OPUS, EVS, and Lyra, the codec proposed in the present invention has obvious advantages in noise reduction effect;

[0064] (2) The optimal training framework proposed in the present invention has a reasonable theoretical basis and is a theoretically optimal structure that can achieve the maximum possible noise reduction at the encoder end, greatly increasing the effective information in the encoded bit stream;

[0065] (3) The optimal training theory proposed in this invention expands the distortion-perception-bitrate constraint theory and can be applied to any noise scenario and generalized distortion scenario, so that the reconstructed audio can achieve the minimum distortion upper bound under the optimal perceptual quality constraint. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0067] Figure 1 Schematic diagram of the optimal training framework in the present invention;

[0068] Figure 2 Detailed structural diagram of the encoder, vector quantizer, and decoder in the present invention;

[0069] Figure 3 Detailed structural diagram of the waveform identifier and spectrum identifier used in the present invention. DETAILED DESCRIPTION

[0070] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0071] Example 1:

[0072] This paper provides an optimal training framework, using a phased training approach. In the first phase, an optimal noise reduction encoder and quantizer are trained, and in the second phase, encoding is performed to achieve maximum possible noise reduction. To demonstrate the optimality of this framework, the paper provides a theoretical formula derivation demonstrating that the optimal training framework can achieve a perceptually constrained minimum distortion upper bound in the presence of noise.

[0073] The method includes the following:

[0074] Optimal training framework: To achieve the maximum possible noise reduction during the encoding process and achieve the minimum distortion upper bound under perceptual constraints, for noisy audio, if the encoded audio contains as little noise as possible, more effective information can be retained during the compression process. The optimal training framework includes an encoder, a residual vector quantizer, a decoder, a waveform-based discriminator, and a spectrum-based discriminator. The training steps are as follows:

[0075] 1. In the first stage, the encoder, quantizer, and decoder are trained based on the multi-scale spectral reconstruction loss to obtain an optimal denoising encoder and vector quantizer.

[0076] 2. In the second stage, the encoder and quantizer are fixed, and the encoder, quantizer, and decoder are regarded as generators. A waveform-based discriminator and a spectrum-based discriminator are introduced. An optimal decoder under the optimal perceptual quality constraint is trained based on adversarial loss, feature matching loss, and multi-scale spectrum reconstruction loss.

[0077] The two training steps above yield an optimal noise reduction encoder, quantizer, and a perceptually constrained optimal decoder. During inference, the noisy audio can be directly input to the encoder for end-to-end encoding and decoding, and the decoder directly outputs the corresponding clean audio.

[0078] Theoretical Proof of the Optimal Training Framework: In order to prove that the training framework proposed in the present invention is optimal, the present invention provides relevant formula derivations for theoretical proof, extending the distortion-perception-rate constraint theorem to audio noise reduction and audio compression. The theoretical proof includes the distortion-perception-rate constraint theorem and two corollaries of the theorem. The distortion-perception-rate constraint theorem proves that in the compression task of determining a discrete signal source, the optimal encoder based on the MSE loss is also the optimal encoder under the optimal perceptual constraint. The present invention proposes two corollaries of the theorem:

[0079] 1. For noisy audio, the minimum MSE encoder after denoising is still the optimal encoder with the best perceptual quality.

[0080] 2. The minimum distortion has an upper bound under the optimal perceptual constraint, which can be obtained by the optimal encoder under generalized distortion.

[0081] Through these two inferences, it can be proved that the optimal training framework proposed in this invention is effective in the joint denoising and compression task.

[0082] The following will further expand on the content and describe in detail the optimal phased training framework and the theoretical proof of this training framework.

[0083] Best training framework:

[0084] The optimal training framework proposed in this invention adopts a generator and a discriminator. The generator includes an encoder, a residual vector quantizer, and a decoder. Figure 2 The discriminator includes waveform-based discriminator and spectrum-based discriminator, as shown in Figure 3 shown.

[0085] The encoder consists of a one-dimensional convolutional layer and four encoding modules. Each encoding module consists of three residual units and a downsampling layer. Finally, a one-dimensional convolutional layer is used to obtain features with a dimension of 256. To ensure real-time inference, all convolutions are causal. The decoder adopts a similar structure, consisting of a one-dimensional convolutional layer and four decoding modules. The decoding module consists of a transposed convolution for upsampling and three residual units. The parameters used by the decoder are the same as those of the encoder. The residual vector quantizer greatly reduces the size of the codebook by cascading multiple quantizers; the quantizer is trained by dropout to achieve an adjustable target bit rate. The present invention uses 8 cascaded quantizers with a codebook size of 1024, which can support compression of a target bit rate of 3kbps-18kbps.

[0086] The waveform-based discriminator takes as input the original waveform, a waveform downsampled by two times, and a waveform downsampled by four times, extracting waveform features at different scales. Each discriminator consists of a normal convolution and four grouped convolutions, each with a group size of 4, a downsampling factor of 4, and a channel multiplier of 4, outputting up to 1024 channels. Finally, two convolutional layers produce the final output. The spectrum-based discriminator first performs an STFT transform on the waveform and then performs a two-dimensional convolution on the resulting time-frequency spectrum to extract features. This discriminator consists of a series of residual units and a two-dimensional convolutional layer, ultimately resulting in a one-dimensional signal in the downsampled time domain.

[0087] The best training framework is Figure 1 As shown, it consists of two steps:

[0088] 1. The first stage trains the encoder E based on multi-scale spectrum reconstruction loss d , quantizer, decoder G d , obtain an optimal noise reduction encoder E d and vector quantizer.

[0089] 2. Second stage fixed encoder E d and quantizer, the encoder E d , quantizer, decoder G p It is regarded as a generator, and a waveform-based discriminator and a spectrum-based discriminator are introduced. An optimal decoder G under the optimal perceptual quality constraint is trained based on adversarial loss, feature matching loss and multi-scale spectrum reconstruction loss. p .

[0090] The generation loss includes generation adversarial loss, feature matching loss and multi-scale spectrum reconstruction loss:

[0091]

[0092] Among them, λ adv =1,λfeat =100,λ dis =30, X is the original audio, For the reconstructed audio, X d The audio reconstructed in the first stage, X d :=G d (E d (X)).

[0093] The multi-scale spectrum reconstruction loss is:

[0094]

[0095] in, is the Mel spectrum of the tth frame, and the window length is s.

[0096] The feature matching loss is:

[0097]

[0098] Among them, K is the number of discriminators, L is the number of identification layers, is the t-th output of the l-th layer of the k-th discriminator.

[0099] The generative adversarial loss is:

[0100]

[0101] The discriminator loss is:

[0102]

[0103] Theoretical proof of the optimal training framework:

[0104] Assume that X is a certain discrete source sequence. In the speech compression task, the encoder E encodes the source X into a sequence Z with a target code rate R, and the decoder G decodes Z to obtain the reconstructed original sequence. Given a target bitrate, the constraint relationship between distortion and perception can be expressed as:

[0105]

[0106]

[0107] Theorem: Assume that the source X does not contain noise, and the distortion Δ is the MSE loss. d ,G d ) is the optimal codec under the condition of P = +∞, then according to the known E d , we get the optimal codec (E d ,G p ). That is, R (I)The optimal encoder for (D, +∞) is also R (I) Optimal encoder for the (D,0) case.

[0108] prove:

[0109] remember The optimal encoder result under MSE

[0110]

[0111] Item 3:

[0112]

[0113] So the third item is 0. Therefore:

[0114]

[0115] Among them, the first term is the MSE of the decoding result of the source and the minimum MSE, and the second term is the MSE of X mse Mapped to the same distribution as the source The resulting MSE is obtained from the distortion-perception-rate constraint theorem:

[0116]

[0117]

[0118] That is, the optimal encoder with the minimum MSE can also be used as the optimal encoder with the best perceptual quality.

[0119] Corollary 1: In the joint noise reduction compression process, the encoder input is the noisy sequence X′, and the reconstructed sequence is a clean sequence with the same distribution as the source. The distortion Δ is the MSE loss. (I) (D, +∞) case, train an optimal encoder to achieve the maximum possible noise reduction, then in R (I) It is also optimal in the (D,0) case.

[0120] prove:

[0121] The minimum MSE encoding and decoding process in the case of noise is:

[0122] X→X'→Z→X mse

[0123] The output of the optimal denoiser is

[0124]

[0125] When X′ is known, X and X mseIndependent of each other, and

[0126] Item 3:

[0127]

[0128] Right now:

[0129] This shows that the minimum distortion achieved by the end-to-end training framework and the phased training framework that first denoises and then compresses the denoised result are the same. In the presence of noise, the distortion-perception-rate constraint theorem still holds true, and the minimum MSE encoder after denoising remains the optimal encoder for optimal perceptual quality.

[0130] Corollary 2: MSE is not usually used as a distortion calculation method in audio compression. Taking multi-scale spectrum loss as an example,

[0131]

[0132] Denotes the spectrum (or mel-spectrogram) with a window length of s at time t frame. Consider generalizing Theory 1 for the MSE distortion case to the L1 norm and logarithmic MSE cases. The minimum distortion of this framework is upper bounded under optimal perceptual constraints and can be obtained from the optimal encoder for the generalized distortion case.

[0133] prove:

[0134] remember is the optimal encoder result under generalized distortion.

[0135] For L1 norm:

[0136] For log MSE:

[0137]

[0138] Under the optimal perception constraint, The angle between the two inner products of the third term is 180°, so the third term is negative, so:

[0139]

[0140] The framework is to optimize the upper bound of the minimum distortion under the optimal perceptual constraint. The first term is the distortion between the source and the minimum distortion decoding result, and the second term is to convert X mse Mapped to the same distribution as the source According to Theory 1, the optimal encoder E under the condition of P = +∞ d, is the upper bound of the minimum distortion when P = 0. Therefore, for multi-scale spectrum loss, E d It can also be used as the optimal encoder when P=0.

[0141] Example 2:

[0142] The present invention also provides a speech signal joint noise reduction compression system, which can be implemented by executing the process steps of the speech signal joint noise reduction compression method. That is, those skilled in the art can understand the speech signal joint noise reduction compression method as an optimal implementation of the speech signal joint noise reduction compression system.

[0143] The speech signal joint noise reduction and compression system provided by the present invention includes: module M1: training an encoder, quantizer, and decoder based on multi-scale spectrum reconstruction loss to obtain an optimal noise reduction encoder and vector quantizer; module M2: fixing the encoder and quantizer, using the encoder, quantizer, and decoder as generators, introducing a waveform-based discriminator and a spectrum-based discriminator, and training a decoder under optimal perceptual quality constraints based on generation loss; module M3: compressing the speech signal through multiple downsampling on the encoder, and denoising and compressing the speech signal on the encoder to achieve high-quality reconstruction.

[0144] The encoder includes a one-dimensional convolution layer and four encoding modules, each encoding module includes three residual units and a downsampling layer; the residual unit extracts audio features while stably retaining effective information and avoiding gradient disappearance; the downsampling layer performs downsampling and compresses the signal; the decoder includes a one-dimensional convolution layer and four decoding modules, each decoding module includes a transposed convolution for upsampling and three residual units; the upsampling transposed convolution performs upsampling and restores the audio dimension from the compressed signal; the decoder uses the same parameters as the encoder; 8 cascaded quantizers are used, the codebook size is 1024, and compression of the target bit rate of 3kbps-18kbps is supported; the waveform-based discriminator inputs the original waveform, the downsampling, and the compression of the original waveform, the downsampling, and the compression of the compressed signal respectively. The waveform is sampled twice and downsampled four times, and waveform features are extracted from different scales. Each discriminator includes an ordinary convolution and four grouped convolutions. The ordinary convolution preliminarily extracts the features of the waveform. The grouped convolution uses a grouping method to further extract features with less computation and parameters. The group size of each grouped convolution is 4, the downsampling factor is 4, the channel multiplier is 4, and a maximum of 1024 channels are output. Finally, two convolution layers are used to gradually compress the features, reducing the number of channels to a specified number to generate the final output. The spectrum-based discriminator first performs STFT transformation on the waveform, and then performs two-dimensional convolution on the time-frequency spectrum obtained by the transformation to extract features. The discriminator consists of a series of residual units and a two-dimensional convolution layer, and finally obtains a one-dimensional signal in the downsampled time domain.

[0145] The generation loss includes generation adversarial loss, feature matching loss and multi-scale spectrum reconstruction loss, and is expressed as:

[0146]

[0147] Among them, X is the original audio, For the reconstructed audio, X d Audio reconstructed for the first stage; L G represents the generator loss; λ adv represents the weight of generating adversarial loss; feat represents the feature matching loss weight; λ dis represents the distortion loss weight; To generate adversarial loss; is the feature matching loss; is the multi-scale spectral reconstruction loss;

[0148] The expression of multi-scale spectrum reconstruction loss is:

[0149]

[0150] in, is the Mel spectrum of the tth frame of the original audio; Reconstruct the Mel spectrum of the tth frame of audio; s is the window length; α s is the weight parameter;

[0151] The expression of feature matching loss is:

[0152]

[0153] Among them, K is the number of discriminators; L is the number of identification layers; is the t-th output of the l-th layer of the k-th discriminator of the original audio; E x [] represents the expectation of x; T k, Represents the total number of output logits of the fully connected layer of the k-th discriminator layer l along the time dimension;

[0154] The expression of generative adversarial loss is:

[0155]

[0156] Among them, T k Represents the total number of logits along the time dimension of the output of the k-th discriminator; Indicates that the audio will be reconstructed The t-th output after inputting the k-th discriminator.

[0157] The expression of the discriminator loss is:

[0158]

[0159] In the speech compression task, the encoder encodes the source sequence into a sequence with a target bit rate R, and the decoder decodes the sequence to obtain the reconstructed original sequence. Given the target bit rate, the constraint relationship between distortion and perception is expressed as:

[0160]

[0161]

[0162] Among them, R (I) () represents the minimum achievable bit rate under given distortion and perception indicators; D represents distortion; P represents perception; Indicates that under the condition that X is known The conditional distribution of represents expectation; Δ() represents the formula for calculating distortion; p x represents the prior distribution of X; d() represents the divergence measure function of the two distributions.

[0163] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0164] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A speech signal joint noise reduction and compression method, characterized in that: include: Step 1: In the first stage, the encoder, quantizer, and decoder are trained based on the multi-scale spectrum reconstruction loss to obtain the optimal denoising encoder and vector quantizer; Step 2: In the second stage, the encoder and quantizer are fixed, and the encoder, quantizer, and decoder are used as generators. A waveform-based discriminator and a spectrum-based discriminator are introduced, and the decoder under the optimal perceptual quality constraint is trained based on the generation loss. The generation loss includes generation adversarial loss, feature matching loss and multi-scale spectrum reconstruction loss, and the expression is: in, For the original audio, For the reconstructed audio, Reconstructed audio for the first stage; represents the generator loss; represents the weight of generating adversarial loss; Represents the feature matching loss weight; represents the distortion loss weight; To generate adversarial loss; is the feature matching loss; is the multi-scale spectral reconstruction loss; Step 3: The encoder compresses the speech signal through multiple downsampling steps. The noisy audio is directly used as the input of the encoder for end-to-end encoding and decoding. The decoder directly outputs the corresponding clean audio.

2. The method for joint noise reduction and compression of speech signals according to claim 1, wherein: The encoder includes a one-dimensional convolutional layer and four encoding modules, each of which includes three residual units and a downsampling layer; the residual unit extracts audio features while stably retaining effective information and preventing gradient vanishing; the downsampling layer performs downsampling and compresses the signal; The decoder includes a one-dimensional convolution layer and four decoding modules, each of which includes a transposed convolution for upsampling and three residual units; the upsampling transposed convolution performs upsampling to restore the audio dimension from the compressed signal; The decoder and the encoder use the same parameters; It uses 8 cascaded quantizers with a codebook size of 1024, supporting compression at a target bit rate of 3kbps-18kbps. The waveform-based discriminators input the original waveform, the waveform downsampled by two times, and the waveform downsampled by four times, respectively, and extract waveform features at different scales. Each discriminator includes one ordinary convolution and four grouped convolutions. The ordinary convolution initially extracts waveform features. The grouped convolution further extracts features using a grouping approach with less computation and parameters. Each grouped convolution has a group size of 4, a downsampling factor of 4, and a channel multiplier of 4, outputting a maximum of 1024 channels. Finally, two convolutional layers gradually compress the features, reducing the number of channels to the specified number, and producing the final output. The spectrum-based discriminator first performs STFT transformation on the waveform and then performs two-dimensional convolution on the transformed time-frequency spectrum to extract features. The discriminator consists of a series of residual units and a two-dimensional convolution layer, and finally obtains a one-dimensional signal in the downsampled time domain.

3. The method for joint noise reduction and compression of speech signals according to claim 1, wherein: The expression of the multi-scale spectrum reconstruction loss in the second stage is: in, is the Mel spectrum of the reconstructed audio frame t; is the Mel spectrum of the tth frame of the reconstructed audio in the first stage; s is the window length; is the weight parameter; The expression of feature matching loss is: Among them, K is the number of discriminators; L is the number of identification layers; is the t-th output of the l-th layer of the k-th discriminator of the original audio; represents the expectation of x; Represents the total number of output logits of the fully connected layer of the k-th discriminator layer l along the time dimension; The expression of generative adversarial loss is: in, Represents the total number of logits along the time dimension of the output of the k-th discriminator; Indicates that the audio will be reconstructed The t-th output after inputting the k-th discriminator.

4. The method for joint noise reduction and compression of speech signals according to claim 3, wherein: The expression of the discriminator loss is: 。 5. The method for joint noise reduction and compression of speech signals according to claim 3, characterized in that: In the speech compression task, the encoder encodes the source sequence into a target bit rate The decoder then decodes the sequence to obtain the reconstructed original sequence. Given a target bit rate, the constraint relationship between distortion and perception is expressed as: in, It represents the minimum achievable bit rate under given distortion and perception indicators; D represents distortion; P represents perception; Indicates that under the condition that X is known The conditional distribution of express expectations; The formula for calculating distortion is represented; represents the prior distribution of X; A function that measures the divergence between two distributions.

6. A speech signal joint noise reduction and compression system, characterized in that: include: Module M1: In the first stage, the encoder, quantizer, and decoder are trained based on the multi-scale spectrum reconstruction loss to obtain the optimal denoising encoder and vector quantizer; Module M2: In the second stage, the encoder and quantizer are fixed, and the encoder, quantizer, and decoder are used as generators. A waveform-based discriminator and a spectrum-based discriminator are introduced, and the decoder under the optimal perceptual quality constraint is trained based on the generation loss. The generation loss includes generation adversarial loss, feature matching loss and multi-scale spectrum reconstruction loss, and is expressed as: in, For the original audio, For the reconstructed audio, Reconstructed audio for the first stage; represents the generator loss; represents the weight of generating adversarial loss; Represents the feature matching loss weight; represents the distortion loss weight; To generate adversarial loss; is the feature matching loss; is the multi-scale spectral reconstruction loss; Module M3: The encoder compresses the speech signal through multiple downsampling. The noisy audio is directly used as the input of the encoder for end-to-end encoding and decoding. The decoder directly outputs the corresponding clean audio.

7. The speech signal joint noise reduction and compression system according to claim 6, characterized in that: The encoder includes a one-dimensional convolutional layer and four encoding modules, each of which includes three residual units and a downsampling layer; the residual unit extracts audio features while stably retaining effective information and preventing gradient vanishing; the downsampling layer performs downsampling and compresses the signal; The decoder includes a one-dimensional convolution layer and four decoding modules, each of which includes a transposed convolution for upsampling and three residual units; the upsampling transposed convolution performs upsampling to restore the audio dimension from the compressed signal; The decoder and the encoder use the same parameters; It uses 8 cascaded quantizers with a codebook size of 1024, supporting compression at a target bit rate of 3kbps-18kbps. The waveform-based discriminators input the original waveform, the waveform downsampled by two times, and the waveform downsampled by four times, respectively, and extract waveform features at different scales. Each discriminator includes one ordinary convolution and four grouped convolutions. The ordinary convolution initially extracts waveform features. The grouped convolution further extracts features using a grouping approach with less computation and parameters. Each grouped convolution has a group size of 4, a downsampling factor of 4, and a channel multiplier of 4, outputting a maximum of 1024 channels. Finally, two convolutional layers gradually compress the features, reducing the number of channels to the specified number, and producing the final output. The spectrum-based discriminator first performs STFT transformation on the waveform and then performs two-dimensional convolution on the transformed time-frequency spectrum to extract features. The discriminator consists of a series of residual units and a two-dimensional convolution layer, and finally obtains a one-dimensional signal in the downsampled time domain.

8. The speech signal joint noise reduction and compression system according to claim 6, characterized in that: The expression of the multi-scale spectrum reconstruction loss in the second stage is: in, is the Mel spectrum of the reconstructed audio frame t; is the Mel spectrum of the tth frame of the reconstructed audio in the first stage; s is the window length; is the weight parameter; The expression of feature matching loss is: Among them, K is the number of discriminators; L is the number of identification layers; is the t-th output of the l-th layer of the k-th discriminator of the original audio; represents the expectation of x; Represents the total number of output logits of the fully connected layer of the k-th discriminator layer l along the time dimension; The expression of generative adversarial loss is: in, Represents the total number of logits along the time dimension of the output of the k-th discriminator; Indicates that the audio will be reconstructed The t-th output after inputting the k-th discriminator.

9. The speech signal joint noise reduction and compression system according to claim 8, characterized in that: The expression of the discriminator loss is: 。 10. The speech signal joint noise reduction and compression system according to claim 8, characterized in that: In the speech compression task, the encoder encodes the source sequence into a target bit rate The decoder then decodes the sequence to obtain the reconstructed original sequence. Given a target bit rate, the constraint relationship between distortion and perception is expressed as: in, It represents the minimum achievable bit rate under given distortion and perception indicators; D represents distortion; P represents perception; Indicates that under the condition that X is known The conditional distribution of express expectations; The formula for calculating distortion is represented; represents the prior distribution of X; A function that measures the divergence between two distributions.

Images