Speech enhancement method and system based on time-frequency graph convolutional network

Abstract

The invention discloses a speech enhancement method and system based on a time-frequency graph convolutional network in the technical field of speech signal processing, and the method comprises the steps: obtaining to-be-enhanced noisy speech data, inputting the to-be-enhanced noisy speech data into an improved metric generative adversarial network (CMGAN), and outputting an enhanced speech time domain signal; the improvement method of the metric generative adversarial network CMGAN comprises the following steps: replacing an encoder module of the metric generative adversarial network CMGAN with a time-frequency graph convolution encoder module; in the time-frequency graph convolution encoder module, a time convolution network and a graph convolution network are used for respectively extracting a long-time dependency relationship and global frequency characteristic information of noise-containing voice data to be enhanced, and dense residual error connection is introduced to fuse multi-scale characteristic information at the same time. The speech enhancement performance is improved under the condition of less model parameter quantity and calculation quantity.

Description

Technical Field

[0001] The present invention relates to a speech enhancement method and system based on a time-frequency graph convolutional network, belonging to the technical field of speech signal processing. Background Art

[0002] Speech enhancement aims to extract pure speech with higher clarity and intelligibility from noisy speech affected by noise or other interference. It is widely used in speech recognition, conference systems, hearing aids, etc. Benefiting from the rapid development of deep learning, speech enhancement methods based on deep neural networks are more effective than traditional methods in dealing with non-stationary and complex noise.

[0003] Speech enhancement systems based on deep learning usually rely on convolutional encoders to extract features of speech signals in order to obtain feature representations of deep neural networks. However, due to the limited receptive field of convolutional neural networks, they can only model speech features in a local range and lack global feature information. In order to expand the receptive field, it is often necessary to stack multiple layers of convolutional neural networks, which not only significantly increases the amount of computation, but also makes it difficult to capture the complex correlations between different frequencies in speech signals. Speech signals are usually composed of a complex combination of multiple frequencies, and the correlations between different frequencies are significantly different between speech and noise. Relying solely on convolution can only capture local adjacent frequency relationships and cannot fully utilize the global correlation between frequencies.

[0004] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgement or any form of suggestion that the information constitutes the prior art already known to ordinary technicians in this field. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to improve the performance of speech enhancement with a smaller number of model parameters and a smaller amount of calculation.

[0006] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0007] In a first aspect, the present invention provides a speech enhancement method based on a time-frequency graph convolutional network, characterized in that it includes:

[0008] Acquire noisy speech data to be enhanced;

[0009] Input the noisy speech data to be enhanced into the improved metric generative adversarial network CMGAN, and output an enhanced speech time domain signal;

[0010] The improved method of the metric generative adversarial network CMGAN includes:

[0011] Replace the encoder module of the metric generative adversarial network CMGAN with a time-frequency graph convolutional encoder module;

[0012] In the time-frequency graph convolution encoder module, a temporal convolutional network and a graph convolutional network are used to respectively extract the long-term dependency and global frequency feature information of the noisy speech data to be enhanced, and a dense residual connection is introduced between each time-frequency graph convolution block to fuse multi-scale feature information.

[0013] Furthermore, before the noisy speech data to be enhanced is input into the improved metric generative adversarial network CMGAN, the method further includes:

[0014] The original clean speech is processed using perceptual contrast stretching PCS to obtain processed clean speech;

[0015] Using the loss function, calculate the difference between the enhanced speech time domain signal output by the improved metric generative adversarial network CMGAN and the processed pure speech to obtain the total loss;

[0016] The weights of the improved metric generative adversarial network CMGAN are updated via back-propagation according to the total loss.

[0017] Furthermore, the perceptual contrast stretching PCS is used to process the original clean speech to obtain the processed clean speech, wherein the processed clean speech is expressed as:

[0018] (1);

[0019] In the formula, Represents the amplitude spectrum of the original pure speech after short-time Fourier transform STFT, where Indicates the amplitude, Represents the original pure speech amplitude spectrum The amplitude spectrum obtained after perceptual contrast stretching PCS processing, (·) represents the logarithmic function, It represents the perceptual score of each frequency band of the original clean speech amplitude spectrum derived based on the frequency band importance function BIF.

[0020] Furthermore, the loss function is used to calculate the difference between the enhanced speech time domain signal output by the improved metric generative adversarial network CMGAN and the processed pure speech to obtain the total loss, wherein the loss function is expressed as:

[0021] (2);

[0022] In the formula, Represents the loss function of measuring the generation of adversarial networks CMGAN, represents the time domain loss term The weight coefficient of , Represents the original pure voice, It represents the enhanced speech time domain signal output by the improved metric generative adversarial network CMGAN, represents the L1 norm, represents the frequency domain loss term The weight coefficient of Represents the reconstruction loss term The weight coefficient of , , , and They represent the amplitude, real part and imaginary part of the pure speech time spectrum obtained by perceptual contrast stretching PCS, short-time Fourier transform STFT and power law compression of the original pure speech, respectively. , and denote the amplitude, real part and imaginary part respectively, , and They represent the amplitude, real part and imaginary part of the speech time-frequency spectrum of the enhanced speech time-domain signal output by the improved metric generative adversarial network CMGAN, respectively. represents the L2 norm, represents the discriminator loss used to improve the PESQ score of enhanced speech The weight coefficient of .

[0023] Furthermore, the improved metric generation adversarial network comprises:

[0024] A preprocessing module, used for performing short-time Fourier transform on the noisy speech data to be enhanced to obtain a complex spectrum in the time-frequency domain, performing power-law compression processing on the amplitude spectrum of the complex spectrum to obtain a processed amplitude spectrum, and forming a three-channel feature map with the processed amplitude spectrum and the real part and imaginary part of the complex spectrum;

[0025] A time-frequency graph convolution encoder module, used for extracting high-dimensional feature representation of the noisy speech data to be enhanced according to the three-channel feature graph, separating noise and retaining effective components in the noisy speech data to be enhanced;

[0026] An enhancement module, configured to output a feature map of a separated noise part using four consecutive two-stage Conformer blocks according to the high-dimensional feature representation;

[0027] The decoder module includes an amplitude decoder and a complex decoder, which is used to decouple the amplitude mask and complex information from the feature map of the separated noise part based on the dilated convolution dense connection network, combine the masked amplitude with the complex information output by the complex decoder to generate a complex spectrum, perform inverse power-law compression on the amplitude spectrum of the complex spectrum, and then generate an enhanced speech time domain signal through inverse Fourier transform.

[0028] Further, the time-frequency graph convolution encoder module includes a graph learning layer located in a first branch, a first convolution block located in a second branch, a plurality of stacked time-frequency graph convolution blocks and a second convolution block, and a first branch and a second branch connected to the plurality of stacked time-frequency graph convolution blocks;

[0029] The method of extracting high-dimensional feature representation of noisy speech data to be enhanced according to the three-channel feature map includes:

[0030] The three-channel feature map is passed through a graph learning layer by using the first branch, and an adjacency matrix is ​​learned according to the frequency dimension relationship between the feature maps to obtain the adjacency matrix and the feature map processed by the graph learning layer, and the adjacency matrix and the feature map are input into a plurality of stacked time-frequency graph convolution blocks passing through the second branch;

[0031] Using the second branch, the three-channel feature map is passed through the first convolution block to obtain a feature map with an increased number of channels, and the feature map with an increased number of channels is input into a plurality of stacked time-frequency map convolution blocks;

[0032] Multiple stacked time-frequency graph convolution blocks are used to extract the global frequency information and long-distance time dependency in the feature graph processed by the adjacency matrix and the graph learning layer, as well as the feature graph after increasing the number of channels. The feature graph processed by multiple stacked time-frequency graph convolution blocks is input into the second convolution block to reduce the frequency dimension, and a high-dimensional feature representation of the noisy speech data to be enhanced is obtained.

[0033] Furthermore, the three-channel feature map is passed through a graph learning layer using the first branch, and an adjacency matrix is ​​learned according to the frequency dimension relationship between the feature maps to obtain the adjacency matrix and the feature map processed by the graph learning layer, including:

[0034] Performing average pooling on the three-channel feature maps in the time dimension to obtain a pooled speech feature map;

[0035] Calculating the similarity of the speech feature graph in the frequency dimension to obtain an adjacency matrix;

[0036] The adjacency matrix is ​​sparsely processed to obtain an adjacency matrix and a feature graph processed by a graph learning layer.

[0037] Furthermore, the time-frequency graph convolution block includes a dilated convolution block and a graph convolution layer;

[0038] The dilated convolution block is sequentially connected to a two-dimensional convolution layer, an Instance Norm layer, and a PReLU activation function, and is used to extract the time dimension dependency in the feature map processed by the graph learning layer and the feature map after the number of channels is increased, while keeping the number of channels of the feature map processed by the graph learning layer and the feature map after the number of channels is increased consistent with the number of channels of the high-dimensional feature representation of the noisy speech data to be enhanced;

[0039] The graph convolution layer is used to extract the frequency dimension dependency between the feature graph processed by the graph learning layer and the feature graph after the number of channels is increased according to the adjacency matrix obtained by the graph learning layer.

[0040] Furthermore, the propagation method between graph convolutional layers is expressed as:

[0041] (3);

[0042] In the formula, Indicates The feature map of the layer output, represents the weight coefficient of the residual connection, Indicates Layer passes through dilated convolutional blocks The obtained feature map, represents the adjacency matrix obtained from the graph learning layer, represents an adjacency matrix with self-loops added, where , I represents the identity matrix, and Respectively represent the adjacency matrix obtained according to the graph learning layer and the adjacency matrix with self-loops added The degree matrix of Indicates The network training parameters of the layer.

[0043] In a second aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention replaces the encoder module of the metric generative adversarial network CMGAN with a time-frequency graph convolutional encoder module, which can more effectively process noisy speech data. The present invention captures the long-term dependencies in the speech signal through the time convolutional network, which helps to process continuous and dynamically changing speech features. At the same time, the global frequency feature information is extracted through the graph convolutional network, which helps to more accurately understand the characteristics of the speech signal at different frequencies. The combination of the two not only improves the learning ability of the metric generative adversarial network CMGAN for speech signal features, but also significantly enhances the robustness of the metric generative adversarial network CMGAN in complex noise environments, thereby improving the effect and accuracy of speech enhancement, and achieving improved speech enhancement performance with fewer model parameters and calculations;

[0046] 2. The present invention further improves the performance of the metric generative adversarial network (CMGAN) by introducing dense residual connections to fuse multi-scale feature information. Dense residual connections help alleviate the gradient vanishing problem in deep networks, promote the effective transmission of information and the reuse of features, and enable the metric generative adversarial network (CMGAN) to more comprehensively utilize feature information at different levels and further optimize the results of speech enhancement;

[0047] 3. The present invention introduces perceptual contrast stretching PCS to process the original pure speech, thereby enhancing the ability of the metric generative adversarial network CMGAN to capture speech detail features, which not only improves the perceptual quality of speech, but also enables the metric generative adversarial network CMGAN to learn more refined and realistic speech features during training. At the same time, the present invention provides a loss function for the metric generative adversarial network CMGAN, which comprehensively considers the differences between multiple speech features, including amplitude spectrum, real part and imaginary part, etc., further improving the accuracy and fidelity of speech enhancement. In addition, the improved metric generative adversarial network and the method provided by the present invention effectively extract high-dimensional feature representations of noisy speech data, and successfully separate the noise and retain the effective components in the speech, thereby significantly improving the quality of speech enhancement.

[0048] 4. The present invention also constructs a network structure of a metric generative adversarial network CMGAN, including a preprocessing module, a time-frequency graph convolutional encoder module, an enhancement module, and a decoder module, etc., to achieve comprehensive processing of noisy speech data. In particular, the time-frequency graph convolutional encoder module successfully extracts global frequency information and long-distance time dependencies by combining a graph learning layer and multiple stacked time-frequency graph convolutional blocks, further enhancing the learning ability of the metric generative adversarial network CMGAN for complex speech features, so that the metric generative adversarial network CMGAN can show strong generalization ability and adaptability when facing different types of noise and speech features. At the same time, by introducing advanced technologies such as dilated convolution dense connection networks, the stability and robustness of the metric generative adversarial network CMGAN are improved, so that the method provided by the present invention has a wider application prospect in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a flow chart of a speech enhancement method based on a time-frequency graph convolutional network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0051] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0052] Example 1

[0053] like Figure 1 As shown, this embodiment introduces a speech enhancement method based on a time-frequency graph convolutional network, which is characterized by comprising:

[0054] Step 1: Obtain the noisy speech data to be enhanced.

[0055] Step 2: Input the noisy speech data to be enhanced into the improved metric generative adversarial network CMGAN, and output the enhanced speech time domain signal.

[0056] Before the noisy speech data to be enhanced is input into the improved metric generative adversarial network CMGAN, it also includes:

[0057] The original clean speech is processed using perceptual contrast stretching PCS to obtain processed clean speech.

[0058] Among them, the processed pure speech is expressed as:

[0059] (1);

[0060] In the formula, Represents the amplitude spectrum of the original pure speech after short-time Fourier transform STFT, where Indicates the amplitude, Represents the original pure speech amplitude spectrum The amplitude spectrum obtained after perceptual contrast stretching PCS processing, (·) represents the logarithmic function, It represents the perceptual score of each frequency band of the original clean speech amplitude spectrum derived based on the frequency band importance function BIF.

[0061] In this embodiment, the perceptual scores of each frequency band of the original pure speech amplitude spectrum derived based on the frequency band importance function BIF are: The values ​​for different frequency bands are shown in the following table:

[0062] Table 1 PCS coefficients corresponding to different frequency bands

[0063]

[0064] Using the loss function, the difference between the enhanced speech time domain signal output by the improved metric generative adversarial network CMGAN and the processed pure speech is calculated to obtain the total loss; where the loss function is expressed as:

[0065] (2);

[0066] In the formula, Represents the loss function of measuring the generation of adversarial networks CMGAN, represents the time domain loss term The weight coefficient of , Represents the original pure voice, It represents the enhanced speech time domain signal output by the improved metric generative adversarial network CMGAN, represents the L1 norm, represents the frequency domain loss term The weight coefficient of Represents the reconstruction loss term The weight coefficient of , , , and They represent the amplitude, real part and imaginary part of the pure speech time spectrum obtained by perceptual contrast stretching PCS, short-time Fourier transform STFT and power law compression of the original pure speech, respectively. , and denote the amplitude, real part and imaginary part respectively, , and They represent the amplitude, real part and imaginary part of the speech time-frequency spectrum of the enhanced speech time-domain signal output by the improved metric generative adversarial network CMGAN, respectively. represents the L2 norm, represents the discriminator loss used to improve the PESQ score of enhanced speech The weight coefficient of .

[0067] The weights of the improved metric generative adversarial network CMGAN are updated via back-propagation according to the total loss.

[0068] Among them, the improved methods for measuring generative adversarial networks (CMGANs) include:

[0069] Replace the encoder module of the metric generative adversarial network CMGAN with a time-frequency graph convolutional encoder module;

[0070] In the time-frequency graph convolution encoder module, the temporal convolutional network and graph convolutional network are used to respectively extract the long-term dependency and global frequency feature information of the noisy speech data to be enhanced. At the same time, dense residual connections are introduced between each time-frequency graph convolution block to fuse multi-scale feature information.

[0071] The improved metric generative adversarial network includes a preprocessing module, a time-frequency graph convolutional encoder module, an enhancement module and a decoder module.

[0072] The preprocessing module is used to perform short-time Fourier transform on the noisy speech data to be enhanced to obtain a complex spectrum in the time-frequency domain, perform power-law compression processing on the amplitude spectrum of the complex spectrum to obtain a processed amplitude spectrum, and form a three-channel feature map with the processed amplitude spectrum and the real and imaginary parts of the complex spectrum.

[0073] In this embodiment, after the noisy speech data to be enhanced passes through the preprocessing module, the three-channel feature map is obtained as follows: ,in, Represents the feature map of the CMGAN that is used as input to improve the metric generative adversarial network, Represents the matrix dimension as The set of real numbers, and Respectively represent the time frame and frequency dimensions of the three-channel feature map, Represents the data batch size of the three-channel feature map.

[0074] The time-frequency graph convolution encoder module is used to extract high-dimensional feature representation of the noisy speech data to be enhanced according to the three-channel feature graph, separate the noise and retain the effective components in the noisy speech data to be enhanced.

[0075] The time-frequency graph convolution encoder module includes a graph learning layer located in a first branch, a first convolution block located in a second branch, a plurality of stacked time-frequency graph convolution blocks and a second convolution block, and a first branch and a second branch connected to the plurality of stacked time-frequency graph convolution blocks.

[0076] The method for extracting high-dimensional feature representation of noisy speech data to be enhanced according to the three-channel feature map includes:

[0077] The first branch is used to pass the three-channel feature map through the graph learning layer, and the adjacency matrix is ​​learned according to the frequency dimension relationship between the feature maps. The adjacency matrix and the feature map processed by the graph learning layer are obtained and input into multiple stacked time-frequency graph convolution blocks passing through the second branch.

[0078] Among them, the first branch is used to pass the three-channel feature map through the graph learning layer, and the adjacency matrix is ​​learned according to the frequency dimension relationship between the feature maps to obtain the adjacency matrix and the feature map processed by the graph learning layer, including:

[0079] Perform average pooling on the three-channel feature maps in the time dimension to obtain a pooled speech feature map;

[0080] The adjacency matrix is ​​obtained by calculating the similarity of the speech feature graph in the frequency dimension;

[0081] The adjacency matrix is ​​sparsely processed to obtain the adjacency matrix and the feature graph processed by the graph learning layer.

[0082] In this embodiment, the three-channel feature map is After being transmitted to the time-frequency graph convolution encoder, the three-channel feature map is firstly trained using the graph learning layer. Perform average pooling in the time dimension to obtain the pooled speech feature map , expressed as:

[0083] (3);

[0084] In the formula, Represents a three-channel feature map The time dimension, Indicates The three-channel feature map of the input time step.

[0085] Then, based on the speech feature map Calculate the similarity in the frequency dimension to obtain the adjacency matrix , which effectively models the correlation between frequencies, where the adjacency matrix The specific calculation method is expressed as:

[0086] (4);

[0087] (5);

[0088] In the formula, Representing speech feature graph In frequency index and The similarity score between the corresponding feature vectors, represents the exponential function, Represents the weight vector of feature fusion The transpose of , where , represents the channel dimension, represents the activation function, , and Represents the speech feature graph In frequency index , and The corresponding eigenvector, Representation and The network training parameters for dimension transformation, Representation and or The network training parameters for dimension transformation, .

[0089] In this embodiment, the adjacency matrix When performing sparse processing, only the adjacency matrix is ​​retained A maximum number of values ​​are preset before each row, and the remaining values ​​are set to 0, thereby reducing the amount of calculation and suppressing noise interference, and obtaining the feature map after the adjacency matrix and the graph learning layer are processed.

[0090] The second branch is used to pass the three-channel feature map through the first convolution block to obtain a feature map with an increased number of channels, and the feature map with an increased number of channels is input into multiple stacked time-frequency map convolution blocks.

[0091] Multiple stacked time-frequency graph convolution blocks are used to extract the global frequency information and long-distance time dependency in the feature graph processed by the adjacency matrix and the graph learning layer, as well as the feature graph after increasing the number of channels. The feature graph processed by multiple stacked time-frequency graph convolution blocks is input into the second convolution block to reduce the frequency dimension, and a high-dimensional feature representation of the noisy speech data to be enhanced is obtained.

[0092] In this embodiment, the first convolution block sequentially connects the two-dimensional convolution layer, the Instance Norm layer, and the PReLU activation function. The two-dimensional convolution uses a convolution kernel size of , a convolutional layer with a step size of 1 to improve the three-channel feature map The channel dimension is , and get the feature map after increasing the number of channels , input into multiple stacked time-frequency graph convolution blocks.

[0093] In this embodiment, the second convolution block sequentially connects the two-dimensional convolution layer, the Instance Norm layer, and the PReLU activation function. The two-dimensional convolution uses a convolution kernel size of , the step length is The convolution layer reduces the frequency dimension of the feature, reduces the amount of calculation, and obtains the high-dimensional feature representation of the noisy speech data to be enhanced. .

[0094] In this embodiment, the time-frequency graph convolution block is connected in the following manner: the input of the previous time-frequency graph convolution block is connected to the output of the next time-frequency graph convolution block, wherein the feature transfer manner between the time-frequency graph convolution blocks is:

[0095] (6);

[0096] In the formula, Indicates The feature matrix output by the time-frequency graph convolution block, represents the feature map output by the first convolutional block, Represents the feature matrix output by the first time-frequency graph convolution block, Indicates The feature matrix output by the time-frequency graph convolution block, represents the time-frequency graph convolution block, Indicates that each feature map is concatenated in the channel dimension.

[0097] Among them, the time-frequency graph convolution block contains a dilated convolution block and a graph convolution layer.

[0098] The dilated convolution block sequentially connects the two-dimensional convolution layer, the Instance Norm layer and the PReLU activation function to extract the time dimension dependency in the feature map processed by the graph learning layer and the feature map after the number of channels is increased, while keeping the number of channels of the feature map processed by the graph learning layer and the feature map after the number of channels is increased consistent with the number of channels of the high-dimensional feature representation of the noisy speech data to be enhanced.

[0099] The graph convolution layer is used to extract the frequency dimension dependency of the feature graph processed by the graph learning layer and the feature graph after increasing the number of channels according to the adjacency matrix obtained by the graph learning layer.

[0100] In this embodiment, the propagation mode between graph convolutional layers is expressed as:

[0101] (7);

[0102] In the formula, Indicates The feature map of the layer output, represents the weight coefficient of the residual connection, Indicates Layer passes through dilated convolutional blocks The obtained feature map is represents the adjacency matrix obtained from the graph learning layer, represents an adjacency matrix with self-loops added, where , I represents the identity matrix, and Respectively represent the adjacency matrix obtained according to the graph learning layer and the adjacency matrix with self-loops added The degree matrix of Indicates The network training parameters of the layer.

[0103] The enhancement module is used to output a feature map of the separated noise part based on the high-dimensional feature representation using four consecutive two-stage Conformer blocks.

[0104] In this embodiment, the high-dimensional feature representation of the noisy speech data to be enhanced is It is transmitted to the enhancement module, and the noise part in the speech is separated through 4 consecutive two-stage Conformer blocks to obtain the feature map of the separated noise part.

[0105] The two-stage Conformer block can capture the time dependency and frequency dependency of the input features respectively.

[0106] High-dimensional feature representation of noisy speech data to be enhanced First, the dimension transformation is performed to obtain the time-dependent feature map to be extracted , which is used in the first Conformer block to capture temporal dependencies. Then, the output feature map of the first Conformer block is and Perform residual connection, where Represents the output feature map of the first Conformer block, and re-transforms the dimension to obtain the frequency-dependent feature map to be extracted , used in the second Conformer block to capture frequency dependency. After the residual connection, the output feature map is finally obtained after the dimension transformation. ,in Represents the output feature map of the two-stage Conformer block.

[0107] Each Conformer block consists of two feedforward neural networks with halved feature dimensions, a multi-head self-attention layer with four heads and a convolution module between the feedforward neural networks. The convolution module sequentially undergoes layer normalization, point-by-point convolution layer, gated linear unit GLU activation function, 1D depth convolution layer, swish activation function, point-by-point convolution layer, and dropout layer.

[0108] In addition, each Conformer block uses residual connections to connect the input with the output.

[0109] The decoder module includes an amplitude decoder and a complex decoder, which is used to decouple the amplitude mask and complex information from the feature map of the separated noise part based on the dilated convolution dense connection network, combine the masked amplitude with the complex information output by the complex decoder to generate a complex spectrum, perform inverse power-law compression on the amplitude spectrum of the complex spectrum, and then generate an enhanced speech time domain signal through inverse Fourier transform.

[0110] In this embodiment, the decoder module extracts output from the feature map of the separated noise part after the two-stage Conformer block in a decoupled manner, and the output path includes an amplitude decoder path and a complex decoder path.

[0111] The goal of the amplitude decoder is to predict the amplitude mask to denoise the amplitude spectrum of the enhanced speech.

[0112] The goal of the complex decoder is to directly predict the real and imaginary parts of the complex spectrum of the enhanced speech.

[0113] Both the amplitude decoder and the complex decoder include a dilated dense network Dilated DenseNet, which uses a convolution kernel size of The sub-pixel convolutional layer mask decoder path and the complex decoder path are used to upsample the frequency dimension back to the frequency dimension of the network input feature map.

[0114] The amplitude decoder uses a convolution block to compress the number of channels of the input feature map to 1, and then predicts the amplitude mask through a convolution layer of the PReLU activation function. The PReLU activation function learns a different slope for each frequency band of the amplitude spectrum. In this embodiment, the initial slope of the PReLU activation function is defined as 0.2.

[0115] The structure of the complex decoder is the same as the amplitude decoder, but no activation function is applied to the output of the complex decoder.

[0116] In this embodiment, the amplitude spectrum of the original noisy speech Amplitude spectrum after amplitude mask enhancement First, the noise phase of the original noisy speech Combined to obtain the complex spectrum of the amplitude spectrum enhanced speech. Then, the complex spectrum of the amplitude spectrum enhanced speech is combined with the output of the complex decoder Add element by element to obtain the complex spectrum of the enhanced speech output by the final network:

[0117] (8);

[0118] (9);

[0119] In the formula, represents the real part of the enhanced speech complex spectrum output by the improved metric generative adversarial network CMGAN, represents the imaginary part of the enhanced speech complex spectrum output by the improved metric generative adversarial network CMGAN, represents the real part of the enhanced speech complex spectrum output by the complex decoder, represents the imaginary part of the enhanced speech complex spectrum output by the complex decoder, represents the sine function, Represents the cosine function.

[0120] Complex spectrum The inverse processing of power-law compression is performed and the inverse short-time Fourier transform is applied to obtain the enhanced speech time domain signal.

[0121] Step 3: Experimentally verify the enhanced speech time domain signal.

[0122] Based on the above steps, experiments were conducted on the Voice Band+DEMAND dataset, using the objective speech quality evaluation PESQ and three evaluation criteria based on the mean opinion score MOS: speech signal quality CSIG, background noise interference CBAK and overall speech quality COVL, to comprehensively evaluate the performance of the enhanced speech time domain signal obtained by the method provided in this embodiment.

[0123] At the same time, the number of parameters of the improved metric generative adversarial network CMGAN and the floating-point operations required per second are compared.

[0124] In this embodiment, the batch size of the initialized improved metric generative adversarial network CMGAN is 4, the sampling rate is 16kHz, and the number of channels is 64.

[0125] The audio data of the original clean speech of the trained improved metric generative adversarial network CMGAN is cropped or spliced ​​to form segments of 2 seconds duration.

[0126] This embodiment adopts short-time Fourier transform, with a window length of 25 ms, a window shift of 6.25 ms, a Hamming window, and a 400-point fast Fourier transform.

[0127] In this embodiment, when training the improved metric generative adversarial network CMGAN, after 120 rounds of training, the parameters of the improved metric generative adversarial network CMGAN are adjusted using the AdamW optimizer, and the learning rate is initially set to 5× , and the learning rate is reduced by half every 30 training cycles.

[0128] Table 2 shows the comparison results of the PESQ, CSIG, CBAK and COVL scores of the method provided in this embodiment and other prior art methods on the Voice Band+DEMAND dataset, as well as the model parameter amount and calculation amount.

[0129] Table 2 Performance comparison results of the method provided in this embodiment and the prior art method

[0130]

[0131] SCP-CMGAN, MP-SENet and UPB-CMGAN are all improved methods based on the CMGAN model. Specifically, SCP-CMGAN improves the training process of the GAN network by introducing a consistency loss function and a self-correction optimization strategy; MP-SENet adopts a method of directly denoising the amplitude spectrum and phase spectrum in parallel; and UPB-CMGAN proposes a strategy for optimizing phase reconstruction, which relaxes the strict requirements for accurate phase and thus improves the performance of the model.

[0132] As can be seen from Table 2, the method provided in this embodiment has the highest PESQ score compared with other prior art methods. Compared with CMGAN, the PESQ score of the method provided in this embodiment is improved by 0.21, and the required floating point operations and model parameters are reduced by 16.7% and 6.6% respectively. Therefore, the method provided in this embodiment can effectively improve the accuracy and reliability of speech enhancement tasks.

[0133] Example 2

[0134] Based on the same inventive concept as Example 1, this example introduces a computer system, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described in Example 1.

[0135] In summary, the present invention replaces the encoder module of the metric generative adversarial network CMGAN with a time-frequency graph convolutional encoder module, which can more effectively process noisy speech data. The present invention captures the long-term dependencies in the speech signal through the time convolutional network, which helps to process continuous and dynamically changing speech features. At the same time, the global frequency feature information is extracted through the graph convolutional network, which helps to more accurately understand the characteristics of the speech signal at different frequencies. The combination of the two not only improves the learning ability of the metric generative adversarial network CMGAN for speech signal features, but also significantly enhances the robustness of the metric generative adversarial network CMGAN in complex noise environments, thereby improving the effect and accuracy of speech enhancement;

[0136] The present invention further improves the performance of the metric generative adversarial network (CMGAN) by introducing dense residual connections to fuse multi-scale feature information. Dense residual connections help alleviate the gradient vanishing problem in deep networks, promote the effective transmission of information and the reuse of features, and enable the metric generative adversarial network (CMGAN) to more comprehensively utilize feature information at different levels and further optimize the results of speech enhancement.

[0137] The present invention introduces perceptual contrast stretching PCS to process the original pure speech, thereby enhancing the ability of the metric generative adversarial network CMGAN to capture speech detail features, which not only improves the perceptual quality of speech, but also enables the metric generative adversarial network CMGAN to learn more refined and realistic speech features during training. At the same time, the present invention provides a loss function for the metric generative adversarial network CMGAN, which comprehensively considers the differences between multiple speech features, including amplitude spectrum, real part and imaginary part, etc., further improving the accuracy and realism of speech enhancement. In addition, the improved metric generative adversarial network, the method provided by the present invention effectively extracts high-dimensional feature representations of noisy speech data, and successfully separates noise and retains effective components in speech, thereby significantly improving the quality of speech enhancement.

[0138] The present invention also constructs a network structure of a metric generative adversarial network CMGAN, including a preprocessing module, a time-frequency graph convolution encoder module, an enhancement module and a decoder module, etc., to achieve comprehensive processing of noisy speech data. In particular, the time-frequency graph convolution network module successfully extracts global frequency information and long-distance time dependencies by combining a graph learning layer and multiple stacked time-frequency graph convolution blocks, further enhancing the learning ability of the metric generative adversarial network CMGAN for complex speech features. This comprehensive processing method enables the metric generative adversarial network CMGAN to show strong generalization ability and adaptability when facing different types of noise and speech features. At the same time, by introducing advanced technologies such as dilated convolution dense connection networks, the stability and robustness of the metric generative adversarial network CMGAN are improved, so that this method has a wider application prospect in practical applications.

[0139] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A speech enhancement method based on time-frequency graph convolutional network, characterized in that: include: Acquire noisy speech data to be enhanced; Input the noisy speech data to be enhanced into the improved metric generative adversarial network CMGAN, and output an enhanced speech time domain signal; The improved method of the metric generative adversarial network CMGAN includes: Replace the encoder module of the metric generative adversarial network CMGAN with a time-frequency graph convolutional encoder module; In the time-frequency graph convolution encoder module, a temporal convolutional network and a graph convolutional network are used to respectively extract the long-term dependency and global frequency feature information of the noisy speech data to be enhanced, and a dense residual connection is introduced between each time-frequency graph convolution block to fuse multi-scale feature information.

2. The speech enhancement method based on time-frequency graph convolutional network according to claim 1 is characterized in that: Before inputting the noisy speech data to be enhanced into the improved metric generative adversarial network CMGAN, the method further includes: The original clean speech is processed using perceptual contrast stretching PCS to obtain processed clean speech; Using the loss function, calculate the difference between the enhanced speech time domain signal output by the improved metric generative adversarial network CMGAN and the processed pure speech to obtain the total loss; The weights of the improved metric generative adversarial network CMGAN are updated via back-propagation according to the total loss.

3. The speech enhancement method based on time-frequency graph convolutional network according to claim 2 is characterized in that: The original clean speech is processed using perceptual contrast stretching PCS to obtain processed clean speech, wherein the processed clean speech is expressed as: （1）； In the formula, Represents the amplitude spectrum of the original pure speech after short-time Fourier transform STFT, where Indicates the amplitude, Represents the original pure speech amplitude spectrum The amplitude spectrum obtained after perceptual contrast stretching PCS processing, (·) represents the logarithmic function, It represents the perceptual score of each frequency band of the original clean speech amplitude spectrum derived based on the frequency band importance function BIF.

4. The speech enhancement method based on time-frequency graph convolutional network according to claim 2 is characterized in that: The loss function is used to calculate the difference between the enhanced speech time domain signal output by the improved metric generative adversarial network CMGAN and the processed pure speech to obtain the total loss, wherein the loss function is expressed as: （2）； In the formula, Represents the loss function of measuring the generation of adversarial networks CMGAN, represents the time domain loss term The weight coefficient of , Represents the original pure voice, It represents the enhanced speech time domain signal output by the improved metric generative adversarial network CMGAN, represents the L1 norm, represents the frequency domain loss term The weight coefficient of Represents the reconstruction loss term The weight coefficient of , , , and They represent the amplitude, real part and imaginary part of the pure speech time spectrum obtained by perceptual contrast stretching PCS, short-time Fourier transform STFT and power law compression of the original pure speech, respectively. , and denote the amplitude, real part and imaginary part respectively, , and They represent the amplitude, real part and imaginary part of the speech time-frequency spectrum of the enhanced speech time-domain signal output by the improved metric generative adversarial network CMGAN, respectively. represents the L2 norm, represents the discriminator loss used to improve the PESQ score of enhanced speech The weight coefficient of .

5. The speech enhancement method based on time-frequency graph convolutional network according to claim 1 is characterized in that: The improved metric generation adversarial network comprises: A preprocessing module, used for performing short-time Fourier transform on the noisy speech data to be enhanced to obtain a complex spectrum in the time-frequency domain, performing power-law compression processing on the amplitude spectrum of the complex spectrum to obtain a processed amplitude spectrum, and forming a three-channel feature map with the processed amplitude spectrum and the real part and imaginary part of the complex spectrum; A time-frequency graph convolution encoder module, used for extracting high-dimensional feature representation of the noisy speech data to be enhanced according to the three-channel feature graph, separating noise and retaining effective components in the noisy speech data to be enhanced; An enhancement module, configured to output a feature map of a separated noise part using four consecutive two-stage Conformer blocks according to the high-dimensional feature representation; The decoder module includes an amplitude decoder and a complex decoder, which is used to decouple the amplitude mask and complex information from the feature map of the separated noise part based on the dilated convolution dense connection network, combine the masked amplitude with the complex information output by the complex decoder to generate a complex spectrum, perform inverse power-law compression on the amplitude spectrum of the complex spectrum, and then generate an enhanced speech time domain signal through inverse Fourier transform.

6. The speech enhancement method based on time-frequency graph convolutional network according to claim 5 is characterized in that: The time-frequency graph convolution encoder module includes a graph learning layer located in a first branch, a first convolution block located in a second branch, a plurality of stacked time-frequency graph convolution blocks and a second convolution block, and a first branch and a second branch connected to the plurality of stacked time-frequency graph convolution blocks; The method of extracting high-dimensional feature representation of noisy speech data to be enhanced according to the three-channel feature map includes: The three-channel feature map is passed through a graph learning layer by using the first branch, and an adjacency matrix is ​​learned according to the frequency dimension relationship between the feature maps to obtain the adjacency matrix and the feature map processed by the graph learning layer, and the adjacency matrix and the feature map are input into a plurality of stacked time-frequency graph convolution blocks passing through the second branch; Using the second branch, the three-channel feature map is passed through the first convolution block to obtain a feature map with an increased number of channels, and the feature map with an increased number of channels is input into a plurality of stacked time-frequency map convolution blocks; Multiple stacked time-frequency graph convolution blocks are used to extract the global frequency information and long-distance time dependency in the feature graph processed by the adjacency matrix and the graph learning layer, as well as the feature graph after increasing the number of channels. The feature graph processed by multiple stacked time-frequency graph convolution blocks is input into the second convolution block to reduce the frequency dimension, and a high-dimensional feature representation of the noisy speech data to be enhanced is obtained.

7. The speech enhancement method based on time-frequency graph convolutional network according to claim 6 is characterized in that: The three-channel feature map is passed through the graph learning layer by using the first branch, and the adjacency matrix is ​​learned according to the frequency dimension relationship between the feature maps to obtain the adjacency matrix and the feature map processed by the graph learning layer, including: Performing average pooling on the three-channel feature maps in the time dimension to obtain a pooled speech feature map; Calculating based on the similarity of the speech feature graph in the frequency dimension to obtain an adjacency matrix; The adjacency matrix is ​​sparsely processed to obtain an adjacency matrix and a feature graph processed by a graph learning layer.

8. The speech enhancement method based on time-frequency graph convolutional network according to claim 6 is characterized in that: The time-frequency graph convolution block includes a dilated convolution block and a graph convolution layer; The dilated convolution block is sequentially connected to a two-dimensional convolution layer, an Instance Norm layer, and a PReLU activation function, and is used to extract the time dimension dependency in the feature map processed by the graph learning layer and the feature map after the number of channels is increased, while keeping the number of channels of the feature map processed by the graph learning layer and the feature map after the number of channels is increased consistent with the number of channels of the high-dimensional feature representation of the noisy speech data to be enhanced; The graph convolution layer is used to extract the frequency dimension dependency between the feature graph processed by the graph learning layer and the feature graph after the number of channels is increased according to the adjacency matrix obtained by the graph learning layer.

9. The speech enhancement method based on time-frequency graph convolutional network according to claim 8 is characterized in that: The propagation method between graph convolutional layers is expressed as: （3）； In the formula, Indicates The feature map of the layer output, represents the weight coefficient of the residual connection, Indicates Layer after dilated convolutional block The obtained feature map is represents the adjacency matrix obtained from the graph learning layer, represents an adjacency matrix with self-loops added, where , I represents the identity matrix, and Respectively represent the adjacency matrix obtained according to the graph learning layer and the adjacency matrix with self-loops added The degree matrix of Indicates The network training parameters of the layer.

10. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images