Speech enhancement method based on fusion of improved generative adversarial network and convex conjugate transformation

Abstract

The invention provides an improved generative adversarial network and convex conjugate transform fusion speech enhancement method, and belongs to the technical field of speech enhancement, and the method comprises the steps: employing a loss function of a discriminator constructed by a convex function, and updating the parameters of a pre-designed discriminator and the parameters of a pre-designed generator; adding a time-frequency attention mechanism in each coding layer and decoding layer of the pre-designed generator, adding a time-frequency attention mechanism between the decoding layers of the pre-designed discriminator, and generating third false voice sample data by adopting the generator corresponding to the final parameters of the generator, the discriminator corresponding to the final discriminator parameter is adopted to discriminate the third false voice sample data, and a voice enhancement result is obtained.According to the method, after convex conjugate transformation and a time-frequency attention mechanism are introduced, the voice denoising effect and the original sound fidelity in a complex noise environment are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech enhancement, and in particular relates to a speech enhancement method combining an improved generative adversarial network with a convex conjugate transformation. Background Art

[0002] Speech enhancement technology is an important research direction in the field of speech signal processing and is widely used in scenarios such as voice communication, speech recognition, intelligent voice assistants, and hearing aids. In these applications, speech signals are often interfered by complex environmental noise, which affects speech quality and intelligibility. Therefore, how to effectively remove noise in a noisy environment while maintaining the clarity and naturalness of speech has become a core challenge for speech enhancement technology. Traditional speech enhancement methods, such as spectral subtraction, adaptive filtering, statistical models, and subspace methods, have achieved certain results in simple noise environments in the early days, but have limited denoising effects when faced with complex noise, different signal-to-noise ratio conditions, and multi-speaker speech. These methods usually have problems such as music noise, speech distortion, and signal detail loss, and are difficult to adapt to complex and changeable noise scenarios. In recent years, deep learning technology, especially generative adversarial networks (GANs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs), has made significant progress in the field of speech enhancement. By learning the characteristics of large-scale data, they have shown good denoising effects in complex noise environments. However, deep learning methods still face challenges such as unstable model training, insufficient adaptability to complex noise, and poor preservation of speech details. For example, GAN is prone to unstable convergence and smoothing of generated speech signals in generative adversarial learning, which weakens the naturalness of speech.

[0003] To address the above challenges, researchers have tried to improve the ability to capture noise features by improving network structure, designing new loss functions, and introducing attention mechanisms, and have achieved certain improvements. Convex optimization, as a mathematical optimization method with advantages such as easy global optimal solution and efficient convergence, has been widely used in signal processing and machine learning. The stability and theoretical guarantee of convex optimization make it an ideal choice for solving model optimization problems. In recent years, some studies have tried to combine convex optimization with deep learning, and improve the training process of deep learning models through the stability advantage of convex optimization, thereby improving the effect of speech enhancement. However, the application of convex optimization in the field of speech enhancement is still in its infancy. How to effectively combine the advantages of convex optimization and deep learning to improve the performance of existing speech enhancement technology is still a direction worthy of in-depth exploration. At present, the challenges faced by speech enhancement technology in complex noisy environments, including the limitations of traditional methods in denoising effect and speech fidelity, and the problems of deep learning methods in model stability, adaptability and data dependence, have prompted researchers to continuously explore new solutions. Combining the stability of convex optimization with the modeling ability of deep learning, improving speech enhancement technology, and improving the speech denoising effect in complex environments are important directions of current research.

[0004] In the field of speech enhancement technology, methods based on deep learning are developing rapidly internationally, especially in the study of speech denoising in complex noise environments. Technologies such as generative adversarial networks (GAN), deep neural networks (DNN), self-supervised learning, and time-frequency attention mechanisms have achieved remarkable results. Leading research institutions and companies around the world have proposed a variety of innovative speech enhancement methods with the help of large-scale data and powerful computing resources. For example, the WaveNet model and the time-frequency domain adaptive GAN algorithm effectively improve the clarity and naturalness of speech signals by learning complex time-frequency features. These methods continuously optimize the model's ability to handle complex noise through adversarial training and attention mechanisms. However, despite the continuous breakthroughs in the effects of these international cutting-edge technologies, their models often have problems such as unstable training, strong dependence on data, and insufficient generalization of noise features, which affect the reliability and universality of practical applications.

[0005] Domestic research institutions have also made significant progress in the field of speech enhancement and actively explored the application of deep learning in complex noisy environments. Domestic research has achieved good results in improving traditional model structures and integrating a variety of deep learning technologies, such as denoising speech signals through convolutional neural networks (CNN), recurrent neural networks (RNN) and Transformer structures. In recent years, domestic research has gradually developed in the direction of multi-task learning and attention mechanism optimization, and has performed well in international speech enhancement competitions. However, domestic research still has certain problems in model training stability, adaptability to complex noise environments, and the ability to retain speech details. In addition, due to the limitations of data resources and computing power, domestic models still have a lot of room for improvement in generalization and adaptability. Summary of the invention

[0006] In view of the shortcomings of the prior art, this application proposes a speech enhancement method that combines an improved generative adversarial network with a convex conjugate transformation.

[0007] In the first aspect, the present application proposes a speech enhancement method by integrating an improved generative adversarial network with a convex conjugate transformation, comprising:

[0008] Step S1: Acquire real speech sample data to be enhanced;

[0009] Step S2: In a generative adversarial network, a pre-designed generator is used to generate first false speech sample data;

[0010] Step S3: using a pre-designed discriminator to discriminate the real speech sample data to be enhanced and the first false speech sample data to obtain a first discrimination result;

[0011] Step S4: according to the first discrimination result, the parameters of the pre-designed generator are fixed, and the loss function of the discriminator constructed by the convex function is adopted to update the parameters of the pre-designed discriminator;

[0012] Step S5: using a pre-designed generator to generate second false voice sample data;

[0013] Step S6: using a pre-designed discriminator to discriminate the real speech sample data to be enhanced and the second false speech sample data to obtain a second discrimination result;

[0014] Step S7: according to the second discrimination result, the parameters of the pre-designed discriminator are fixed, and the loss function of the discriminator constructed by a convex function is adopted to update the parameters of the pre-designed generator;

[0015] Step S8: when the number of iterations is less than a predetermined iteration threshold, return to step S2, and regenerate the first false voice sample data according to the updated pre-designed generator parameters;

[0016] Step S9: when the number of iterations is equal to or greater than a predetermined iteration threshold, output the final parameters of the generator and the parameters of the discriminator;

[0017] Step S10: using a generator corresponding to the parameters of the final generator to generate third false speech sample data, and using a discriminator corresponding to the parameters of the final discriminator to discriminate the third false speech sample data to obtain a speech enhancement result.

[0018] In the generative adversarial network, the optimization objective function is calculated as follows:

[0019]

[0020] Among them, D(x) is the score of the discriminator D on the input data, f * (D(G(z))) is the minimization conjugate function, V(D,G) is the optimization objective function of the discriminator D and the generator G, P is the distribution of real speech sample data data (x) is the expectation of the downsampled x, P data (x) The distribution of real speech sample data, The expected value of the real speech sample data x, For noise distribution P z (z) is the expectation of the downsampled z, The expected value of the input noise z, P z (z) The distribution of the input noise z of the generator adopts Gaussian distribution or uniform distribution. D(G(z)) is the score given by the discriminator D to the sample G(z) generated by the generator G.

[0021] The pre-designed generator includes: multiple encoding layers and corresponding decoding layers, a time-frequency attention mechanism is added to each encoding layer and decoding layer of the pre-designed generator, and the output of the encoder of the pre-designed generator is concatenated with the original input of the decoding layer, and the concatenated result is used as the input of the decoding layer of the pre-designed generator; the pre-designed discriminator includes multiple decoding layers, and a time-frequency attention mechanism is added between the decoding layers of the pre-designed discriminator.

[0022] The pre-designed generator uses a loss function constructed by a convex function, and the calculation formula is as follows:

[0023]

[0024] Among them, V GAN(G) is the objective function of the generator G in the generative adversarial network, x is the real speech sample data, G(z,x) is the false speech sample data generated by the generator when the input noise is z and the real speech sample data is x, λ is the weight hyperparameter in the loss function, D(G(z,x),x) is the output of the discriminator when the false speech sample data generated by the generator is G(z,x) and the real speech sample data is x, is the expected value of the reconstruction error between the generated data and the real speech sample data, P data (x) The distribution of real speech sample data x, P z (z) is the noise distribution.

[0025] The pre-designed discriminator comprises:

[0026]

[0027] Among them, V GAN (D) is the optimization objective function of the discriminator D, D(x,x c ) is the discriminator D inputting the real sample x and context information x c The score of G(z,x c ) is the output of the generator G, that is, according to the input noise z and context information x c Generated fake samples, is the average judgment of the discriminator D on the data generated by the generator G, P is the distribution of real speech sample data data (x,x c ) Real speech sample data x and x c expectations.

[0028] The loss function of the discriminator constructed by the convex function is calculated as follows:

[0029]

[0030] ∥G(z,x)-x∥1

[0031] Where D(x) is the output of the discriminator, G(z) is the false speech sample data generated by the generator, z is the input noise, V(D,G) is the optimization objective function of the discriminator D and the generator G, x c is the context information, as input condition, P is the distribution of real speech sample data data (x,x c ) Real speech sample data x and x c expectations, is the average judgment of the discriminator D on the data generated by the generator G, is the expected value of the reconstruction error between the generated data and the real speech sample data, x is the real speech sample data, λ is the weight hyperparameter in the loss function, P z (z) is the noise distribution, P data (x c )Context information x c The true distribution of P data (x) the distribution of real speech sample data x, For x and x c The joint distribution P data (x,x c ) downsampling expectation.

[0032] In a second aspect, the present application proposes an electronic device comprising: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the speech enhancement method of fusing the improved generative adversarial network with the convex conjugate transformation.

[0033] In a third aspect, the present application proposes a computer-readable storage medium storing executable instructions, which, when executed, enable a processor to execute the speech enhancement method of integrating the improved generative adversarial network with the convex conjugate transformation.

[0034] In a fourth aspect, the present application proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the speech enhancement method of integrating the improved generative adversarial network with the convex conjugate transformation.

[0035] Beneficial effects:

[0036] This application proposes a speech enhancement method that combines an improved generative adversarial network with a convex conjugate transform, which improves the speech denoising effect and the fidelity of the original sound in a complex noise environment. After the convex conjugate transform is introduced, the retention of high-frequency details is improved; after the time-frequency attention mechanism is introduced, the high-frequency harmonics are clearer and more prominent, and the frequency resolution is improved, which significantly improves the naturalness and clarity of the speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flow chart of a speech enhancement method combining an improved generative adversarial network and a convex conjugate transformation according to an embodiment of the present application;

[0038] Figure 2 A flowchart of a speech enhancement method combining an improved generative adversarial network and a convex conjugate transformation according to an embodiment of the present application;

[0039] Figure 3 Spectrogram of the speech enhancement generative adversarial network of the embodiment of the present application;

[0040] Figure 4 The spectrogram of the embodiment of the present application that introduces a convex conjugate transformation into the speech enhancement generative adversarial network loss function;

[0041] Figure 5 Spectrogram of adding time-frequency attention mechanism to speech enhancement generative adversarial network structure in the embodiment of the present application

[0042] Figure 6 The network structure diagram of the embodiment of the present application adds a time-frequency attention mechanism to the speech enhancement generative adversarial network structure, where (a) is the structure of the pre-designed generator and (b) is the structure of the pre-designed discriminator. DETAILED DESCRIPTION

[0043] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0044] The data set used in this embodiment comes from the public data set of Valentini et al., which contains speech data of 30 speakers, 28 of which are used for training and 2 for testing. Specifically, the training set contains 10 different noise types (including 2 artificial noises and 8 noises from the Demand database), generated under 4 different signal-to-noise ratios (SNR: 15, 10, 5, 0dB). The test set contains 5 different noise types (all from the Demand database), generated under 4 different SNR conditions (17.5, 12.5, 7.5, 2.5dB). The speech in the data set comes from the Voice Bank corpus, and the speech enhancement method based on convex conjugate transformation and deep learning of the present invention is adopted.

[0045] Embodiment 1:

[0046] This embodiment proposes a speech enhancement method that combines an improved generative adversarial network with a convex conjugate transformation. Figure 1 , Figure 2 As shown, including:

[0047] Step S1: Acquire real speech sample data to be enhanced;

[0048] In this embodiment, a public dataset from Valentini et al. is used, which includes speech data of 30 speakers, of which 28 bits are used for training, which in this embodiment are used to train a pre-designed generator and a pre-designed discriminator. In this embodiment, the real speech sample data to be enhanced uses 2 bits from the public dataset.

[0049] Step S2: In a generative adversarial network, a pre-designed generator is used to generate first false speech sample data;

[0050] In the Generative Adversarial Network (GAN) structure, the generator is used to generate denoised speech signals, the discriminator is used to distinguish the generated speech from the real speech, and the generator is optimized through adversarial learning; the goal of the generator G is to generate sufficiently realistic denoised speech signals to deceive the discriminator so that it cannot distinguish the generated signal from the real clean speech signal; the goal of the generator G is to generate samples that are as close to the real data as possible while minimizing the difference with the real data. The loss function of the pre-designed generator consists of two parts:

[0051]

[0052] Part I:

[0053]

[0054] This item represents the generator trying to make the output of the discriminator close to 1, that is, to make the discriminator think that the generated sample is a real sample. The generator deceives the discriminator by minimizing this loss, so that the generated samples are considered real by the discriminator.

[0055] Part II:

[0056]

[0057] This is the L1 regularization loss, which is used to ensure that the generated samples The distance between the real sample x and the real sample x is as small as possible. L1 loss makes the generator not only focus on deceiving the discriminator, but also ensure that the generated samples are closer to the real data at the pixel level or feature level.

[0058] Step S3: using a pre-designed discriminator to discriminate the real speech sample data to be enhanced and the first false speech sample data to obtain a first discrimination result;

[0059] In this embodiment, the goal of the pre-designed discriminator D is to continuously improve its classification ability of the real clean speech signal and the denoised speech signal generated by the generator, ensuring that the two can be accurately distinguished; the goal of the discriminator D is to distinguish the real samples from the generated samples as accurately as possible, maximize its judgment score for the real samples, and minimize the wrong judgment score for the generated samples.

[0060] The loss function of the discriminator is as follows:

[0061]

[0062] Part I:

[0063]

[0064] This term indicates that the discriminator hopes to accurately classify the real sample x so that the output of the discriminator for the real sample is as close to 1 as possible. By maximizing this loss, the discriminator can identify the real sample and classify it correctly.

[0065] Part II:

[0066]

[0067] This term indicates that the discriminator processes the samples G(z,x c ), we hope that its output is as close to 0 as possible, so as to correctly identify these samples as "fake". The discriminator improves the ability to distinguish generated samples by maximizing this part of the loss.

[0068] In the confrontation between the discriminator and the generator, the discriminator continuously improves its classification ability, striving to accurately distinguish between real samples and generated samples. The generator continuously optimizes the generation strategy, trying to "cheat" the discriminator and make it mistakenly believe that the generated samples are real. This adversarial training forces the generator to generate more and more realistic data while improving the discriminator's identification ability.

[0069] Step S4: according to the first discrimination result, the parameters of the pre-designed generator are fixed, and the loss function of the discriminator constructed by the convex function is adopted to update the parameters of the pre-designed discriminator;

[0070] Step S5: using a pre-designed generator to generate second false voice sample data;

[0071] Step S6: using a pre-designed discriminator to discriminate the real speech sample data to be enhanced and the second false speech sample data to obtain a second discrimination result;

[0072] Step S7: according to the second discrimination result, the parameters of the pre-designed discriminator are fixed, and the loss function of the discriminator constructed by a convex function is adopted to update the parameters of the pre-designed generator;

[0073] Step S8: when the number of iterations is less than a predetermined iteration threshold, return to step S2, and regenerate the first false voice sample data according to the updated pre-designed generator parameters;

[0074] Step S9: when the number of iterations is equal to or greater than a predetermined iteration threshold, output the final parameters of the generator and the parameters of the discriminator;

[0075] In this embodiment, during the iteration process, through the minimization-maximization game, the generator gradually learns to generate more realistic samples, and the discriminator continuously improves its classification accuracy.

[0076] When using a real dataset, the goal of the generator is to generate a denoised speech signal based on the noise sample, while the task of the discriminator is to distinguish the denoised speech generated by the generator from the real denoised speech signal. Each training iteration is divided into two stages:

[0077] Fix the parameters of the generator and optimize the discriminator: In this stage, the parameters of the discriminator are optimized so that it can better distinguish between real samples and generated samples. The discriminator gradually improves its ability to distinguish between generated samples and real samples through learning.

[0078] Fix the parameters of the discriminator and optimize the generator: In this stage, the parameters of the generator are optimized to make the samples generated by the generator gradually close to the real data, so as to "fool" the discriminator and make it difficult for the discriminator to distinguish between real and fake samples.

[0079] By alternately optimizing the generator and the discriminator, the generator continuously improves the quality of the generated denoised signal, while the classification accuracy of the discriminator gradually increases. In this process, the confrontation between the discriminator and the generator can be seen as a "minimization-maximization game", where the generator tries to minimize the discriminator's misjudgment rate of generated samples, while the discriminator tries to maximize the ability to distinguish between real samples and generated samples.

[0080] In this minimization-maximization game process, the generator tries to generate samples that are closer and closer to the real data, while the discriminator strives to accurately distinguish these generated samples from the real samples. This game process drives the entire network model to continuously learn and improve, achieving higher generation effects.

[0081] The game strategy of the generator G: The generator continuously tries to generate high-quality samples so that the discriminator cannot easily distinguish between real data and fake data. The learning goal of the generator is to gradually learn the ability to generate real data distribution by minimizing the judgment error rate of the discriminator.

[0082] Game strategy of the discriminator D: The discriminator tries to continuously optimize its classification ability by distinguishing between real data and generated data, and maximize its accuracy in distinguishing between real samples and generated samples. This process can help the discriminator recognize increasingly complex generated data and improve its classification ability.

[0083] Using a loss function based on convex conjugate transformation in training the GAN model provides significant advantages, specifically in the following aspects:

[0084] Improve training stability: Traditional GANs often encounter gradient vanishing or gradient exploding problems during training, which leads to unstable training of the generator and discriminator. By introducing a loss function based on convex conjugate transformation, the optimization path can be smoother, making the training of the generator and discriminator more stable and avoiding common crashes or non-convergence phenomena during training.

[0085] Accelerate model convergence: Using the convex conjugate transformation loss function can make the optimization problem simpler and more controllable, which means that the model can achieve better convergence in a shorter training time. Compared with the long and unstable training process in the traditional GAN ​​model, the loss function based on the convex conjugate transformation can effectively accelerate the convergence of the model and reduce unnecessary iterations.

[0086] Improve the generation effect: Figure 3 As shown, by using the convex conjugate transformation loss function, Figure 3 The generated results in Figure 2 The original SEGAN graph is smoother and closer to the real speech spectrum. This means that the GAN model can better retain important frequency information, reduce distortion, and enhance the fidelity and naturalness of speech when generating high-quality speech signals.

[0087] Reduce the risk of model divergence and mode collapse: The introduction of convex conjugate transformation effectively reduces the possibility of model divergence during training, and avoids the phenomenon that the generator frequently falls into mode collapse when fighting against the discriminator. This further improves the generalization ability of the GAN model, so that the generated results remain consistent and high quality under a variety of different input conditions.

[0088] In summary, the loss function based on convex conjugate transformation helps solve the key problems in GAN training, making the generated speech enhancement results more natural and delicate, while greatly improving the efficiency and stability of the training process.

[0089] In step S4 and step S7, the loss function of the discriminator constructed by a convex function is used. The process of constructing the loss function is described in detail below:

[0090] Design a loss function based on convex conjugate transformation, which uses convex functions to transform the model parameter optimization problem into a convex optimization problem to improve the stability and convergence speed of model training;

[0091] In order to improve the instability in GAN training, a loss function based on convex conjugate transformation is proposed. By converting the optimization problem of GAN into a convex optimization problem, the introduction of convex conjugate transformation ensures a smoother optimization path and makes the training more stable, especially in the competition between the generator and the discriminator, which effectively avoids the problem of gradient vanishing or non-convergence.

[0092] The goal of this embodiment is to find the optimal generator G(z) through the following optimization problem, so that the generated distribution P G Closer to the actual data distribution P data :

[0093]

[0094] Among them, function f is a convex function, x is the real speech sample data, P G (x) is the distribution of data generated by generator G, P data (x) represents the real data distribution, D f (P data ||P G ) represents f-divergence, which is used to measure the real data distribution P data and generate data distribution P G Different f functions correspond to different divergence measures, and the choice of these convex functions determines the nature of the optimization.

[0095] The Fenchel conjugate is introduced to help decompose the optimization problem. Through the Fenchel conjugate transformation, the optimization problem of the discriminator can be equivalently transformed into the optimization problem of the generator, thereby avoiding the direct optimization of the non-convex generator-discriminator adversarial problem. The core idea of ​​Fenchel conjugate is to transform complex non-convex problems into a series of convex optimization sub-problems, ensuring the training stability of GAN. This embodiment can derive its corresponding conjugate function f from the convex function f(u) * (t). The conjugate function is defined as:

[0096]

[0097] Among them, sup stands for supremum, which refers to the largest value or the smallest upper bound in the set, and domf stands for the domain of the function f(u), that is, the range of values ​​of u.

[0098] Due to the reversibility of Fenchel conjugation, we can express the convex function f(u) as:

[0099]

[0100] Therefore, a generator function f(u) is designed to be defined as a convex function, namely:

[0101]

[0102] Among them, u is the input variable of the function, and u is used to describe the input characteristics of the generator; the generator function satisfies the condition f(1) = 0 and f(u) is a convex function, which ensures that when the two distributions are consistent, their divergence is zero. By solving the conjugate function, we get:

[0103]

[0104] Substituting the convex function and the conjugate function into the f-divergence formula, we can get:

[0105]

[0106] Where f* is the Fenchel conjugate function of the convex function f, domf* is the domain of f*, and the sum of the maximum values ​​is greater than the maximum value of the sum. In the formula, T represents a class of functions that satisfy χ→R, so T(x) can be used to replace t in the formula; and T(x) can be expressed as follows:

[0107]

[0108]

[0109] Among them, D w (x) is the discriminator function, a(·) is the activation function, χ is the input data space, For real number domain, T(x) can be regarded as a discriminator with a special activation function a(·); many GAN variants can be derived using different generator functions f and their corresponding activation functions a(·). First, maximize the equation Regarding the lower bound of T(x), the approximate divergence is then minimized so that the distribution learned by the generator is more similar to the distribution of the real data.

[0110] The discriminator outputs one of the parameters of the core T(x). By introducing the convex optimization framework, the discriminator is no longer just a binary classifier, but guides the optimization of the generator by learning a distribution difference metric. The optimization goal of the discriminator is to minimize the divergence between different distributions, and the output T(x) is used to measure the closeness between the generated data and the real data. By minimizing the divergence function, the generator can be further guided to generate samples that are more consistent with the real data distribution.

[0111] In the convex optimization framework, the optimization goal is to approximate the true distribution by minimizing the loss function, which can be expressed in the following form:

[0112]

[0113] Among them, there are two expectations: the first expectation It represents the expectation of the output of the discriminator D(x) under the real data distribution. By measuring the real data, the model can ensure that the generated samples are closer to the real samples. It measures the expected value of the output data of the generator G(z).z The discriminator D(G(z)) gives the score of the generated sample, which further guides the optimization of the generator. The core of convex optimization lies in its stable gradient calculation and update direction, which can effectively avoid the training difficulties caused by the instability of the loss function in traditional GAN.

[0114] In the standard GAN, when the generator learns the distribution P g and the actual data distribution P data When there is no intersection between them, that is, P g Distance P data When the probability is still very far away, he will still judge the samples generated by the generator as false with a high degree of confidence. At this time, the objective function value will be a constant, and the gradient will be 0 during back propagation, which will eventually cause the gradient to disappear.

[0115] Based on these problems, a loss function based on convex conjugate transformation is proposed to replace the cross entropy loss function in the original GAN. It is a strictly convex function, so it provides a smooth gradient signal during the optimization process. Compared with the traditional GAN ​​loss, this loss function in the form of square error can provide a more stable optimization path and avoid the problem of gradient disappearance or gradient explosion.

[0116] Propose an equation similar to the following:

[0117]

[0118] Therefore, the loss function based on convex conjugate transformation is as follows:

[0119]

[0120] Among them, D(x) represents the output of the discriminator, G(z) represents the sample generated by the generator, and z represents a random vector that obeys a certain distribution. This choice of loss function not only makes the generator have a better gradient signal when adversarial to the discriminator (compared to the Jensen-Shannon divergence of traditional GAN), but also has stronger convergence when generating realistic speech signals.

[0121]

[0122] Among them, p(x) represents the real data distribution, and q(x) represents the generated sample distribution. The above formula represents the f-divergence corresponding to the loss function designed based on the convex conjugate transformation. This divergence combines the distribution Q of the generated samples and the distribution P of the real samples to measure the difference between the samples generated by the generator and the real samples.

[0123] The design of the convex conjugate transformation loss function provides a stable theoretical basis for the application of GAN in complex tasks. By introducing f-divergence, the model can establish a more stable training path between the discriminator and the generator, avoiding the gradient vanishing and non-convergence phenomena common in traditional GAN. In the speech enhancement task, the use of the convex conjugate transformation design not only improves the generation ability of the model, but also significantly speeds up the convergence speed during the training process, and can generate high-quality clear speech in complex noisy environments.

[0124] The following is a detailed description of the introduction process of the attention mechanism:

[0125] The time-frequency attention mechanism can reduce information loss and over-smoothing, improve the retention of speech signal details, and achieve higher speech fidelity and naturalness. In the generative adversarial network, the encoder compresses the speech signal layer by layer into feature maps, and the decoder gradually reconstructs the speech signal based on these feature maps. In order to improve the effect of speech enhancement, the time-frequency attention mechanism can be embedded in the jump connection between the encoder and the decoder. The network structure diagram is shown in the figure below. Figure 6 The core idea is to process the feature map by weighting the frequency dimension in each layer of jump connection, reduce information loss, highlight key frequency features, and improve the naturalness and fidelity of speech.

[0126] The pre-designed generator includes: multiple encoding layers and corresponding decoding layers, a time-frequency attention mechanism is added to each encoding layer and decoding layer of the pre-designed generator, and the output of the encoder of the pre-designed generator is concatenated with the original input of the decoding layer, and the concatenated result is used as the input of the decoding layer of the pre-designed generator; the pre-designed discriminator includes multiple decoding layers, and a time-frequency attention mechanism is added between the decoding layers of the pre-designed discriminator.

[0127] A time-frequency attention mechanism is added to the encoding layer and decoding layer of the speech enhancement generative adversarial network. This mechanism captures the non-local correlation in the frequency dimension through skip connections and optimizes the ability to capture time-frequency features, especially the harmonic characteristics of speech, which helps the network better restore the harmonic features masked by noise; Figure 6As shown in (a), the pre-designed generator includes: 11 encoding layers and corresponding 10 decoding layers, a time-frequency attention mechanism is added to each encoding layer and decoding layer of the pre-designed generator, and the output of the encoder of the pre-designed generator is spliced ​​with the original input of the decoding layer, and the splicing result is used as the input of the decoding layer of the pre-designed generator; for example: the real speech sample data with noise is input into encoding layer 1, the output of encoding layer 1 is input into the first layer of time-frequency attention mechanism, the output of the first layer of time-frequency attention mechanism is input into encoding layer 2, and so on, the output of encoding layer 11 is input into the eleventh layer of time-frequency attention mechanism, the output in the eleventh layer of time-frequency attention mechanism is compressed into a thought vector c, the output vector c of the encoder is spliced ​​with the potential vector z, and the spliced ​​result is used as the input of the decoding layer 10, and so on, so as to realize the jump connection. Similarly, in this embodiment, the pre-designed discriminator includes 10 decoding layers, and the time-frequency attention mechanism is added between the decoding layers of the pre-designed discriminator, such as Figure 6 (b) as shown.

[0128] Among them, the encoding layer is equivalent to feature extraction: local frequency features in the time-frequency graph are extracted through convolution operations. In the processing of time-frequency graphs, feature extraction is a key step, and convolutional neural networks (CNNs) are usually used to capture the characteristics of local frequency components in time-frequency graphs. In this step, a one-dimensional convolution (Conv1D) is used to slide along the frequency dimension (channel dimension) to extract local features in different frequency ranges. The encoder performs a series of convolution operations on the input noisy speech signal. Each convolution extracts different features of the speech signal and compresses the high-dimensional speech data into gradually reduced feature maps. These feature maps not only contain the global structural information of the speech, but also cover detailed features, especially the time-frequency features of the speech. In the final layer, the "thought vector" generated by the encoder is a compressed deep feature representation that condenses the key information in the speech.

[0129] The convolution operation takes the following form:

[0130] F conv1 =Conv1D(X)

[0131] Where: X is the input time-frequency graph, the dimension is [B, C, T'], where B is the batch size, C is the frequency dimension (number of channels), and T' is the time step. conv1 It is the local frequency feature extracted by convolution.

[0132] The role of the convolution kernel is to slide on the frequency dimension through a fixed window size (kernel size) to extract local frequency features within the window. For example, if the convolution kernel size is 9, it will extract features from 9 adjacent frequency points at each time step. This local operation can capture local dependencies between frequencies, such as formants and phoneme features in speech.

[0133] In addition, the convolution operation is usually followed by an activation function (ReLU) and batch normalization (BatchNorm) to enhance the nonlinear representation ability of the model and speed up training. This part of the operation provides the basic frequency characteristics for the subsequent attention mechanism.

[0134] The time-frequency attention mechanism is equivalent to attention weighting: based on the feature map generated by the encoder, the time-frequency attention mechanism is applied to the jump connections of each layer. The core of the attention mechanism is to extract time-frequency features through convolution operations and strengthen the key frequency components in the speech through frequency dimension weighting. Generate an attention map on the frequency dimension and multiply it element by element with the input feature map. After feature extraction, the generated frequency feature map F conv1 contains local frequency information, but the importance of these features is uneven. In order to emphasize the key frequency components, the module introduces an attention mechanism to weight the features. The operation of the attention mechanism generates an attention weight map A(t,f) to represent the attention distribution on each frequency dimension. The generation of the attention map can be expressed as:

[0135] A(t,f)=softmax(Conv1D(F conv1 ))

[0136] Among them, the Softmax function normalizes the generated convolution result to the range of 0 to 1, ensuring that the sum of the weights of each frequency point is 1.

[0137] Next, the attention weight map is combined with the original feature map F conv1 Perform element-by-element multiplication to form a weighted feature map, which is the output of the time-frequency attention mechanism:

[0138] F att =A(t,f)⊙F conv1

[0139] Among them, A(t,f) is the attention weight map, which represents the weight of each frequency point. ⊙ represents the element-by-element multiplication operation, ensuring that the weighting only acts on a specific frequency dimension.

[0140] This operation enables the model to dynamically adjust the weights of frequency features, enhancing the focus on key frequencies while suppressing the influence of irrelevant or noise frequencies. For example, some frequency components in a speech signal (such as formants) may be more important than other frequency components, so these frequencies will be given greater weight.

[0141] Finally, a jump connection is performed, which is equivalent to feature fusion and reconstruction. The weighted feature map and the original input are concatenated, and further convolution processing is performed to output the optimized features.

[0142] In generating the attention-weighted feature map F att After that, the time-frequency attention module does not directly use this weighted feature map as the final output. Instead, it concatenates the weighted feature map with the original input feature X to form a richer feature map F cat :

[0143] F cat =[F att ,X]

[0144] This concatenation operation combines the attention-weighted features with the original features, ensuring that the original time-frequency graph information is not lost while retaining the attention-weighted enhanced features.

[0145] Next, the module performs a convolution operation on the concatenated feature map F cat Further convolution processing is performed to integrate the information of the two and output the final feature map F out :

[0146] F out =Conv1DD(F cat )

[0147] This convolution operation plays the role of reconstructing features, compressing and optimizing the concatenated high-dimensional features, and outputting the optimized time-frequency features F out , whose dimension is the same as the input time-frequency feature map. This process ensures efficient integration and representation of features while retaining the local capture ability of the convolution operation and the global enhancement effect of the attention mechanism.

[0148] The time-frequency attention module first extracts local frequency features, then uses the attention mechanism to dynamically adjust the weights of each frequency point, and finally concatenates and reconstructs the weighted features with the original features to output an optimized feature representation. This design can better capture the time-frequency features in speech signals, especially the attention to local important components in the frequency dimension, making the model more accurate and efficient in processing speech signals.

[0149] The weighted feature map is passed to the decoder through the jump connection, and the decoder gradually reconstructs the speech signal through deconvolution operations. In this process, thanks to the support of the time-frequency attention mechanism, the decoder can not only use the low-level features retained by the jump connection, but also refine the features of the frequency dimension through the attention mechanism. Therefore, the decoder can more efficiently restore clear and high-fidelity speech signals.

[0150] The introduction of convex conjugate transformation ensures stability and gradient transfer during training: convex conjugate transformation provides a smoother and more stable gradient signal in the loss function, avoiding the common gradient vanishing problem when generating high-frequency details. Compared with the loss function of traditional GAN, the loss function designed by convex conjugate transformation ensures the gradient transfer of the generator, making the generation effect of the model more accurate in the high-frequency part, and can better restore the high-frequency details and harmonic characteristics of the speech signal.

[0151] The time-frequency attention mechanism improves the ability to focus on and capture high-frequency features: The time-frequency attention mechanism allows the model to dynamically adjust the weights of each frequency band in the frequency dimension to ensure that high-frequency details receive more attention. When processing speech signals containing complex noise, the time-frequency attention mechanism can effectively emphasize the important components of the high-frequency area, enhance the clarity and details of the speech signal, and significantly improve the model's enhancement effect in the high-frequency part.

[0152] Step S10: using a generator corresponding to the parameters of the final generator to generate third false speech sample data, and using a discriminator corresponding to the parameters of the final discriminator to discriminate the third false speech sample data to obtain a speech enhancement result.

[0153] Finally, the final generator and discriminator are used for speech enhancement.

[0154] Experimental results verify:

[0155] In this experiment, SEGAN was used for effect comparison. SEGAN (Speech Enhancement Generative Adversarial Network) is a type of GAN network that has achieved certain results in the field of speech enhancement, especially in the low-frequency and mid-frequency bands, which can effectively remove noise and enhance speech signals. However, in the high-frequency part, the enhancement effect of SEGAN is relatively poor, mainly manifested in the loss of high-frequency details, reduced speech clarity, and blurred harmonic structure. The reasons for this phenomenon include the following:

[0156] Gradient vanishing and instability: SEGAN’s traditional GAN ​​structure may face the problem of gradient vanishing during training, especially for the generation of high-frequency details. When the generator cannot obtain smooth and stable gradients in the high-frequency part, the generated signal often lacks clear high-frequency details, resulting in poor performance of the enhanced speech in the high-frequency band.

[0157] Weak ability to capture high-frequency information: SEGAN lacks an effective mechanism to identify and highlight key features in high-frequency areas when processing high-frequency components. High-frequency details are a key component of speech, affecting the clarity and naturalness of speech, but SEGAN lacks a dedicated mechanism to focus on these high-frequency features, resulting in smoother results in high-frequency parts and low detail restoration.

[0158] In order to overcome the shortcomings of SEGAN in enhancing the high-frequency part, the convex conjugate transformation and time-frequency attention mechanism are introduced. The combination of these two technologies significantly improves the enhancement effect of speech in the high-frequency part:

[0159] The introduction of the time-frequency attention mechanism has the following advantages in speech enhancement:

[0160] 1. High-frequency detail preservation and harmonic enhancement: The time-frequency attention mechanism significantly improves the preservation of high-frequency information. Compared with the SEGAN baseline model, after the introduction of the time-frequency attention mechanism, high-frequency harmonics are clearer and more prominent, the frequency resolution is improved, and the naturalness and clarity of the speech are significantly improved.

[0161] 1) Qualitative analysis: Figure 3 , Figure 4 and Figure 5 The comparison of spectrograms shown shows how different methods perform in terms of preserving high frequency information and enhancing harmonic characteristics:

[0162] a) Figure 3 (SEGAN baseline model): In the enhanced spectrogram, especially in the high-frequency part (above 5000Hz) after 2 seconds, the high-frequency details of the speech are relatively blurred and some details are lost. This shows that SEGAN has limited ability to retain high-frequency information in its original state.

[0163] b) Figure 4 (SEGAN+convex conjugate transformation): After introducing the convex conjugate transformation into the SEGAN loss function, the spectrogram also has some improvements compared to the baseline model. Although the enhancement effect of the high-frequency part is not as obvious as the time-frequency attention mechanism, the retention of high-frequency details has been improved, especially in the processing of high-frequency information after 2.5 seconds.

[0164] c) Figure 5(SEGAN+time-frequency attention mechanism): After the introduction of the time-frequency attention mechanism, the high-frequency components in the spectrogram are significantly enhanced. Especially in the high-frequency band (the part after 2 seconds), the harmonic information is clearer, and the naturalness and detail retention of the speech are significantly improved. This shows that the time-frequency attention mechanism can better focus on the key information in the time-frequency domain, thereby improving the model's ability to process complex speech signals.

[0165] 2) In order to further verify the effect of model improvement, the experiment selected the speech file p257_387.wav and compared the signal-to-noise ratio (SNR) of the enhanced speech in the frequency band above 5000Hz. SNR is an important indicator of speech enhancement effect.

[0166] The higher the value, the better the quality of the enhanced speech. The experimental results are shown in the following table:

[0167] Table 1 Comparative experimental results

[0168]

[0169] From the results, it can be seen that the addition of the time-frequency attention mechanism significantly improves the SNR in the high-frequency band to 3.72, verifying the advantage of this method in enhancing the clarity of high-frequency speech. The convex conjugate transformation also brings improvement, but the effect is not as significant as the time-frequency attention mechanism.

[0170] 2. Frequency selectivity enhancement: The time-frequency attention mechanism can effectively highlight the key frequency area by weighting different frequencies, thereby giving priority to restoring important frequency information masked by noise in a complex noise environment. Figure 4 In the figure, the brightness distribution of the overall spectrum is more balanced, indicating that the time-frequency attention mechanism can enhance specific frequencies and reduce the interference of noise on the frequency dimension.

[0171] 3. Reduce smoothing phenomenon: In the original enhanced image of SEGAN, some frequency bands are processed too smoothly and details are seriously lost, while the time-frequency attention mechanism can enhance the details. Figure 4 In the AVC, over-smoothing of the spectrum is reduced and more details are retained, making the speech clearer and more natural.

[0172] 4. Improved speech intelligibility: Through weighted processing of the frequency dimension, the time-frequency attention mechanism not only restores speech details but also improves speech intelligibility. In complex noisy environments, this mechanism can better restore speech features.

[0173] This makes the final output enhanced speech easier to understand.

[0174] In general, after adding the time-frequency attention mechanism, the quality of speech enhancement is significantly improved, especially in terms of high-frequency detail preservation, frequency weighted selectivity, reduction of over-smoothing and improvement of intelligibility. This also verifies the effectiveness of the present invention.

[0175] Embodiment 2:

[0176] This embodiment proposes an electronic device, comprising: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the speech enhancement method of fusing the improved generative adversarial network with the convex conjugate transformation.

[0177] The electronic device may be a mobile phone, a computer or a tablet computer, etc., including a memory and a processor, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the speech enhancement method of the improved generative adversarial network and the convex conjugate transformation fusion as described in the embodiment is implemented. It can be understood that the electronic device may also include an input / output (I / O) interface and a communication component.

[0178] The processor is used to execute all or part of the steps in the speech enhancement method of improving the generation of adversarial networks and convex conjugate transformation fusion as described in the above embodiment. The memory is used to store various types of data, which may include instructions of any application or method in the electronic device, as well as data related to the application.

[0179] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the speech enhancement method of the improved generative adversarial network and convex conjugate transformation fusion described in the above embodiment.

[0180] Embodiment 3:

[0181] This embodiment provides a computer-readable storage medium storing executable instructions. When the instructions are executed and implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0182] The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the speech enhancement method for fusing an improved generative adversarial network with a convex conjugate transformation as described in the various embodiments of the present application.

[0183] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (for example, SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR abbreviation, memory data register) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, CD, server, APP (Application, abbreviation of application software) application store and other media that can store program verification codes, on which computer programs are stored. When the computer program is executed by the processor, the various steps of the above-mentioned improved speech enhancement method of fusion of generative adversarial network and convex conjugate transformation can be implemented.

[0184] Embodiment 4:

[0185] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the speech enhancement method of integrating the improved generative adversarial network with the convex conjugate transformation.

[0186] Based on such understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a computer program product.

[0187] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0188] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A speech enhancement method combining improved generative adversarial network and convex conjugate transformation, characterized in that: include: Step S1: Acquire real speech sample data to be enhanced; Step S2: In a generative adversarial network, a pre-designed generator is used to generate first false speech sample data; Step S3: using a pre-designed discriminator to discriminate the real speech sample data to be enhanced and the first false speech sample data to obtain a first discrimination result; Step S4: according to the first discrimination result, the parameters of the pre-designed generator are fixed, and the loss function of the discriminator constructed by the convex function is adopted to update the parameters of the pre-designed discriminator; Step S5: using a pre-designed generator to generate second false voice sample data; Step S6: using a pre-designed discriminator to discriminate the real speech sample data to be enhanced and the second false speech sample data to obtain a second discrimination result; Step S7: according to the second discrimination result, the parameters of the pre-designed discriminator are fixed, and the loss function of the discriminator constructed by a convex function is adopted to update the parameters of the pre-designed generator; Step S8: when the number of iterations is less than a predetermined iteration threshold, return to step S2, and regenerate the first false voice sample data according to the updated pre-designed generator parameters; Step S9: when the number of iterations is equal to or greater than a predetermined iteration threshold, output the final parameters of the generator and the parameters of the discriminator; Step S10: using a generator corresponding to the parameters of the final generator to generate third false speech sample data, and using a discriminator corresponding to the parameters of the final discriminator to discriminate the third false speech sample data to obtain a speech enhancement result.

2. The speech enhancement method of the improved generative adversarial network and convex conjugate transformation fusion according to claim 1 is characterized in that: In the generative adversarial network, the optimization objective function is calculated as follows: Among them, D(x) is the score of the discriminator D on the input data, f * (D(G(z))) is the minimization conjugate function, V(D,G) is the optimization objective function of the discriminator D and the generator G, P is the distribution of real speech sample data data (x) is the expectation of the downsampled x, P data (x) The distribution of real speech sample data, The expected value of the real speech sample data x, For noise distribution P z (z) is the expectation of the downsampled z, The expected value of the input noise z, P z (z) The distribution of the input noise z of the generator adopts Gaussian distribution or uniform distribution. D(G(z)) is the score given by the discriminator D to the sample G(z) generated by the generator G.

3. The speech enhancement method of the improved generative adversarial network and convex conjugate transformation fusion according to claim 1 is characterized in that: The pre-designed generator includes: multiple encoding layers and corresponding decoding layers, a time-frequency attention mechanism is added to each encoding layer and decoding layer of the pre-designed generator, and the output of the encoder of the pre-designed generator is concatenated with the original input of the decoding layer, and the concatenated result is used as the input of the decoding layer of the pre-designed generator; the pre-designed discriminator includes multiple decoding layers, and a time-frequency attention mechanism is added between the decoding layers of the pre-designed discriminator.

4. The speech enhancement method of the improved generative adversarial network and convex conjugate transformation fusion according to claim 1 is characterized in that: The pre-designed generator uses a loss function constructed by a convex function, and the calculation formula is as follows: Among them, V GAN (G) is the objective function of the generator G in the generative adversarial network, x is the real speech sample data, G(z,x) is the false speech sample data generated by the generator when the input noise is z and the real speech sample data is x, λ is the weight hyperparameter in the loss function, D(G(z,x),x) is the output of the discriminator when the false speech sample data generated by the generator is G(z,x) and the real speech sample data is x, is the expected value of the reconstruction error between the generated data and the real speech sample data, Px ata (x) The distribution of real speech sample data x, P z (z) is the noise distribution.

5. The speech enhancement method of improved generative adversarial network and convex conjugate transformation fusion according to claim 1, characterized in that: The pre-designed discriminator comprises: Among them, V GAN (D) is the optimization objective function of the discriminator D, D(x,x c ) is the discriminator D inputting the real sample x and context information x c The score of G(z,x c ) is the output of the generator G, that is, according to the input noise z and context information x c Generated fake samples, is the average judgment of the discriminator D on the data generated by the generator G, P is the distribution of real speech sample data data (x,x c ) Real speech sample data x and x c expectations.

6. The speech enhancement method of improved generative adversarial network and convex conjugate transformation fusion according to claim 1, characterized in that: The loss function of the discriminator constructed by the convex function is calculated as follows: Where D(x) is the output of the discriminator, G(z) is the false speech sample data generated by the generator, z is the input noise, V(D,G) is the optimization objective function of the discriminator D and the generator G, x c is the context information, as input condition, P is the distribution of real speech sample data data (f i x,x c ) Real speech sample data x and x c expectations, is the average judgment of the discriminator D on the data generated by the generator G, is the expected value of the reconstruction error between the generated data and the real speech sample data, x is the real speech sample data, λ is the weight hyperparameter in the loss function, P z (z) is the noise distribution, P data (x c )Context information x c The true distribution of P data (x) the distribution of real speech sample data x, For x and x c The joint distribution P data (x,x c ) downsampling expectation.

7. An electronic device, characterized in that: include: One or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the speech enhancement method of integrating the improved generative adversarial network and the convex conjugate transformation as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: It stores executable instructions, which, when executed, enable the processor to execute the speech enhancement method of integrating the improved generative adversarial network and the convex conjugate transformation as described in any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the speech enhancement method of integrating the improved generative adversarial network and the convex conjugate transformation as described in any one of claims 1 to 6 is implemented.

Images