Speech enhancement method based on improved generative adversarial network and convex conjugate transformation fusion

Abstract

The application provides a speech enhancement method improved by fusing a generative adversarial network and a convex conjugate transformation, and belongs to the technical field of speech enhancement. The method comprises the following steps: a loss function of a discriminator constructed by using a convex function is adopted to update parameters of a pre-designed discriminator and parameters of a pre-designed generator; a time-frequency attention mechanism is added to each coding layer and decoding layer of the pre-designed generator, and the time-frequency attention mechanism is added between decoding layers of the pre-designed discriminator; a third false speech sample data is generated by using a generator corresponding to final generator parameters; a discriminator corresponding to final discriminator parameters is used to discriminate the third false speech sample data, and a speech enhancement result is obtained. After the convex conjugate transformation and the time-frequency attention mechanism are introduced, the speech de-noising effect and original sound fidelity in a complex noise environment are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of speech enhancement, and particularly relates to a speech enhancement method combining improved generative adversarial network and convex conjugate transformation. BACKGROUND

[0002] Speech enhancement technology is an important research direction in the field of speech signal processing, and is widely used in speech communication, speech recognition, intelligent voice assistants and hearing aid devices, etc. In these applications, speech signals are often disturbed by complex environmental noise, affecting the quality and intelligibility of speech. Therefore, how to effectively remove noise in a noisy environment while maintaining the clarity and naturalness of speech has become the core challenge of speech enhancement technology. Traditional speech enhancement methods, such as spectral subtraction, adaptive filtering, statistical models and subspace methods, although have achieved certain results in early simple noise environments, but in the face of complex noise, different signal-to-noise ratio conditions and multi-speaker speech, the de-noising effect is limited. These methods often have problems such as musical noise, speech distortion and loss of signal details, and are difficult to adapt to complex and variable noise scenes. In recent years, deep learning technology, especially generative adversarial network (GAN), convolutional neural network (CNN) and recurrent neural network (RNN), has made significant progress in the field of speech enhancement, and has shown good de-noising effect in complex noise environment by learning the characteristics of large-scale data. However, deep learning methods still face challenges such as unstable model training, insufficient adaptability to complex noise and poor preservation of speech details. For example, GAN is prone to convergence instability and smoothing of generated speech signals in generative adversarial learning, which weakens the naturalness of speech.

[0003] To address these challenges, researchers have attempted to enhance the ability to capture noise features by improving network structures, designing new loss functions, and introducing attention mechanisms, achieving some improvement effects. Convex optimization, as a mathematical optimization method with global optimal solution, easy to solve and efficient convergence advantages, is widely used in signal processing and machine learning. The stability and theoretical guarantee of convex optimization make it an ideal choice for solving model optimization problems. In recent years, some research attempts to combine convex optimization with deep learning, to improve the training process of deep learning models through the stability advantages of convex optimization, thereby improving the effect of speech enhancement. However, the application of convex optimization in the field of speech enhancement is still in its infancy, and how to effectively combine the advantages of convex optimization and deep learning to improve the performance of existing speech enhancement technology is still a direction worth exploring. Currently, speech enhancement technology faces challenges in complex noise environments, including the limitations of traditional methods in de-noising effect and speech fidelity, as well as the problems of deep learning methods in model stability, adaptability, and data dependency, prompting researchers to continuously explore new solutions. Combining the stability of convex optimization and the modeling ability of deep learning, improving speech enhancement technology, and enhancing speech de-noising effect in complex environments are important directions of current research.

[0004] In the field of speech enhancement technology, international methods centered on deep learning have developed rapidly, especially in the study of speech de-noising in complex noise environments, with techniques such as generative adversarial networks (GAN), deep neural networks (DNN), self-supervised learning, and time-frequency attention mechanisms achieving significant results. Global leading research institutions and enterprises have proposed a variety of innovative speech enhancement methods with large-scale data and powerful computing resources. For example, the WaveNet model and the time-frequency domain adaptive GAN algorithm effectively improve the clarity and naturalness of speech signals by learning complex time-frequency features. These methods continuously optimize the model's ability to handle complex noise through adversarial training and attention mechanisms. However, despite the continuous breakthroughs in effectiveness of these international cutting-edge technologies, their models often have problems such as unstable training, strong dependence on data, and insufficient generalization ability for noise features, affecting the reliability and universality of practical applications.

[0005] Domestic research institutions have also made significant progress in the field of speech enhancement, actively exploring the application of deep learning in complex noise environments. Domestic research has achieved good results in improving traditional model structures and integrating various deep learning techniques, such as using convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer structures for speech signal denoising. In recent years, domestic research has gradually developed towards multi-task learning, attention mechanism optimization, and other directions, and has performed well in international speech enhancement competitions. However, domestic research still has certain problems in model training stability, adaptability to complex noise environments, and ability to preserve speech details. In addition, due to limitations in data resources and computing power, domestic models still have room for improvement in generalization ability and adaptability. SUMMARY

[0006] To address the deficiencies of the prior art, the present application proposes an improved speech enhancement method that integrates a generative adversarial network and a convex conjugate transformation.

[0007] In a first aspect, the present application proposes an improved speech enhancement method that integrates a generative adversarial network and a convex conjugate transformation, comprising:

[0008] Step S1: Obtain real speech sample data to be enhanced;

[0009] Step S2: In the generative adversarial network, a pre-designed generator is used to generate first fake speech sample data;

[0010] Step S3: A pre-designed discriminator is used to discriminate between the real speech sample data to be enhanced and the first fake speech sample data, obtaining a first discrimination result;

[0011] Step S4: According to the first discrimination result, the parameters of the pre-designed generator are fixed, the loss function of the discriminator constructed by the convex function is used to update the parameters of the pre-designed discriminator;

[0012] Step S5: A pre-designed generator is used to generate second fake speech sample data;

[0013] Step S6: A pre-designed discriminator is used to discriminate between the real speech sample data to be enhanced and the second fake speech sample data, obtaining a second discrimination result;

[0014] Step S7: According to the second discrimination result, the parameters of the pre-designed discriminator are fixed, the loss function of the discriminator constructed by the convex function is used to update the parameters of the pre-designed generator;

[0015] Step S8: If the number of iterations is less than a predetermined iteration threshold, return to Step S2 to regenerate the first fake speech sample data based on the updated parameters of the pre-designed generator;

[0016] Step S9: output the final parameters of the generator and the parameters of the discriminator in the case of the number of iterations being equal to or greater than a predetermined iteration threshold;

[0017] Step S10: generate third fake speech sample data by using the generator corresponding to the final parameters of the generator, and perform discrimination on the third fake speech sample data by using the discriminator corresponding to the final parameters of the discriminator to obtain a speech enhancement result.

[0018] In the generative adversarial network, the optimization objective function is calculated as follows:

[0019]

[0020] wherein D(x) is the score of the input data by the discriminator D, f * (D(G(z))) is the minimum conjugate function, V(D, G) is the optimization objective function of the discriminator D and the generator G, is the expectation of x down-sampled under the distribution P data (x) of the real speech sample data, P data (x) is the distribution of the real speech sample data, is the expectation value of the real speech sample data x, is the expectation of z down-sampled under the distribution P z (z) of the noise, is the expectation value of the input noise z, P z (z) is the distribution of the input noise z of the generator, which adopts a Gaussian distribution or a uniform distribution, and D(G(z)) is the score of the sample G(z) generated by the generator G by the discriminator D.

[0021] The pre-designed generator comprises a plurality of encoding layers and corresponding decoding layers, a time-frequency attention mechanism is added in each encoding layer and decoding layer of the pre-designed generator, the output of the encoder of the pre-designed generator is connected in parallel with the original input of the decoding layer, and the parallel connection result is taken as the input of the decoding layer of the pre-designed generator; the pre-designed discriminator comprises a plurality of decoding layers, and a time-frequency attention mechanism is added between the decoding layers of the pre-designed discriminator.

[0022] The pre-designed generator adopts a loss function constructed by a convex function, and the calculation formula is as follows:

[0023]

[0024] wherein V GAN(G) is the objective function of the generator G in the generative adversarial network, x is the real speech sample data, G(z, x) is the false speech sample data generated by the generator when the input noise is z and the real speech sample data is x, λ is a weight hyperparameter in the loss function, D(G(z, x), x) is the output of the discriminator when the false speech sample data generated by the generator is G(z, x) and the real speech sample data is x, is the expected value of the reconstruction error between the generated data and the real speech sample data, P data (x) is the distribution of the real speech sample data x, P z (z) is the noise distribution.

[0025] The pre-designed discriminator includes:

[0026]

[0027] Wherein, V GAN (D) is the optimization objective function of the discriminator D, D(x, x c ) is the score of the discriminator D to the input real sample x and the context information x c , G(z, x c ) is the output of the generator G, that is, the false sample generated according to the input noise z and the context information x c , is the average judgment of the discriminator D to the generated data of the generator G, is the expected value of the real speech sample data distribution P data (x, x c ) real speech sample data x and x c .

[0028] The loss function of the discriminator constructed by the convex function is calculated as follows:

[0029]

[0030] ∥G(z, x)-x∥1

[0031] Wherein, D(x) is the output of the discriminator, G(z) is the false speech sample data generated by the generator, z is the input noise, V(D, G) is the optimization objective function of the discriminator D and the generator G, x c is the context information as the input condition, is the expected value of the real speech sample data distribution P data (x, x c ) real speech sample data x and x c , is the average judgment of the discriminator D to the generated data of the generator G, is the expected value of the reconstruction error between the generated data and the real speech sample data, x is the real speech sample data, λ is the weight hyperparameter in the loss function, P z (z) is the noise distribution, P data (x c )Context information x c The true distribution of P data (x) the distribution of real speech sample data x, For x and x c The joint distribution P data (x,x c ) downsampling expectation.

[0032] In a second aspect, the present application proposes an electronic device comprising: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the speech enhancement method of the improved generative adversarial network and convex conjugate transformation fusion.

[0033] In a third aspect, the present application proposes a computer-readable storage medium storing executable instructions, which, when executed, enable a processor to execute the speech enhancement method of integrating the improved generative adversarial network with the convex conjugate transformation.

[0034] In a fourth aspect, the present application proposes a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the speech enhancement method of the improved generative adversarial network and convex conjugate transformation fusion.

[0035] Beneficial effects:

[0036] This application proposes a speech enhancement method that integrates an improved generative adversarial network and a convex conjugate transform, which improves the speech denoising effect and the fidelity of the original sound in complex noisy environments. Specifically, after the introduction of the convex conjugate transform, the retention of high-frequency details is improved; after the introduction of the time-frequency attention mechanism, the high-frequency harmonics are clearer and more prominent, and the frequency resolution is improved, which significantly improves the naturalness and clarity of the speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flowchart of a speech enhancement method combining an improved generative adversarial network and a convex conjugate transformation according to an embodiment of the present application;

[0038] Figure 2 A flowchart of a speech enhancement method combining an improved generative adversarial network and a convex conjugate transformation according to an embodiment of the present application;

[0039] Figure 3 Spectrogram of the speech enhancement generative adversarial network of the embodiment of the present application;

[0040] Figure 4 The spectrogram of the application embodiment introduces a convex conjugate transformation to the loss function of the speech enhancement generative adversarial network;

[0041] Figure 5 The spectrogram of the application embodiment adds a time-frequency attention mechanism to the structure of the speech enhancement generative adversarial network

[0042] Figure 6 The network structure diagram of the application embodiment adds a time-frequency attention mechanism to the structure of the speech enhancement generative adversarial network, wherein (a) is the structure of the pre-designed generator, and (b) is the structure of the pre-designed discriminator. DETAILED DESCRIPTION

[0043] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings and examples.

[0044] The data set used in this embodiment is from the public data set of Valentini et al., containing speech data of 30 speakers, of which 28 are used for training and 2 are used for testing. Specifically, the training set contains 10 different noise types (including 2 artificial noises and 8 noises from the Demand database), generated under 4 different signal-to-noise ratios (SNR: 15, 10, 5, 0 dB). The test set contains 5 different noise types (all from the Demand database), generated under 4 different SNRs (17.5, 12.5, 7.5, 2.5 dB). The speech in the data set comes from the Voice Bank corpus, and the speech enhancement method based on convex conjugate transformation and deep learning of the application is used.

[0045] Embodiment 1:

[0046] This embodiment proposes an improved speech enhancement method of fusion of generative adversarial network and convex conjugate transformation, as shown in Figure 1 、 Figure 2 , which includes:

[0047] Step S1: obtaining real speech sample data to be enhanced;

[0048] In this embodiment, the public data set from Valentini et al. is used, containing speech data of 30 speakers, of which 28 are used for training, and in this embodiment, the pre-designed generator and the pre-designed discriminator are used for training. In this embodiment, the real speech sample data to be enhanced uses 2 from the public data set.

[0049] Step S2: in the generative adversarial network, a pre-designed generator is used to generate first fake speech sample data;

[0050] In the structure of a generative adversarial network (GAN), a generator is used to generate a denoised speech signal, a discriminator is used to distinguish the generated speech from the real speech, and the generator is optimized through adversarial learning; the goal of the generator G is to generate a denoised speech signal that is realistic enough to deceive the discriminator and make it unable to distinguish the generated signal from the real clean speech signal; the goal of the generator G is to generate a sample as close as possible to the real data while minimizing the difference between the generated sample and the real data. The loss function of the pre-designed generator contains two parts:

[0051]

[0052] The first part is:

[0053]

[0054] This term represents the generator's attempt to make the output of the discriminator close to 1, i.e., to make the discriminator think that the generated sample is a real sample. The generator deceives the discriminator by minimizing this loss, so that the generated sample is considered real by the discriminator.

[0055] The second part is:

[0056]

[0057] This term is an L1 regularization loss, which is used to ensure that the distance between the generated sample and the real sample x is as small as possible. The L1 loss makes the generator not only focus on deceiving the discriminator, but also ensure that the generated sample is closer to the real data at the pixel level or feature level.

[0058] Step S3: using a pre-designed discriminator to distinguish the real speech sample data to be enhanced and the first false speech sample data, and obtaining a first discrimination result;

[0059] In this embodiment, the goal of the pre-designed discriminator D is to continuously improve its classification ability for real clean speech signals and denoised speech signals generated by the generator, to ensure that it can accurately distinguish between the two; the goal of the discriminator D is to accurately distinguish between real samples and generated samples as much as possible, to maximize its judgment score for real samples and to minimize its false judgment score for generated samples.

[0060] The loss function of the discriminator is as follows:

[0061]

[0062] The first part is:

[0063]

[0064] This term indicates that the discriminator wants to accurately classify the real sample x, so that the output of the discriminator for the real sample is as close to 1 as possible. By maximizing this loss, the discriminator can identify real samples and correctly classify them.

[0065] Second part:

[0066]

[0067] This term indicates that the discriminator wants to accurately classify the real sample x, so that the output of the discriminator for the real sample is as close to 1 as possible. By maximizing this loss, the discriminator can identify real samples and correctly classify them. c ) when processing the samples G(z, x generated by the generator, so that it can correctly identify these samples as "fake". The discriminator improves its ability to distinguish generated samples by maximizing the loss of this part.

[0068] In the adversarial process of the discriminator and the generator, the discriminator constantly improves its classification ability and strives to accurately distinguish real samples and generated samples. The generator, on the other hand, constantly optimizes its generation strategy and tries to "deceive" the discriminator, making it mistakenly believe that the generated samples are real. This adversarial training forces the generator to generate increasingly realistic data while improving the discriminator's ability to identify.

[0069] Step S4: According to the first discrimination result, fix the parameter of the pre-designed generator, use the loss function of the discriminator constructed by the convex function to update the parameter of the pre-designed discriminator;

[0070] Step S5: Use the pre-designed generator to generate second fake speech sample data;

[0071] Step S6: Use the pre-designed discriminator to discriminate the real speech sample data to be enhanced and the second fake speech sample data, and obtain a second discrimination result;

[0072] Step S7: According to the second discrimination result, fix the parameter of the pre-designed discriminator, use the loss function of the discriminator constructed by the convex function to update the parameter of the pre-designed generator;

[0073] Step S8: If the number of iterations is less than the predetermined iteration threshold, return to step S2 to generate first fake speech sample data according to the updated parameter of the pre-designed generator;

[0074] Step S9: If the number of iterations is equal to or greater than the predetermined iteration threshold, output the final parameters of the generator and the discriminator;

[0075] In this embodiment, in the iteration process, the generator gradually learns to generate more realistic samples through minimax game, and the discriminator constantly improves its classification accuracy.

[0076] In using actual data sets, the goal of the generator is to generate denoised speech signals from noisy samples, while the task of the discriminator is to distinguish between denoised speech generated by the generator and real denoised speech signals. Each training iteration is divided into two stages:

[0077] Fixing the parameters of the generator, optimizing the discriminator: This stage optimizes the parameters of the discriminator to better distinguish between real samples and generated samples. Through learning, the discriminator gradually improves its ability to distinguish between generated samples and real samples.

[0078] Fixing the parameters of the discriminator, optimizing the generator: In this stage, the parameters of the generator are optimized to gradually approach the real data generated by the generator to "trick" the discriminator, making it difficult for the discriminator to distinguish between real and fake samples.

[0079] By alternating the optimization of the generator and the discriminator, the generator continuously improves the quality of the denoised signal generation, while the classification accuracy of the discriminator gradually improves. In this process, the confrontation between the discriminator and the generator can be seen as a "minimax game", where the generator tries to minimize the misclassification rate of the discriminator for generated samples, while the discriminator tries to maximize the ability to distinguish between real samples and generated samples.

[0080] In this minimax game process, the generator tries to generate samples that are increasingly close to real data, while the discriminator strives to accurately distinguish between these generated samples and real samples. This game process drives the entire network model to continuously learn and improve, achieving higher generation results.

[0081] Game strategy of generator G: The generator continuously tries to generate high-quality samples so that the discriminator cannot easily distinguish between real data and fake data. The learning goal of the generator is to gradually learn the ability to generate real data distribution by minimizing the error rate of the discriminator.

[0082] Game strategy of discriminator D: The discriminator tries to optimize its classification ability by distinguishing between real data and generated data, maximizing its accuracy in distinguishing between real samples and generated samples. This process helps the discriminator recognize increasingly complex generated data and improve its classification ability.

[0083] The use of loss functions based on convex conjugate transformation in the training of GAN models provides significant advantages, which are manifested in the following aspects:

[0084] Improved training stability: Traditional GANs often encounter vanishing or exploding gradients during training, leading to unstable training of the generator and discriminator. By introducing a loss function based on convex conjugate transformations, we ensure a smoother optimization path, making the training of the generator and discriminator more stable and avoiding common crashes or non-convergence during training.

[0085] Accelerate model convergence: Using a convex conjugate transformation loss function makes the optimization problem simpler and more controllable, meaning the model can achieve better convergence in a shorter training time. Compared to the lengthy and unstable training process of traditional GAN ​​models, the convex conjugate transformation-based loss function can effectively accelerate model convergence and reduce unnecessary iterations.

[0086] Improve the generation effect: Figure 3 As shown, by using the convex conjugate transformation loss function, Figure 3 The generated results in Figure 2 The original SEGAN graph is smoother and closer to the real speech spectrum. This means that the GAN model can better preserve important frequency information, reduce distortion, and enhance the fidelity and naturalness of speech when generating high-quality speech signals.

[0087] Reducing the risk of model divergence and mode collapse: The introduction of convex conjugate transformations effectively reduces the possibility of model divergence during training, preventing the generator from frequently falling into mode collapse when competing against the discriminator. This further improves the generalization ability of the GAN model, ensuring that the generated results remain consistent and high-quality under a variety of input conditions.

[0088] In summary, the loss function based on convex conjugate transformation helps solve the key problems in GAN training, making the generated speech enhancement results more natural and delicate, while greatly improving the efficiency and stability of the training process.

[0089] In step S4 and step S7, the loss function of the discriminator is constructed using a convex function. The process of constructing the loss function is described in detail below:

[0090] Design a loss function based on convex conjugate transformation. The loss function uses convex functions to transform the model parameter optimization problem into a convex optimization problem to improve the stability and convergence speed of model training.

[0091] In order to improve the instability in GAN training, a loss function based on convex conjugate transformation is proposed. By converting the GAN optimization problem into a convex optimization problem, the introduction of convex conjugate transformation ensures a smoother optimization path and makes the training more stable, especially in the competition between the generator and the discriminator, effectively avoiding the problem of gradient vanishing or non-convergence.

[0092] The objective of this embodiment is to find the optimal generator G(z) that makes the generated distribution P G closer to the real data distribution P data :

[0093]

[0094] where the function f is a convex function, x is the real speech sample data, P G (x) is the distribution of the generated data by the generator G, P data (x) represents the real data distribution, D f (P data ||P G ) represents the f-divergence, which is used to measure the difference between the real data distribution P data and the generated data distribution P G . Different f functions correspond to different divergence measures, and the choice of these convex functions determines the nature of the optimization.

[0095] The introduction of Fenchel conjugate helps to decompose the optimization problem. Through Fenchel conjugate transformation, the optimization problem of the discriminator can be equivalently transformed into the optimization problem of the generator, thereby avoiding the direct optimization of the non-convex generator-discriminator adversarial problem. The core idea of Fenchel conjugate is to transform a complex non-convex problem into a series of convex optimization sub-problems, ensuring the stability of GAN training. This embodiment can derive the corresponding conjugate function f * (t) from the convex function f(u). The definition of the conjugate function is:

[0096]

[0097] where sup represents the supremum, which is the largest value or the smallest upper bound in a set, and domf represents the domain of the function f(u), i.e., the range of values of u.

[0098] Due to the reversibility of Fenchel conjugate, we can express the convex function f(u) as:

[0099]

[0100] Therefore, a generator function f(u) is designed to be a convex function, i.e.:

[0101]

[0102] where u is the input variable of the function, and u is used to describe the input characteristics of the generator; the generator function satisfies the condition f(1) = 0 and f(u) is a convex function, which ensures that the divergence is zero when the two distributions are consistent. By solving the conjugate function, we get:

[0103]

[0104] By bringing the convex function and the conjugate function into the f divergence formula, we can get:

[0105]

[0106] where f* is the Fenchel conjugate function of the convex function f, domf* is the domain of f*, and the sum of the maximum values is greater than the maximum value of the sum. In the formula, T represents a class of functions that satisfy χ→R, so T(x) can be used instead of t in the formula; and T(x) can be represented by the following formula:

[0107]

[0108]

[0109] where D w (x) is the discriminator function, a(·) is the activation function, χ is the input data space, is the real number field, T(x) can be regarded as a discriminator with a special activation function a(·); many variants of GANs can be derived by using different generator functions f and the corresponding activation functions a(·). First, maximize the equation the lower bound of T(x), and then minimize the approximate divergence, so that the distribution learned by the generator is more similar to the distribution of the real data.

[0110] The output T(x) of the discriminator is one of the core parameters. By introducing the framework of convex optimization, the discriminator is no longer just a binary classifier, but guides the optimization of the generator by learning a distribution difference measure. The optimization goal of the discriminator is to minimize the divergence between different distributions, and the output T(x) is used to measure the closeness of the generated data to the real data. By minimizing the divergence function, the generator can be further guided to generate samples that conform to the real data distribution.

[0111] Under the framework of convex optimization, the optimization goal is to approximate the real distribution by minimizing the loss function, which can be represented in the following form:

[0112]

[0113] where it contains two expectation terms: the first term expectation represents the expectation of the output of the discriminator D(x) under the real data distribution. By measuring the real data, the model can ensure that the generated samples can be more approximate to the real samples. The second expectation measures the expected value of the output data of the generator G(z). By generating from the noise distribution Pz The generated sample, the discriminator D(G(z)) gives the score of the generated sample, further guiding the optimization of the generator. The core of convex optimization lies in its stable gradient calculation and update direction, which can effectively avoid the training difficulties caused by the instability of the loss function in traditional GAN.

[0114] In the standard GAN, when the distribution P g learned by the generator is far away from the real data distribution P data , the discriminator will still judge the generated sample as false with high confidence, which will cause the objective function value to be a constant, the gradient to be 0 during backpropagation, and finally lead to gradient disappearance. g data

[0115] Based on these problems, a loss function based on convex conjugate transformation is proposed to replace the cross-entropy loss function in the original GAN. Since the designed convex function is a strictly convex function, it provides a smooth gradient signal during optimization. Compared with the traditional GAN loss, this square error form of loss function can provide a more stable optimization path, avoiding the problems of gradient disappearance or gradient explosion.

[0116] The proposed loss function is similar to the following equation:

[0117]

[0118] Therefore, the loss function based on convex conjugate transformation is as follows:

[0119]

[0120] where D(x) represents the output of the discriminator, G(z) represents the sample generated by the generator, and z represents a random vector following a certain distribution. The choice of this loss function makes the generator have better gradient signals when it is against the discriminator (compared to the Jensen-Shannon divergence of traditional GAN), and has stronger convergence when generating realistic speech signals.

[0121]

[0122] where p(x) represents the real data distribution and q(x) represents the generated sample distribution. The above calculation formula represents the f-divergence corresponding to the loss function designed based on convex conjugate transformation. This divergence combines the distribution Q of the generated sample and the distribution P of the real sample, and is used to measure the difference between the generated sample and the real sample.

[0123] ​​The design of the convex conjugate transformation loss function provides a stable theoretical basis for the application of GAN in complex tasks. By introducing the f-divergence, the model can establish a more stable training path between the discriminator and the generator, avoiding the common gradient disappearance and non-convergence phenomena in traditional GAN. In the speech enhancement task, the use of convex conjugate transformation design not only improves the generation ability of the model, but also significantly speeds up the convergence speed in the training process, and can generate high-quality clear speech in complex noise environments.

[0124] The introduction process of the attention mechanism is described in detail below:

[0125] Through the time-frequency attention mechanism, information loss and over-smoothing phenomenon are reduced, speech signal detail preservation is improved, and higher speech fidelity and naturalness are achieved. In the generative adversarial network, the encoder compresses the speech signal into feature maps layer by layer, and the decoder reconstructs the speech signal according to these feature maps. In order to improve the effect of speech enhancement, the time-frequency attention mechanism can be embedded in the skip connection of the encoder and the decoder, and the network structure diagram is as Figure 6 The core idea is to weight the feature map in each layer of skip connection through the frequency dimension, reduce information loss, highlight key frequency features, and improve the naturalness and fidelity of speech.

[0126] The pre-designed generator includes a plurality of encoding layers and corresponding decoding layers, and the time-frequency attention mechanism is added to each encoding layer and decoding layer of the pre-designed generator, and the output of the encoder of the pre-designed generator is connected with the original input of the decoding layer, and the connection result is used as the input of the decoding layer of the pre-designed generator. The pre-designed discriminator includes a plurality of decoding layers, and the time-frequency attention mechanism is added between the decoding layers of the pre-designed discriminator.

[0127] The time-frequency attention mechanism is added to the encoding layer and the decoding layer in the speech enhancement generative adversarial network, which captures the non-local correlation in the frequency dimension through the skip connection, optimizes the capture ability of the time-frequency feature, especially the harmonic characteristics of the speech, which helps the network better recover the harmonic features hidden by noise; as Figure 6(a) shown, wherein the pre-designed generator includes: 11 encoding layers and corresponding 10 decoding layers, a time-frequency attention mechanism is added in each encoding layer and decoding layer of the pre-designed generator, and the output of the encoder of the pre-designed generator is spliced with the original input of the decoding layer, and the spliced result is taken as the input of the decoding layer of the pre-designed generator; for example: a noisy real speech sample data is input into the encoding layer 1, the output of the encoding layer 1 is input into the first layer time-frequency attention mechanism, the output of the first layer time-frequency attention mechanism is input into the encoding layer 2, and the encoding layer 2 is input into the encoding layer 2, and the output of the encoding layer 11 is input into the eleventh layer time-frequency attention mechanism, the output in the eleventh layer time-frequency attention mechanism is compressed into a thought vector c, the output vector c of the encoder is spliced with the latent vector z, and the spliced result is taken as the input of the decoding layer 10, and the like. In this embodiment, the pre-designed discriminator includes 10 decoding layers, and a time-frequency attention mechanism is added between the decoding layers of the pre-designed discriminator, as shown in Figure 6 (b) shown.

[0128] Wherein, the encoding layer is equivalent to feature extraction: local frequency features in the time-frequency graph are extracted through convolution operation. In the processing of the time-frequency graph, feature extraction is a key step, and a convolutional neural network (CNN) is usually used to capture the features of local frequency components in the time-frequency graph. In this step, one-dimensional convolution (Conv1D) is used to slide along the frequency dimension (channel dimension) to extract local features in different frequency ranges. The encoder performs a series of convolution operations on the input noisy speech signal. Each convolution extracts different features of the speech signal, compressing high-dimensional speech data into progressively smaller feature maps. These feature maps not only contain global structural information of the speech, but also cover detailed features, especially the time-frequency features of the speech. In the final layer, the "thought vector" generated by the encoder is a compressed deep feature representation that condenses the key information in the speech.

[0129] The form of convolution operation is as follows:

[0130] F conv1 = Conv1D(X)

[0131] Wherein: X is the input time-frequency graph, with a dimension of [B, C, T'], where B is the batch size, C is the frequency dimension (channel number), and T' is the time step. F conv1 is the local frequency feature extracted by convolution.

[0132] The role of the convolution kernel is to slide through a fixed window size (kernel size) in the frequency dimension, extracting local frequency features within the window. For example, if the convolution kernel size is 9, it will perform feature extraction on 9 adjacent frequency points at each time step. This local operation can capture local dependencies between frequencies, such as formants in speech, phoneme features, etc.

[0133] In addition, after the convolution operation, an activation function (ReLU) and batch normalization (BatchNorm) are usually used to enhance the model's nonlinear representation ability and speed up the training. This part of the operation provides the basic frequency features for the subsequent attention mechanism.

[0134] The time-frequency attention mechanism is equivalent to attention weighting: based on the feature map generated by the encoder, the time-frequency attention mechanism is applied to the skip connection of each layer. The core of the attention mechanism is to extract time-frequency features through convolution operation and strengthen the key frequency components in speech by weighting in the frequency dimension. The attention map in the frequency dimension is generated and multiplied element-wise with the input feature map. After feature extraction, the generated frequency feature map F conv1 contains local frequency information, but the importance of these features is uneven. In order to emphasize the key frequency components, the module introduces an attention mechanism to weight the features. The operation of the attention mechanism generates an attention weight map A(t,f) representing the attention distribution in each frequency dimension. The generation of the attention map can be represented as:

[0135] A(t,f) = softmax(Conv1D(F conv1 ))

[0136] Where the Softmax function normalizes the generated convolution result to the range of 0 to 1, ensuring that the weight sum of each frequency point is 1.

[0137] Next, the attention weight map is multiplied element-wise with the original feature map F conv1 to form a weighted feature map, which is the output of the time-frequency attention mechanism:

[0138] F att = A(t,f) ⊙ F conv1

[0139] Where A(t,f) is the attention weight map, representing the weight of each frequency point. ⊙ represents the element-wise multiplication operation, ensuring that the weighting only acts on the specific frequency dimension.

[0140] This operation enables the model to dynamically adjust the weights of frequency features, enhancing the focus on key frequencies while suppressing the influence of irrelevant or noisy frequencies. For example, certain frequency components in a speech signal, such as formants, may be more important than others, so these frequencies will be given greater weight.

[0141] Finally, a skip connection is made, equivalent to feature fusion and reconstruction, which concatenates the weighted feature maps and the original input, and performs further convolution processing to output the optimized features.

[0142] After generating the attention-weighted feature map F att , the time-frequency attention module does not directly use this weighted feature map as the final output. Instead, it concatenates the weighted feature map with the original input feature X to form a more comprehensive feature map F cat :

[0143] F cat = [F att , X]

[0144] This concatenation operation combines the attention-weighted features and the original features, ensuring that the original time-frequency information is not lost while retaining the enhanced features from attention weighting.

[0145] Next, the module performs further convolution processing on the concatenated feature map F cat to integrate the information from both and output the final feature map F out :

[0146] F out = Conv1D (F cat )

[0147] This convolution operation serves as a feature reconstruction, compressing and optimizing the high-dimensional concatenated features to output the optimized time-frequency features F out , which have the same dimension as the input time-frequency feature map. This process ensures efficient integration and representation of features while retaining the local capture ability of convolution operations and the global enhancement effect of attention mechanisms.

[0148] The time-frequency attention module first extracts local frequency features, then dynamically adjusts the weights of each frequency point using attention mechanisms, and finally concatenates the weighted features with the original features and reconstructs them to output optimized feature representations. This design better captures the time-frequency features in speech signals, especially focusing on local important components in the frequency dimension, making the model more accurate and efficient in processing speech signals.

[0149] The weighted feature map is transmitted to the decoder through a jump connection, and the decoder gradually reconstructs the speech signal through a deconvolution operation. In this process, due to the addition of the time-frequency attention mechanism, the decoder not only retains the low-level features preserved by the jump connection, but also refines the features in the frequency dimension through the attention mechanism. Therefore, the decoder can more efficiently restore a clear and high-fidelity speech signal.

[0150] The introduction of the convex conjugate transformation ensures the stability and gradient transmission in the training process: the convex conjugate transformation provides a smoother and more stable gradient signal in the loss function, avoiding the common problem of gradient disappearance when generating high-frequency details. Compared with the loss function of the traditional GAN, the loss function designed by the convex conjugate transformation ensures the gradient transmission of the generator, so that the model can generate more accurate high-frequency parts and better restore the high-frequency details and harmonic characteristics of the speech signal.

[0151] The time-frequency attention mechanism improves the attention and capture ability of high-frequency features: the time-frequency attention mechanism allows the model to dynamically adjust the weights of each frequency band in the frequency dimension, ensuring that high-frequency details receive more attention. When processing speech signals containing complex noise, the time-frequency attention mechanism can effectively emphasize the important components in the high-frequency region, enhancing the clarity and details of the speech signal, so that the model's enhancement effect in the high-frequency part is significantly improved.

[0152] Step S10: using the final generator corresponding to the parameters of the generator to generate third fake speech sample data, using the final discriminator corresponding to the parameters of the discriminator to discriminate the third fake speech sample data, and obtaining a speech enhancement result.

[0153] Finally, the final generator and discriminator are used for speech enhancement.

[0154] The experimental results verify that:

[0155] In this experiment, SEGAN is used for comparative experiments, and SEGAN (Speech Enhancement Generative Adversarial Network) is a kind of GAN network, which has achieved certain results in the field of speech enhancement, especially in the low and medium frequency bands, it can better remove noise and enhance speech signals. However, in the high frequency part, the enhancement effect of SEGAN is relatively poor, mainly manifested in the loss of high-frequency details, the reduction of speech clarity and the blurring of harmonic structure. The reasons for this phenomenon include the following points:

[0156] Gradient Vanishing and Instability: The traditional GAN structure of SEGAN may face the problem of gradient vanishing during training, especially for the generation process of high-frequency details. When the generator cannot obtain smooth and stable gradients in the high-frequency part, the generated signal often lacks clear high-frequency details, resulting in poor performance of the enhanced speech in the high-frequency band.

[0157] Weak High-Frequency Information Capture Ability: SEGAN lacks effective mechanisms to identify and highlight key features in high-frequency regions when processing high-frequency components. High-frequency details are crucial components of speech, affecting its clarity and naturalness. However, SEGAN lacks a dedicated mechanism to focus on these high-frequency features, resulting in a relatively smooth and low-detail restoration in the high-frequency part of the generated results.

[0158] To overcome the deficiencies of SEGAN in the high-frequency part of the enhancement effect, convex conjugate transformation and time-frequency attention mechanism are introduced. The combination of these two technologies significantly improves the enhancement effect of speech in the high-frequency part:

[0159] The introduction of time-frequency attention mechanism has the following advantages in speech enhancement:

[0160] 1. High-frequency detail preservation and harmonic enhancement: The time-frequency attention mechanism significantly improves the preservation of high-frequency information. Compared with the SEGAN baseline model, the introduction of the time-frequency attention mechanism makes the high-frequency harmonics clearer and more prominent, with improved frequency resolution, resulting in significant improvement in the naturalness and clarity of the speech.

[0161] 1) Qualitative analysis: As shown in the spectrogram comparisons in Figure 3 , Figure 4 and Figure 5 , the performance differences of different methods in preserving high-frequency information and enhancing harmonic characteristics are shown:

[0162] a) Figure 3 (SEGAN baseline model): In the enhanced spectrogram, especially in the high-frequency part (above 5000 Hz) after 2 seconds, the high-frequency details of the speech are relatively fuzzy, and some detail information is lost. This indicates that SEGAN has limited ability to preserve high-frequency information in its original state.

[0163] b) Figure 4 (SEGAN+convex conjugate transformation): After introducing the convex conjugate transformation into the SEGAN loss function, the spectrogram has improved compared to the baseline model. Although the enhancement effect in the high-frequency part is not as obvious as the time-frequency attention mechanism, the preservation of high-frequency details has improved, especially in the processing of high-frequency information after 2.5 seconds.

[0164] c) Figure 5(SEGAN+time-frequency attention mechanism): After introducing the time-frequency attention mechanism, the high-frequency components in the spectrogram are significantly enhanced. Especially in the high-frequency band (after 2 seconds), the harmonic information is clearer, and the naturalness and detail preservation of the speech are significantly improved. This shows that the time-frequency attention mechanism can better focus on the key information in the time-frequency domain, thus improving the model's processing ability for complex speech signals.

[0165] 2) To further verify the effect of model improvement, the experiment selects the speech file p257_387.wav and compares the signal-to-noise ratio (SNR) of the enhanced speech above 5000Hz. SNR is a

[0166] important indicator of speech enhancement effect, the higher the value, the better the quality of the enhanced speech. The experimental results are shown in the following table:

[0167] Table 1 Comparison of experimental results

[0168]

[0169] From the results, we can see that the addition of the time-frequency attention mechanism significantly improves the SNR of the high-frequency band to 3.72, verifying the advantage of this method in high-frequency speech clarity enhancement, while the convex conjugate transformation also brings improvement, but the effect is not as significant as the time-frequency attention mechanism.

[0170] 2. Frequency selective enhancement: The time-frequency attention mechanism can effectively highlight key frequency regions by weighting different frequencies, thus prioritizing the recovery of important frequency information obscured by noise in complex noise environments. In the Figure 4 , the brightness distribution of the overall spectrum is more balanced, indicating that the time-frequency attention mechanism can enhance specific frequencies and reduce noise interference in the frequency dimension.

[0171] 3. Reduce smoothing phenomenon: In the original SEGAN enhancement graph, some frequency bands are too smooth and details are lost, while the time-frequency attention mechanism can enhance the details. In the Figure 4 , the over-smoothing phenomenon of the spectrum is reduced, and more details are preserved, making the speech clearer and more natural.

[0172] 4. Speech intelligibility improvement: Through the weighted processing of the frequency dimension, the time-frequency attention mechanism not only restores the details of the speech, but also improves the intelligibility of the speech. In complex noise environments, this mechanism can better restore the characteristics of the speech,

[0173] making the final output of the enhanced speech more easily understood.

[0174] In summary, the quality of speech enhancement is significantly improved after adding the time-frequency attention mechanism, especially in terms of preserving high-frequency details, frequency-weighted selectivity, reducing over-smoothing, and improving intelligibility. This also verifies the effectiveness of the present application.

[0175] Embodiment 2

[0176] The electronic device can be a mobile phone, a computer, or a tablet computer, etc., comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the improved speech enhancement method of fusing the generative adversarial network and the convex conjugate transformation.

[0177] The electronic device can be a mobile phone, a computer, or a tablet computer, etc., comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the improved speech enhancement method of fusing the generative adversarial network and the convex conjugate transformation.

[0178] The processor is configured to execute all or part of the steps of the improved speech enhancement method of fusing the generative adversarial network and the convex conjugate transformation as described in the above embodiments. The memory is configured to store various types of data, which may, for example, include instructions of any application program or method in the electronic device, and application-related data.

[0179] The processor can be an Application Specific Integrated Cricuit (ASIC), a Digital Signal Processor (DSP), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic elements, and is configured to execute the improved speech enhancement method of fusing the generative adversarial network and the convex conjugate transformation as described in the above embodiments.

[0180] Embodiment 3

[0181] The present embodiment provides a computer-readable storage medium storing executable instructions, which, when executed, can be stored in a computer-readable storage medium if implemented in the form of a software functional unit and sold or used as an independent product.

[0182] The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the improved speech enhancement method of fusing the generative adversarial network and the convex conjugate transformation.

[0183] The storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD (Secure Digital Memory Card) or a DX (Memory Data Register, MDR) memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an APP (Application) application store, etc., which can store a program check code, and stores a computer program thereon, which can implement the steps of the improved speech enhancement method of fusing the generative adversarial network and the convex conjugate transformation when executed by a processor.

[0184] Embodiment 4

[0185] The embodiment provides a computer program product, which includes a computer program or instructions, and the computer program or instructions are executed by a processor to implement the improved speech enhancement method of fusing the generative adversarial network and the convex conjugate transformation.

[0186] Based on such understanding, the technical solution of the present application or the part of the technical solution which essentially contributes to the prior art or the part of the technical solution can be embodied in the form of a computer program product.

[0187] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments mainly describes the difference from other embodiments.

[0188] The protection scope of the present application is not limited to the above-described embodiments. Obviously, those skilled in the art can make various modifications and changes to the present disclosure without departing from the scope and spirit of the present disclosure. If these modifications and changes belong to the scope of the claims of the present disclosure and the equivalent technology thereof, the present disclosure also intends to include these modifications and changes.

Claims

1. A speech enhancement method improved by fusing a generative adversarial network and a convex conjugate transformation, characterized in that, The method comprises the following steps: Step S1: obtaining real speech sample data to be enhanced; Step S2: in a generative adversarial network, a pre-designed generator is used to generate first fake speech sample data; Step S3: a pre-designed discriminator is used to discriminate the real speech sample data to be enhanced and the first fake speech sample data, to obtain a first discrimination result; Step S4: according to the first discrimination result, the parameters of the pre-designed generator are fixed, a loss function of the discriminator constructed by a convex function is used to update the parameters of the pre-designed discriminator; Step S5: the pre-designed generator is used to generate second fake speech sample data; Step S6: the pre-designed discriminator is used to discriminate the real speech sample data to be enhanced and the second fake speech sample data, to obtain a second discrimination result; Step S7: according to the second discrimination result, the parameters of the pre-designed discriminator are fixed, the loss function of the discriminator constructed by the convex function is used to update the parameters of the pre-designed generator; Step S8: in the case that the number of iterations is less than a predetermined iteration threshold, returning to step S2, the first fake speech sample data is regenerated according to the updated parameters of the pre-designed generator; Step S9: in the case that the number of iterations is equal to or greater than the predetermined iteration threshold, outputting the final parameters of the generator and the parameters of the discriminator; Step S10: third fake speech sample data is generated by a generator corresponding to the final parameters of the generator, and the third fake speech sample data is discriminated by a discriminator corresponding to the final parameters of the discriminator, to obtain a speech enhancement result.

2. The improved speech enhancement method of claim 1, wherein, In the generative adversarial network, the objective function is calculated as follows: where D(x) is the score of the input data by the discriminator D, f * (D(G(z))) is the minimum conjugate function, and V(D, G) is the optimization objective function of the discriminator D and the generator G, is the distribution of the real speech sample data P data (x) is the expectation of x down-sampled by P data (x) is the distribution of the real speech sample data, is the expectation of the real speech sample data x, is the expectation of the noise distribution P z (z) is the expectation of z down-sampled by P is the expectation of the input noise z, z (z) is the distribution of the input noise z of the generator, which adopts a Gaussian distribution or a uniform distribution, and D(G(z)) is the score of the sample G(z) generated by the generator G by the discriminator D.

3. The improved speech enhancement method of claim 1, wherein, The pre-designed generator comprises a plurality of encoding layers and corresponding decoding layers, a time-frequency attention mechanism is added in each encoding layer and decoding layer of the pre-designed generator, the output of the encoder of the pre-designed generator is connected in parallel with the original input of the decoding layer, and the parallel connection result is taken as the input of the decoding layer of the pre-designed generator; the pre-designed discriminator comprises a plurality of decoding layers, and a time-frequency attention mechanism is added between the decoding layers of the pre-designed discriminator.

4. The improved speech enhancement method of claim 1, wherein, The pre-designed generator adopts a loss function constructed by a convex function, and the calculation formula is as follows: wherein V GAN (G) is the objective function of the generator G in the generative adversarial network, x is the real speech sample data, G(z, x) is the fake speech sample data generated by the generator when the input noise is z and the real speech sample data is x, λ is a weight hyperparameter in the loss function, D(G(z, x), x) is the output of the discriminator when the fake speech sample data generated by the generator is G(z, x) and the real speech sample data is x, is the expected value of the reconstruction error between the generated data and the real speech sample data, Px ata (x) is the distribution of the real speech sample data x, P z (z) is the noise distribution.

5. The improved speech enhancement method of claim 1, wherein, The pre-designed discriminator comprises: wherein V GAN (D) is the optimization objective function of the discriminator D, D(x, x c ) is the score of the discriminator D on the input real sample x and the context information x c , G(z, x c ) is the output of the generator G, i.e. the fake sample generated according to the input noise z and the context information x c , is the average judgment of the discriminator D on the generated data of the generator G, is the expectation on the real speech sample data distribution P data (x, x c ) real speech sample data x and x c .

6. The improved speech enhancement method of claim 1, wherein, The loss function of the discriminator constructed by the convex function is calculated as follows: Among them, D(x) is the output of the discriminator, G(z) is the false speech sample data generated by the generator, z is the input noise, V(D,G) is the optimization objective function of the discriminator D and the generator G, x c is context information, as input conditions, is the distribution of real speech sample data P data (f i x,x c ) Real speech sample data x and x c expectations, is the average judgment of the discriminator D on the generated data of the generator G, is the expected value of the reconstruction error between the generated data and the real speech sample data, x is the real speech sample data, λ is the weight hyperparameter in the loss function, P z (z) is the noise distribution, P data (x c )Context information x c The true distribution of P data (x) the distribution of real speech sample data x, For x and x c The joint distribution P data (x,x c ) downsampling expectation.

7. An electronic device, comprising: The method comprises the following steps: One or more processors and a memory for storing instructions, when the instructions are executed by the one or more processors, the one or more processors execute the improved generative adversarial network and convex conjugate transformation fusion speech enhancement method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer program or instructions are executed by the processor to implement the improved generative adversarial network and convex conjugate transformation fusion speech enhancement method of any one of claims 1-6.

9. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the improved generative adversarial network and convex conjugate transformation fusion speech enhancement method of any one of claims 1-6.

Images