Training method for speech enhancement network, speech enhancement method, and electronic device

Abstract

A training method for a speech enhancement network, a speech enhancement method, and an electronic device. The training method comprises: performing speech effectiveness classification on respective enhanced speech frames, and on the basis of the classification results of the enhanced speech frames, generating an effectiveness distribution for sample enhanced speech; by means of the effectiveness distribution, determining the speech classification accuracy of the speech enhancement network; and measuring the degree of change in speech effectiveness of each enhanced speech frame compared to that before noise reduction, and on this basis, determining speech enhancement accuracy according to noise reduction accuracy and the speech classification accuracy. In this way, the method can focus on improving the ability of the speech enhancement network to suppress noise in non-speech segments. When the trained speech enhancement network is used to perform noise reduction on speech to be processed, if the speech comprises non-speech segments, the trained speech enhancement network can effectively reduce the phenomenon of residual noise and improve the quality of speech enhancement. The invention can be widely applied in various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

Description

Speech enhancement network training method, speech enhancement method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 17, 2023, with application number 2023110441086 and application name “Speech enhancement method, training method of speech enhancement network and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to speech enhancement technology. Background Art

[0003] Currently, speech enhancement technology is widely used in various scenarios. With the rapid development of artificial intelligence, the application of AI-based speech enhancement networks in this field is increasing. In related technologies, when speech enhancement networks perform speech enhancement on noisy speech containing non-speech segments (i.e., segments without valid human voice, such as silence or noise), residual noise is easily present, thereby reducing the quality of speech enhancement.

[0004] Summary of the Invention

[0005] The following is an overview of the subject matter described in detail in this application. This overview is not intended to limit the scope of protection of the claims.

[0006] The embodiments of the present application provide a training method for a speech enhancement network, a speech enhancement method, and an electronic device, which can effectively reduce the occurrence of noise residue when performing speech enhancement processing on noisy speech containing non-speech segments, thereby improving the quality of speech enhancement.

[0007] In one aspect, an embodiment of the present application provides a method for training a speech enhancement network, performed by an electronic device, comprising:

[0008] Acquire a sample clean speech and a sample noisy speech, and mix the sample clean speech and the sample noisy speech into a sample noisy speech;

[0009] Denoising the sample noisy speech based on a speech enhancement network to obtain a sample enhanced speech;

[0010] Dividing the sample enhanced speech into multiple enhanced speech frames, classifying the speech effectiveness of each of the enhanced speech frames, and generating a effectiveness distribution of the sample enhanced speech according to the classification results of each of the enhanced speech frames;

[0011] The noise reduction accuracy of the speech enhancement network is determined based on the sample enhanced speech and the sample clean speech, the speech classification accuracy of the speech enhancement network is determined based on the validity distribution, the speech enhancement accuracy of the speech enhancement network is determined based on the noise reduction accuracy and the speech classification accuracy, and the speech enhancement network is trained based on the speech enhancement accuracy.

[0012] On the other hand, an embodiment of the present application further provides a speech enhancement method, which is performed by an electronic device and includes:

[0013] Get the voice to be processed;

[0014] The speech to be processed is denoised based on a speech enhancement network to obtain corresponding enhanced speech; the speech enhancement network is trained using the speech enhancement network training method provided in an embodiment of the present application.

[0015] On the other hand, an embodiment of the present application further provides a training device for a speech enhancement network, comprising:

[0016] A first sample speech mixing module is used to obtain a sample clean speech and a sample noisy speech, and mix the sample clean speech and the sample noisy speech into a sample noisy speech;

[0017] A first sample speech enhancement module is used to reduce noise on the sample noisy speech based on a speech enhancement network to obtain a sample enhanced speech;

[0018] a first effectiveness classification module, configured to divide the sample enhanced speech into a plurality of enhanced speech frames, classify the speech effectiveness of each of the enhanced speech frames, and generate an effectiveness distribution of the sample enhanced speech according to the classification results of each of the enhanced speech frames;

[0019] The first network training module is used to determine the noise reduction accuracy of the speech enhancement network based on the sample enhanced speech and the sample clean speech, determine the speech classification accuracy of the speech enhancement network based on the validity distribution, determine the speech enhancement accuracy of the speech enhancement network based on the noise reduction accuracy and the speech classification accuracy, and train the speech enhancement network based on the speech enhancement accuracy.

[0020] On the other hand, an embodiment of the present application further provides a speech enhancement device, comprising:

[0021] An acquisition module is used to acquire the speech to be processed;

[0022] The speech enhancement module is used to reduce noise on the speech to be processed based on a speech enhancement network to obtain corresponding enhanced speech; the speech enhancement network is trained by the speech enhancement network training method provided in the embodiment of the present application.

[0023] On the other hand, an embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned speech enhancement network training method or speech enhancement method.

[0024] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned speech enhancement network training method or speech enhancement method.

[0025] In another aspect, embodiments of the present application further provide a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to implement the aforementioned speech enhancement network training method or speech enhancement method.

[0026] Embodiments of the present application include at least the following beneficial effects: by dividing the sample enhanced speech into multiple enhanced speech frames, classifying the speech validity of each enhanced speech frame, and generating a validity distribution of the sample enhanced speech based on the classification results of each enhanced speech frame, since the validity distribution can indicate whether each enhanced speech frame generated by the speech enhancement network is a non-speech segment, the speech classification accuracy of the speech enhancement network determined by the validity distribution can measure the recognition accuracy of the currently trained speech enhancement network for speech segments (i.e., speech signals containing valid human voices) and non-speech segments (i.e., speech signals not containing valid human voices, such as silence signals, noise signals, etc.). On this basis, the speech enhancement accuracy of the speech enhancement network is determined based on the noise reduction accuracy and the speech classification accuracy. Training the speech enhancement network based on the speech enhancement accuracy can improve the speech enhancement network's recognition ability for speech segments and non-speech segments, thereby correspondingly focusing on improving the noise suppression ability of the non-speech segments. When performing noise reduction on the processed speech based on the trained speech enhancement network, for the processed speech containing non-speech segments, the trained speech enhancement network can effectively reduce the phenomenon of residual noise in the non-speech segments, thereby improving the quality of speech enhancement.

[0027] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or may be understood by practicing the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0029] FIG1 is a schematic diagram of an optional implementation environment provided in an embodiment of the present application;

[0030] FIG2 is a schematic diagram of an optional flow chart of a method for training a speech enhancement network according to an embodiment of the present application;

[0031] FIG3 is a schematic diagram of functional modules of the overall system framework for applying the training method provided in an embodiment of the present application;

[0032] FIG4 is a schematic diagram of the structural design of a neural network model inference module provided in an embodiment of the present application;

[0033] FIG5 is a structural diagram of a speech enhancement network based on an end-to-end model according to an embodiment of the present application;

[0034] FIG6 is a schematic diagram of an optional process for obtaining speech enhancement accuracy according to an embodiment of the present application;

[0035] FIG7 is a schematic diagram of another optional process for obtaining speech enhancement accuracy according to an embodiment of the present application;

[0036] FIG8 is a schematic diagram of another optional process for obtaining speech enhancement accuracy according to an embodiment of the present application;

[0037] FIG9 is a schematic diagram of another optional process for obtaining speech enhancement accuracy according to an embodiment of the present application;

[0038] FIG10 is a schematic diagram of an optional process for obtaining conversion loss according to an embodiment of the present application;

[0039] FIG11 is a schematic diagram of the PESQ score results of the test process provided in an embodiment of the present application;

[0040] FIG12 is a schematic diagram of the SI-SNR score results of the test process provided in an embodiment of the present application;

[0041] FIG13 is a schematic diagram of the MOS_OVL score results of the test process provided in an embodiment of the present application;

[0042] FIG14 is a schematic diagram of an optional flow chart of a speech enhancement method provided in an embodiment of the present application;

[0043] FIG15 is a schematic diagram of an optional structure of a training device for a speech enhancement network provided in an embodiment of the present application;

[0044] FIG16 is a schematic diagram of an optional structure of a speech enhancement device provided in an embodiment of the present application;

[0045] FIG17 is a partial structural block diagram of a terminal provided in an embodiment of the present application;

[0046] FIG18 is a partial structural block diagram of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] It should be noted that, in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present application needs to obtain target object attribute information, the target object's separate permission or separate consent will be obtained by means of a pop-up window or jumping to a confirmation page. After clearly obtaining the target object's separate permission or separate consent, the necessary target object-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0049] Currently, in various application scenarios such as calls and video conferencing, the audio signal processing chain typically includes multiple audio processing steps. Typically, after the audio signal undergoes speech enhancement and noise reduction processing, the enhanced signal is often sent to the Automatic Gain Control (AGC) module. This module can adjust the loudness of the audio stream, suppressing parts with excessive volume and compensating for segments with insufficient volume, thereby reducing volume fluctuations. This poses a problem. When the audio stream passes through the noise reduction module, if there is significant residual noise in the non-speech segments (non-speech segments refer to speech segments that do not include valid human voices, where valid human voices can be understood as audio signals with signal strength greater than a preset intensity threshold and meeting signal continuity requirements), the AGC module is likely to amplify the residual noise signals in these segments. This increases the noise energy, and due to the discontinuity of the residual noise, it will ultimately reduce speech fluency and listening quality.

[0050] Therefore, in the related art, when a speech enhancement network performs speech enhancement processing on noisy speech containing non-speech segments, residual noise is likely to occur, thereby reducing the quality of speech enhancement.

[0051] Based on this, embodiments of the present application provide a speech enhancement network training method, a speech enhancement method, and an electronic device. These methods effectively reduce residual noise when performing speech enhancement processing on noisy speech containing non-speech segments, thereby improving the quality of speech enhancement. Furthermore, without introducing additional computational effort, embodiments of the present application enhance the noise reduction effect of the speech enhancement algorithm, particularly significantly improving the ability to suppress noise in non-speech segments.

[0052] 1 , which is a schematic diagram of an optional implementation environment provided by an embodiment of the present application, the implementation environment includes a terminal 101 and a server 102 , wherein the terminal 101 and the server 102 are connected via a communication network.

[0053] For example, the server 102 can obtain a sample clean speech and a sample noisy speech, mix the sample clean speech and the sample noisy speech into a sample noisy speech, perform noise reduction on the sample noisy speech based on the speech enhancement network to obtain a sample enhanced speech, divide the sample enhanced speech into multiple enhanced speech frames, classify the speech effectiveness of each enhanced speech frame, generate an effectiveness distribution of the sample enhanced speech based on the classification results of each enhanced speech frame, determine the noise reduction accuracy of the speech enhancement network based on the sample enhanced speech and the sample clean speech, determine the speech classification accuracy of the speech enhancement network based on the effectiveness distribution, determine the speech enhancement accuracy of the speech enhancement network based on the noise reduction accuracy and the speech classification accuracy, and train the speech enhancement network based on the speech enhancement accuracy. Subsequently, the terminal 101 can send the speech to be processed to the server 102, and the server 102 performs noise reduction on the speech to be processed based on the trained speech enhancement network to obtain the target enhanced speech. After obtaining the target enhanced speech, the server 102 can send the target enhanced speech to the terminal 101, and the terminal 101 can further process or play the target enhanced speech.

[0054] Server 102 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Furthermore, server 102 can be a node server in a blockchain network.

[0055] The terminal 101 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 and the server 102 may be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment of the present application.

[0056] For example, the speech enhancement method in the embodiments of the present application is applicable to a variety of specific scenarios, such as call noise reduction, video conferencing, speech recognition front-end, live broadcast and on-demand applications, etc. Specifically: in the call noise reduction scenario, there may be noise interference in the phone call, which affects the call quality. The trained speech enhancement network can effectively reduce the noise residual in the non-speech segment, improve the clarity and intelligibility of the speech, and thus improve the call quality. In the video conferencing scenario, participants usually use microphones for voice communication. However, there may be various noises in the conference environment, such as background noise and computer fan noise. By applying the trained speech enhancement network, these noises can be effectively suppressed and the speech recognition accuracy and auditory experience of the participants can be improved. In the speech recognition front-end scenario, such as mobile phone smart voice assistants and in-vehicle voice assistants, noise processing in the speech front-end is an important link. By applying the trained speech enhancement network, the negative impact of noise on speech recognition can be reduced, and the accuracy and stability of speech recognition can be improved. In the live broadcast and on-demand application, audio quality is crucial to the user experience. By applying the trained speech enhancement network, the clarity and quality of the audio can be improved, noise interference can be reduced, and the user can obtain a better auditory experience.

[0057] The method provided in the embodiments of the present application can be applied to different scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.

[0058] Referring to Figure 2, Figure 2 is an optional flow chart of a training method for a speech enhancement network provided in an embodiment of the present application. The training method for the speech enhancement network can be executed by an electronic device, for example, by the server 102 in Figure 1 above. The speech enhancement method includes but is not limited to the following steps 201 to 205.

[0059] Step 201: obtaining a sample clean speech and a sample noisy speech, and mixing the sample clean speech and the sample noisy speech into a sample noisy speech.

[0060] In one possible implementation, sample clean speech refers to a clear speech signal that is not interfered with by noise. For example, a preset speech quality evaluation standard (such as an evaluation standard based on the signal-to-noise ratio (SNR) in the time domain or frequency domain, an evaluation standard based on linear prediction coefficients (LPC), etc.) can be used to evaluate the quality of the speech signal, and the speech signal that meets the preset clean speech conditions (such as SNR, LPC in the corresponding preset clean signal interval, etc.) is used as the sample clean speech. These speech sounds are usually recorded or obtained from a pre-established speech database. The sample clean speech is a pure sample. In order to obtain a high-quality pure sample, the presence of background noise is usually avoided as much as possible. A professional microphone can be used for recording while maintaining good sound quality and diversity of speech content.

[0061] In one possible implementation, sample noisy speech refers to speech signals interfered with by different types of noise. Similarly, for example, a preset speech quality evaluation standard (such as an evaluation standard based on SNR in the time domain or frequency domain, an evaluation standard based on LPC, etc.) can be used to evaluate the quality of the speech signal, and speech signals that meet the preset noisy speech conditions (such as SNR, LPC in the corresponding preset noise signal range, etc.) are used as sample noisy speech. These speech sounds can be collected from the real world, for example, by using microphones to record background noise in daily life in different environments, or they can be extracted from noise databases, such as simulated noise, car noise, coffee shop ambient noise, and other environmental noise. Furthermore, the collected noisy speech should cover a variety of environments and noise types to ensure that the training model has better generalization capabilities.

[0062] In one possible implementation, the noisy speech sample is generated by mixing clean and noisy speech samples. Specifically, the clean speech sample is mixed with some noisy speech samples, making the clean speech sample noisy. The noisy speech sample can be used to simulate real-world speech. In scenarios such as phone calls, video conferencing, speech recognition front-ends, and live-on-demand applications, the speech captured by the device is generally noisy. By mixing these noise-laden speech samples, they can be used for subsequent model training.

[0063] In one possible implementation, there are several ways to mix the sample clean speech and the sample noisy speech into the sample noisy speech. For example, the sample clean speech signal and the sample noisy speech signal can be added proportionally, and the signal-to-noise ratio can be adjusted by controlling the energy ratio between the two to form the desired sample noisy speech; or, the sample clean speech signal and the sample noisy speech signal can be amplitude-adjusted separately, and then multiplied to obtain a mixed signal, and the signal-to-noise ratio can be controlled by adjusting the amplitude adjustment parameter to form the desired sample noisy speech; or, the sample noisy speech signal can be processed through a filter and then added to the sample clean speech signal to form the desired sample noisy speech; or, the sample clean speech signal and the sample noisy speech signal can be short-time Fourier transformed, mixed in the frequency domain, and then inversely transformed to obtain a mixed signal in the time domain to form the desired sample noisy speech; or, a deep learning model, such as a generative adversarial network (GAN) or an autoencoder, can be used to learn how to mix the sample clean speech and the sample noisy speech to form the desired sample noisy speech by training the model. It should be noted that a suitable hybrid method can be selected in different scenarios and application requirements to obtain the required sample noisy speech, and the embodiments of the present application do not impose specific limitations.

[0064] Step 202: De-noise the sample noisy speech based on the speech enhancement network to obtain sample enhanced speech.

[0065] Among them, the speech enhancement network is a neural network model for processing noisy speech in an embodiment of the present application, which is used to weaken the noise signal in the noisy speech, thereby enhancing the effective speech signal in the noisy speech, that is, it aims to reduce the interference of noise on the speech signal through learning, thereby improving the clarity and audibility of the speech. The speech enhancement network can adopt a deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), which can be used for speech noise reduction. After noise reduction of the input noisy speech, it outputs an enhanced speech signal.

[0066] In one possible implementation, sample-enhanced speech refers to the enhanced speech signal obtained by processing the noisy sample speech through a speech enhancement network. This process can be achieved by inputting the noisy sample speech into the speech enhancement network and obtaining the noise reduction output from the speech enhancement network. This noise reduction output is the sample-enhanced speech. The sample-enhanced speech should have low noise interference and high speech clarity.

[0067] In one possible implementation, the sample noisy speech needs to be frequency-domain transformed before being input into a speech enhancement network for feature processing. The short-time cosine spectrum estimate of the sample noisy speech is determined based on the extracted transformation mask, and finally the sample enhanced speech is obtained by inverse transformation. Specifically, the following steps may be performed: frequency-domain transform the sample noisy speech to obtain the original frequency-domain features of the sample noisy speech; based on the speech enhancement network, the original frequency-domain features are mapped multiple times to obtain mapping features; time-series information is extracted from the mapping features to obtain time-series features; the mapping features are concatenated with the time-series features to obtain concatenated features; the concatenated features are mapped multiple times to obtain a transformation mask; the original frequency-domain features are modulated based on the transformation mask to obtain target frequency-domain features; and the target frequency-domain features are inversely transformed from the frequency-domain transform to obtain the sample enhanced speech.

[0068] In one possible implementation, before the sample noisy speech is input into the speech enhancement network, it needs to be transformed in the frequency domain. The purpose of performing the frequency domain transformation on the sample noisy speech is to convert the speech signal in the time domain into a frequency domain representation, so as to obtain richer frequency domain information for better speech enhancement and other related speech processing tasks, and finally obtain the original frequency domain features of the sample noisy speech.

[0069] In one possible implementation, frequency domain transformation can be performed in a variety of ways. For example, frequency domain transformation can be performed using a Fast Fourier Transform (FFT). By using the Fast Fourier Transform (FFT) to convert a time domain signal into a frequency domain signal, the noisy speech sample signal can be converted from the time domain to the frequency domain. This allows the energy distribution of the speech signal at different frequencies to be obtained, enabling more detailed analysis and processing of the noisy speech sample signal. Alternatively, the frequency domain features of the noisy speech sample can be extracted using a Discrete Cosine Transform (DCT) operation.

[0070] In one possible implementation, before performing the frequency domain transformation, the sample noisy speech signal needs to be resampled, and then the frequency domain transformation is performed. This is illustrated here using the discrete cosine transform as an example. First, the sample noisy speech signal needs to be resampled. All audio data of different sampling rates need to be resampled to 48kHz to ensure that audio data of different sampling rates can be uniformly processed and analyzed in subsequent processing, avoiding the problem of sampling rate mismatch. After the resampling operation is completed, the long audio signal in the signal is then subjected to time domain framing and windowing processing, and the signal is localized in the time domain. Through framing, the original audio signal can be made to exhibit temporal stability, facilitating subsequent frequency domain analysis. The original audio signal can be divided into multiple frames of short signals of fixed length according to a single frame length of 1024 and a frame shift of 512 (overlap 512). Each frame signal is modulated using a Hamming window to prevent spectrum leakage and maintain the accuracy and stability of the frequency domain analysis. After the framing and windowing operations are completed, a discrete cosine transform (DCT) is performed on the modulated signal to extract frequency domain features. This yields a frequency domain representation of the noisy speech sample, representing the original frequency domain features of the noisy speech sample. The combination of audio signal framing and windowing with the DCT is also known as the short-time discrete cosine transform (SDCT).

[0071] In one possible implementation, after receiving the frequency domain representation (i.e., raw frequency domain features) of the input noisy speech signal, the speech enhancement network can perform feature processing on the raw frequency domain features. The goal of the speech enhancement network is to obtain a short-time cosine estimate of the input speech signal by performing feature processing on the raw frequency domain features, and then convert it into enhanced speech. Therefore, the speech enhancement network requires various modules to perform mapping, time sequence extraction, and splicing. Finally, the output of the speech enhancement network is modulated and inversely transformed to generate an enhanced speech signal.

[0072] In one possible implementation, the speech enhancement network is provided with a multi-layer structure. The following describes in sequence the processes of mapping, time sequence extraction, and splicing performed by each module of the speech enhancement network in an embodiment of the present application:

[0073] Mapping: The speech enhancement network is provided with a functional module for mapping input features. This module is used to map the original frequency domain features to obtain mapped features. The module can introduce nonlinear changes, thereby increasing the expressiveness and discrimination of the features, better capturing the effective information in the original frequency domain features, and providing a more discriminative representation. Exemplarily, the module that maps the original frequency domain features in the speech enhancement network is an encoder module. The encoder module is mainly composed of a series of EncConv2d structures with a two-dimensional convolution (Conv2d) kernel. The convolution kernel size (kernel size) of each EncConv2d layer is set to (5, 2), where 5 represents the frequency domain field of view, that is, when performing the convolution operation, each convolution kernel will consider the feature information of the five frequency domain positions before and after, and 2 represents the time domain field of view, that is, each convolution kernel will consider the features of the two adjacent frames of signals. The purpose of this is to allow the processing of the current frame to refer to the information of the previous frame. By introducing the time domain field of view, the temporal relationship between signals can be better captured, and the analysis and processing of the signal features of each frame will refer to the previous frame signal. In addition, the convolution stride of each EncConv2d layer is set to (2, 1). This means that when performing the convolution operation, the frequency domain dimension of the feature map is halved layer by layer, while the time domain dimension remains unchanged. This setting helps to reduce the dimension of the feature map and reduce the amount of computation while retaining important frequency domain features. By halving the frequency domain dimension layer by layer, it can effectively represent the input signal while playing the role of dimensionality reduction and reducing the amount of computation.

[0074] Timing extraction: The speech enhancement network is provided with a functional module for extracting timing information based on mapping features, extracting the dynamic changes of audio in the time dimension. The extraction of timing features helps to model the time-varying characteristics of audio signals and capture the timing relationship of audio. Exemplarily, the speech enhancement network can set up models such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs) for timing extraction. Specifically, the embodiment of the present application uses a recurrent neural network module (RNNs) composed of a stack of gated recurrent units (GRUs). The function of RNNs is mainly to extract and analyze the timing information between audio signal frames. RNNs receive the mapping features output from the last layer of EncConv2d, extract and analyze the timing information, and obtain timing features.

[0075] Splicing: The speech enhancement network is provided with a functional module for splicing mapping features and timing features, which is used to combine the mapping features and timing features and fuse their information so that the resulting spliced ​​features can contain information in both the frequency domain and the time domain, that is, the resulting spliced ​​features provide a richer and more comprehensive audio representation. Exemplarily, the module for splicing mapping features and timing features in the speech enhancement network is a decoder module, which is mainly composed of a series of DecTConv2d. Each DecTConv2d layer uses a transposed two-dimensional convolution (ConvTranspo se2d) as the main operation and has the same parameters as the corresponding EncConv2d layer in the encoder to achieve signal dimensionality restoration. Therefore, after the encoder module receives the original frequency domain features represented by the short-time cosine transform of the noisy speech, it extracts high-dimensional features layer by layer through a series of EncConv2d layers, and passes the corresponding output mapping features to the DecTConv2d layer through skip connections. RNNs receive the output features from the last EncConv2d layer, extract and analyze the timing information, and pass it as input to the decoder. The decoder splices the mapping features with the timing features to obtain spliced ​​features.

[0076] Multiple mappings: The embodiments of the present application can perform multiple mappings on the splicing features, introduce more nonlinear transformations, further extract and enhance the useful information in the splicing features, and increase their representation ability and discrimination. Exemplarily, the decoder module in the speech enhancement network can perform multiple mappings on the splicing features, and generate a transformation mask based on the splicing features after multiple mappings. The transformation mask is a mask vector used to modulate the original frequency domain features. It can change the frequency domain properties of the features by controlling the gain, phase and other information of the spectrum. The purpose of generating the transformation mask is to perform directional adjustment and optimization on the original frequency domain features to achieve the target enhancement effect. Therefore, after the decoder module receives the outputs from the RNNs and the encoder module, it undergoes a layer-by-layer dimensionality increase process to finally generate a cosine transform mask.

[0077] In one possible implementation, after obtaining the transformation mask, it is necessary to modulate the original frequency domain features based on the transformation mask to obtain the target frequency domain features, and perform an inverse transformation of the frequency domain transformation on the target frequency domain features to obtain sample enhanced speech. The following describes the subsequent modulation and inverse transformation processes in the embodiment of the present application in sequence:

[0078] Modulation: In the embodiment of the present application, a functional module is provided for modulating the original frequency domain features based on a transformation mask, which is used to modulate the original frequency domain features using the generated transformation mask, that is, to change the amplitude, phase and other information of the original frequency domain features in a manner guided by the transformation mask. For example, the modulation operation can enhance certain frequency bands of the target signal or suppress the frequency bands of noise as required to achieve the effect of sample enhancement. Therefore, after obtaining the transformation mask of the speech signal, the short-time cosine spectrum of the original noisy speech, that is, the original frequency domain features, can be modulated to obtain an estimate of the short-time cosine spectrum of the sample noisy speech, and used as the target frequency domain features obtained by modulation.

[0079] Inverse transform: The embodiment of the present application is provided with a functional module for inverse transform of the modulated target frequency domain features, which is used to convert them back to the time domain signal. In this way, an enhanced speech signal, i.e., sample enhanced speech, can be obtained, so that the speech signal is improved and optimized in the frequency domain, thereby improving the clarity and robustness of the speech. Therefore, after obtaining the short-time cosine spectrum of the sample noisy speech, i.e., the target frequency domain features, the embodiment of the present application finally performs the inverse short-time cosine transform (iSDCT) corresponding to the SDCT, and a time domain estimate of the enhanced speech signal can be obtained as the final sample enhanced speech.

[0080] Below, the process of obtaining the sample enhanced speech is described in detail in conjunction with the overall system framework of the speech enhancement method applied in the embodiment of the present application:

[0081] In one possible implementation, referring to Figure 3, Figure 3 is a functional module diagram of the overall system framework for the application of the training method for the speech enhancement network provided in an embodiment of the present application. The overall system framework is provided with three modules, namely, an audio signal pre-processing and feature extraction module (corresponding to the above-mentioned frequency domain transformation process), a neural network model inference module (corresponding to the above-mentioned mapping, timing information extraction, splicing, and multiple mapping processes performed based on the speech enhancement network) and a post-processing speech generation module (corresponding to the above-mentioned modulation and inverse transformation process).

[0082] In the pre-processing and feature extraction module, the sample noisy sample speech signal x is first processed. n Resampling is performed to resample all audio data of different sampling rates to 48kHz. After the resampling operation is completed, the long audio signal in the signal is then subjected to time domain frame division and windowing. According to the single frame length of 1024 and the frame shift of 512 (overlap 512), the original audio signal is divided into multiple frames of short signals with fixed lengths, and each frame signal is modulated using a Hamming window to prevent spectrum leakage. After the frame division and windowing operation is completed, the modulated signal is subjected to DCT operation to extract the frequency domain features to obtain the sample noisy speech signal xn The original frequency domain feature X k The combination of audio signal framing and windowing with cosine transform operation is also called SDCT.

[0083] For the neural network model inference module, refer to Figure 4, which is a schematic diagram of the structural design of the neural network model inference module provided in an embodiment of the present application. The neural network model inference module mainly includes an encoder, a recurrent neural network module and a decoder. The encoder part is mainly composed of an EncConv2d structure with a two-dimensional convolution (Conv2d) as the kernel. The convolution kernel size of each layer of EncConv2d is (5,2), which means that the frequency domain field of view is 5 and the time domain field of view is 2. The analysis and processing of each frame signal feature will refer to the previous frame signal. The convolution step is (2,1), which can reduce the number of signal frequency domain features by half layer by layer, while the number of time domain frames remains unchanged, which plays a role in reducing dimensionality and reducing the amount of calculation. The encoder part is mainly composed of DecTConv2d with a transposed two-dimensional convolution (ConvTranspose2d) as the kernel. The parameters of DecTConv2d of each layer are the same as those of the corresponding EncConv2d, which realizes the restoration of the signal dimension. Between the encoder and the decoder, the embodiment of the present application uses a recurrent neural network module RNNs composed of a stack of GRUs. The main function of RNNs is to extract and analyze the temporal information between audio signal frames. Therefore, the workflow of the neural network model inference module is as follows: the encoder receives the short-time cosine transform representation of the sample noisy speech from the signal preprocessing module, that is, the original frequency domain feature X k , then EncConv2d extracts high-dimensional features layer by layer, and the corresponding output is given to DecTConv2d through a skip connection. RNNs accept the output features from the last layer of EncConv2d of the encoder, perform temporal information extraction and analysis, and give the input to the decoder. The decoder accepts the output from RNNs and the encoder, and after layer-by-layer dimensionality increase processing, finally obtains the cosine transform mask

[0084] In the post-processing speech generation module, the original frequency domain features are modulated based on the transformation mask to obtain the short-time cosine spectrum estimation of the sample noisy speech signal, which is used as the target frequency domain feature obtained by modulation. Target frequency domain characteristics The expression is as follows:

[0085] Get the target frequency domain features After that, the target frequency domain features are finally By performing the iSDCT operation corresponding to SDCT, the time domain estimation value of the enhanced speech signal can be obtained as the final sample enhanced speech signal.

[0086] In the above implementation, the sample noisy speech is first transformed into the frequency domain to obtain the corresponding original frequency domain features, converting the sample noisy speech signal from the time domain to the frequency domain, and obtaining the energy distribution of the sample noisy speech at different frequencies, which facilitates subsequent more refined analysis and processing based on the original frequency domain features. Then, through the speech enhancement network, the original frequency domain features are mapped multiple times to obtain mapping features, which effectively represent the input signal while reducing the dimensionality and reducing the amount of computation. Time series information is extracted from the mapping features to obtain time series features, which are used to extract and analyze the time series information between audio signal frames. The mapping features are spliced ​​with the time series features to obtain spliced ​​features, so that the resulting spliced ​​features contain information in both the frequency domain and the time domain, providing a richer and more comprehensive audio representation. The spliced ​​features are mapped multiple times to obtain a transform mask for directional adjustment and optimization of the original frequency domain features to achieve the target enhancement effect. Furthermore, the original frequency domain features are modulated based on the transform mask, so that the speech signal is improved and optimized in the frequency domain, and the target frequency domain features after target enhancement are obtained. Finally, an inverse transform of the frequency domain transform is performed based on the target frequency domain features to obtain a sample enhanced speech. After the above processing, the sample enhanced speech has higher clarity and robustness.

[0087] In one possible implementation, in addition to using the speech enhancement network of the above framework, an end-to-end model can also be used as a speech enhancement network to obtain sample enhanced speech. Exemplarily, the end-to-end model directly uses the original input as the input of the network and outputs the final enhanced speech result, omitting the intermediate feature processing and step-by-step processing process. The following describes the speech enhancement network using the end-to-end model in the embodiment of the present application:

[0088] In one possible implementation, refer to FIG5 , which is a structural diagram of a speech enhancement network based on an end-to-end model provided in an embodiment of the present application. The end-to-end model is provided with an input layer, a feature extraction layer, an encoding layer, a decoding layer, and an output layer. During training, a sample noisy speech can be used as the input of the network and input into the input layer. Subsequently, the feature extraction layer uses a series of convolutional layers and pooling layers to extract features from the input for learning local patterns and spectral features in the speech signal, and forwards it to the encoding layer. In the encoding layer, a recurrent neural network (such as LSTM, GRU) or a convolutional neural network is used to encode the input feature sequence, capture contextual information and temporal relationships, and output the encoding result to the decoding layer. The decoding layer decodes the encoded feature sequence through a reverse recurrent neural network or a convolutional neural network and generates a sample enhanced speech. Finally, the output layer post-processes the speech signal output by the decoding layer (such as denormalization, amplitude adjustment, etc.) to obtain the final sample enhanced speech.

[0089] Step 203: Divide the sample enhanced speech into multiple enhanced speech frames, classify the speech effectiveness of each enhanced speech frame, and generate the effectiveness distribution of the sample enhanced speech according to the classification results of each enhanced speech frame.

[0090] Among them, enhanced speech frames are speech segments obtained by framing the sample enhanced speech. In speech signal processing, in order to effectively extract and process features, the embodiment of the present application divides the long sample enhanced speech into shorter continuous segments, which are called enhanced speech frames. By framing the sample enhanced speech into multiple enhanced speech frames, the long speech signal can be decomposed into a series of short speech frames. Each enhanced speech frame contains local information of the speech signal, which facilitates the subsequent classification and evaluation of the speech effectiveness of each enhanced speech frame.

[0091] In a possible implementation, there are multiple ways to divide the sample enhanced speech into multiple enhanced speech frames. For example, fixed-frame-length framing can be used to divide the enhanced speech signal into fixed-length frames at equal intervals. The frame length and frame shift parameters are first determined. For example, each frame has a signal length of 20 milliseconds, and adjacent frames are separated by a 10-millisecond interval. The enhanced speech signal is then overlapped and sampled according to the frame shift length to obtain a continuous frame sequence. Alternatively, sliding window framing can be used to slide a sliding window across the speech signal to extract frames. The window length and window shift step are first determined. Typically, the window length is equal to the frame length. Starting from the starting position of the enhanced speech signal, the window is slid according to the window shift step, and the signal within the window is extracted as a frame. The window is then continuously slid to extract subsequent frames until the signal ends. Alternatively, dynamic energy threshold framing can be used to determine frame boundaries based on the energy characteristics of the speech signal. The short-term energy of the enhanced speech signal is first calculated. Typically, the signal is framed and the energy of each frame is calculated. An energy threshold is then set to detect frames whose energy exceeds the threshold as valid speech, while frames whose energy is below the threshold are considered silence or background noise. Valid frame boundaries in the enhanced speech signal are then determined based on the energy threshold. The above-mentioned framing method does not represent a limitation on the specific framing method for the embodiments of the present application. The embodiments of the present application can select a suitable framing method under different scenarios and application requirements, and parameters need to be adjusted according to actual conditions to obtain an enhanced speech frame sequence that meets the requirements.

[0092] Among them, speech validity refers to information about whether the enhanced speech frame contains a valid speech signal. The valid speech signal here refers to an audio signal whose energy exceeds a preset energy threshold. The valid speech signal can be, for example, a valid human voice signal. It should be noted that the enhanced speech frame containing a valid speech signal has substantially higher energy, and a speech frame with higher energy is usually a clear human voice segment. Therefore, the enhanced speech frame containing a valid speech signal can be correspondingly considered as a speech segment containing a valid human voice signal; on the contrary, the enhanced speech frame not containing a valid speech signal has substantially lower energy, and a speech frame with lower energy is usually a noisy speech segment. Therefore, the enhanced speech frame not containing a valid speech signal can be correspondingly considered as a non-speech segment not containing a valid human voice signal.

[0093] In the speech signal processing in the embodiment of the present application, speech validity classification is used to determine whether each enhanced speech frame belongs to a speech segment (i.e., a speech segment containing the above-mentioned valid speech signal) or a non-speech segment (i.e., a speech segment not containing the above-mentioned valid speech signal). If a certain enhanced speech frame is determined to belong to a speech segment through speech validity classification, it means that the enhanced speech frame is valid and is a valid speech frame. Conversely, if a certain enhanced speech frame is determined to belong to a non-speech segment (such as silence or noise) through speech validity classification, it means that the enhanced speech frame is not valid and is not a valid speech frame. By performing speech validity classification on each enhanced speech frame, the enhanced speech frame where the valid speech signal (such as the valid human voice signal) is located can be identified, thereby realizing the classification of a large number of enhanced speech frames.

[0094] As mentioned above, each enhanced speech frame contains many non-speech segments, such as silence or noise. By performing speech validity classification on each enhanced speech frame, it is possible to determine which enhanced speech frames contain information about valid speech signals and distinguish between speech and non-speech. Furthermore, after classifying the enhanced speech frames into speech and non-speech segments, noise suppression can be performed on the non-speech segments to identify the frames containing noise. This helps to further reduce the impact of noise, improve the speech enhancement effect, and facilitate subsequent model training. Furthermore, by performing speech validity classification on each enhanced speech frame, the starting and ending positions of the speech segments can be determined, thereby enabling the detection of speech boundaries. This is of great significance for tasks such as speech recognition and speech synthesis, and is not specifically limited here.

[0095] The validity distribution is a feature generated based on the validity classification results of each enhanced speech frame. It is used to indicate which parts of the sample enhanced speech are non-speech segments. For example, if the validity classification result of a certain enhanced speech frame indicates that the enhanced speech frame belongs to a speech segment, it can be represented as "1", otherwise it is represented as "0". Accordingly, the validity classification results of multiple enhanced speech frames can be combined into a validity distribution, such as "101010101010". The validity distribution can indicate which parts of the sample enhanced speech generated by the speech enhancement network are non-speech segments. These features are very important for the calculation and training of the validity classification loss of the speech enhancement network. They can help the network better understand and process non-speech segments, thereby improving the quality of speech enhancement and reducing residual noise.

[0096] In one possible implementation, speech validity detection can be performed on multiple enhanced speech frames using methods such as short-time Fourier transform (STFT), amplitude spectrum, power spectrum, and mel spectrum. These frequency domain representation methods can perform speech validity detection on multiple enhanced speech frames, analyze and compare their enhancement effects or quality, and ultimately obtain classification results for each enhanced speech frame. This embodiment of the present application does not impose specific limitations on this.

[0097] Step 204: Determine the noise reduction accuracy of the speech enhancement network based on the sample enhanced speech and the sample clean speech, determine the speech classification accuracy of the speech enhancement network based on the effectiveness distribution, determine the speech enhancement accuracy of the speech enhancement network based on the noise reduction accuracy and the speech classification accuracy, and train the speech enhancement network based on the speech enhancement accuracy.

[0098] Denoise accuracy measures the difference between the sample-enhanced speech and the sample-clean speech obtained after processing by the speech enhancement network. It indicates the degree of error or distortion incurred by the speech enhancement network when reducing the noise of the sample-noisy speech to the sample-enhanced speech. This indicates the accuracy of the speech enhancement network's denoising of the sample-noisy speech. This denoise accuracy can be measured by the difference between the denoised sample-enhanced speech and the sample-clean speech unmixed with the sample-noise speech. Calculating the difference between the sample-enhanced speech and the sample-clean speech reveals the noise reduction effectiveness and signal conversion capabilities of the speech enhancement network.

[0099] In one possible implementation, the embodiment of the present application can determine the noise reduction loss of the speech enhancement network based on the sample enhanced speech and the sample clean speech in a variety of ways, and the noise reduction loss can correspondingly characterize the above-mentioned noise reduction accuracy. For example, the mean square error loss can be used to calculate the noise reduction loss. The mean square error loss is a commonly used indicator for evaluating the difference between the generated result and the target result. By minimizing the noise reduction loss, the speech enhancement network can minimize the error or distortion when converting the sample noisy speech into the sample enhanced speech. During the training process, optimization algorithms such as gradient descent can be used to gradually reduce the noise reduction loss and update the parameters of the speech enhancement network so that it is gradually optimized and close to the optimal state. Through multiple iterative training, the speech enhancement network can gradually learn the mapping relationship between the sample clean speech and the sample enhanced speech, thereby improving the noise reduction effect. It should be understood that the noise reduction loss is negatively correlated with the noise reduction accuracy, that is, the higher the noise reduction accuracy, the smaller the value of the noise reduction loss.

[0100] It's important to note that noise reduction accuracy only considers the difference between the clean and enhanced speech samples, which limits its effectiveness in measuring the speech enhancement network's processing of non-speech segments. To more comprehensively evaluate the performance of the speech enhancement network, it's necessary to combine speech classification accuracy with speech enhancement accuracy to further optimize network training.

[0101] Specifically, speech classification accuracy measures the accuracy of the speech enhancement network in classifying speech segments (including speech segments with valid speech signals) and non-speech segments (excluding speech segments with valid speech signals) in the noisy speech sample during the speech enhancement process. Speech classification accuracy is derived from the validity distribution, which characterizes the distribution of speech segments and non-speech segments in the enhanced speech sample. It measures the speech enhancement network's recognition and processing performance for non-speech segments. More specifically, the speech classification loss can be determined based on the difference in speech validity between the generated enhanced speech frames and the clean speech frames in the clean speech sample. This speech classification loss reflects the speech classification accuracy and represents the error incurred by the speech enhancement network when classifying the validity of speech frames in the noisy speech sample. By training the speech enhancement network with reference to the speech classification loss, the network can better suppress noise in non-speech segments, thereby improving noise reduction and the quality of the generated enhanced speech. The speech classification loss helps the network learn more accurate speech / non-speech classification and avoids issues such as residual noise. It should be understood that the speech classification loss is negatively correlated with the speech classification accuracy; that is, the higher the speech classification accuracy, the smaller the speech classification loss.

[0102] In one possible implementation, the embodiments of the present application can determine the speech classification loss of the speech enhancement network based on the effectiveness distribution in various ways. A certain metric method can be used, for example, to calculate the speech classification loss of the speech enhancement network using cross entropy loss or mean square error loss, without specific limitation herein.

[0103] In one possible implementation, the noise reduction accuracy and speech classification accuracy are combined to determine the speech enhancement accuracy of the speech enhancement network. That is, the noise reduction loss and speech classification loss are combined to determine an overall loss that reflects the speech enhancement accuracy of the speech enhancement network. Speech enhancement accuracy comprehensively considers the impact of the speech enhancement network on the sound quality of the enhanced speech frames and the effectiveness of the speech to determine the optimization direction of the speech enhancement network during training, thereby more comprehensively optimizing the network's performance during training. It should be understood that the overall loss is also negatively correlated with the speech enhancement accuracy; that is, the higher the speech enhancement accuracy, the smaller the overall loss.

[0104] In one possible implementation, the noise reduction loss and the speech classification loss can be added together with certain weights to obtain the final overall loss. During the training process, the overall loss is used as the optimization objective function, and the parameters of the speech enhancement network are updated through the back-propagation algorithm to minimize the overall loss. Specifically, a gradient descent method or other optimization algorithm can be used to minimize the overall loss determined based on the noise reduction loss and the speech classification loss, and iterative training can be performed based on the training data to continuously improve the performance of the speech enhancement network and enhance the speech enhancement network's ability to suppress non-speech segment noise. When noise reduction is performed on the processed speech based on the trained speech enhancement network, the trained speech enhancement network can effectively reduce the occurrence of residual noise in the processed speech containing non-speech segments, thereby improving the quality of speech enhancement.

[0105] In summary, in an embodiment of the present application, the sample enhanced speech is divided into multiple enhanced speech frames, the speech validity of each enhanced speech frame is classified, and the validity distribution of the sample enhanced speech is generated based on the classification results of each enhanced speech frame. Since the validity distribution can indicate whether each enhanced speech frame generated by the speech enhancement network is a non-speech segment, the speech classification accuracy of the speech enhancement network determined by the validity distribution can be used to measure the recognition accuracy of the currently trained speech enhancement network for speech segments (i.e., speech signals containing valid human voices) and non-speech segments (i.e., speech signals that do not contain valid human voices, such as silence signals, noise signals, etc.). On this basis, the speech enhancement accuracy of the speech enhancement network is determined based on the noise reduction accuracy and the speech classification accuracy. Training the speech enhancement network based on the speech enhancement accuracy can improve the speech enhancement network's recognition ability for speech segments and non-speech segments, and accordingly focus on improving the noise suppression ability of non-speech segments. When noise reduction is performed on the processed speech based on the trained speech enhancement network, for the processed speech containing non-speech segments, the trained speech enhancement network can effectively reduce the phenomenon of residual noise in the non-speech segments, thereby improving the quality of speech enhancement.

[0106] In a possible implementation, the speech effectiveness of each enhanced speech frame is classified according to the time domain energy parameter of the enhanced speech frame. The following is an explanation of how to classify the speech effectiveness in step 203:

[0107] In one possible implementation, the speech effectiveness of each enhanced speech frame can be classified based on a time-domain energy parameter. Specifically, the method includes determining a time-domain energy parameter for each enhanced speech frame, where the time-domain energy parameter indicates the speech energy of the enhanced speech frame in the time domain; and classifying the speech effectiveness of each enhanced speech frame based on the time-domain energy parameter and a preset energy threshold.

[0108] The time domain energy parameter is a parameter used to indicate the speech energy of the enhanced speech frame in the time domain. The time domain energy is the result of calculating the energy of the speech signal over time. There are multiple ways to determine the time domain energy parameter of each enhanced speech frame. For example, the short-time energy method can be used to calculate the time domain energy parameter by calculating the square sum of the sample values ​​within the speech frame. Specifically, for a given speech frame, its sample values ​​are regarded as a vector, and the square sum of the vector is calculated as the energy parameter. Alternatively, the short-time amplitude spectrum method can be used to calculate the energy parameter by analyzing the speech signal in the frequency domain. First, the speech frame is Fourier transformed to obtain a short-time amplitude spectrum, and then the short-time amplitude spectrum is squared and accumulated to obtain the time domain energy parameter. Alternatively, the autocorrelation method can be used to calculate the energy parameter using the autocorrelation function of the speech signal. The autocorrelation function reflects the similarity between the signal and itself at different time points. The time domain energy parameter can be obtained by calculating the peak or area of ​​the autocorrelation function. In addition, the wavelet transform method can be used to analyze the speech frame using wavelet transform and extract energy information from the wavelet coefficients to obtain the time domain energy parameter. In the embodiments of the present application, there is no specific limitation on the process of obtaining the time domain energy parameters.

[0109] In one possible implementation, the speech validity of each enhanced speech frame can be classified based on a time-domain energy parameter and a preset energy threshold. The preset energy threshold is a threshold that is pre-set or experimentally determined in the embodiment of the present application and is used to determine whether the energy of the speech frame meets the limit value required for validity. For each enhanced speech frame, its time-domain energy parameter can be compared with the preset energy threshold. If the time-domain energy parameter of the enhanced speech frame is higher than the preset energy threshold, it is considered to be a valid speech frame; conversely, if the time-domain energy parameter of the speech frame is lower than the preset energy threshold, it is considered to be a non-speech frame or a noise frame.

[0110] The time-domain energy parameter includes the single-frame average energy, and the speech validity of each enhanced speech frame can be classified by comparing the single-frame average energy with the comprehensive average energy of the sampled clean speech. Specifically, the comprehensive average energy of the sampled clean speech is determined, and the comprehensive average energy is weighted according to a preset energy threshold to obtain a weighted average energy; when the single-frame average energy is greater than the weighted average energy, the enhanced speech frame is classified as a valid speech frame; or, when the single-frame average energy is less than or equal to the weighted average energy, the enhanced speech frame is classified as not a valid speech frame.

[0111] In a possible implementation, the single-frame average energy refers to the average energy of the sample values ​​in a speech frame. In the embodiment of the present application, the sum of the squares of the sample values ​​in the speech frame can be calculated to represent the energy, that is, the sample values ​​of each speech frame are squared and summed to obtain the single-frame average energy used to characterize the energy of the speech frame. The overall energy in the speech frame can be evaluated by the single-frame average energy. In the embodiment of the present application, the time domain energy parameter is taken as the single-frame average energy as an example. On the premise of meeting the requirements of the embodiment of the present application, the single-frame energy median value or the single-frame energy maximum value in the single-frame energy value can also be selected as the time domain energy parameter, and no specific limitation is made to this.

[0112] In one possible implementation, the comprehensive average energy is an energy metric for the clean speech sample and is a parameter used to calculate and classify the energy of the clean speech sample. In this embodiment of the present application, since the clean speech sample does not contain non-speech segments or noise, the comprehensive average energy can represent the average energy of each frame in the speech.

[0113] In one possible implementation, the comprehensive average energy is weighted according to a preset energy threshold. The purpose of obtaining a weighted average energy is to better adapt to speech characteristics in different environments. The preset energy threshold is a reference value set based on actual application requirements and relevant field experience. It can reflect the typical energy range of valid speech. By weighting the comprehensive average energy, a reference energy parameter suitable for measuring the effectiveness of speech frames in specific environments can be obtained. In real-world scenarios, speech signals are often contaminated by noise. The preset energy threshold can be used to adjust the comprehensive average energy of clean speech samples to obtain a weighted average energy that is suitable for determining speech characteristics in different noise environments. For example, when the ambient noise is strong, the preset energy threshold can be increased accordingly. This makes it easier to classify enhanced speech frames containing valid speech as valid frames. Furthermore, the energy of a speech signal may fluctuate significantly over time. The weighted average energy comprehensively considers the energy characteristics of the entire speech segment and better captures the dynamic changes of the speech signal. The weighted average energy can reduce the fluctuation of the energy of individual speech frames, making the classification results more stable and reliable. Therefore, by weighting the comprehensive average energy according to the preset energy threshold, the accuracy and robustness of the judgment on the effectiveness of the enhanced speech frame can be improved, and it can better adapt to different noise environments and speech signal changes, thereby achieving a more reliable speech enhancement effect.

[0114] In one possible implementation, since the sample enhanced speech is divided into frames to form multiple enhanced speech frames, then in these enhanced speech frames, if there is no non-speech segment or noise, due to the presence of other non-speech segments or noise, the energy of the current enhanced speech frame will be greater than the overall average energy, that is, the other non-speech segments or noise will lower the overall average energy; conversely, if there is a non-speech segment or noise in the enhanced speech frame, due to the presence of these non-speech segments or noise, the energy of the current frame will be reduced. Therefore, by comparing the single-frame average energy and the weighted average energy, it can be determined whether the enhanced speech frame contains valid speech. Specifically, if the single-frame average energy is greater than the weighted average energy, it means that the energy of the enhanced speech frame is higher than the average energy of the pure speech, and the classification result of the speech validity of the enhanced speech frame can be determined as the enhanced speech frame being a valid speech frame; conversely, if the single-frame average energy is less than or equal to the weighted average energy, it means that the energy of the enhanced speech frame is low, and the classification result of the speech validity of the enhanced speech frame can be determined as the enhanced speech frame not being a valid speech frame.

[0115] The following describes in detail the process of determining whether an enhanced speech frame is a valid speech frame based on the average energy of a single frame through a specific embodiment:

[0116] First, the embodiment of the present application provides a frame-level speech validity determination method, assuming that the sample pure speech signal s n , n=1,2,3…N is a 48kHz discrete time sampling signal. Some segments of the signal are non-speech segments, that is, segments where no one is speaking. Such segments have low energy and can even be considered as silent segments. In the embodiment of the present application, the sample enhanced speech is framed according to a fixed frame length and frame shift (1024, 512). This means that the speech is divided into multiple frames, each containing 1024 sampling points, and there is an overlap of 512 sampling points between adjacent frames. The frame processing is similar to the framing process described above, where the enhanced speech signal s i,j middle, Indicates that s n The frame number after framing, j represents the jth sampling point of the i-th frame signal, and N represents the number of frames. Therefore, we can get:

[0117] The comprehensive average energy P of the sample pure speech s for:

[0118] The average energy of a single frame P i for:

[0119] Preset energy threshold: ε = 0.01

[0120] For example, in addition to 0.01, the preset energy threshold ε may also take other values, which may be obtained through experiments and may be adjusted according to the training effect of the network.

[0121] Therefore, the weighted average energy is obtained by weighting the comprehensive average energy according to the preset energy threshold as P s ·ε, so the classification result of the speech effectiveness of the enhanced speech frame is:

[0122] By comparing the single-frame average energy and the weighted average energy, it is possible to determine whether the frame signal belongs to a valid speech segment with a high accuracy. When the single-frame average energy is greater than the weighted average energy, V i If it is 1, it means that the enhanced speech frame is a valid speech frame. Otherwise, if the average energy of a single frame is less than or equal to the weighted average energy, V i If it is 0, it means that the enhanced speech frame is not a valid speech frame. i A classification result of the speech effectiveness of the enhanced speech frame may be determined.

[0123] Based on the above process, the classification results of the speech effectiveness of each enhanced speech frame are given Simplified representation:

[0124] Among them, the classification results of the speech effectiveness of each enhanced speech frame Can also be written as V s,i or V is , represents the speech effectiveness classification result of the i-th enhanced speech frame.

[0125] In this implementation, by weighting the comprehensive average energy of the clean speech samples using a preset energy threshold to obtain a weighted average energy, this method can better adapt to different noise environments and speech signal variations, and set the corresponding speech validity threshold. Furthermore, by determining whether the enhanced speech frame's single-frame average energy is greater than the weighted average energy, the validity of the enhanced speech frame is determined, thereby improving the accuracy and robustness of the speech validity judgment of the enhanced speech frame.

[0126] In the above embodiment, the process of determining whether an enhanced speech frame is a valid speech frame based on the calculated single-frame average energy in the embodiment of the present application is described. In addition, the embodiment of the present application can also calculate the single-frame short-time energy of the enhanced speech frame and determine whether the enhanced speech frame is a valid speech frame based on the single-frame short-time energy. The process of determining whether an enhanced speech frame is a valid speech frame based on the single-frame short-time energy is described below:

[0127] The time domain energy parameter may further include a single-frame short-time energy, and the number of preset energy thresholds may be multiple, and the multiple preset energy thresholds include a first energy threshold and a second energy threshold, wherein the first energy threshold is greater than the second energy threshold. Therefore, the speech validity of each enhanced speech frame may be classified by comparing the single-frame short-time energy with at least one of the first energy threshold and the second energy threshold. Specifically, when the single-frame short-time energy is greater than the first energy threshold, the classification result of the speech validity of the enhanced speech frame is determined to be that the enhanced speech frame is a valid speech frame, or when the single-frame short-time energy is less than or equal to the second energy threshold, the classification result of the speech validity of the enhanced speech frame is determined to be that the enhanced speech frame is not a valid speech frame.

[0128] Single-frame short-time energy refers to the short-time energy of each enhanced speech frame. Short-time energy is a metric used to measure the local energy of a speech or audio signal. It can be obtained by calculating the energy of each signal frame. The purpose of calculating short-time energy is to reflect the energy changes of the speech or audio signal over a short period of time. In passages with strong sounds, the short-time energy value is higher, while in passages with silence or low speech, the short-time energy value is lower.

[0129] For example, the short-time energy of a single frame can be calculated using the amplitude square method. For example, the sampling points of each enhanced speech frame are squared, and the results are then accumulated and normalized as necessary to obtain the required short-time energy of a single frame. Therefore, the amount of short-time energy of a single frame is related to the sampling points. The calculation of the short-time energy of each frame usually requires combining appropriate frame length and frame shift parameters. The frame length determines the number of sampling points contained in each frame, while the frame shift indicates the number of overlapping points between adjacent frames. Such framing can divide the speech or audio signal into multiple local segments, thereby better capturing the short-time energy changes of the signal.

[0130] In a possible implementation, the first energy threshold and the second energy threshold are two pre-set energy thresholds used to determine whether each enhanced speech frame is a valid speech frame based on the single-frame short-time energy of each enhanced speech frame. Among them, the first energy threshold is greater than the second energy threshold. The first energy threshold is a threshold set to indicate that the short-time energy is large to a certain extent (that is, it indicates that the short-time energy is strong). When there is a single-frame short-time energy greater than the first energy threshold, it means that the short-time energy of this single frame is large enough and the sound in this frame is strong. Therefore, if the single-frame short-time energy of the enhanced speech frame is greater than the first energy threshold, it can be determined that the classification result of the speech validity of the enhanced speech frame is that the enhanced speech frame belongs to a valid speech frame. Conversely, the second energy threshold is a threshold set to indicate that the short-time energy is small to a certain extent (that is, it indicates that the short-time energy is low). When there is a single-frame short-time energy less than or equal to the second energy threshold, it means that the short-time energy of this single frame is small enough and the sound in this frame is low, and it is unlikely to have valid sound. Therefore, if the single-frame short-time energy of the enhanced speech frame is less than the second energy threshold, it can be determined that the classification result of the speech validity of the enhanced speech frame is that the enhanced speech frame does not belong to a valid speech frame. If the single-frame short-time energy is between the first energy threshold and the second energy threshold, further judgment is required.

[0131] In a possible implementation, when the short-time energy of a single frame is less than or equal to a first energy threshold and greater than a second energy threshold, the speech validity of each enhanced speech frame can be classified by comparing the short-time average zero-crossing rate of the enhanced speech frame with a preset zero-crossing rate threshold. Specifically, when the short-time energy of a single frame is less than or equal to the first energy threshold and greater than the second energy threshold, the short-time average zero-crossing rate of the enhanced speech frame is obtained, and when the short-time average zero-crossing rate is greater than the preset zero-crossing rate threshold, the classification result of the speech validity of the enhanced speech frame is determined to be that the enhanced speech frame is a valid speech frame; or when the short-time energy of a single frame is less than or equal to the first energy threshold and greater than the second energy threshold, the short-time average zero-crossing rate of the enhanced speech frame is obtained, and when the short-time average zero-crossing rate is less than or equal to the preset zero-crossing rate threshold, the classification result of the speech validity of the enhanced speech frame is determined to be that the enhanced speech frame is not a valid speech frame.

[0132] In one possible implementation, the short-time average zero-crossing rate (SRR) describes how often a signal crosses zero in a short period of time. Here, zero is a point where energy is zero. In voice activity detection, the SRR can help distinguish between speech and non-speech segments. Speech segments typically have a higher SRR because the higher frequency of vocal cord vibration during pronunciation results in a higher frequency of positive and negative zero crossings. Non-speech segments, however, lack acoustic signals and therefore have a lower SRR.

[0133] In one possible implementation, the short-term average zero-crossing rate is calculated by evaluating the sampling points of each frame to determine whether a positive or negative zero crossing has occurred. A zero crossing occurs when the signal changes from a positive value to a negative value or vice versa. The short-term average zero-crossing rate represents the average number of zero crossings per sample over a period of time. Specifically, the calculation of the short-term average zero-crossing rate for each frame typically requires first comparing the frame's sampling point with the previous sampling point to detect whether a zero crossing has occurred. The number of zero crossings that occur in each frame is then accumulated and normalized if necessary.

[0134] In one possible implementation, a high short-term average zero-crossing rate indicates a high frequency of signal change. Specifically, the signal waveform frequently crosses zero within a short period of time, indicating that the signal contains a high number of high-frequency components or rapidly changing audio content. Therefore, the signal is more likely to be valid speech containing speech fragments. Conversely, a low short-term average zero-crossing rate indicates a low frequency of signal change, a relatively stable waveform, and fewer zero-crossings. This indicates that the signal contains a high number of low-frequency components or relatively stable audio content. Signals with a low short-term average zero-crossing rate may include non-speech fragments or noise.

[0135] In one possible implementation, the preset zero-crossing rate threshold is a threshold value pre-set in an embodiment of the present application for evaluating the short-time average zero-crossing rate. The setting of the preset zero-crossing rate threshold can be determined through experimentation and experience, and the threshold can be adjusted during training based on the training situation. The embodiment of the present application can adjust the preset zero-crossing rate threshold based on specific circumstances and needs. In different application scenarios, the evaluation of the short-time average zero-crossing rate may vary, and therefore the preset zero-crossing rate threshold may also vary.

[0136] In one possible implementation, when the short-time energy of a single frame is less than or equal to a first energy threshold and greater than a second energy threshold, it is necessary to obtain the short-time average zero-crossing rate of the enhanced speech frame and use the short-time average zero-crossing rate to determine the validity of the enhanced speech frame. When the short-time average zero-crossing rate is greater than a preset zero-crossing rate threshold, the enhanced speech frame is classified as a valid speech frame. Conversely, when the short-time average zero-crossing rate is less than or equal to the preset zero-crossing rate threshold, the enhanced speech frame is classified as not a valid speech frame.

[0137] It should be noted that the above method of first comparing the short-time energy of a single frame with the first energy threshold and the second energy threshold, and then comparing the short-time average zero-crossing rate with the preset zero-crossing rate threshold when the short-time energy of the single frame is between the first energy threshold and the second energy threshold, belongs to the dual-threshold method used in the embodiment of the present application. Through the dual-threshold method, the embodiment of the present application can effectively determine the classification result of the speech effectiveness of the enhanced speech frame.

[0138] In addition, the embodiment of the present application can also determine the classification result of the speech effectiveness of the enhanced speech frame by using the autocorrelation method, spectral entropy method, ratio method or logarithmic spectrum distance method, as follows:

[0139] In one possible implementation, a short-time autocorrelation method can be used to determine the classification result of the speech effectiveness of the enhanced speech frame. First, it is necessary to determine the autocorrelation function. The autocorrelation function has some properties, such as it is an even function. It is necessary to assume that the sequence is periodic, and its autocorrelation function is also a periodic function with the same period. Specifically, in the embodiment of the present application, the discrete sampling points of the speech signal waveform on the enhanced speech frame can be obtained first, and according to the preset number of delay points, they are input into the autocorrelation function to calculate the autocorrelation value of the enhanced speech frame. At the same time, in order to avoid the influence of absolute energy in the process, the current autocorrelation value needs to be normalized according to the autocorrelation value at the signal revelation moment.

[0140] Specifically, there are many ways to determine the classification results of the speech effectiveness of the enhanced speech frame through the autocorrelation function. For example, by calculating the autocorrelation function of the speech signal, the pitch period of the speech waveform sequence can be extracted. The pitch period is one of the important features in the speech signal and is very critical for tasks such as speech recognition and acoustic modeling. The pitch period of the speech waveform sequence refers to the time interval of the repetition of the pitch between consecutive speech frames. If the pitch period is successfully extracted, the embodiment of the present application can determine that there is periodicity of sound vibration in the enhanced speech signal, which indicates that the speech signal may contain valid speech information; or, because there is a difference in the autocorrelation function of the speech signal and the noise signal, this difference can be used to distinguish the speech signal from the noise signal, thereby effectively performing speech endpoint detection. When When the maximum value of the autocorrelation function, that is, the maximum autocorrelation value, exceeds a certain autocorrelation threshold, the enhanced speech frame can be determined to be a valid speech frame; alternatively, the autocorrelation function maximum value method can be used to determine the starting point and ending point of the speech signal. When the maximum value of the autocorrelation function is greater than or less than the set threshold, it can be determined as the endpoint of the speech signal. Since speech signals usually have higher energy and more spectral information, compared with noise signals, the starting point and ending point of the speech signal after endpoint detection will exhibit different characteristics. Based on the determined endpoint of the speech signal, the validity of the speech can be further determined, that is, the speech signal contains valid speech information during this period.

[0141] In one possible implementation, spectral entropy can be used to determine the speech effectiveness classification results for the enhanced speech frames. Entropy represents the degree of order in information. In information theory, entropy describes the uncertainty of the outcome of random events. Specifically, information entropy is used as a measure of information selectivity and uncertainty in the signal emitted by a source. The entropy of speech and noise differ significantly. Spectral entropy is a feature that offers some selectivity, reflecting the distribution probability of speech and noise across the entire signal segment.

[0142] Specifically, in the embodiments of the present application, the enhanced speech frames can be windowed and framed, and FFT transformed to obtain spectral information for each frame. The spectral entropy value of each frame is then calculated to obtain a spectral entropy sequence. Based on the statistical characteristics of the spectral entropy sequence, an appropriate threshold is determined as a criterion for judging the validity of the speech signal. The spectral entropy sequence is then binarized based on the threshold, with portions above the threshold marked as speech segments and portions below the threshold marked as non-speech segments. This allows the classification result of the speech validity of the enhanced speech frames to be determined.

[0143] In one possible implementation, the energy entropy ratio can also be used to determine the classification result of the speech effectiveness of the enhanced speech frame. The ratio is the energy entropy ratio. Spectral entropy is a statistical feature of the speech signal spectrum that can reflect the complexity of the speech signal and the uniformity of the spectral distribution. In the speech segment, the spectral entropy value is small because the spectral distribution of the human voice is relatively uniform and complex, while in the noise segment, the spectral entropy value is large because the spectral distribution of the noise is relatively uneven and simple. Therefore, by comparing the spectral entropy values, the speech segment and the noise segment can be better distinguished. In the embodiment of the present application, by calculating the energy entropy ratio, that is, the ratio of the energy entropy of the current frame to the energy entropy of the noise segment, the difference between the speech segment and the noise segment can be further highlighted. In the speech segment, the energy entropy is relatively high; while in the noise segment, the energy entropy is relatively low. Therefore, by setting a suitable energy entropy ratio threshold, the part with an energy entropy ratio higher than the threshold can be determined as a speech segment, thereby determining the effectiveness of the speech.

[0144] Specifically, in the embodiment of the present application, the spectral information of the enhanced speech frame can be calculated, and the spectral entropy value of the spectral information of each frame can be calculated to obtain a spectral entropy sequence. The spectral entropy values ​​of the speech segment and the noise segment can then be calculated, which can be obtained by statistics of a training sample set or experience. Then, the energy entropy ratio of the enhanced speech frame is calculated, and an appropriate threshold is set. The portion above the threshold is marked as a speech segment, and the portion below the threshold is marked as a non-speech segment, thereby obtaining a classification result of the speech effectiveness of the enhanced speech frame.

[0145] In one possible implementation, the log spectrum distance can also be used to determine the classification result of the speech effectiveness of the enhanced speech frame. In the embodiment of the present application, the speech quality of each frame can be measured by performing an FFT transform on the speech signal of each frame and calculating its log spectrum distance, thereby determining the effectiveness of the signal. The log spectrum distance determines the effectiveness of the signal by comparing the distance between the log spectrum of each frame of the speech signal and the average noise spectrum. In the speech frame, the energy of the speech signal will be expressed as a higher log spectrum value, while in the noise frame, the energy of the noise signal is relatively low and the log spectrum value is small. Therefore, by calculating the log spectrum distance, the speech frame and the noise frame can be better distinguished. The log spectrum distance utilizes the statistical characteristics of the speech signal in the frequency domain. The speech signal has a certain spectral distribution pattern, while the spectrum of the noise signal is relatively chaotic. By calculating the distance between the log spectra, the difference between the speech and the noise can be quantified, thereby determining the effectiveness of the speech.

[0146] In addition, the average noise spectrum can be used as a reference in the calculation of the logarithmic spectral distance. By using the average noise spectrum, the impact of noise on speech recognition can be suppressed, making the logarithmic spectral distance more accurately reflect the quality of speech. Since the characteristics of noise are uneven and relatively simple spectral distribution, by comparing with the average noise spectrum, the characteristics of the speech signal can be better extracted.

[0147] Specifically, in the embodiment of the present application, the spectral information of the enhanced speech frame can be calculated, the modulus value of the spectrum is taken, and then the logarithm is taken to obtain the logarithmic spectrum. Then, the logarithmic spectrum distance between the logarithmic spectrum of each frame and the average noise spectrum is calculated, and based on experience or experimental results, a suitable threshold is set to distinguish the speech frame from the noise frame. The part above the threshold is marked as a speech frame, and the part below the threshold is marked as a non-speech segment, thereby obtaining a classification result of the speech effectiveness of the enhanced speech frame.

[0148] It can be understood that compared with the aforementioned methods of determining the classification results of the speech effectiveness of the enhanced speech frames, the classification results of the speech effectiveness of the enhanced speech frames based on the time domain energy parameters can be determined more intuitively, thereby effectively improving the efficiency of determining the classification results of the speech effectiveness.

[0149] In one possible implementation, the speech classification accuracy of the speech enhancement network can be determined in a variety of ways. The following describes the process of determining the speech classification accuracy of the speech enhancement network in step 204:

[0150] In one possible implementation, the speech classification accuracy of the speech enhancement network can be calculated using the validity distribution labels. Specifically, the method includes obtaining a validity distribution label indicating the speech validity of each frame in the sample clean speech; and determining the speech classification accuracy of the speech enhancement network based on the similarity between the validity distribution and the validity distribution label.

[0151] In one possible implementation, the validity distribution label is a label for indicating the speech validity of each frame in the sample clean speech. It can indicate whether each speech frame in the sample clean speech contains a speech signal (with validity) or a non-speech signal (without validity). The speech enhancement network can measure the speech enhancement effect according to the validity distribution label during training and calculate the speech classification loss. By using it in combination with the validity distribution, the validity distribution label can help the speech enhancement network learn to correctly classify the speech and non-speech parts in noisy speech to reduce noise residue. The speech classification loss is determined based on the similarity between the validity distribution and the validity distribution label. In the embodiment of the present application, the speech classification loss can be calculated in a variety of ways, and by minimizing the speech classification loss, the speech enhancement network can better suppress the noise of the non-speech segment and improve the quality of speech enhancement, and enable the speech enhancement network to more accurately judge the speech and non-speech parts, thereby generating a clearer and more natural target enhanced speech.

[0152] In one possible implementation, the similarity between the validity distribution and the validity distribution label can be measured by calculating the mean absolute error, mean squared error, and binary cross entropy, thereby determining the speech classification loss of the speech enhancement network. Specifically, the binary cross entropy between the validity distribution and the validity label feature, the mean absolute error between the validity distribution and the validity label feature, and the mean squared error between the validity distribution and the validity label feature can be determined; the binary cross entropy, mean absolute error, and mean squared error can be weighted to obtain the speech classification loss of the speech enhancement network.

[0153] In one possible implementation, the validity label feature is obtained based on the validity distribution label, and the validity label feature corresponds to the validity distribution. It should be noted that since the sample contains a validity distribution label used to characterize the validity of each speech frame in the sample clean speech, the validity distribution labels of each frame of the sample clean speech are combined to form a validity label feature. The validity distribution is generated based on the classification results of each enhanced speech frame. In other words, the classification results of each enhanced speech frame indicate the classification status of each frame. Combining the classification status of each sample enhanced speech frame can form a validity distribution.

[0154] In one possible implementation, the mean absolute error (MABS) between the validity distribution and the validity label feature is also known as the L1 loss, denoted as L1 v , the required mean absolute error can be obtained by calculating the average of the absolute differences between the validity distribution and the validity label features.

[0155] In one possible implementation, the mean squared error (MSE) between the validity distribution and the validity label feature is also known as the L2 loss, denoted as L2 v , we can get the required mean square error by calculating the square of the difference between the two validity distributions and the validity label features, and then averaging the squares of the differences.

[0156] In one possible implementation, the binary cross entropy (BCE) loss between the validity distribution and the validity label feature is denoted as L BCE , we can calculate the cross entropy between the validity distribution and the validity label feature. The binary cross entropy L in the embodiment of the present application BCE It can be obtained by the following formula:

[0157] L BCE = -[p×logq+(1-p)×log(1-q)]

[0158] Among them, p is the validity label feature V s The classification results in , q is the validity distribution In the classification result, when the indicated frame signal is valid, p and q are both 1, otherwise they are 0.

[0159] Finally, the speech classification loss can be obtained in the embodiment of the present application: As shown in the following formula:

[0160] Among them, K1 is the coefficient of L1 loss, K2 is the coefficient of L2 loss, and K3 is the coefficient of BCE loss.

[0161] In this implementation, the speech classification accuracy of the speech enhancement network is calculated using validity labels that indicate the speech validity of each frame in the sample clean speech. This can help the speech enhancement network learn to correctly classify the speech and non-speech parts in noisy speech, thereby reducing noise residue and improving the quality of speech enhancement. This enables the speech enhancement network to more accurately judge the speech and non-speech parts, thereby generating clearer and more natural target enhanced speech.

[0162] In one possible implementation, the speech classification accuracy of the speech enhancement network can also be determined using a contrastive loss approach. Specifically, the method includes determining the distribution similarity between the validity distributions corresponding to two corresponding enhanced speech samples, where the two corresponding enhanced speech samples are obtained by performing noise reduction by the speech enhancement network on two noisy speech samples included in a speech pair, where the two noisy speech samples included in the speech pair are generated based on the same clean speech sample; and determining the speech classification accuracy of the speech enhancement network based on the distribution similarity.

[0163] In one possible implementation, distribution similarity is an indicator used to measure the degree of similarity between two validity distributions. A sample speech pair is two noisy speech samples obtained by mixing the same sample clean speech with different sample noisy speech samples. When determining the speech classification loss used to characterize the speech classification accuracy of a speech enhancement network, the distribution similarity between the validity distributions of the two sample enhanced speech samples obtained by denoising the two noisy speech samples in the sample speech pair can be calculated. For example, the distribution similarity can be calculated using a measurement method such as cosine similarity or Euclidean distance. By comparing the validity distributions of the two sample enhanced speech samples, a distribution similarity value can be obtained to measure whether the validity of the two sample enhanced speech samples is similar. This value can be used to determine the contrast loss of the speech enhancement network and used as the speech classification loss. The speech enhancement network is trained by minimizing this loss to improve the noise reduction effect and speech intelligibility.

[0164] In one possible implementation, in the process of determining the speech classification loss of the speech enhancement network using the contrast loss approach, it is necessary to first configure two sample noisy speech samples obtained by mixing the same sample clean speech into a sample speech pair. Subsequently, the speech enhancement network is used to perform noise reduction processing on the two sample noisy speech samples of the sample speech pair to obtain two sample enhanced speech samples. Each sample enhanced speech is then framed and the speech effectiveness of each enhanced speech frame is classified. Based on the classification results, an effectiveness distribution of each sample enhanced speech is generated. For the effectiveness distributions of the two sample enhanced speech samples, the distribution similarity between them is calculated. The distribution similarity can be calculated using cosine similarity, Euclidean distance, or other suitable similarity measurement methods. Finally, the speech classification loss of the speech enhancement network is determined based on the distribution similarity. Generally, the higher the distribution similarity, the more similar the effectiveness of the two sample enhanced speech samples is, indicating that the noise reduction effect of the speech enhancement network is better. Therefore, the distribution similarity can be used as part of the speech classification loss.

[0165] In this implementation, using a contrastive loss approach, the distribution similarity between the effectiveness distributions of two corresponding enhanced speech samples is calculated, and the speech classification accuracy of the speech enhancement network is determined based on this distribution similarity. Training the speech enhancement network based on this speech classification accuracy helps it learn to accurately distinguish between speech and non-speech components, thereby improving speech enhancement effectiveness.

[0166] In one possible implementation, the noise reduction accuracy of the speech enhancement network can be determined in a variety of ways. The following describes the process of determining the noise reduction accuracy of the speech enhancement network in step 204:

[0167] In one possible implementation, the noise reduction accuracy of the speech enhancement network can be determined by calculating the scale-invariant signal-to-noise ratio, mean absolute error, and mean square error. Specifically, the scale-invariant signal-to-noise ratio between the enhanced speech sample and the clean speech sample, the mean absolute error between the enhanced speech sample and the clean speech sample, and the mean square error between the enhanced speech sample and the clean speech sample can be determined; the scale-invariant signal-to-noise ratio, mean absolute error, and mean square error can be weighted to obtain the noise reduction accuracy of the speech enhancement network.

[0168] In one possible implementation, the mean absolute error (MABS) between the sample enhanced speech and the sample pure speech is also known as the L1 loss, denoted as L1 t , the required mean absolute error can be obtained by calculating the average of the absolute differences between the sample enhanced speech and the sample pure speech.

[0169] In one possible implementation, the mean squared error (MSE) between the enhanced speech sample and the pure speech sample is also known as the L2 loss, denoted as L2 t , we can calculate the square of the difference between the two samples of enhanced speech and the sample of pure speech, and then average the square of the difference to get the required mean square error.

[0170] In one possible implementation, the Scale-Invariant Signal-to-Noise Ratio (SI-SNR) loss between the sample enhanced speech and the sample clean speech is denoted as L SI-SNR , which can be used to compare the amplitude ratio of the sample enhanced speech signal and the sample clean speech signal to measure the quality of the enhancement effect. For example, since the sample enhanced speech is generated by the model, its overall energy is not necessarily the same as that of the sample clean speech. Therefore, it is necessary to calculate a scale coefficient to adjust the volume of the sample enhanced speech so that it has similar energy to the sample clean speech. Then, the sample enhanced speech is multiplied by the scale coefficient to obtain the estimated source signal. The error of the estimated signal is obtained by calculating the difference between the estimated source signal and the sample clean speech, and the power of the sample clean speech and the power of the sample enhanced speech are calculated. Finally, the scale-invariant signal-to-noise ratio L in the embodiment of the present application is calculated. SI-SNR It can be obtained by the following formula:

[0171] L SI-SNR =-10×log 10(power_s / power_e)

[0172] Among them, power_s is the sample pure speech s n The power of power_e is the sample enhanced speech power.

[0173] Finally, the noise reduction loss used to reflect the noise reduction accuracy can be obtained in the embodiment of the present application. As shown in the following formula:

[0174] Among them, K3 is the coefficient of L1 loss, K4 is the coefficient of L2 loss, and K5 is the coefficient of SI-SNR loss.

[0175] The following describes the process of obtaining the speech enhancement accuracy:

[0176] In the embodiment of the present application, a speech enhancement model is trained based on a deep learning speech enhancement and noise reduction solution. A noisy speech signal is generated by mixing a clean speech data set and a noise data set. The signal-to-noise ratio of noisy speech in different noise environments is simulated by controlling the noise mixing ratio. Assuming that the sample clean speech signal is s n , the sample noise signal is d n , the corresponding short-time cosine transforms are S k and D k , where the sample enhanced speech is The target frequency domain feature is Then we have:

[0177] The sample noisy speech signal is: x n =s n +d n

[0178] The short-time cosine is expressed as: X k =S k +D k

[0179] The ideal mask output by the network model is:

[0180] Function used to reflect the overall loss of speech enhancement accuracy:

[0181] Finally, the overall loss obtained is L. Training the speech enhancement network based on the overall loss can focus on improving the noise suppression ability of the speech enhancement network for non-speech segments. When denoising the processed speech based on the trained speech enhancement network, for the processed speech containing non-speech segments, the trained speech enhancement network can effectively reduce the phenomenon of residual noise, thereby improving the quality of speech enhancement.

[0182] In one possible implementation, the sample enhanced speech and other clean speech can be configured as a first discriminant speech pair and input into a first discriminator. The noise reduction accuracy of the speech enhancement network is determined based on the scoring result output by the first discriminator. Specifically, the following steps can be performed: obtaining clean speech other than the sample clean speech, configuring the sample enhanced speech and other clean speech as a first discriminant speech pair, and inputting them into the first discriminator; performing a truthfulness score on the first discriminant speech pair based on the first discriminator to obtain a first scoring result; determining a first adversarial loss based on the first scoring result, and determining the noise reduction accuracy of the speech enhancement network based on the first adversarial loss.

[0183] The purpose of acquiring clean speech other than the sampled clean speech is to increase the diversity and generalization of the training data. This clean speech can be the remaining clean speech from the same batch of sampled clean speech. Including this clean speech allows the speech enhancement network to better learn different types of speech features and noise reduction patterns, improving its adaptability in real-world scenarios.

[0184] Specifically, the speech enhancement network serves as a generator, aiming to generate speech output similar to the sampled clean speech. The first discriminator, part of a generative adversarial network (GAN), serves as a discriminator, and its goal is to discriminate the authenticity of the input first discriminant speech pair. During training, backpropagation is used to optimize the weights of the generator, making the generated speech closer to the clean speech, thereby deceiving the first discriminator. Therefore, the denoising loss used to reflect the denoising accuracy of the speech enhancement network is determined based on the first adversarial loss. This allows the generator's adversarial and transformation capabilities to be taken into account during training, resulting in more realistic and high-quality generated speech.

[0185] The first discriminant speech pair refers to a speech pair consisting of the sample-enhanced speech and other clean speech. This first discriminant speech pair is subsequently input into the first discriminator for authenticity scoring. The first discriminant speech pair is used to evaluate the authenticity of the speech output by the generator. By pairing the sample-enhanced speech with other clean speech and inputting it into the first discriminator for scoring, the difference between the speech output by the generator and the clean speech can be determined.

[0186] The authenticity score refers to the score performed by the first discriminator on the first discriminant speech pair, which is used to measure the authenticity of the speech in the input speech pair, that is, to measure whether the two speech sounds in the input speech pair are the same speech. The higher the score, the more similar the two speech sounds in the speech pair are, and the more likely they are to be the same speech. Based on the first score, a first adversarial loss can be calculated. In the embodiment of the present application, the least squares loss or cross-entropy loss can be used as the first adversarial loss to measure the difference between the generated sample-enhanced speech and other pure speech sounds.

[0187] In one possible implementation, there are multiple ways to determine the denoising loss of the speech enhancement network based on the first adversarial loss. For example, the denoising loss can be calculated in a weighted manner together with other loss values, and the speech enhancement network serving as the generator is optimized through adversarial training during training. For example, the weights of the generator are updated by maximizing the first adversarial loss between the generator output speech and other clean speech, including defining the denoising loss of the generator as maximizing the score of the first discriminator on the generated speech based on the first scoring result, so that the generator can learn to generate output that is closer to real speech; or, the denoising loss is further determined by calculating the gradient information of the generator output speech with respect to the first discriminator. First, the gradient of the generated speech with respect to the first discriminator score needs to be calculated, and this gradient is used as the target denoising loss of the generator to encourage the generator to generate output that is closer to real speech; or, a feature matching loss is introduced between the generator output and the real speech, and the denoising loss is determined by comparing the similarity of speech features. Specifically, the feature matching loss is compared with the statistical characteristics of the generator output and the real speech in certain feature spaces, such as spectral shape, speech rate, etc., and the feature matching loss is used as the denoising loss of the generator.

[0188] Among them, the embodiment of the present application determines the noise reduction accuracy of the speech enhancement network based on the first adversarial loss, and can simultaneously take into account the adversarial and conversion capabilities of the generator during the training process, thereby making the speech generated by the speech enhancement network more realistic and high-quality.

[0189] In one possible implementation, a second discriminator can be added to further discriminate noisy speech, thereby determining the noise reduction accuracy of the speech enhancement network. Specifically, the following steps can be performed: based on the sample enhanced speech, a reference noisy speech is separated from the sample noisy speech; the reference noisy speech and the sample noisy speech are configured as a second discriminant speech pair and input into the second discriminator; the second discriminant speech pair is scored for authenticity based on the second discriminator to obtain a second scoring result; a second adversarial loss is determined based on the second scoring result, and the noise reduction accuracy of the speech enhancement network is determined based on the first and second adversarial losses.

[0190] Among them, the purpose of introducing the second discriminator is to enhance the performance of the speech enhancement network and improve the ability to separate noise. By using the second discriminator to score the authenticity of the second discriminant speech pair, the learning process of the speech enhancement network can be guided, so that it can better restore pure speech and reduce noise components.

[0191] Specifically, the speech enhancement network serves as a generator, and its goal is to generate speech output similar to the sample clean speech. The second discriminator serves as a discriminator. The second discriminator is a model used to evaluate the degree of similarity between the generated speech and the sample noisy speech. It scores the authenticity of the generated speech based on the input second discriminant speech pair (including the reference noisy speech and the sample noisy speech). By scoring the second discriminant speech pair, the second discriminator can help determine the quality of the generated speech.

[0192] The second discriminant speech pair refers to a pair of speech used in the speech enhancement method, consisting of a reference noisy speech and a sample noisy speech. The reference noisy speech is the noise portion separated from the sample enhanced speech, while the sample noisy speech is the noisy speech used to mix with the sample clean speech to produce the sample noisy speech. The second discriminant speech pair is scored by passing the reference noisy speech and the sample noisy speech as input to a second discriminator. The second discriminator assesses the realism of the generated speech based on the similarity between the two speech samples. By comparing the second scoring results, the degree of similarity between the generated speech and the noisy speech can be determined, and then the second adversarial loss is calculated and determined, guiding the training process of the speech enhancement network.

[0193] Among them, the authenticity score refers to the second discriminator scoring the second discriminant speech pair, which is used to measure the authenticity of the speech in the input speech pair, that is, to measure whether the two voices in the input speech pair are the same speech, which will not be repeated here.

[0194] In one possible implementation, there are multiple ways to determine the noise reduction accuracy of the speech enhancement network based on the first adversarial loss and the second adversarial loss. For example, the first adversarial loss and the second adversarial loss can be weighted together with other loss values ​​to obtain a noise reduction loss that reflects the noise reduction accuracy, and the speech enhancement network used as a generator can be optimized through adversarial training during the training process. I will not go into details here.

[0195] In the above implementation, by introducing the second discriminator, the degree of proximity between the reference noise speech and the sample noise speech in the generated sample noisy speech can be determined, and then the second adversarial loss can be determined. The second adversarial loss is used to guide the training process of the speech enhancement network, so that the speech enhancement network can better restore the pure speech and reduce the noise component.

[0196] The following describes the process of determining the speech classification loss and noise reduction loss in different ways in the embodiments of the present application, as well as the process of obtaining the overall loss, with reference to the accompanying drawings.

[0197] Referring to Figure 6, Figure 6 is a schematic diagram of an optional process for obtaining the overall loss provided in an embodiment of the present application. In this example, the calculation of the speech classification loss needs to be performed in combination with the validity distribution and the validity distribution label. Specifically, after the sample clean speech and the sample noisy speech are mixed, the sample noisy speech is obtained. After the sample noisy speech is input into the speech enhancement network for processing, the sample enhanced speech is obtained. The sample enhanced speech is framed to obtain n enhanced speech frames including enhanced speech frame 1, enhanced speech frame 2 to enhanced speech frame n, etc., where n is greater than 1. The speech validity of the n enhanced speech frames is classified to obtain the validity distribution. Each frame of the sample clean speech contains m validity distribution labels including validity distribution label 1, validity distribution label 2 to validity distribution label m, etc., where m is greater than 1. The validity label feature is obtained by combining the m validity distribution labels. Finally, the speech classification loss is calculated based on the validity classification feature and the validity label feature. The specific calculation process will not be repeated here.

[0198] In this embodiment, the noise reduction loss needs to be calculated based on the clean and enhanced speech samples. Specifically, after obtaining the enhanced speech samples, the scale-invariant signal-to-noise ratio, mean absolute error, and mean squared error (MSE) during the training process need to be calculated for the clean and enhanced speech samples, respectively. The noise reduction loss is then calculated based on the scale-invariant signal-to-noise ratio, mean absolute error, and MSE. The specific calculation process is not detailed here.

[0199] Finally, after obtaining the noise reduction loss and speech classification loss, the overall loss can be weighted and the parameters of the speech enhancement network can be adjusted according to the overall loss.

[0200] Among them, by determining the noise reduction loss and speech classification loss separately in a supervised manner, the accuracy and reliability of the noise reduction loss and speech classification loss can be improved respectively, thereby improving the accuracy and reliability of the overall loss as a whole and improving the training effect of the speech enhancement network.

[0201] In addition, refer to Figure 7, which is a schematic diagram of another optional process for obtaining the overall loss provided in an embodiment of the present application. In this example, the calculation of the speech classification loss needs to be carried out in combination with the validity distribution and the validity distribution label. Specifically, after the sample clean speech and the sample noisy speech are mixed, the sample noisy speech is obtained. After the sample noisy speech is input into the speech enhancement network for processing, the sample enhanced speech is obtained. The sample enhanced speech is framed to obtain n enhanced speech frames including enhanced speech frame 1, enhanced speech frame 2 to enhanced speech frame n, etc., where n is greater than 1. The validity of the n enhanced speech frames is classified to obtain the validity distribution. Each frame of the sample clean speech contains m validity distribution labels including validity distribution label 1, validity distribution label 2 to validity distribution label m, etc., where m is greater than 1. The validity label feature is obtained by combining the m validity distribution labels. Finally, the speech classification loss is calculated based on the validity classification feature and the validity label feature. The specific calculation process will not be repeated here.

[0202] In this embodiment, the noise reduction loss needs to be calculated based on other clean speech samples and the sample enhanced speech. Specifically, after obtaining the other clean speech samples, the other clean speech samples and the sample enhanced speech are combined to form a first discriminant speech pair. This first discriminant speech pair is input into the first discriminator for realism scoring, obtaining a first scoring result. Based on the first scoring result, a first adversarial loss is calculated. Finally, the noise reduction loss is determined based on the first adversarial loss. The specific calculation process is not repeated here.

[0203] Finally, after obtaining the noise reduction loss and speech classification loss, the overall loss can be weighted and the parameters of the speech enhancement network can be adjusted according to the overall loss.

[0204] Among them, determining the noise reduction loss in an unsupervised manner can improve the efficiency of determining the noise reduction loss, while determining the speech classification loss in a supervised manner can improve the accuracy and reliability of the speech classification loss, thereby taking into account the efficiency, accuracy and reliability of determining the overall loss as a whole, and improving the training effect of the speech enhancement network.

[0205] In addition, refer to Figure 8, which is another optional process diagram for obtaining the overall loss provided by an embodiment of the present application. In this example, the calculation of the speech classification loss needs to be based on comparative learning of different sample noisy speeches. Specifically, the same sample pure speech needs to be mixed with two different sample noisy speeches, for example, sample noisy speech 1 is mixed to obtain sample noisy speech 1, and sample noisy speech 2 is mixed to obtain sample noisy speech 2. The two obtained sample noisy speeches are configured as sample speech pairs and are respectively input into the speech enhancement network, and sample enhanced speech 1 and sample enhanced speech 2 are output respectively. Subsequently, the distribution similarity between the validity distributions corresponding to the two sample enhanced speeches is determined, and the speech classification loss of the speech enhancement network is determined based on the distribution similarity. The specific calculation process will not be repeated here.

[0206] In this embodiment, the noise reduction loss needs to be calculated based on the clean and enhanced speech samples. Specifically, after obtaining the enhanced speech samples, the scale-invariant signal-to-noise ratio, mean absolute error, and mean squared error (MSE) during the training process need to be calculated for the clean and enhanced speech samples, respectively. The noise reduction loss is then calculated based on the scale-invariant signal-to-noise ratio, mean absolute error, and MSE. The specific calculation process is not detailed here.

[0207] Finally, after obtaining the noise reduction loss and speech classification loss, the overall loss can be weighted and the parameters of the speech enhancement network can be adjusted according to the overall loss.

[0208] Among them, determining the noise reduction loss in a supervised manner can improve the accuracy and reliability of the noise reduction loss, while determining the speech classification loss in an unsupervised manner can improve the efficiency of determining the speech classification loss, thereby taking into account the efficiency, accuracy and reliability of determining the overall loss as a whole, and improving the training effect of the speech enhancement network.

[0209] In addition, refer to Figure 9, which is a schematic diagram of another optional process for obtaining the overall loss provided by an embodiment of the present application. In this example, the calculation of the speech classification loss needs to be obtained based on comparative learning of different sample noisy speeches. Specifically, the same sample pure speech needs to be mixed with two different sample noisy speeches, for example, sample noisy speech 1 is mixed to obtain sample noisy speech 1, and sample noisy speech 2 is mixed to obtain sample noisy speech 2. The two obtained sample noisy speeches are configured as sample speech pairs and are respectively input into the speech enhancement network, and sample enhanced speech 1 and sample enhanced speech 2 are output respectively. Subsequently, the distribution similarity between the validity distributions corresponding to the two sample enhanced speeches is determined, and the speech classification accuracy of the speech enhancement network is determined based on the distribution similarity. The specific calculation process will not be repeated here.

[0210] In this embodiment, it is necessary to calculate the noise reduction loss based on other clean speech and sample enhanced speech other than the sample clean speech. Specifically, after obtaining the other clean speech other than the sample clean speech, the other clean speech and the sample enhanced speech are combined to form a first discriminant speech pair. Specifically, the first discriminant speech pair can also be formed based on sample enhanced speech 1 and sample enhanced speech 2, and the first discriminant speech pair is input into the first discriminator for authenticity scoring to obtain a first scoring result. The first adversarial loss is calculated based on the first scoring result, and finally the noise reduction loss is determined based on the first adversarial loss. The specific calculation process is not repeated here.

[0211] Finally, after obtaining the noise reduction loss and speech classification loss, the overall loss can be weighted and the parameters of the speech enhancement network can be adjusted according to the overall loss.

[0212] Among them, determining the noise reduction loss and speech classification loss separately in an unsupervised manner can improve the efficiency of determining the noise reduction loss and speech classification accuracy respectively, thereby improving the efficiency of determining the overall loss as a whole and improving the training effect of the speech enhancement network.

[0213] In addition, refer to Figure 10, which is a schematic diagram of an optional process for obtaining noise reduction loss provided in an embodiment of the present application. In this example, the noise reduction loss can also be calculated in combination with the first adversarial loss and the second adversarial loss. In this embodiment, it is necessary to calculate the first adversarial loss based on other clean voices other than the sample clean voice and the sample enhanced voice. Specifically, after obtaining other clean voices other than the sample clean voice, the other clean voices and the sample enhanced voice are combined to form a first discriminant voice pair. Specifically, the first discriminant voice pair can also be formed according to the sample enhanced voice 1 and the sample enhanced voice 2, and the first discriminant voice pair is input into the first discriminator for authenticity scoring to obtain a first scoring result, and the first adversarial loss is calculated based on the first scoring result. The specific calculation process will not be repeated here.

[0214] In this embodiment, the second adversarial loss is calculated based on the reference noisy speech and the sample noisy speech obtained by separating the sample noisy speech. Specifically, the reference noisy speech is first separated from the sample noisy speech based on the sample enhanced speech. The reference noisy speech and the sample noisy speech are then configured as a second discriminant speech pair. The second discriminant speech pair is input into the second discriminator for realism scoring, resulting in a second scoring result. The second adversarial loss is then calculated based on the second scoring result. The specific calculation process is not further described here.

[0215] Finally, after obtaining the first adversarial loss and the second adversarial loss, the denoising loss can be weighted and the overall loss can be obtained based on the newly determined denoising loss and the speech classification loss to adjust the parameters of the speech enhancement network.

[0216] It should be noted that the above-mentioned speech classification loss and noise reduction loss can not only be obtained according to the above method, but can also be obtained by combining losses such as L1 loss, L2 loss, BCE loss and SI-SNR loss, and selecting appropriate loss weights to obtain speech classification loss and noise reduction loss, which will not be repeated here.

[0217] Among them, by combining the first adversarial loss and the second adversarial loss to obtain the noise reduction loss, the accuracy and reliability of the noise reduction loss can be effectively improved, thereby further improving the accuracy and reliability of the overall loss.

[0218] Through the above-mentioned speech enhancement method, the embodiment of the present application can be tested before application. This embodiment performs speech noise separation on a self-built test data set and generates a batch of test data according to the signal-to-noise ratio range of [-10, 30] dB, with a step of 2 dB, for a total of 1000 test data sets. The objective speech quality evaluation (PESQ), the scale-invariant signal-to-noise ratio parameter (SI-SNR), and the objective auditory mean opinion score (MOS_OVL) that simulates the subjective audio quality perception parameter are selected as effect evaluation indicators. Among them, PESQ is used to measure the difference between the speech quality after sound processing or transmission and the original pure speech, SI-SNR is used to measure the ratio of clean signal to noise between the separated speech signal and the original speech signal, and MOS_OVL is used to compare the difference between the original speech signal and the processed speech signal, giving an evaluation score for measuring the quality of speech.

[0219] Exemplarily, the corresponding results are shown in Figures 11, 12 and 13, where Figure 11 is a schematic diagram of the PESQ score results of the test process provided in an embodiment of the present application, Figure 12 is a schematic diagram of the SI-SNR score results of the test process provided in an embodiment of the present application, and Figure 13 is a schematic diagram of the MOS_OVL score results of the test process provided in an embodiment of the present application. In Figures 11 to 13, the signal-to-noise ratio (SNR) is an indicator used to measure the relative strength between the signal and the noise, and is measured in decibels (dB). The signal-to-noise ratio / decibel (SNR / dB) is used to measure the relative strength or quality difference between the speech signal and the noise. A higher SNR value indicates less noise interference, while a lower SNR value indicates more noise interference. In the figure, noisy speech (noisy) represents the original speech signal without noise reduction processing. This set of data is usually used as a benchmark to compare the effects of other processing methods; other processed speech, that is, speech signal without Voice Activity Detection (w / o VAD), represents the speech signal that has not been processed by the speech enhancement method in the embodiment of the present application during the noise reduction process; the target enhanced speech can be expressed as Proposed, which represents the speech signal obtained after noise reduction processing using the speech enhancement method. It can be seen that the embodiment of the present application has higher scores than other data in terms of PESQ score results, SI-SNR score results and MOS_OVL score results. Therefore, the embodiment of the present application can effectively suppress noise and improve speech quality, and after adding the VAD auxiliary loss function, the noise reduction effect of the model is significantly improved.

[0220] It will be appreciated that, although the various steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless clearly stated in the present embodiment, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0221] Referring to Figure 14, Figure 14 is an optional flow chart of the voice enhancement method provided in an embodiment of the present application. The voice enhancement method can be executed by an electronic device, for example, it can be executed by the server 102 in Figure 1 above. The voice enhancement method includes but is not limited to the following steps 1401 to 1402.

[0222] Step 1401: Acquire the speech to be processed.

[0223] In one possible implementation, the processed speech refers to the original speech signal that requires speech enhancement processing. It may contain noise, distortion, or other interference. Similar to the noisy speech samples used during training, the processed speech typically also contains some noise and represents real-world speech. For example, the processed speech can be real-world speech captured during phone calls, video conferencing, speech recognition front-ends, live streaming and on-demand applications, and other scenarios.

[0224] Step 1402: De-noise the speech to be processed based on the trained speech enhancement network to obtain corresponding enhanced speech; the speech enhancement network is trained using the speech enhancement network training method provided in an embodiment of the present application.

[0225] In one possible implementation, the enhanced speech is the enhanced speech signal obtained after processing by the speech enhancement network. The noise of the processed speech is reduced based on the trained speech enhancement network, and the output obtained is the enhanced speech. Through the processing of the speech enhancement network, the noise in the processed speech can be effectively reduced, the speech quality can be improved, and the residual noise in the non-speech segment can be reduced, thereby enhancing the clarity and intelligibility of the speech.

[0226] It should be noted that the speech enhancement network is trained using the speech enhancement network training method introduced above.

[0227] This embodiment of the present application divides the sample enhanced speech into multiple enhanced speech frames, classifies the speech effectiveness of each enhanced speech frame, and generates an effectiveness distribution for the sample enhanced speech based on the classification results of each enhanced speech frame. Because the effectiveness distribution can indicate whether each enhanced speech frame generated by the speech enhancement network is a non-speech segment, the speech classification accuracy of the speech enhancement network, determined by the effectiveness distribution, can be used to measure the recognition accuracy of the currently trained speech enhancement network for speech segments (i.e., speech signals containing valid human voices) and non-speech segments (i.e., speech signals not containing valid human voices, such as silence and noise). Based on this, the speech enhancement accuracy is determined based on the noise reduction accuracy and speech classification accuracy. Training the speech enhancement network based on the speech enhancement accuracy can improve the speech enhancement network's ability to recognize speech and non-speech segments, thereby focusing on improving its noise suppression capabilities for non-speech segments. When noise reduction is performed on the processed speech based on the trained speech enhancement network, the trained speech enhancement network can effectively reduce the residual noise in the non-speech segments of the processed speech, thereby improving the quality of speech enhancement.

[0228] It will be appreciated that, although the various steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless clearly stated in the present embodiment, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0229] 15 , which is a schematic diagram of an optional structure of a training device for a speech enhancement network according to an embodiment of the present application, the speech enhancement device 1500 includes:

[0230] The first sample speech mixing module 1501 is used to obtain a sample clean speech and a sample noisy speech, and mix the sample clean speech and the sample noisy speech into a sample noisy speech;

[0231] A first sample speech enhancement module 1502 is configured to reduce noise on the sample noisy speech based on a speech enhancement network to obtain a sample enhanced speech;

[0232] A first effectiveness classification module 1503 is configured to divide the sample enhanced speech into multiple enhanced speech frames, classify the speech effectiveness of each enhanced speech frame, and generate an effectiveness distribution of the sample enhanced speech based on the classification results of each enhanced speech frame;

[0233] The first network training module 1504 is used to determine the noise reduction accuracy of the speech enhancement network based on the sample enhanced speech and the sample clean speech, determine the speech classification accuracy of the speech enhancement network based on the effectiveness distribution, determine the speech enhancement accuracy of the speech enhancement network based on the noise reduction accuracy and the speech classification accuracy, and train the speech enhancement network based on the speech enhancement accuracy.

[0234] Furthermore, the first network training module 1504 is specifically configured to:

[0235] Obtaining a validity distribution label for indicating the speech validity of each frame in the sample clean speech;

[0236] The speech classification accuracy of the speech enhancement network is determined based on the validity distribution and the similarity between the validity distribution labels.

[0237] Furthermore, two noisy speech samples obtained by mixing the same sample clean speech are configured as a sample speech pair, and the first network training module 1504 is further used to:

[0238] Determine the distribution similarity between the validity distributions corresponding to two corresponding sample enhanced speech sounds; the two corresponding sample enhanced speech sounds are obtained by respectively performing denoising on two sample noisy speech sounds included in a sample speech pair by a speech enhancement network, and the two sample noisy speech sounds included in the sample speech pair are generated based on the same sample clean speech; determine the speech classification accuracy of the speech enhancement network based on the distribution similarity.

[0239] Furthermore, the first validity classification module 1503 is specifically configured to:

[0240] Determining a time domain energy parameter of each enhanced speech frame, wherein the time domain energy parameter is used to indicate the speech energy of the enhanced speech frame in the time domain;

[0241] The speech effectiveness of each enhanced speech frame is classified according to the time domain energy parameter and the preset energy threshold.

[0242] Furthermore, the time domain energy parameter includes the single-frame average energy. The first validity classification module 1503 is further configured to:

[0243] Determine the comprehensive average energy of the sample clean speech, and weight the comprehensive average energy according to a preset energy threshold to obtain a weighted average energy;

[0244] When the single-frame average energy is greater than the weighted average energy, the classification result of the speech validity of the enhanced speech frame is determined to be that the enhanced speech frame is a valid speech frame; or, when the single-frame average energy is less than or equal to the weighted average energy, the classification result of the speech validity of the enhanced speech frame is determined to be that the enhanced speech frame is not a valid speech frame.

[0245] Furthermore, the time domain energy parameter includes a single-frame short-time energy, the number of preset energy thresholds is multiple, the multiple preset energy thresholds include a first energy threshold and a second energy threshold, the first energy threshold is greater than the second energy threshold, and the first validity classification module 1503 is further used to:

[0246] When the single-frame short-time energy is greater than the first energy threshold, determining that the classification result of the speech validity of the enhanced speech frame is that the enhanced speech frame is a valid speech frame;

[0247] Alternatively, when the short-time energy of the single frame is less than or equal to the first energy threshold and greater than the second energy threshold, obtaining the short-time average zero-crossing rate of the enhanced speech frame; when the short-time average zero-crossing rate is greater than a preset zero-crossing rate threshold, determining that the classification result of the speech validity of the enhanced speech frame is that the enhanced speech frame is a valid speech frame;

[0248] Alternatively, when the short-time energy of the single frame is less than or equal to the first energy threshold and greater than the second energy threshold, obtaining the short-time average zero-crossing rate of the enhanced speech frame; when the short-time average zero-crossing rate is less than or equal to a preset zero-crossing rate threshold, determining that the classification result of the speech validity of the enhanced speech frame is that the enhanced speech frame is not a valid speech frame;

[0249] Alternatively, when the short-time energy of a single frame is less than or equal to the second energy threshold, the classification result of determining the speech validity of the enhanced speech frame is that the enhanced speech frame does not belong to a valid speech frame.

[0250] Furthermore, the first network training module 1504 is further configured to:

[0251] determining a scale-invariant signal-to-noise ratio between the sample enhanced speech and the sample clean speech, a mean absolute error between the sample enhanced speech and the sample clean speech, and a mean square error between the sample enhanced speech and the sample clean speech;

[0252] The scale-invariant signal-to-noise ratio, mean absolute error, and mean square error are weighted to obtain the noise reduction accuracy of the speech enhancement network.

[0253] Furthermore, the first network training module 1504 is further configured to:

[0254] Acquire other clean speech except the sample clean speech, configure the sample enhanced speech and the other clean speech as a first discriminant speech pair, and input them into the first discriminator;

[0255] Performing a authenticity score on the first discriminant speech pair based on the first discriminator to obtain a first scoring result;

[0256] A first adversarial loss is determined according to the first scoring result, and a noise reduction accuracy of the speech enhancement network is determined according to the first adversarial loss.

[0257] Furthermore, the first network training module 1504 is further configured to:

[0258] Based on the sample enhanced speech, the reference noisy speech is separated from the sample noisy speech;

[0259] Configuring the reference noise speech and the sample noise speech as a second discriminant speech pair and inputting the pair into the second discriminator;

[0260] Performing a authenticity score on the second discriminant speech pair based on the second discriminator to obtain a second scoring result;

[0261] A second adversarial loss is determined according to the second scoring result, and the noise reduction accuracy of the speech enhancement network is determined according to the first adversarial loss and the second adversarial loss.

[0262] Furthermore, the first sample speech enhancement module 1502 is specifically configured to:

[0263] Perform frequency domain transformation on the sample noisy speech to obtain the original frequency domain features of the sample noisy speech;

[0264] Based on the speech enhancement network, the original frequency domain features are mapped multiple times to obtain mapping features, the time series information of the mapping features is extracted to obtain time series features, the mapping features and the time series features are spliced ​​to obtain splicing features, and the splicing features are mapped multiple times to obtain a transformation mask;

[0265] Modulate the original frequency domain features based on the transformation mask to obtain the target frequency domain features;

[0266] The target frequency domain features are subjected to an inverse transform of the frequency domain transform to obtain sample enhanced speech.

[0267] 16 , which is a schematic diagram of an optional structure of a speech enhancement device provided in an embodiment of the present application, the speech enhancement network training device 1600 includes:

[0268] An acquisition module 1601 is used to acquire speech to be processed;

[0269] The speech enhancement module 1602 is used to reduce noise on the speech to be processed based on a speech enhancement network to obtain corresponding enhanced speech; the speech enhancement network is trained using the speech enhancement network training method provided in the embodiment of the present application.

[0270] The electronic device for executing the above-mentioned speech enhancement method or speech enhancement network training method provided in the embodiment of the present application can be a terminal. Referring to FIG17 , FIG17 is a partial structural block diagram of the terminal provided in the embodiment of the present application, and the terminal includes: a camera assembly 1710, a memory 1720, an input unit 1730, a display unit 1740, a sensor 1750, an audio circuit 1760, a wireless fidelity (WiFi) module 1770, a processor 1780, and a power supply 1790. It will be understood by those skilled in the art that the terminal structure shown in FIG17 does not constitute a limitation of the terminal, and the terminal may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0271] The camera assembly 1710 can be used to capture images or videos. Optionally, the camera assembly 1710 includes a front camera and a rear camera. Typically, the front camera is provided on the front panel of the terminal, and the rear camera is provided on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0272] The memory 1720 may be used to store software programs and modules. The processor 1780 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the memory 1720 .

[0273] The input unit 1730 may be configured to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1730 may include a touch panel 1731 and other input devices 1732.

[0274] The display unit 1740 may be configured to display input information or provided information and various menus of the terminal. The display unit 1740 may include a display panel 1741 .

[0275] The audio circuit 1760 , the speaker 1761 , and the microphone 1762 may provide an audio interface.

[0276] The power source 1790 may be AC ​​power, DC power, disposable batteries, or rechargeable batteries.

[0277] The number of sensors 1750 can be one or more, and the one or more sensors 1750 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0278] The accelerometer can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal. For example, the accelerometer can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 1780 can control the display unit 1740 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer. The accelerometer can also be used to collect game or user motion data.

[0279] The gyroscope sensor can detect the device's orientation and rotation angle. It can also work with the accelerometer to capture the user's 3D movements. Based on the data collected by the gyroscope sensor, the processor 1780 can implement the following functions: motion sensing (such as changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0280] The pressure sensor can be set on the side frame of the terminal and / or the lower layer of the display unit 1740. When the pressure sensor is set on the side frame of the terminal, it can detect the user's grip signal of the terminal, and the processor 1780 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor. When the pressure sensor is set on the lower layer of the display unit 1740, the processor 1780 controls the operability controls on the UI interface based on the user's pressure operation on the display unit 1740. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0281] The optical sensor is used to collect ambient light intensity. In one embodiment, the processor 1780 can control the display brightness of the display unit 1740 based on the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1740 is increased; when the ambient light intensity is low, the display brightness of the display unit 1740 is decreased. In another embodiment, the processor 1780 can also dynamically adjust the shooting parameters of the camera assembly 1710 based on the ambient light intensity collected by the optical sensor.

[0282] In this embodiment, the processor 1780 included in the terminal can execute the speech enhancement method or the training method of the speech enhancement network in the previous embodiment.

[0283] The electronic device provided in the embodiments of the present application for executing the above-mentioned speech enhancement method or speech enhancement network training method can also be a server. Referring to FIG. 18 , FIG. 18 is a partial structural block diagram of a server provided in the embodiments of the present application. The server 1800 may vary significantly due to different configurations or performances, and may include one or more central processing units (CPUs) 1822 (e.g., one or more processors) and a memory 1832, and one or more storage media 1830 (e.g., one or more mass storage devices) storing application programs 1842 or data 1844. The memory 1832 and storage medium 1830 may be temporary storage or permanent storage. The program stored in the storage medium 1830 may include one or more modules (not shown), each of which may include a series of instruction operations on the server 1800. Furthermore, the CPU 1822 may be configured to communicate with the storage medium 1830 to execute the series of instruction operations in the storage medium 1830 on the server 1800.

[0284] The server 1800 may also include one or more power supplies 1826, one or more wired or wireless network interfaces 1850, one or more input and output interfaces 1858, and / or one or more operating systems 1841, such as Windows Server 2003. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM etc.

[0285] The processor in the server 1800 can be used to execute a speech enhancement method or a training method for a speech enhancement network.

[0286] An embodiment of the present application further provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the speech enhancement method or speech enhancement network training method of each of the aforementioned embodiments.

[0287] The present application also provides a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to implement the aforementioned speech enhancement method or speech enhancement network training method.

[0288] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchanged where appropriate to describe the embodiments of the present application, for example, they can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0289] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0290] It should be understood that in the description of the embodiments of the present application, multiple (or multiple items) means more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.

[0291] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0292] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0293] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0294] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0295] It should also be understood that the various implementation methods provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0296] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for training a speech enhancement network, performed by an electronic device, comprising: Acquire a sample clean speech and a sample noisy speech, and mix the sample clean speech and the sample noisy speech into a sample noisy speech; Based on the speech enhancement network, the sample noisy speech is denoised to obtain a sample enhanced speech; Dividing the sample enhanced speech into multiple enhanced speech frames, classifying the speech effectiveness of each of the enhanced speech frames, and generating the effectiveness distribution of the sample enhanced speech according to the classification results of each of the enhanced speech frames; Determine the noise reduction accuracy of the speech enhancement network according to the sample enhanced speech and the sample clean speech, determine the speech classification accuracy of the speech enhancement network according to the validity distribution, determine the speech enhancement accuracy of the speech enhancement network according to the noise reduction accuracy and the speech classification accuracy, and train the speech enhancement network based on the speech enhancement accuracy.

2. The method according to claim 1, wherein determining the speech classification accuracy of the speech enhancement network according to the effectiveness distribution comprises: Acquire a validity distribution label for indicating the speech validity of each frame in the sample clean speech; The speech classification accuracy of the speech enhancement network is determined based on the similarity between the validity distribution and the validity distribution label.

3. The method according to claim 1 or 2, wherein determining the speech classification accuracy of the speech enhancement network according to the effectiveness distribution comprises: Determine the distribution similarity between the validity distributions respectively corresponding to the two sample enhanced speech with corresponding relationship; the two sample enhanced speech with corresponding relationship are obtained by respectively reducing the noise of the two sample noisy speech included in the sample speech pair by the speech enhancement network, and the two sample noisy speech included in the sample speech pair are generated based on the same sample clean speech; The speech classification accuracy of the speech enhancement network is determined according to the distribution similarity.

4. The method according to any one of claims 1 to 3, wherein the classifying the speech effectiveness of each of the enhanced speech frames comprises: Determining a time domain energy parameter of each of the enhanced speech frames, wherein the time domain energy parameter is used to indicate the speech energy size of the enhanced speech frame in the time domain; The speech effectiveness of each of the enhanced speech frames is classified according to the time domain energy parameter and the preset energy threshold.

5. The method according to any one of claims 1 to 4, wherein the time domain energy parameter comprises a single frame average energy, and the classifying the speech effectiveness of each of the enhanced speech frames according to the time domain energy parameter and a preset energy threshold comprises: Determine the comprehensive average energy of the sample clean speech, and weight the comprehensive average energy according to the preset energy threshold to obtain a weighted average energy; When the single-frame average energy is greater than the weighted average energy, determining the classification result of the speech validity of the enhanced speech frame as the enhanced speech frame being a valid speech frame; Alternatively, when the single frame average energy is less than or equal to the weighted average energy, the classification result of determining the speech validity of the enhanced speech frame is that the enhanced speech frame does not belong to for valid speech frames.

6. The method according to any one of claims 1 to 5, wherein the time domain energy parameter comprises a single frame short-time energy, the number of the preset energy thresholds is multiple, the multiple preset energy thresholds comprise a first energy threshold and a second energy threshold, the first energy threshold is greater than the second energy threshold, and the classifying the speech effectiveness of each of the enhanced speech frames according to the time domain energy parameter and the preset energy threshold comprises: When the single-frame short-time energy is greater than the first energy threshold, determining that the classification result of the speech validity of the enhanced speech frame is that the enhanced speech frame belongs to a valid speech frame; Alternatively, when the short-time energy of the single frame is less than or equal to the first energy threshold and greater than the second energy threshold, the short-time average zero-crossing rate of the enhanced speech frame is obtained, and when the short-time average zero-crossing rate is greater than a preset zero-crossing rate threshold, the classification result of the speech validity of the enhanced speech frame is determined to be that the enhanced speech frame belongs to a valid speech frame; Alternatively, when the short-time energy of the single frame is less than or equal to the first energy threshold and greater than the second energy threshold, the short-time average zero-crossing rate of the enhanced speech frame is obtained, and when the short-time average zero-crossing rate is less than or equal to the preset zero-crossing rate threshold, the classification result of the speech validity of the enhanced speech frame is determined to be that the enhanced speech frame does not belong to a valid speech frame; Alternatively, when the single-frame short-time energy is less than or equal to the second energy threshold, the classification result of determining the speech validity of the enhanced speech frame is that the enhanced speech frame does not belong to a valid speech frame.

7. The method according to any one of claims 1 to 6, wherein determining the noise reduction accuracy of the speech enhancement network according to the sample enhanced speech and the sample clean speech comprises: Determining a scale-invariant signal-to-noise ratio between the sample enhanced speech and the sample clean speech, a mean absolute error between the sample enhanced speech and the sample clean speech, and a mean square error between the sample enhanced speech and the sample clean speech; The scale-invariant signal-to-noise ratio, the mean absolute error and the mean square error are weighted to obtain the noise reduction accuracy of the speech enhancement network.

8. The method according to any one of claims 1 to 7, wherein determining the noise reduction accuracy of the speech enhancement network according to the sample enhanced speech and the sample clean speech comprises: Acquire other clean speech except the sample clean speech, configure the sample enhanced speech and the other clean speech as a first discriminant speech pair, and input them into a first discriminator; Performing a authenticity score on the first discriminator for the first discriminator speech pair to obtain a first score result; A first adversarial loss is determined according to the first scoring result, and a noise reduction accuracy of the speech enhancement network is determined according to the first adversarial loss.

9. The method according to any one of claims 1 to 8, wherein determining the noise reduction accuracy of the speech enhancement network according to the first adversarial loss comprises: Based on the sample enhanced speech, separating the reference noisy speech from the sample noisy speech; Configuring the reference noise speech and the sample noise speech as a second discriminant speech pair, and inputting the pair into a second discriminator; Performing a authenticity score on the second discriminator speech pair based on the second discriminator to obtain a second score result; Determine a second adversarial loss according to the second scoring result, and determine a second adversarial loss according to the first adversarial loss and the second adversarial loss. The loss determines the denoising accuracy of the speech enhancement network.

10. The method according to any one of claims 1 to 9, wherein the step of performing noise reduction on the sample noisy speech based on a speech enhancement network to obtain the sample enhanced speech comprises: Performing frequency domain transformation on the sample noisy speech to obtain original frequency domain features of the sample noisy speech; Based on the speech enhancement network, the original frequency domain features are mapped multiple times to obtain mapping features, time series information is extracted from the mapping features to obtain time series features, the mapping features are spliced ​​with the time series features to obtain spliced ​​features, and the spliced ​​features are mapped multiple times to obtain a transformation mask; Modulating the original frequency domain feature based on the transformation mask to obtain a target frequency domain feature; An inverse transformation of the frequency domain transformation is performed on the target frequency domain feature to obtain the sample enhanced speech.

11. A speech enhancement method, performed by an electronic device, comprising: Get the voice to be processed; Based on the speech enhancement network, the speech to be processed is subjected to noise reduction to obtain corresponding enhanced speech; The speech enhancement network is trained by the speech enhancement network training method according to any one of claims 1 to 10.

12. A training device for a speech enhancement network, comprising: A first sample speech mixing module is used to obtain a sample clean speech and a sample noisy speech, and mix the sample clean speech and the sample noisy speech into a sample noisy speech; A first sample speech enhancement module, used for performing noise reduction on the sample noisy speech based on a speech enhancement network to obtain a sample enhanced speech; A first effectiveness classification module, used for dividing the sample enhanced speech into multiple enhanced speech frames, classifying the speech effectiveness of each of the enhanced speech frames, and generating the effectiveness distribution of the sample enhanced speech according to the classification results of each of the enhanced speech frames; The first network training module is used to determine the noise reduction accuracy of the speech enhancement network according to the sample enhanced speech and the sample clean speech, determine the speech classification accuracy of the speech enhancement network according to the effectiveness distribution, determine the speech enhancement accuracy of the speech enhancement network according to the noise reduction accuracy and the speech classification accuracy, and train the speech enhancement network based on the speech enhancement accuracy.

13. A speech enhancement device, comprising: An acquisition module, used to acquire the speech to be processed; A speech enhancement module, used for performing noise reduction on the speech to be processed based on a speech enhancement network to obtain corresponding enhanced speech; The speech enhancement network is trained by the speech enhancement network training method according to any one of claims 1 to 10.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the training method of the speech enhancement network described in any one of claims 1 to 10, or implements the speech enhancement method described in claim 11.

15. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the training method of a speech enhancement network as described in any one of claims 1 to 10, or implements the speech enhancement method as described in claim 11.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the training method of the speech enhancement network described in any one of claims 1 to 10 is implemented, or the speech enhancement method described in claim 11 is implemented.

Images