Post-filtering method and system for speech enhancement network

Abstract

Disclosed in embodiments of the present application are a post-filtering method and system for a speech enhancement network. The technical solution provided in the embodiments of the present application comprises: acquiring a first speech masking value of a microphone signal output by an ENC module, wherein the first speech masking value is generated on the basis of the microphone signal and a loudspeaker signal to be played, and the microphone signal is aligned with the loudspeaker signal to be played; determining a first residual echo masking value of the microphone signal, calculating a noise masking value of the microphone signal on the basis of the first speech masking value and the first residual echo masking value, and performing overestimation on the first residual echo masking value to obtain a second residual echo masking value; and outputting a second speech masking value on the basis of the noise masking value and the second residual echo masking value, to synthesize a speech enhancement signal of the microphone signal on the basis of the second speech masking value. By using the foregoing technical means, echo residuals in a speech signal can be filtered out, reliability of the speech enhancement network is improved, the speech enhancement effect is improved, and voice call experience of users is optimized.

Description

A post-filtering method and system for speech enhancement network

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 10, 2023, with application number 202311312141.2, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of speech processing technology, and in particular to a post-filtering method and system for a speech enhancement network. Background Art

[0003] Currently, in scenarios such as live streaming and voice interaction on mobile devices, users are in full-duplex communication. During full-duplex communication, when the user's phone is in speaker mode, the sound played by the phone's speaker will be collected by the phone's microphone and then form an acoustic echo. In addition, when users communicate, their voices are often interfered with by the surrounding noise. The presence of echo and noise greatly reduces the quality of voice calls. To this end, the ENC module (Echo and Noise Cancellation) of the voice enhancement network is usually used during voice processing to perform echo and noise cancellation to improve the user's voice call quality.

[0004] However, due to limitations in computing power and memory, the number of parameters and computational complexity of mobile models is typically relatively small. The ENC module is prone to performance degradation due to mismatches between training data and real-world data. This can cause persistent echo residue in speech signals processed by the speech enhancement network, impacting the user's voice call experience.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a post-filtering method and system for a speech enhancement network, which can perform echo over-estimation on a speech signal and solve the technical problem of residual echo in the speech signal in the speech enhancement network.

[0007] In a first aspect, an embodiment of the present application provides a post-filtering method for a speech enhancement network, comprising:

[0008] Obtaining a first speech masking value of the microphone signal output by the ENC module, where the first speech masking value is generated based on the microphone signal and the signal to be played by the speaker, and the microphone signal and the signal to be played by the speaker are aligned;

[0009] Determining a first residual echo masking value of the microphone signal, calculating a noise masking value of the microphone signal based on the first speech masking value and the first residual echo masking value, and overestimating the first residual echo masking value to obtain a second residual echo masking value;

[0010] A second speech masking value is output according to the noise masking value and the second residual echo masking value to synthesize a speech enhancement signal of the microphone signal based on the second speech masking value.

[0011] In a second aspect, an embodiment of the present application provides a post-filtering system for a speech enhancement network, comprising:

[0012] an acquisition module configured to acquire a first speech masking value of the microphone signal output by the ENC module, where the first speech masking value is generated based on the microphone signal and the signal to be played by the speaker, and the microphone signal and the signal to be played by the speaker are aligned;

[0013] an over-estimation module configured to determine a first residual echo masking value of the microphone signal, calculate a noise masking value of the microphone signal based on the first speech masking value and the first residual echo masking value, and over-estimate the first residual echo masking value to obtain a second residual echo masking value;

[0014] The output module is configured to output a second speech masking value according to the noise masking value and the second residual echo masking value, so as to synthesize a speech enhancement signal of the microphone signal based on the second speech masking value.

[0015] In a third aspect, an embodiment of the present application provides a post-filtering device for a speech enhancement network, comprising:

[0016] memory and one or more processors;

[0017] The memory is configured to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the post-filtering method of the speech enhancement network as described in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a computer processor, they are configured to perform the post-filtering method of the speech enhancement network as described in the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the post-filtering method of the speech enhancement network as described in the first aspect.

[0021] The embodiment of the present application obtains the first voice masking value of the microphone signal output by the ENC module, the first voice masking value is generated based on the microphone signal and the signal to be played by the speaker, and the microphone signal and the signal to be played by the speaker are aligned; determines the first residual echo masking value of the microphone signal, calculates the noise masking value of the microphone signal based on the first voice masking value and the first residual echo masking value, over-estimates the first residual echo masking value to obtain a second residual echo masking value; outputs the second voice masking value according to the noise masking value and the second residual echo masking value, and synthesizes the voice enhancement signal of the microphone signal based on the second voice masking value. By adopting the above technical means, by determining the residual echo masking value of the microphone signal and over-estimating it, the voice masking value of the microphone signal is corrected, so that the voice enhancement signal generated thereby can filter out the echo residue in the voice signal, improve the reliability of the voice enhancement network, improve the voice enhancement effect, and optimize the user's voice call experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a flow chart of a post-filtering method for a speech enhancement network provided in an embodiment of the present application;

[0023] FIG2 is a schematic diagram of voice enhancement of user audio in an embodiment of the present application;

[0024] FIG3 is a signal processing flow chart of a speech enhancement network according to an embodiment of the present application;

[0025] FIG4 is a schematic diagram of the input and output of the post-filtering module in an embodiment of the present application;

[0026] FIG5 is a flowchart of the correction of the speech masking value in an embodiment of the present application;

[0027] FIG6 is a flowchart of the correction judgment of the speech masking value in an embodiment of the present application;

[0028] FIG7 is a flowchart of determining an over-estimated strength of a first residual echo masking value in an embodiment of the present application;

[0029] FIG8 is a schematic structural diagram of a post-filtering system of a speech enhancement network provided in an embodiment of the present application;

[0030] FIG9 is a schematic structural diagram of a post-filtering device of a speech enhancement network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. It should also be noted that, for ease of description, only parts related to the present application, not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0032] The post-filtering method of the speech enhancement network provided in this application aims to post-filter the echo and noise cancellation results of the ENC module in the speech enhancement network so that the speech enhancement signal can filter out the echo residue in the speech signal and improve the user's voice call experience.

[0033] In related speech enhancement networks, in order to solve the echo and noise problems in speech signals, an ENC module is usually used in the speech enhancement network to perform echo and noise elimination. After decades of development, especially the advancement of deep learning technology, the upper limit of echo and noise elimination algorithms has been significantly improved. However, deep learning algorithms are based on data-driven, and often have degradation problems due to the inability of training data to match real data. This is especially evident for mobile applications with limited computing resources. The performance degradation of the echo and noise elimination model will cause the speech signal processed by the ENC module to still have persistent echo residues, affecting the user's voice call experience. Based on this, a post-filtering method for a speech enhancement network in an embodiment of the present application is provided to solve the technical problem of residual echo in the speech signal in the speech enhancement network.

[0034] Example:

[0035] FIG1 is a flowchart of a post-filtering method for a speech enhancement network provided in an embodiment of the present application. The post-filtering method for a speech enhancement network provided in this embodiment can be performed by a post-filtering device of the speech enhancement network. The post-filtering device of the speech enhancement network can be implemented via software and / or hardware. The post-filtering device of the speech enhancement network can be composed of two or more physical entities, or a single physical entity. Generally speaking, the post-filtering device of the speech enhancement network can be a processing device such as a mobile phone, tablet, or computer.

[0036] The following description will be made using a mobile phone as an example of the main body of the post-filtering method for performing the speech enhancement network. Referring to FIG1 , the post-filtering method for the speech enhancement network specifically includes:

[0037] S110: Obtain a first speech masking value of the microphone signal output by the ENC module, where the first speech masking value is generated based on the microphone signal and a signal to be played by a loudspeaker, and the microphone signal and the signal to be played by the loudspeaker are aligned.

[0038] In the embodiment of the present application, the residual echo existing in the speech signal after being processed by the ENC module is processed by correcting the speech masking value output by the ENC module to implement post-filtering of the speech signal, thereby achieving the effect of eliminating the residual echo.

[0039] For example, referring to Figure 2, when a user's mobile phone is used to make a voice call, the microphone collects the user's audio and inputs the microphone signal into the voice enhancement network for voice enhancement. During the voice enhancement process, the post-filtering method of the voice enhancement network of the embodiment of the present application is used to eliminate the residual echo of the microphone signal. The generated voice enhancement signal is then output to the audio encoder, and after encoding, it is sent to the client communicating with the user's mobile phone. The client on the other end can then hear clear, echo-free user audio, improving the user's voice call experience.

[0040] Referring to Figure 3, when using a speech enhancement network to perform speech enhancement on user audio, the speech enhancement network has two input signals, namely the time domain signal recorded by the user's microphone and the time domain signal to be played by the speaker. The time domain signal recorded by the user's microphone is defined as the microphone signal, and the time domain signal to be played by the speaker is defined as the speaker's to-be-played signal. It is understandable that since the echo during a voice call is mainly caused by the microphone collecting the audio played by the speaker, the embodiment of the present application inputs the speaker's to-be-played signal into the speech enhancement network for speech enhancement to eliminate the echo in the microphone signal. It should be noted that since the microphone signal collects the sound played by the speaker, the microphone signal needs to be aligned with the speaker's to-be-played signal to achieve accurate echo cancellation processing.

[0041] As shown in Figure 3, the input signal is first processed by the DFT (Discrete Fourier Transform) of the signal analysis module, and then transformed from the time domain to the frequency domain. The frequency domain signal is then processed by the DSP (Digital Signal Processing) pre-processing and the ENC module. Among them, the DSP pre-processing and ENC module processing include delay estimation, adaptive filtering and feature extraction operations of the ENC module. The ENC module outputs the frequency masking value (defined as the first speech masking value) and the estimated residual echo in the microphone signal. Then, the post-filtering module corrects these frequency masking values ​​based on the post-filtering method of the speech enhancement network of the embodiment of the present application to obtain the corrected frequency masking values. Next, the frequency masking value is multiplied by the frequency signal output by the adaptive filtering module through the signal synthesis module, and then the discrete Fourier inverse transform is performed to obtain the enhanced time domain microphone signal. In this way, speech enhancement is completed and the elimination of residual echo is achieved.

[0042] Furthermore, as shown in Figure 4, a schematic diagram of the signal input and output of the post-filtering module in the speech enhancement network is given. Wherein, D represents the mic frequency domain signal, that is, the microphone signal; X0 represents the unaligned loudspeaker signal to be played, and X represents the loudspeaker signal to be played aligned with D; E represents the signal after adaptive filtering (AF), which contains residual echo, noise and near-end speech (that is, microphone signal), and R represents the residual echo estimated by the ENC module; gain represents the near-end speech masking value mask output by the ENC module, which is defined as the first speech masking value of the microphone signal. Based on the above signal input, the post-filtering module outputs the near-end speech masking value mask corrected by post-processing based on the post-filtering method of the speech enhancement network of the present application, which is defined as the second speech masking value of the microphone signal. After the residual echo is processed by the post-filtering module, the corrected second speech masking value can be used for signal synthesis to obtain a clear and echo-free microphone signal.

[0043] It should be noted that, by observing the degradation behavior of the ENC model on the mobile terminal, it was found that the residual echo estimated by the ENC model was jittering around the true value. Therefore, the residual echo estimated by the ENC module can be overestimated to a certain extent, so as to ensure that the estimated residual echo is greater than the true value, thereby ensuring that there will be no echo leakage, which increases the robustness of the algorithm. Based on this, the embodiment of the present application obtains a first speech masking value of the microphone signal and determines the residual echo masking value in the microphone signal, over-estimates the residual echo masking value, and uses the over-estimated residual echo masking value to correct the first speech masking value to obtain a second speech masking value.

[0044] Specifically, referring to FIG5 , a flowchart of correcting a speech masking value according to an embodiment of the present application is provided, wherein, after obtaining a first speech masking value of a microphone signal output by an ENC module, the post-filtering module subsequently determines whether to correct the first speech masking value.

[0045] 6 , after obtaining the first speech masking value of the microphone signal output by the ENC module, the method further includes:

[0046] S1101, performing echo judgment on the microphone signal and performing designated non-stationary frame judgment on the loudspeaker to-be-played signal;

[0047] S1102. If it is determined that the microphone signal has an echo and the signal to be played by the speaker is a specified non-stationary frame, enter the over-estimation process of the first residual echo masking value; otherwise, directly output the first speech masking value to synthesize a speech enhancement signal of the microphone signal based on the first speech masking value.

[0048] Since the post-filtering module primarily corrects the speech masking value for microphone signals containing echoes, the first speech masking value output by the ENC module can be directly used to synthesize the speech enhancement signal for portions of the microphone signal that do not contain echoes. Furthermore, if the speaker's pending signal is a stationary frame, this will not affect the microphone signal, and therefore, the speech masking value for this portion of the microphone signal will not be corrected. The first speech masking value is only corrected if it is determined that the microphone signal has an echo and the speaker's pending signal is a specified non-stationary frame, indicating that the current microphone signal has an echo.

[0049] It should be noted that in actual applications, the judgment condition for triggering the speech masking value correction can be adaptively adjusted according to the actual processing requirements of speech enhancement. The embodiment of the present application does not impose fixed restrictions on the specific judgment conditions, and will not be described in detail here.

[0050] The echo judgment of the microphone signal includes:

[0051] Obtain an adaptive filtering signal generated based on the microphone signal and the signal to be played by the speaker, and calculate the power spectra of the microphone signal, the signal to be played by the speaker, and the adaptive filtering signal; determine the first coherence coefficient of the microphone signal and the signal to be played by the speaker based on the power spectrum, as well as the second coherence coefficient of the microphone signal and the adaptive filtering signal; calculate the first frame average mutual coherence coefficient based on the first coherence coefficient, and calculate the second frame average mutual coherence coefficient based on the second coherence coefficient, and perform echo judgment of the microphone signal in combination with the first frame average mutual coherence coefficient and the second frame average mutual coherence coefficient.

[0052] The embodiment of the present application adopts a sensitive echo detection algorithm, which first determines the coherence of each frequency point of the microphone signal and the signal to be played by the speaker (X and D) and the microphone signal and the adaptive filter signal (D and E), further calculates the respective frame average coherence, and finally combines the corresponding threshold information to perform microphone echo judgment. Among them, the power spectrum of the microphone signal, the signal to be played by the speaker, and the adaptive filter signal is first calculated: P x (l,k)=α·P x (l-1,k)+(1-α)·|x(l,k)| 2 P d (l,k)=α·P d (l-1,k)+(1-α)·|d(l,k)| 2 P e (l,k)=α·P e (l-1,k)+(1-α)·|e(l,k)| 2

[0053] Among them, P x (l,k) represents the power spectrum of the microphone signal, P d (l,k) represents the power spectrum of the microphone signal, P e (l,k) represents the power spectrum of the microphone signal, l is the frame index, k is the frequency index, and α is the inter-frame smoothing coefficient, which ranges from 0 to 1.

[0054] Based on the above signal power spectrum, calculate the cross power spectrum of the microphone signal and the signal to be played by the speaker: P xd (l,k)=α·P xd (l-1,k)+(1-α)·x * (l,k)·d(l,k)

[0055] And the cross power spectrum of the microphone signal and the adaptive filter signal: P de (l,k)=α·P de (l-1,k)+(1-α)·d * (l,k)·e(l,k)

[0056] Then the first coherence coefficient of the microphone signal and the signal to be played by the loudspeaker is calculated as:

[0057] The second coherence coefficient between the microphone signal and the adaptive filter signal is:

[0058] The above coherence coefficients are averaged in the main frequency band to obtain the frame average mutual coherence coefficient. Among them, the first frame average mutual coherence coefficient is:

[0059] The average mutual interference coefficient of the second frame is:

[0060] Based on the above frame average mutual interference coefficient, it is possible to determine whether the microphone signal has an echo in combination with the corresponding threshold range. The judgment logic is:

[0061] Indicates the absence of echo;

[0062] Indicates the presence of an echo;

[0063] Else maintains the previous frame state;

[0064] The empirical values ​​of the corresponding thresholds are T0=0.1, T1=0.98, T2=0.2, and T3=0.95.

[0065] On the other hand, the designated non-stationary frame judgment of the signal to be played by the loudspeaker is performed, including:

[0066] A minimum-value-based noise floor estimation is performed on the signal to be played by the loudspeaker, and a designated non-stationary frame of the signal to be played by the loudspeaker is determined based on the estimated noise floor energy.

[0067] There are two reasons for performing non-stationary frame detection on the speaker's signal to be played. First, the speech enhancement network uses the ENC model, which can suppress noise. The stationary frames in the speaker's signal to be played are often treated as noise by the model. Second, during a two-way call, the echo perception generated by the non-stationary frames is more obvious. This application expects to only post-process the non-stationary echo, so non-stationary frame detection of the speaker's signal to be played is performed. The specific judgment principle is to first perform a minimum-based background noise estimation on the speaker's signal to be played to obtain the corresponding background noise energy, and then determine whether the current frame energy of the aligned speaker's signal to be played is greater than the background noise energy than the corresponding threshold value T4. If so, it is considered to be a non-stationary frame, otherwise it is a stationary frame.

[0068] 5 , if the current microphone signal is judged to have an echo and the signal to be played by the speaker is a non-stationary frame, the first speech masking value correction process is entered. Otherwise, the post-filter module transparently transmits the signal and directly outputs the first speech masking value gain of the ENC module.

[0069] It should be noted that, in actual applications, the above-mentioned echo judgment and non-stationary frame judgment processes can adaptively set other judgment conditions and thresholds. The embodiment of the present application does not impose fixed restrictions on the specific judgment process, which will not be described in detail here.

[0070] S120: Determine a first residual echo masking value of the microphone signal, calculate a noise masking value of the microphone signal based on the first speech masking value and the first residual echo masking value, and over-estimate the first residual echo masking value to obtain a second residual echo masking value.

[0071] Furthermore, as shown in FIG5 , when determining to enter the correction process of the first speech masking value, the residual echo masking value in the microphone signal is first determined, which is defined as the first residual echo masking value. The first speech masking value is corrected by overestimating the first residual echo masking value.

[0072] Determining a first residual echo masking value of the microphone signal includes:

[0073] An adaptive filtering signal generated based on the microphone signal and the signal to be played by the speaker, as well as a residual echo estimation signal output by the ENC module, is obtained, and a first residual echo masking value of the microphone signal is determined based on a ratio of the residual echo estimation signal to the adaptive filtering signal.

[0074] Since the post-filtering module only wants to over-estimate the residual echo estimation signal R estimated by the ENC module, it is necessary to eliminate the interference of the near-end microphone signal noise. Therefore, this application first splits the first speech masking value gain(l,k) of the near-end microphone signal directly output by the ENC module into the noise masking value mask_noise of the near-end microphone signal and the first residual echo masking value mask_echo. The specific calculation method is: mask_echo(l,k)=R(l,k) / E(l,k) mask_noise(l,k)=1-gain(l,k)-mask_echo(l,k)

[0075] Based on the determined first residual echo masking value, an estimation operation may be adaptively performed to obtain a corresponding second residual echo masking value.

[0076] Optionally, referring to FIG. 7 , before estimating the first residual echo masking value, the method further includes:

[0077] S1201, calculating a frame signal-to-echo ratio based on a first speech masking value and a first residual echo masking value in a microphone signal;

[0078] S1202: Based on the frame signal-to-echo ratio, determine the over-estimation strength of the first residual echo masking value according to the comparison result, using the set threshold.

[0079] Through actual testing, it was found that the instability of the residual echo estimation of the ENC module is related to the frame signal-to-echo ratio. The lower the frame signal-to-echo ratio, the more unstable it tends to be. Obviously. Observing the degradation of the ENC module's echo suppression, it exhibits significant degradation under low frame signal-to-echo ratios. Therefore, the frame signal-to-echo ratio is used as a feature here to control the over-estimation intensity of the residual echo output by the ENC module. Therefore, the embodiment of the present application adjusts the degree of over-estimation of the first residual echo masking value based on the frame signal-to-echo ratio. In addition, due to the coherence between the microphone signal and the signal to be played by the speaker, it is also a feature of the robustness of the post-filtering algorithm. Therefore, this coherence feature can be combined to over-estimate the echo estimated by the network.

[0080] The frame-level frame signal-to-return ratio is calculated from the frame signal-to-return ratio of all frequencies or the frame signal-to-return ratio of some core frequencies. The calculation formula of the frame signal-to-return ratio is:

[0081] Where N is the number of discrete Fourier transform points. Because the ENC module's echo suppression capability is more susceptible to degradation when the frame signal-to-echo ratio is low, a strong overestimation of the first residual echo masking value is performed when frameser(l) < T5; otherwise, a weak overestimation is performed. The empirical value of threshold T5 is 1.

[0082] Furthermore, the first residual echo masking value is estimated, including:

[0083] A first coherence coefficient between a microphone signal and a signal to be played by a loudspeaker is obtained, and a first residual echo masking value is over-estimated based on the first coherence coefficient and a set empirical value, wherein the set empirical value is set according to an over-estimation strength of the first residual echo masking value.

[0084] Based on the first coherence coefficient R calculated when performing echo judgment on the microphone signal xd (l,k). First, sort the core frequency band from large to small in 0-D, and then get the point 0.75*D of the sort, which is recorded as R xd Low(l). Then the first coherence coefficient R xd (l,k) is weighted and the calculation formula is as follows:

[0085] in, k=0,1,2,...,N-1

[0086] After obtaining the first coherence coefficient R xd The weighted result of (l,k) After that, the first residual echo masking value mask_echo is overestimated to obtain the second residual echo masking value mask_echo_overdrive(l,k), which is calculated as follows:

[0087] Among them, the over-estimation strength is set according to the frame signal-to-echo ratio. For strong over-estimation, Wecho takes a larger value, with an empirical value of 2.5; for weak over-estimation, Wecho takes a smaller value, with an empirical value of 1.05. It should be noted that the reason for making a weak estimation here is that the residual echo estimation signal R output by the ENC module is jittering around the true value. Slightly raising it can achieve the purpose of reducing the leakage echo. xd The purpose of weighting (l,k) is to suppress the nonlinear harmonics in the echo to a certain extent.

[0088] In this way, by overestimating the first residual echo masking value, the residual echo estimation signal R output by the ENC module can be corrected to avoid the situation where the residual echo is not completely eliminated due to estimation jitter.

[0089] S130: Output a second speech masking value according to the noise masking value and the second residual echo masking value, so as to synthesize a speech enhancement signal of the microphone signal based on the second speech masking value.

[0090] Finally, based on the second residual echo masking value determined above and combined with the previously calculated noise masking value, the corrected voice masking value of the microphone signal can be calculated, which is defined as the second voice masking value. The calculation formula for the second voice masking value gain_out(l,k) is: gain_out(l,k) = 1 - mask_noise(l,k) - mask_echo_overdrive(l,k)

[0091] Based on this second speech masking value, the signal synthesis module multiplies the second speech masking value with the frequency signals output by the adaptive filtering module, and then performs an inverse discrete Fourier transform to obtain an enhanced time-domain microphone signal, namely the speech enhancement signal. Because the second speech masking value corrects and eliminates jitter in the ENC module's echo signal estimation process, the synthesized speech enhancement signal can eliminate residual echo and improve voice call quality.

[0092] Therefore, by observing the degradation behavior of the ENC module in the speech enhancement network, when a non-stationary echo is detected, the residual echo estimated by the network is overestimated to varying degrees based on the frame signal-to-echo ratio and signal coherence characteristics estimated based on the network output mask. This can significantly reduce the leakage echo problem caused by the network directly outputting the mask, thereby ensuring the user's call quality.

[0093] In the above, by obtaining the first voice masking value of the microphone signal output by the ENC module, the first voice masking value is generated based on the microphone signal and the signal to be played by the speaker, and the microphone signal and the signal to be played by the speaker are aligned; determining the first residual echo masking value of the microphone signal, calculating the noise masking value of the microphone signal based on the first voice masking value and the first residual echo masking value, over-estimating the first residual echo masking value to obtain the second residual echo masking value; outputting the second voice masking value according to the noise masking value and the second residual echo masking value, so as to synthesize the voice enhancement signal of the microphone signal based on the second voice masking value. By adopting the above technical means, by determining the residual echo masking value of the microphone signal and over-estimating it, the voice masking value of the microphone signal is corrected, so that the voice enhancement signal generated thereby can filter out the echo residue in the voice signal, improve the reliability of the voice enhancement network, improve the voice enhancement effect, and optimize the user's voice call experience.

[0094] Based on the above embodiment, FIG8 is a schematic diagram of the structure of a post-filtering system of a speech enhancement network provided by this application. Referring to FIG8 , the post-filtering system of the speech enhancement network provided by this embodiment specifically includes: an acquisition module 21, an over-estimation module 22, and an output module 23.

[0095] The acquisition module 21 is configured to acquire a first speech masking value of the microphone signal output by the ENC module, where the first speech masking value is generated based on the microphone signal and the signal to be played by the speaker, and the microphone signal and the signal to be played by the speaker are aligned;

[0096] The over-estimation module 22 is configured to determine a first residual echo masking value of the microphone signal, calculate a noise masking value of the microphone signal based on the first speech masking value and the first residual echo masking value, and over-estimate the first residual echo masking value to obtain a second residual echo masking value;

[0097] The output module 23 is configured to output a second speech masking value according to the noise masking value and the second residual echo masking value, so as to synthesize a speech enhancement signal of the microphone signal based on the second speech masking value.

[0098] Specifically, after obtaining the first speech masking value of the microphone signal output by the ENC module, the method further includes:

[0099] Perform echo judgment on the microphone signal and perform specified non-stationary frame judgment on the signal to be played by the speaker;

[0100] If it is determined that the microphone signal has an echo and the signal to be played by the speaker is a specified non-stationary frame, the over-estimation process of the first residual echo masking value is entered; otherwise, the first speech masking value is directly output to synthesize a speech enhancement signal of the microphone signal based on the first speech masking value.

[0101] The echo judgment of the microphone signal includes:

[0102] Obtaining an adaptive filtering signal generated based on the microphone signal and the signal to be played by the speaker, and calculating the power spectra of the microphone signal, the signal to be played by the speaker, and the adaptive filtering signal;

[0103] Determining a first coherence coefficient between the microphone signal and the signal to be played by the loudspeaker, and a second coherence coefficient between the microphone signal and the adaptive filtering signal based on the power spectrum;

[0104] A first frame average mutual coherence coefficient is calculated based on the first coherence coefficient, a second frame average mutual coherence coefficient is calculated based on the second coherence coefficient, and the first frame average mutual coherence coefficient and the second frame average mutual coherence coefficient are combined to perform echo judgment of the microphone signal.

[0105] Performing specified non-stationary frame judgment on the signal to be played by the speaker, including:

[0106] A minimum-value-based noise floor estimation is performed on the signal to be played by the loudspeaker, and a designated non-stationary frame of the signal to be played by the loudspeaker is determined based on the estimated noise floor energy.

[0107] Specifically, determining a first residual echo masking value of a microphone signal includes:

[0108] An adaptive filtering signal generated based on the microphone signal and the signal to be played by the speaker, as well as a residual echo estimation signal output by the ENC module, is obtained, and a first residual echo masking value of the microphone signal is determined based on a ratio of the residual echo estimation signal to the adaptive filtering signal.

[0109] Specifically, before estimating the first residual echo masking value, the method further includes:

[0110] Calculating a frame signal-to-echo ratio based on a first speech masking value and a first residual echo masking value in the microphone signal;

[0111] The threshold value set based on the frame signal-to-echo ratio comparison is determined according to the comparison result, and the over-estimation strength of the first residual echo masking value is determined.

[0112] The first residual echo masking value is estimated, including:

[0113] A first coherence coefficient between a microphone signal and a signal to be played by a loudspeaker is obtained, and a first residual echo masking value is over-estimated based on the first coherence coefficient and a set empirical value, wherein the set empirical value is set according to an over-estimation strength of the first residual echo masking value.

[0114] In the above, by obtaining the first voice masking value of the microphone signal output by the ENC module, the first voice masking value is generated based on the microphone signal and the signal to be played by the speaker, and the microphone signal and the signal to be played by the speaker are aligned; determining the first residual echo masking value of the microphone signal, calculating the noise masking value of the microphone signal based on the first voice masking value and the first residual echo masking value, over-estimating the first residual echo masking value to obtain the second residual echo masking value; outputting the second voice masking value according to the noise masking value and the second residual echo masking value, so as to synthesize the voice enhancement signal of the microphone signal based on the second voice masking value. By adopting the above technical means, by determining the residual echo masking value of the microphone signal and over-estimating it, the voice masking value of the microphone signal is corrected, so that the voice enhancement signal generated thereby can filter out the echo residue in the voice signal, improve the reliability of the voice enhancement network, improve the voice enhancement effect, and optimize the user's voice call experience.

[0115] The post-filtering system of the speech enhancement network provided in the embodiment of the present application can be configured to execute the post-filtering method of the speech enhancement network provided in the above embodiment, and has corresponding functions and beneficial effects.

[0116] Based on the above practical example, an embodiment of the present application further provides a post-filtering device for a speech enhancement network. Referring to FIG9 , the post-filtering device for the speech enhancement network includes: a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the post-filtering method for the speech enhancement network described in any embodiment of the present application (for example, the acquisition module, over-estimation module, and output module in the post-filtering system of the speech enhancement network). The communication module is configured to perform data transmission. The processor executes the software programs, instructions, and modules stored in the memory to execute various functional applications and data processing of the device, thereby implementing the post-filtering method for the speech enhancement network described above. The input device can be configured to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device can include a display device such as a display screen. The post-filtering device for the speech enhancement network provided above can be configured to execute the post-filtering method for the speech enhancement network provided in the above embodiment, and has corresponding functions and beneficial effects.

[0117] Based on the above embodiments, embodiments of the present application further provide a computer-readable storage medium storing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are configured to execute a post-filtering method for a speech enhancement network. The storage medium can be any of various types of memory devices or storage devices. Of course, the computer-executable instructions of the computer-readable storage medium provided in embodiments of the present application are not limited to the post-filtering method for a speech enhancement network described above, and can also execute related operations in the post-filtering method for a speech enhancement network provided in any embodiment of the present application.

[0118] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the post-filtering method of the speech enhancement network described in each embodiment of the present application.

Claims

1. A post-filtering method for a speech enhancement network, wherein: include: Obtaining a first speech masking value of a microphone signal output by an ENC module, wherein the first speech masking value is generated based on the microphone signal and a signal to be played by a loudspeaker, and the microphone signal and the signal to be played by the loudspeaker are aligned; Determine a first residual echo masking value of the microphone signal, calculate a noise masking value of the microphone signal based on the first speech masking value and the first residual echo masking value, and over-estimate the first residual echo masking value to obtain a second residual echo masking value; A second speech masking value is output according to the noise masking value and the second residual echo masking value to synthesize a speech enhancement signal of the microphone signal based on the second speech masking value.

2. The post-filtering method of the speech enhancement network according to claim 1, wherein: Before estimating the first residual echo masking value, the method further includes: Calculating a frame signal-to-echo ratio according to the first speech masking value and the first residual echo masking value in the microphone signal; Based on a threshold set by the frame signal-to-echo ratio comparison, an over-estimation strength of the first residual echo masking value is determined according to a comparison result.

3. The post-filtering method of the speech enhancement network according to claim 2, wherein: The first residual echo masking value is estimated, including: A first coherence coefficient between the microphone signal and the signal to be played by the speaker is obtained, and the first residual echo masking value is over-estimated based on the first coherence coefficient and a set empirical value, wherein the set empirical value is set according to an over-estimation strength of the first residual echo masking value.

4. The post-filtering method of the speech enhancement network according to claim 1, wherein: After obtaining the first speech masking value of the microphone signal output by the ENC module, the method further includes: Performing echo judgment on the microphone signal and performing designated non-stationary frame judgment on the signal to be played by the loudspeaker; When it is determined that there is an echo in the microphone signal and the signal to be played by the speaker is a specified non-stationary frame, the over-estimation process of the first residual echo masking value is entered; otherwise, the first speech masking value is directly output to synthesize the speech enhancement signal of the microphone signal based on the first speech masking value.

5. The post-filtering method of the speech enhancement network according to claim 4, wherein: Performing echo determination on the microphone signal includes: Acquire an adaptive filtering signal generated based on the microphone signal and the signal to be played by the speaker Calculate the power spectrum of the microphone signal, the signal to be played by the loudspeaker and the adaptive filtering signal; Determine a first coherence coefficient between the microphone signal and the signal to be played by the speaker, and a second coherence coefficient between the microphone signal and the adaptive filtering signal based on the power spectrum; A first frame average mutual coherence coefficient is calculated based on the first coherence coefficient, a second frame average mutual coherence coefficient is calculated based on the second coherence coefficient, and the echo of the microphone signal is determined by combining the first frame average mutual coherence coefficient and the second frame average mutual coherence coefficient.

6. The post-filtering method of the speech enhancement network according to claim 4, wherein: Performing a designated non-stationary frame judgment on the signal to be played by the loudspeaker includes: A minimum value-based background noise estimation is performed on the signal to be broadcasted by the loudspeaker, and a designated non-stationary frame judgment of the signal to be broadcasted by the loudspeaker is performed based on the estimated background noise energy.

7. The post-filtering method of the speech enhancement network according to claim 1, wherein: Determining a first residual echo masking value of the microphone signal includes: An adaptive filtering signal generated based on the microphone signal and the signal to be played by the speaker, and a residual echo estimation signal output by the ENC module are obtained, and a first residual echo masking value of the microphone signal is determined based on a ratio of the residual echo estimation signal to the adaptive filtering signal.

8. A post-filtering system for a speech enhancement network, wherein: include: An acquisition module is configured to acquire a first speech masking value of the microphone signal output by the ENC module, wherein the first speech masking value is generated based on the microphone signal and a signal to be played by a loudspeaker, and the microphone signal and the signal to be played by the loudspeaker are aligned; an over-estimation module, configured to determine a first residual echo masking value of the microphone signal, calculate a noise masking value of the microphone signal based on the first speech masking value and the first residual echo masking value, and over-estimate the first residual echo masking value to obtain a second residual echo masking value; The output module is configured to output a second speech masking value according to the noise masking value and the second residual echo masking value, so as to synthesize a speech enhancement signal of the microphone signal based on the second speech masking value.

9. A post-filtering device for a speech enhancement network, wherein: include: memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the post-filtering method of the speech enhancement network as described in any one of claims 1-7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, which, when executed by a computer processor, are configured to execute the post-filtering method of a speech enhancement network as described in any one of claims 1-7.

11. A computer program product, wherein: The computer program product includes instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the post-filtering method of the speech enhancement network as described in any one of claims 1-7.

Images