A voice dereverberation method, device and electronic equipment

Abstract

The application provides a voice dereverberation method, device and electronic equipment, and belongs to the technical field of voice signal processing. The voice dereverberation method comprises the following steps: inputting reverberation voice signals collected by each voice collection sensor in a voice collection array into a dereverberation device to obtain an estimated voice spectrum signal and an estimated reverberation component corresponding to the reverberation voice signals; and removing residual reverberation in the estimated voice spectrum signal through a residual reverberation suppressor to obtain a dereverberation voice signal, so that a large amount of residual reverberation remaining in the reverberation voice signal after the removal of the reverberation can be removed, the quality of the voice signal is improved, and a clearer dereverberation voice signal is obtained.

Description

Technical Field

[0001] This application belongs to the field of speech signal processing technology, specifically relating to a speech de-eruption method, apparatus, and electronic device. Background Technology

[0002] Reverberation is an unavoidable acoustic phenomenon in indoor environments, causing the reverberant speech signal acquired by voice acquisition equipment to contain a large amount of reverberation components, thus affecting the overall voice communication effect.

[0003] Related technologies employ filter-based dereverberators to remove the reverberation component from reverberant speech signals. However, due to computational complexity limitations, the reverberant speech signal after dereverberation still retains a significant amount of residual reverberation. Summary of the Invention

[0004] In view of this, this application provides a speech de-reverberation method, apparatus and electronic device to solve the problem that the reverberated speech signal still retains a large amount of residual reverberation after reverberation removal.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] A first aspect provides a speech denoising method applied to a speech acquisition device, the speech acquisition device being equipped with a speech acquisition array consisting of a plurality of speech acquisition sensors; and the speech acquisition device implementing a denoiser for removing the reverberation component of the reverberant speech signal acquired by the speech acquisition sensors, and a residual reverberation suppressor for removing the residual reverberation component in the reverberant speech signal after removing the reverberation component; the method includes:

[0007] The reverberant speech signals collected by each speech acquisition sensor in the speech acquisition array are input into the dereverberator to obtain the estimated speech spectrum signal and the estimated reverberation component corresponding to the reverberant speech signal.

[0008] The estimated speech spectrum signal and the estimated reverberation component are passed through the residual reverberation suppressor to remove residual reverberation from the estimated speech spectrum signal, resulting in a de-reverberated speech signal.

[0009] Optionally, the dereverberator employs an adaptive weighted prediction error algorithm based on an adaptive filter.

[0010] Optionally, the step of removing residual reverberation from the estimated speech spectrum signal and the estimated reverberation component using the residual reverberation suppressor to obtain a dereverberated speech signal includes:

[0011] The estimated speech spectrum signal and the estimated reverberation component are input into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal. The residual reverberation in the estimated speech spectrum signal is then removed based on the residual reverberation factor to obtain a de-reverberated speech signal.

[0012] Optionally, the residual reverberation suppressor is a residual reverberation suppressor implemented on the speech acquisition device using a signal processing method based on an adaptive filter;

[0013] The step of inputting the estimated speech spectrum signal and the estimated reverberation component into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal includes:

[0014] The estimated speech spectrum signal and the estimated reverberation component are input into the residual reverberation suppressor. The residual reverberation factor corresponding to the estimated speech spectrum signal is obtained by using a signal processing method based on an adaptive filter and calculating based on the power spectral density of the estimated reverberation component.

[0015] Optionally, the adaptive filter is a Wiener filter; the residual reverberation factor is a weighting parameter of the Wiener filter.

[0016] Optionally, the signal processing method based on an adaptive filter calculates the residual reverberation factor corresponding to the estimated speech spectrum signal based on the power spectral density of the estimated reverberation component, including:

[0017] Based on the power spectral density of the estimated reverberation component, calculate the power spectral density of the residual reverberation component that is proportional to the power spectral density of the estimated reverberation component.

[0018] The power spectral density of the residual reverberation component is calculated using the average smoothing method to obtain the estimated power spectral density of the residual reverberation component.

[0019] The ratio of the power spectral density of the estimated speech spectrum signal to that of the estimated residual reverberation component is calculated to obtain the posterior direct reverberation power ratio.

[0020] The a prior direct reverberation power ratio is obtained by calculating the posterior direct reverberation power ratio using a decision-oriented method.

[0021] The weighting parameters of the Wiener filter are calculated based on the prior direct reverberation power ratio.

[0022] Optionally, calculating the power spectral density of the residual reverberation component, which is proportional to the power spectral density of the estimated reverberation component, based on the power spectral density of the estimated reverberation component, includes:

[0023] The power spectral density of the residual reverberation component is calculated using the following formula.

[0024]

[0025] Where μ is a preset proportionality constant that is greater than zero. To estimate the reverberation components The power spectral density.

[0026] Optionally, the step of calculating the power spectral density of the residual reverberation component using the average smoothing method to obtain the estimated power spectral density of the residual reverberation component includes:

[0027] The power spectral density of the estimated residual reverberation component is calculated using the following formula.

[0028]

[0029] Where p is the smoothing factor. This is the power spectral density of the estimated residual reverberation component from the previous frame.

[0030] Optionally, the step of calculating the posterior direct reverberation power ratio using a decision-oriented method to obtain the prior direct reverberation power ratio includes:

[0031] The a priori direct reverberation power ratio is calculated using the following formula.

[0032]

[0033] Where q is the smoothing factor; For the a posteriori direct reverberation power ratio; The posterior direct reverberation power ratio of the previous frame; ξ is the weighting parameter of the Wiener filter used to suppress the residual reverberation component of the estimated speech spectrum signal from the previous frame. min This is the lower boundary of the a priori direct reverberation power ratio.

[0034] Optionally, calculating the weight parameters of the Wiener filter based on the prior direct reverberation power ratio includes:

[0035] The weight parameters of the Wiener filter are calculated using the following formula.

[0036]

[0037] in, The a priori direct reverberation power ratio.

[0038] Optionally, removing residual reverberation from the estimated speech spectrum signal based on the residual reverberation factor includes:

[0039] The estimated speech spectrum signal is multiplied by the weighting parameters of the corresponding Wiener filter to obtain the dereverberated speech signal.

[0040] Optionally, the residual reverberation suppressor is a mask-based neural network; the mask-based neural network is a neural network used to calculate a mask as the residual reverberation factor corresponding to the estimated speech spectrum signal.

[0041] Optionally, the neural network includes a complex number of convolutional subnetworks and a complex number of subband recurrent subnetworks. The complex number of convolutional subnetworks is used to obtain the frequency correlation of the input at each time point, and the complex number of subband recurrent subnetworks is used to obtain the temporal correlation of the input at each frequency point.

[0042] Optionally, the step of inputting the estimated speech spectrum signal and the estimated reverberation component into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal includes:

[0043] The estimated speech spectrum signal and the estimated reverberation component are divided into real and imaginary parts and then concatenated.

[0044] The concatenation results are sequentially input into the complex convolutional subnetwork and the complex subband recurrent subnetwork;

[0045] The output of the complex subband cyclic subnetwork is converted to the target dimension size, and the mask corresponding to the estimated speech spectrum signal is calculated.

[0046] Optionally, the complex convolutional subnetwork includes 2-dimensional complex convolutional layers, complex block normalization layers, and complex activation function layers.

[0047] Optionally, the activation function layer uses the PReLU activation function.

[0048] Optionally, the complex subband cyclic subnetwork adopts a 2-layer complex long short-term memory model for each frequency point.

[0049] Optionally, the complex long short-term memory model includes 256 hidden units.

[0050] Optionally, removing residual reverberation from the estimated speech spectrum signal based on the residual reverberation factor includes:

[0051] The estimated speech spectrum signal is multiplied by the mask to obtain the dereverberated speech signal.

[0052] Secondly, a speech de-reverberation device is provided, the device comprising:

[0053] The first dereverberation module is used to input the reverberated speech signals collected by each speech acquisition sensor in the speech acquisition array into the dereverberator to obtain the estimated speech spectrum signal and the estimated reverberation component corresponding to the reverberated speech signal.

[0054] The second dereverberation module removes residual reverberation from the estimated speech spectrum signal and the estimated reverberation component by passing them through the residual reverberation suppressor to obtain a dereverberated speech signal.

[0055] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0056] Fourthly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect.

[0057] In the technical solution of this application embodiment, the reverberant speech signal acquired by the speech acquisition sensor is processed by a dereverberator to remove reverberation, and then by a residual reverberation factor to remove residual reverberation. This process removes a large amount of residual reverberation from the reverberant speech signal after reverberation removal, thereby improving the quality of the speech signal and obtaining a clearer dereverberant speech signal. Attached Figure Description

[0058] Figure 1 This is a schematic flowchart illustrating a speech dereverberation method according to an exemplary embodiment of this application;

[0059] Figure 2 This is a schematic diagram of a speech dereverberation method according to an exemplary embodiment of this application;

[0060] Figure 3 This is a schematic diagram of a neural network architecture shown in an exemplary embodiment of this application;

[0061] Figure 4 This is a schematic diagram of the structure of a speech de-reverberation device shown in an exemplary embodiment of this application;

[0062] Figure 5 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0064] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0065] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0066] It should be noted that the voice acquisition device can be an electronic device equipped with a voice acquisition array consisting of several voice acquisition sensors. Each voice acquisition sensor acquires a reverberant speech signal when collecting voice signals indoors. This may include components via direct transmission paths and early reflection paths. Components of the late reflection path and noise components

[0067] Specifically, the reverberant speech signal can be obtained using the following formula.

[0068]

[0069] Where m is the m-th voice acquisition sensor in the voice acquisition array; n is used to indicate the frame where the reverberant voice signal is located; and ω is used to indicate the frequency point where the reverberant voice signal is located.

[0070] Based on the above analysis, the reverberant speech signal Components of the late reflection path in and noise components The reverberation component is the reverberation component in the acquired reverberant speech signal.

[0071] In related technologies, an adaptive filter-based dereverberation is used to remove the reverberation component from the reverberant speech signal. However, due to limitations in computational complexity, the reverberant speech signal after removing the reverberation component still retains a large amount of reverberation, which is called residual reverberant.

[0072] In view of this, this specification proposes to add a residual reverberant suppression (RRS) after the dereverberator to remove residual reverberation in the reverberant speech signal.

[0073] In implementation, the reverberant speech signals collected by each speech acquisition sensor can be input into a dereverberator to obtain an estimated speech spectrum and estimated reverberation components corresponding to the reverberant speech signal; the estimated speech spectrum and estimated reverberation components are then passed through the residual reverberation suppressor to remove residual reverberation from the estimated speech spectrum signal to obtain a dereverberant speech signal.

[0074] In the above technical solution, the acquired reverberant speech signal is passed sequentially through a de-reverberator and a residual reverberation suppressor. The residual reverberation suppressor removes the residual reverberation component in the reverberant speech signal after the reverberation component has been removed, thereby effectively suppressing the residual reverberation in the speech signal and improving the quality of the speech signal.

[0075] It should be noted that the signals in this specification are all illustrative examples of signals in the short-time Fourier transform (STFT) domain.

[0076] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0077] Please see Figure 1 and Figure 2 , Figure 1 and Figure 2This application illustrates a speech de-reverberation method provided in an embodiment. The method can be applied to a speech acquisition device equipped with a speech acquisition array consisting of a plurality of speech acquisition sensors; and the speech acquisition device implements a de-reverberator for removing the reverberation component of the reverberant speech signal acquired by the speech acquisition sensors, and a residual reverberation suppressor for removing the residual reverberation component in the reverberant speech signal after removing the reverberation component.

[0078] It should be noted that the dereverberator and residual reverberation suppressor can be a hardware-based speech signal processor or a software program for implementing speech signal processing; no specific limitation is made here.

[0079] The speech de-reverberation method includes the following steps.

[0080] S110. Input the reverberant speech signals collected by each speech acquisition sensor in the speech acquisition array into the de-reverberator to obtain the estimated speech spectrum signal and the estimated reverberation component corresponding to the reverberant speech signal.

[0081] The voice acquisition device can acquire reverberant voice signals collected by various voice acquisition sensors. After performing a short-time Fourier transform on the acquired reverberant voice signals, the transformed reverberant voice signals can be... The signal is input to a de-reverb unit for de-reverb processing to remove the reverb component from the reverb speech signal.

[0082] After removing the reverberation component from a reverberant speech signal, a de-reverberator can output an estimated speech spectrum signal after removing the reverberation component. and the estimated reverberation component in the reverberant speech signal

[0083] The estimated speech spectrum signal can be obtained by removing the estimated reverberation component from the reverberant speech signal, as expressed by the following formula:

[0084]

[0085] The dereverberator can employ a dereverberation algorithm based on an adaptive filter; for example, it can employ an adaptive filter-based weighted prediction error (AWPE) algorithm, or a multichannel adaptive weighted prediction error (MultichannelAWPE) algorithm based on an adaptive filter.

[0086] S120. The estimated speech spectrum signal and the estimated reverberation component are passed through the residual reverberation suppressor to remove the residual reverberation in the estimated speech spectrum signal, thereby obtaining a de-reverberated speech signal.

[0087] Since the estimated speech spectrum signal obtained after the dereverberation process, which has removed the estimated reverberation component, still retains a large amount of residual reverberation, in order to remove the residual reverberation in the estimated speech spectrum signal, the estimated speech spectrum signal and the estimated reverberation component can be input into a residual reverberation suppressor for residual reverberation suppression processing, so as to further remove the residual reverberation in the estimated speech spectrum signal.

[0088] After removing residual reverberation from the estimated speech spectrum signal, the residual reverberation suppressor can output a dereverberanted speech signal.

[0089] It should be noted that the residual reverberation suppressor can be implemented in various ways, and no particular limitation is made in this specification.

[0090] For example, in one implementation, residual reverberation in the estimated speech spectrum signal can be removed by calculating the residual reverberation factor corresponding to the estimated speech spectrum signal. In this implementation, the estimated speech spectrum signal and the estimated reverberation component can be input into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal, and the residual reverberation in the estimated speech spectrum signal can be removed based on the residual reverberation factor to obtain a dérated speech signal.

[0091] Specifically, the aforementioned residual reverberation factor may include the residual reverberation factor corresponding to each frame and each frequency point of the estimated speech spectrum signal; or, it may include the residual reverberation factor corresponding to each frequency point of the estimated speech spectrum signal.

[0092] It should be noted that in practical applications, the residual reverberation factor can be a value between 0 and 1.

[0093] After inputting the estimated speech spectrum signal and the estimated reverberation component into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal, the estimated speech spectrum signal at each frame and each frequency point can be multiplied with the corresponding residual reverberation factor to obtain the de-reverberated speech signal at the corresponding frame and frequency point.

[0094] In the technical solution of this application embodiment, the reverberant speech signal acquired by the speech acquisition sensor is processed by a dereverberator to remove reverberation, and then by a residual reverberation factor to remove residual reverberation. This process removes a large amount of residual reverberation from the reverberant speech signal after reverberation removal, thereby improving the quality of the speech signal and obtaining a clearer dereverberant speech signal.

[0095] Based on the above embodiments, the residual reverberation suppressor in this application embodiment is used to remove residual reverberation from the estimated speech spectrum signal by calculating the residual reverberation factor corresponding to the estimated speech spectrum signal based on the input estimated speech spectrum signal and the estimated reverberation component, and removing the residual reverberation from the estimated speech spectrum signal based on the residual reverberation factor to obtain a de-reverberated speech signal.

[0096] It should be noted that there are various ways to implement a residual reverberation suppressor on a voice acquisition device, and this specification does not impose any particular limitations; only two specific implementation methods are given in this application. These are a signal processing (SP) method based on an adaptive filter and a neural network (NN) based on a mask.

[0097] In one embodiment shown, the aforementioned residual reverberation suppressor can be implemented on a speech acquisition device using a signal processing method based on an adaptive filter. If the residual reverberation suppressor is a residual reverberation suppressor implemented on a speech acquisition device using a signal processing method based on an adaptive filter, the residual reverberation factor corresponding to the estimated speech spectrum signal can be obtained by calculating based on the power spectrum density (PSD) of the estimated reverberation component using the signal processing method based on an adaptive filter.

[0098] The type of adaptive filter is not specifically limited in this specification and can be set according to actual needs.

[0099] In one embodiment shown, the adaptive filter can be a Wiener filter; correspondingly, the residual reverberation factor corresponding to the estimated speech spectrum signal calculated by the signal processing method based on the Wiener filter can be the weighting parameter of the Wiener filter. In this case, the residual reverberation suppressor adopts a Wiener filter-based signal processing method. It can calculate the weight parameters of the Wiener filter corresponding to the estimated speech spectrum signal by using the input estimated speech spectrum signal and the estimated reverberation component, and calculate the dereverberated speech signal based on the weight parameters of the Wiener filter.

[0100] The specific calculation process is illustrated in the following example.

[0101] 1. Due to the estimation of the power spectral density of the reverberant component and the power spectral density of the residual reverberant component. Since they are proportional, the power spectral density of the residual reverberation component, which is proportional to the power spectral density of the estimated reverberation component, can be calculated based on the power spectral density of the estimated reverberation component.

[0102] Specifically, the power spectral density of the residual reverberation component can be calculated using the following formula.

[0103]

[0104] Where μ is a preset proportionality constant greater than 0, which can be a fixed value. To estimate the reverberation components The power spectral density.

[0105] 2. The power spectral density of the residual reverberation component can be obtained by averaging smoothing. The calculation is equivalent to averaging the power spectral density of the residual reverberation component to obtain an estimated power spectral density of the residual reverberation component.

[0106] Specifically, the power spectral density of the residual reverberation component can be estimated using the following formula.

[0107]

[0108] Where p is the smoothing factor. This is the power spectral density of the estimated residual reverberation component from the previous frame.

[0109] 3. Calculate the ratio of the power spectral density of the estimated speech spectrum signal to that of the estimated residual reverberation component to obtain the posterior direct-to-reverberation power ratio (posteriori DRR).

[0110] Specifically, the a posteriori direct reverberation power ratio can be calculated using the following formula.

[0111]

[0112] 4. The a prior direct reverberation power ratio is obtained by calculating the posterior direct reverberation power ratio using a decision-directed approach;

[0113] Specifically, the a priori direct reverberation power ratio can be calculated using the following formula.

[0114]

[0115] Where q is the smoothing factor; For the a posteriori direct reverberation power ratio; The posterior direct reverberation power ratio of the previous frame; ξ is the weighting parameter of the Wiener filter used to suppress the residual reverberation component of the estimated speech spectrum signal from the previous frame. min This is the lower boundary of the a priori direct reverberation power ratio.

[0116] 5. Calculate the weighting parameters of the Wiener filter based on the prior direct reverberation power ratio.

[0117] The weight parameters of the Wiener filter can be calculated using the following formula.

[0118]

[0119] in, The a priori direct reverberation power ratio.

[0120] 6. Multiply the estimated speech spectrum signal by the corresponding weight parameters of the Wiener filter to obtain the dereverberant speech signal.

[0121] Specifically, the de-reverberated speech signal can be calculated using the following formula.

[0122]

[0123] Based on the technical solution of the above embodiments, the voice acquisition device can employ an AWPE for reverberation removal and a residual reverberation suppressor (SP-RRS) implemented using a Wiener filter-based signal processing method for removing residual reverberation. That is, the voice de-reverberation method is implemented using the AWPE+SP-RRS method. Among them, the residual reverberation suppressor implemented using the Wiener filter-based signal processing method can calculate the weight parameters of the Wiener filter corresponding to the estimated speech spectrum signal output by the AWPE, and remove the residual reverberation in the estimated speech spectrum signal based on the weight parameters of the Wiener filter, thereby improving the quality of the voice signal.

[0124] In one embodiment shown, the aforementioned residual reverberation suppressor can also be implemented on the speech acquisition device using a mask-based neural network. It should be noted that the mask-based neural network is a neural network used to calculate the mask as the residual reverberation factor corresponding to the estimated speech spectrum signal.

[0125] If the above residual reverberation suppressor is implemented on the speech acquisition device using a mask-based neural network, the estimated speech spectrum signal and the estimated reverberation component can be input into the neural network, and a mask corresponding to the estimated speech spectrum signal can be output. Then, the residual reverberation in the estimated speech spectrum signal can be removed according to the mask to obtain the de-reverberated signal.

[0126] The network architecture of the neural network can be constructed according to actual needs, and no special limitations are made in this specification.

[0127] In one embodiment shown, the mask-based neural network may include a complex-valued convolutional sub-network and a complex-valued sub-band recurrent sub-network. The complex-valued convolutional sub-network can be used to obtain the frequency dependence of the input at various time points, and the complex-valued sub-band recurrent sub-network can be used to obtain the temporal dependencies of the input at various frequency points.

[0128] Please see Figure 3 , Figure 3 This illustrates the overall architecture of a mask-based neural network provided in an embodiment of this application.

[0129] First, the input estimated speech spectrum signal and estimated reverberation component can be concatenated along a preset feature dimension; for example, such as Figure 3 As shown, the estimated speech spectrum signal and the estimated reverberation component are divided into real part and image part respectively and then concatenated.

[0130] The above splicing results are input into a complex convolutional subnetwork for processing. The complex convolutional subnetwork is used to obtain the frequency correlation of the input at each time point.

[0131] The specific architecture of the complex convolutional subnetwork can be constructed according to actual needs, and this specification does not impose any particular limitations. For example, the complex convolutional subnetwork may include a 2D complex convolutional layer, a complex batch normalization layer, and a complex activation function layer.

[0132] The number of convolutional layers in the complex convolutional subnet can be configured according to actual needs, and this specification does not impose any special limitations. For example, 32 convolutional kernels of size 31×2 can be used, with a stride of 2×1. In practical applications, larger convolutional kernels can also be used to construct convolutional layers to obtain a wider range of frequency correlations.

[0133] The activation function used in the activation function layer can be set according to actual needs, and this specification does not impose any special limitations. For example, the Parametric Rectified Linear Unit (PReLU) activation function can be used.

[0134] After the complex convolutional subnetwork processes the above-mentioned concatenation result, the output of the complex convolutional subnetwork can be used as the input of the complex subband cyclic subnetwork for further processing. The complex subband cyclic subnetwork is used to obtain the temporal correlation of the input at various frequency points.

[0135] The architecture of the complex subband cyclic subnetwork can be set according to actual needs, and this specification does not impose any special limitations. For example, the complex subband cyclic subnetwork may include a two-layer complex long short-term memory (LSTM) model for each frequency point, and the complex LSTM model may include a preset number of hidden units; for example, including 256 hidden units, etc.

[0136] After further processing the output of the complex convolutional subnetwork by the complex subband recurrent subnetwork, the output of the complex subband recurrent subnetwork can be converted to the target dimension. The conversion process can be customized according to actual needs; for example, it can employ methods such as... Figure 3 The linear processing method shown processes the data into real and imaginary parts and calculates the mask corresponding to the estimated speech spectrum signal.

[0137] Finally, the estimated speech spectrum signal is multiplied by the mask calculated by the neural network to obtain the dereverberated speech signal.

[0138] It should be noted that the above-mentioned neural network needs to be pre-trained. There are various ways to train the neural network. For example, during the training process, the scale-invariant signal-to-noise ratio (SI-SNR) loss function can be used in the time domain to calculate the estimated speech spectrum signal to approximate the dereverberated speech signal.

[0139] Based on the technical solution of the above embodiments, the speech acquisition device can use AWPE for reverberation removal and a mask-based neural network residual reverberation suppressor (NN-RRS) for residual reverberation removal, that is, the speech de-reverberation method is implemented by using AWPE+NN-RRS; wherein, the mask-based neural network can calculate the mask corresponding to the estimated speech spectrum signal output by AWPE, and remove the residual reverberation in the estimated speech spectrum signal based on the mask, thereby improving the quality of the speech signal.

[0140] Corresponding to the aforementioned embodiments of the speech dereverberation method, this application also provides embodiments of a speech dereverberation device.

[0141] like Figure 4 As shown, the voice de-reverberation device includes: a first de-reverberation module 401 and a second de-reverberation module 402.

[0142] The first dereverberation module 401 is used to input the reverberant speech signals collected by each speech acquisition sensor in the speech acquisition array into the dereverberator to obtain an estimated speech spectrum signal and an estimated reverberation component corresponding to the reverberant speech signal; the second dereverberation module 402 removes the residual reverberation in the estimated speech spectrum signal by passing the estimated speech spectrum signal and the estimated reverberation component through the residual reverberation suppressor to obtain a dereverberated speech signal.

[0143] Optionally, the dereverberator employs an adaptive weighted prediction error algorithm based on an adaptive filter.

[0144] Optionally, the step of removing residual reverberation from the estimated speech spectrum signal and the estimated reverberation component using the residual reverberation suppressor to obtain a dereverberated speech signal includes:

[0145] The estimated speech spectrum signal and the estimated reverberation component are input into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal. The residual reverberation in the estimated speech spectrum signal is then removed based on the residual reverberation factor to obtain a de-reverberated speech signal.

[0146] In the technical solution of this application embodiment, the reverberant speech signal acquired by the speech acquisition sensor is processed by a dereverberator to remove reverberation, and then by a residual reverberation factor to remove residual reverberation. This process removes a large amount of residual reverberation from the reverberant speech signal after reverberation removal, thereby improving the quality of the speech signal and obtaining a clearer dereverberant speech signal.

[0147] Optionally, the residual reverberation suppressor is a residual reverberation suppressor implemented on the speech acquisition device using a signal processing method based on an adaptive filter;

[0148] The second de-reverberation module 402 is used to input the estimated speech spectrum signal and the estimated reverberation component into the residual reverberation suppressor, and to calculate the residual reverberation factor corresponding to the estimated speech spectrum signal using a signal processing method based on an adaptive filter based on the power spectral density of the estimated reverberation component.

[0149] Optionally, the adaptive filter is a Wiener filter; the residual reverberation factor is a weighting parameter of the Wiener filter.

[0150] Optionally, the second dereverberation module 402 is used for:

[0151] Based on the power spectral density of the estimated reverberation component, calculate the power spectral density of the residual reverberation component that is proportional to the power spectral density of the estimated reverberation component.

[0152] The power spectral density of the residual reverberation component is calculated using the average smoothing method to obtain the estimated power spectral density of the residual reverberation component.

[0153] The ratio of the power spectral density of the estimated speech spectrum signal to that of the estimated residual reverberation component is calculated to obtain the posterior direct reverberation power ratio.

[0154] The a prior direct reverberation power ratio is obtained by calculating the posterior direct reverberation power ratio using a decision-oriented method.

[0155] The weighting parameters of the Wiener filter are calculated based on the prior direct reverberation power ratio.

[0156] Optionally, calculating the power spectral density of the residual reverberation component, which is proportional to the power spectral density of the estimated reverberation component, based on the power spectral density of the estimated reverberation component, includes:

[0157] The power spectral density of the residual reverberation component is calculated using the following formula.

[0158]

[0159] Where μ is a preset proportionality constant that is greater than zero. To estimate the reverberation components The power spectral density.

[0160] Optionally, the second dereverberation module 402 is used to calculate the power spectral density of the estimated residual reverberation component using the following formula.

[0161]

[0162] Where p is the smoothing factor. This is the power spectral density of the estimated residual reverberation component from the previous frame.

[0163] Optionally, the second dereverberation module 402 is used to calculate the prior direct reverberation power ratio using the following formula.

[0164]

[0165] Where q is the smoothing factor; For the a posteriori direct reverberation power ratio; The posterior direct reverberation power ratio of the previous frame; ξ is the weighting parameter of the Wiener filter used to suppress the residual reverberation component of the estimated speech spectrum signal from the previous frame. min This is the lower boundary of the a priori direct reverberation power ratio.

[0166] Optionally, the second dereverberation module 402 is used to calculate the weight parameters of the Wiener filter using the following formula.

[0167]

[0168] in, The a priori direct reverberation power ratio.

[0169] In this embodiment of the application, the weight parameters of the Wiener filter corresponding to the estimated speech spectrum signal output by the Wiener filter can be calculated by the signal processing method based on the Wiener filter, and the residual reverberation in the estimated speech spectrum signal can be removed based on the weight parameters of the Wiener filter, thereby improving the quality of the speech signal.

[0170] Based on the above embodiments, optionally, the residual reverberation suppressor is a mask-based neural network; the mask-based neural network is a neural network used to calculate a mask as the residual reverberation factor corresponding to the estimated speech spectrum signal.

[0171] Optionally, the neural network includes a complex number of convolutional subnetworks and a complex number of subband recurrent subnetworks. The complex number of convolutional subnetworks is used to obtain the frequency correlation of the input at each time point, and the complex number of subband recurrent subnetworks is used to obtain the temporal correlation of the input at each frequency point.

[0172] Optionally, the step of inputting the estimated speech spectrum signal and the estimated reverberation component into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal includes:

[0173] The estimated speech spectrum signal and the estimated reverberation component are divided into real and imaginary parts and then concatenated.

[0174] The concatenation results are sequentially input into the complex convolutional subnetwork and the complex subband recurrent subnetwork;

[0175] The output of the complex subband cyclic subnetwork is converted to the target dimension size, and the mask corresponding to the estimated speech spectrum signal is calculated.

[0176] Optionally, the complex convolutional subnetwork includes 2-dimensional complex convolutional layers, complex block normalization layers, and complex activation function layers.

[0177] Optionally, the activation function layer uses the PReLU activation function.

[0178] Optionally, the complex subband cyclic subnetwork adopts a 2-layer complex long short-term memory model for each frequency point.

[0179] Optionally, the complex long short-term memory model includes 256 hidden units.

[0180] Optionally, removing residual reverberation from the estimated speech spectrum signal based on the residual reverberation factor includes:

[0181] The estimated speech spectrum signal is multiplied by the mask to obtain the dereverberated speech signal.

[0182] The embodiments of this application use a mask-based neural network to calculate a mask corresponding to the estimated speech spectrum signal output by AWPE, and remove residual reverberation in the estimated speech spectrum signal based on the mask, thereby improving the quality of the speech signal.

[0183] The embodiments of the speech de-reverberation device of this application can be applied to electronic devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the electronic device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5The diagram shown is a hardware structure diagram of the electronic device containing the speech de-reverberation device of this application. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device is located in the embodiment may also include other hardware depending on the actual function of the electronic device, which will not be described in detail here.

[0184] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0185] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0186] This application also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps of the speech de-reverberation method described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0187] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0188] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0189] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0190] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0191] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0192] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0193] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0194] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0195] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A speech dereverberation method, characterized in that, The method is applied to a voice acquisition device, which is equipped with a voice acquisition array consisting of a plurality of voice acquisition sensors; and the voice acquisition device implements a de-reverberator for removing the reverberation component of the reverberant voice signal acquired by the voice acquisition sensors, and a residual reverberation suppressor for removing the residual reverberation component in the reverberant voice signal after removing the reverberation component; the method includes: The reverberant speech signals collected by each speech acquisition sensor in the speech acquisition array are input into the dereverberator to obtain the estimated speech spectrum signal and the estimated reverberation component corresponding to the reverberant speech signal. The estimated speech spectrum signal and the estimated reverberation component are input into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal. The residual reverberation in the estimated speech spectrum signal is then removed based on the residual reverberation factor to obtain a de-reverberated speech signal.

2. The method according to claim 1, characterized in that, The dereverberator employs an adaptive weighted prediction error algorithm based on an adaptive filter.

3. The method according to claim 1, characterized in that, The residual reverberation suppressor is a residual reverberation suppressor implemented on the voice acquisition device using a signal processing method based on an adaptive filter; The step of inputting the estimated speech spectrum signal and the estimated reverberation component into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal includes: The estimated speech spectrum signal and the estimated reverberation component are input into the residual reverberation suppressor. The residual reverberation factor corresponding to the estimated speech spectrum signal is obtained by using a signal processing method based on an adaptive filter and calculating based on the power spectral density of the estimated reverberation component.

4. The method according to claim 3, characterized in that, The adaptive filter is a Wiener filter; the residual reverberation factor is the weighting parameter of the Wiener filter.

5. The method according to claim 4, characterized in that, The signal processing method based on an adaptive filter calculates the residual reverberation factor corresponding to the estimated speech spectrum signal based on the power spectral density of the estimated reverberation component, including: Based on the power spectral density of the estimated reverberation component, calculate the power spectral density of the residual reverberation component that is proportional to the power spectral density of the estimated reverberation component. The power spectral density of the residual reverberation component is calculated using the average smoothing method to obtain the estimated power spectral density of the residual reverberation component. The ratio of the power spectral density of the estimated speech spectrum signal to that of the estimated residual reverberation component is calculated to obtain the posterior direct reverberation power ratio. The a prior direct reverberation power ratio is obtained by calculating the posterior direct reverberation power ratio using a decision-oriented method. The weighting parameters of the Wiener filter are calculated based on the prior direct reverberation power ratio.

6. The method according to claim 5, characterized in that, The step of calculating the power spectral density of the residual reverberation component, which is proportional to the power spectral density of the estimated reverberation component, based on the power spectral density of the estimated reverberation component includes: The power spectral density of the residual reverberation component is calculated using the following formula. : in, A preset proportionality constant that is greater than zero. To estimate the reverberation components The power spectral density.

7. The method of claim 6, characterized in that, The calculation of the power spectral density of the residual reverberation component using the average smoothing method to obtain the estimated power spectral density of the residual reverberation component includes: The power spectral density of the estimated residual reverberation component is calculated using the following formula. : in, As a smoothing factor, This is the power spectral density of the estimated residual reverberation component from the previous frame.

8. The method according to claim 5, characterized in that, The calculation of the posterior direct reverberation power ratio using a decision-oriented method to obtain the prior direct reverberation power ratio includes: The a priori direct reverberation power ratio is calculated using the following formula. : in, It is a smoothing factor; For the a posteriori direct reverberation power ratio; The posterior direct reverberation power ratio of the previous frame; The weight parameters of the Wiener filter are used to suppress the residual reverberation component of the estimated speech spectrum signal from the previous frame. This is the lower boundary of the a priori direct reverberation power ratio.

9. The method according to claim 5, characterized in that, The calculation of the weight parameters of the Wiener filter based on the prior direct reverberation power ratio includes: The weight parameters of the Wiener filter are calculated using the following formula. : in, The a priori direct reverberation power ratio.

10. The method according to claim 5, characterized in that, The step of removing residual reverberation from the estimated speech spectrum signal based on the residual reverberation factor includes: The estimated speech spectrum signal is multiplied by the weighting parameters of the corresponding Wiener filter to obtain the dereverberated speech signal.

11. The method according to claim 1, characterized in that, The residual reverberation suppressor is a mask-based neural network; the mask-based neural network is a neural network used to calculate the mask as the residual reverberation factor corresponding to the estimated speech spectrum signal.

12. The method according to claim 11, characterized in that, The neural network includes complex convolutional subnetworks and complex subband recurrent subnetworks. The complex convolutional subnetworks are used to obtain the frequency correlation of the input at each time point, and the complex subband recurrent subnetworks are used to obtain the temporal correlation of the input at each frequency point.

13. The method according to claim 12, characterized in that, The step of inputting the estimated speech spectrum signal and the estimated reverberation component into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal includes: The estimated speech spectrum signal and the estimated reverberation component are divided into real and imaginary parts and then concatenated. The concatenation results are sequentially input into the complex convolutional subnetwork and the complex subband recurrent subnetwork; The output of the complex subband cyclic subnetwork is converted to the target dimension size, and the mask corresponding to the estimated speech spectrum signal is calculated.

14. The method according to claim 13, characterized in that, The complex convolutional subnetwork includes 2-dimensional complex convolutional layers, complex block normalization layers, and complex activation function layers.

15. The method according to claim 14, characterized in that, The activation function layer uses the PReLU activation function.

16. The method according to claim 13, characterized in that, The complex subband cyclic subnetwork employs a 2-layer complex long short-term memory model for each frequency point.

17. The method according to claim 16, characterized in that, The complex long short-term memory model includes 256 hidden units.

18. The method according to claim 13, characterized in that, The step of removing residual reverberation from the estimated speech spectrum signal based on the residual reverberation factor includes: The estimated speech spectrum signal is multiplied by the mask to obtain the dereverberated speech signal.

19. A speech de-reverberation device, characterized in that, An apparatus is applied to a voice acquisition device, the voice acquisition device being equipped with a voice acquisition array consisting of a plurality of voice acquisition sensors; and the voice acquisition device implementing a de-reverberation unit for removing the reverberation component of the reverberant voice signal acquired by the voice acquisition sensors, and a residual reverberation suppressor for removing the residual reverberation component in the reverberant voice signal after removing the reverberation component; the apparatus includes: The first dereverberation module is used to input the reverberated speech signals collected by each speech acquisition sensor in the speech acquisition array into the dereverberator to obtain the estimated speech spectrum signal and the estimated reverberation component corresponding to the reverberated speech signal. The second dereverberation module is used to input the estimated speech spectrum signal and the estimated reverberation component into the residual reverberation suppressor to obtain the residual reverberation factor corresponding to the estimated speech spectrum signal, and remove the residual reverberation in the estimated speech spectrum signal based on the residual reverberation factor to obtain the dereverberated speech signal.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-18.

21. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-18.

Images