Hybrid method and system for real-time speech dereverberation
By combining multi-beamforming and signal-to-noise ratio weighting with frequency domain coherence coefficient processing, the trade-off between computational complexity and dereverberation performance in existing technologies is solved, achieving real-time speech dereverberation and making it suitable for microphone array conference microphone devices.
Patent Information
- Application Number
- CN202111567537.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-12-20
AI Technical Summary
Existing technologies struggle to strike a good balance between computational complexity and dereverberation performance, especially in conference microphone devices with limited computing resources, where real-time speech dereverberation is difficult to achieve.
By employing multi-beamforming and signal-to-noise ratio weighting, and combining the normalized frequency domain coherence coefficients of the current frame and historical frames, two-stage dereverberation is performed through spectral subtraction and Wiener filtering, which reduces computational complexity and improves the accuracy of mid-to-late stage reverberation component estimation.
While reducing computational complexity, it significantly improves dereverberation performance, making it suitable for microphone array conference microphone devices with limited computing resources, and achieving real-time voice dereverberation.
Smart Images

Figure CN114255777B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech processing technology, and in particular to a hybrid method and system for real-time speech dereverberation. Background Art
[0002] In recent years, users have put forward more requirements for voice quality in real-time conferences. Reverberation is a key factor affecting voice clarity and intelligibility. The reverberation level of the environment in which users attend meetings varies greatly. Some conference rooms that have been well acoustically treated have very low reverberation levels, but other conference rooms with glass on all four sides and no specific acoustic treatment have heavy reverberation, which poses a great challenge to voice processing. At present, dereverberation methods mainly include deconvolution method, weighted linear prediction filter method, reverberation suppression method similar to noise reduction principle, and speech dereverberation method based on deep learning. Among the above methods, deconvolution method, weighted linear prediction filter method and speech dereverberation method based on deep learning are all difficult to run in real time due to the large amount of calculation, especially in conference microphone devices with weak main chip computing power and tight resources. Relatively speaking, the reverberation suppression method using the principle of noise reduction has a small amount of calculation. The difficulty lies in how to reasonably construct a model to accurately estimate and suppress the reverberation component in the voice. It is worth mentioning that the conference microphone equipment on the market generally adopts microphone array configuration, and the beamforming module in array signal processing naturally has a certain reverberation suppression capability.
[0003] In the prior art, a single-channel speech dereverberation method and device with patent number CN201210201879.7 uses an AR (autoregressive) model or an MA (moving average) model or an ARMA (autoregressive moving average) model to estimate the power spectrum of the mid- and late-stage reverberation components of the current frame from several previous frame signals, and then removes them by spectral subtraction. Although the coefficients in the model can be solved by the Yule-Walker equation or the Burg algorithm, the computational complexity is reduced to a certain extent compared to other types of dereverberation methods, but in a strong reverberation environment, as the model coefficients increase, the computational complexity will continue to increase, and it is also difficult to apply to conference hardware pickup equipment with scarce computing resources.
[0004] A method and device for de-reverberation of speech data with patent number CN201510401640.8 in the prior art is equivalent to a simplified version of the above invention patent, which only uses a frame before the current frame to estimate the mid- and late-stage reverberation components of the current frame, and then estimates the spectral subtraction gain coefficient at the current moment, and finally averages and weights the spectral subtraction gain coefficients within a certain time window to obtain the final spectral subtraction gain coefficient. Compared with the previous method, this method has been greatly simplified, and the computational complexity has been significantly reduced, but it is obviously not enough to estimate the mid- and late-stage reverberation components using only a single frame before the current frame, and the average weighted processing method of the spectral subtraction gain coefficients within a certain time window is also slightly rough.
[0005] Therefore, there is an urgent need to provide a dereverberation method that can achieve a better compromise between computational complexity and dereverberation performance and reduce the stringent requirements on hardware resources. Summary of the invention
[0006] To this end, the technical problem to be solved by the present invention is to overcome the problems existing in the prior art and propose a hybrid method and system for real-time speech dereverberation, which not only significantly reduces the amount of calculation brought by directly solving ARMA (autoregressive moving average) similar models, but also improves the accuracy of estimating mid- and late-stage reverberation components, and achieves a relatively good compromise between computational complexity and dereverberation performance, thereby achieving a balance between computational complexity and dereverberation effect and reducing the stringent requirements on hardware resources.
[0007] In order to solve the above technical problems, the present invention provides a hybrid method for real-time speech dereverberation, comprising the following steps:
[0008] S1: Acquire original speech signals of multiple microphone channels in a real-time conference scene, perform windowing, framing and fast Fourier transform preprocessing on the original speech signals, and obtain frequency domain speech signals of multiple microphone channels;
[0009] S2: performing fixed beamforming in multiple beam directions for frequency domain speech signals of multiple microphone channels to obtain multiple beam output frequency domain signals;
[0010] S3: performing beam fusion processing based on signal-to-noise ratio weighting on multiple beam output frequency domain signals to obtain a single-channel frequency domain speech signal after the first stage of dereverberation, and calculating a normalized frequency domain coherence coefficient of a current frame and a plurality of frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation;
[0011] S4: Estimate the mid- and late-stage reverberation power spectrum components of the current frame based on the normalized frequency domain coherence coefficient, suppress the mid- and late-stage reverberation power spectrum components of the current frame, obtain the speech spectrum after reverberation suppression, perform inverse fast Fourier transform on the speech spectrum after reverberation suppression, and obtain the dereverberated time domain speech signal.
[0012] In one embodiment of the present invention, fixed beamforming is performed on frequency domain speech signals of multiple microphone channels in multiple beam directions, including:
[0013] The frequency domain signals of multiple microphone channels are summed according to the beam direction and beamforming weight to obtain multiple beam output frequency domain signals Among them, X m (f), m = 1, 2, ..., M represents the complex spectrum of each microphone, W mb (f), b = 1, 2, ..., B represents the complex beamforming weight, Y b (f), b=1, 2, ..., B represents the output frequency domain signal of beamforming.
[0014] In one embodiment of the present invention, in S2, the division of the beam direction is pre-set. For a uniform circular array, the beam direction selects the direction of each mic. For a uniform linear array, the forward 180° horizontal azimuth angle is evenly divided into several beam ranges.
[0015] In one embodiment of the present invention, in S3, the normalized frequency domain coherence coefficient of the current frame and a plurality of frame signals in a previous period of time is calculated based on the single-channel frequency domain speech signal after the first stage of dereverberation, including:
[0016] The calculation formula of the normalized frequency domain coherence coefficient is as follows:
[0017] Y i P SD (f) = α2*Y i P SD (f)+(1-α2)*Y i (f)*conj(Y i (f)), i = P, ..., P - Q + 1
[0018] Y i C PSD (f) = α2*Y i C PSD (f)+(1-α2)*Y i (f)*conj(Y P(f)), i = P-1, ..., P-Q+1
[0019]
[0020] Among them, Y i (f) and Y i P SD (f) are the spectrum and autopower spectrum of the i-th frame, i = P, ..., P-Q+1, Y i C PSD (f), i≠P is the cross power spectrum between the i-th frame and the P-th frame, Coh i (f), i≠P is the normalized spectral coherence coefficient between the i-th frame and the P-th frame.
[0021] In one embodiment of the present invention, in S4, estimating the mid-late reverberation power spectrum component of the current frame includes:
[0022] The calculation formula of the mid-late reverberation power spectrum component of the current frame is: Where i = P-R+2, ..., P-Q+1, which corresponds to the historical frame used to estimate the mid- and late-stage reverberation components, Coh i (f), i≠P is the normalized spectral coherence coefficient between the i-th frame and the P-th frame.
[0023] In one embodiment of the present invention, in S4, after the mid-late reverberation power spectrum is calculated, a strong constraint is imposed on the mid-late reverberation power spectrum so that the estimated reverberation power spectrum is not greater than the instantaneous autopower spectrum of the current frame.
[0024] In addition, the present invention also provides a hybrid system for real-time speech dereverberation, comprising:
[0025] A data preprocessing module is used to obtain the original speech signals of multiple microphone channels in a real-time conference scene. The data preprocessing module is used to perform windowing, framing and fast Fourier transform preprocessing on the original speech signals to obtain frequency domain speech signals of multiple microphone channels.
[0026] A fixed beam forming module, wherein the fixed beam forming module is used to perform fixed beam forming in multiple beam directions for frequency domain speech signals of multiple microphone channels to obtain multiple beam output frequency domain signals;
[0027] A beam fusion module, wherein the beam fusion module is used to perform beam fusion processing based on signal-to-noise ratio weighting on multiple beam output frequency domain signals to obtain a single-channel frequency domain speech signal after the first stage of dereverberation, and calculate a normalized frequency domain coherence coefficient of a current frame and a plurality of frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation;
[0028] A nonlinear dereverberation module, the nonlinear dereverberation module is used to estimate the mid- and late-stage reverberation power spectrum components of the current frame based on the normalized frequency domain coherence coefficient, suppress the mid- and late-stage reverberation power spectrum components of the current frame, obtain the speech spectrum after reverberation suppression, perform inverse fast Fourier transform on the speech spectrum after reverberation suppression, and obtain the dereverberated time domain speech signal.
[0029] In one embodiment of the present invention, the data preprocessing module includes:
[0030] The Fourier transform unit is used to perform fast Fourier transform on the original speech signal after windowing and framing processing, and convert the time domain signal into a frequency domain signal.
[0031] In one embodiment of the present invention, the beam fusion module includes:
[0032] A normalized frequency domain coherence coefficient calculation unit is used to calculate the normalized frequency domain coherence coefficients of the current frame and several frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation.
[0033] In one embodiment of the present invention, the nonlinear dereverberation module comprises:
[0034] A mid-late reverberation power spectrum calculation unit is used to estimate the mid-late reverberation power spectrum components of the current frame.
[0035] The above technical solution of the present invention has the following advantages compared with the prior art:
[0036] The present invention provides a hybrid method for real-time speech dereverberation for a microphone array conference pickup device. The hybrid method first performs beamforming processing in multiple specific directions to obtain a multi-beam signal after the first stage of dereverberation, then performs signal-to-noise ratio weighting on the outputs in the above different directions to obtain a single-channel signal after the first stage of dereverberation, then uses several frames in a time window before the current frame to estimate the mid- and late-stage reverberation components of the current frame, and finally performs spectral subtraction or Wiener filtering processing to obtain a speech signal after the second stage of dereverberation; wherein after the first stage of dereverberation processing, the mid- and high-frequency components can be In order to obtain a more obvious dereverberation effect, in the second-level single-channel dereverberation stage, by calculating the frequency domain normalized coherence coefficient between the current frame and several frames in the previous window, and estimating the contribution of a frame before the current frame to the mid- and late-stage reverberation components of the current frame, it not only significantly reduces the amount of calculation brought by directly solving ARMA (autoregressive moving average)-like models, but also improves the accuracy of estimating mid- and late-stage reverberation components, and achieves a better compromise between computational complexity and dereverberation performance, thereby achieving a balance between computational complexity and dereverberation effect and reducing the stringent requirements on hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.
[0038] Figure 1 It is a flow chart of the hybrid method for real-time speech dereverberation of the present invention.
[0039] Figure 2 It is a schematic diagram of fixed beam forming of the present invention.
[0040] Figure 3 It is a schematic diagram of multi-beam signal-to-noise ratio fusion weighting of the present invention.
[0041] Figure 4 It is a schematic diagram of reverberation component estimation of the present invention.
[0042] Figure 5 It is a schematic diagram of calculating the normalized frequency domain coherence coefficient of the present invention.
[0043] Figure 6 It is a schematic diagram of power spectrum estimation of late reverberation components in the present invention.
[0044] Figure 7 It is a hardware structure diagram of the hybrid system for real-time speech dereverberation of the present invention.
[0045] Figure 8 It is a speech spectrum diagram before dereverberation of the present invention.
[0046] Fig. 9 It is a speech spectrum diagram after dereverberation of the present invention.
[0047] The reference numerals are described as follows: 10, data preprocessing module; 20, fixed beam forming module; 30, beam fusion module; 40, nonlinear dereverberation module. DETAILED DESCRIPTION
[0048] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0049] Embodiment 1
[0050] See also Figure 1 As shown, this embodiment provides a hybrid method for real-time speech dereverberation, comprising the following steps:
[0051] S1: Acquire original speech signals of multiple microphone channels in a real-time conference scene, perform windowing, framing and fast Fourier transform preprocessing on the original speech signals, and obtain frequency domain speech signals of multiple microphone channels;
[0052] S2: performing fixed beamforming in multiple beam directions for frequency domain speech signals of multiple microphone channels to obtain multiple beam output frequency domain signals;
[0053] S3: performing beam fusion processing based on signal-to-noise ratio weighting on multiple beam output frequency domain signals to obtain a single-channel frequency domain speech signal after the first stage of dereverberation, and calculating a normalized frequency domain coherence coefficient of a current frame and a plurality of frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation;
[0054] S4: Estimate the mid- and late-stage reverberation power spectrum components of the current frame based on the normalized frequency domain coherence coefficient, suppress the mid- and late-stage reverberation power spectrum components of the current frame, obtain the speech spectrum after reverberation suppression, perform inverse fast Fourier transform on the speech spectrum after reverberation suppression, and obtain the dereverberated time domain speech signal.
[0055] In a hybrid method for real-time speech dereverberation disclosed in the present invention, the present invention provides a hybrid method for real-time speech dereverberation for a microphone array conference pickup device. The hybrid method first performs beamforming processing in multiple specific directions to obtain a multi-beam signal after the first stage of dereverberation, and then performs signal-to-noise ratio weighting on the outputs in the above different directions to obtain a single-channel signal after the first stage of dereverberation. Then, several frames within a time window before the current frame are used to estimate the mid- and late-stage reverberation components of the current frame, and finally spectral subtraction or Wiener filtering is performed to obtain a speech signal after the second stage of dereverberation; wherein the first stage After dereverberation processing, the mid- and high-frequency frequencies can obtain a relatively obvious dereverberation effect. In the second-level single-channel dereverberation stage, by calculating the frequency domain normalized coherence coefficient between the current frame and several frames in the previous window, and estimating the contribution of a certain frame before the current frame to the mid- and late-stage reverberation components of the current frame, it not only significantly reduces the amount of calculation brought by directly solving ARMA (autoregressive moving average)-like models, but also improves the accuracy of estimating mid- and late-stage reverberation components. A relatively good compromise is achieved between computational complexity and dereverberation performance, thereby achieving a balance between computational complexity and dereverberation effect and reducing the stringent requirements on hardware resources.
[0056] In a hybrid method for real-time speech dereverberation disclosed in the present invention, for S1 of the above-mentioned implementation mode, when the original speech signal is windowed and framed and pre-processed by fast Fourier transform, the window function firstly used can be a Hamming window, a Hanning window, a Blackman window, etc., and the range of the speech frame is generally 10 to 30 milliseconds. For example, the duration of each frame can be 10ms, 16ms, 20ms, 30ms, etc., and then the original speech signal of each microphone channel is fast Fourier transformed to convert the time domain signal into a frequency domain signal.
[0057] Specifically, assuming an M-element microphone array, taking the mth microphone as an example, the digital signal collected by it is x m (n). The window function sequence is defined as h(n), and the time domain data obtained by windowing and framing is X wm (n) = X m (n)*h(n), for X wm (n) Perform fast Fourier transform and obtain the corresponding complex spectrum as X m (f) = fft(x wm (n)), m=1,2,…,M.
[0058] In the hybrid method for real-time speech dereverberation disclosed in the present invention, for S2 of the above-mentioned embodiment, reference can be made to Figure 2 As shown in the figure, the frequency domain signals of multiple microphone channels are summed according to the beam direction and beamforming weight to obtain multiple beam output frequency domain signals Among them, Xm (f), m = 1, 2, ..., M represents the complex spectrum of each microphone, W mb (f), b = 1, 2, ..., B represents the complex beamforming weight, Y b (f), b=1, 2, ..., B represents the output frequency domain signal of beamforming.
[0059] Among them, the division of the above beam directions is pre-set. For a uniform circular array, the beam direction can select the direction of each mic. For a uniform linear array, the 180° horizontal azimuth angle can be evenly divided into several beam ranges. For other irregular arrays, specific beam directions can be specified according to the application scenario requirements. In addition, the above beamforming weight design method can adopt CBF (conventional beamforming), MVDR (minimum variance distortionless response), LCMV (linear constrained minimum variance), SD (super directivity) and differential beamforming and other design schemes. For example, beamforming is calculated using the following formula: The calculation of the multi-beam output speech spectrum based on signal-to-noise ratio weighting is adopted Or use Where φ(f) is the original input signal spectrum X of any microphone channel m (f) Phase spectrum.
[0060] In the hybrid method for real-time speech dereverberation disclosed in the present invention, for S4 of the above-mentioned embodiment, reference can be made to Figure 3 As shown, Y b (f), b = 1, 2, ..., B represents the output spectrum of each beam direction, Y c (f) represents the output spectrum after multi-beam fusion. Beam fusion can be implemented in the following two ways: Method 1: Among them, |Y b (f)| indicates Y b (f), φ(f) represents the spectrum of the original input signal of any microphone channel X m (f) Phase spectrum; Method 2: The above-mentioned background noise power spectrum and signal-to-noise ratio estimation may adopt the MCRA (minimum controlled recursive averaging) method of the prior art. The specific content of the MCRA method is not elaborated in detail in the present invention.
[0061] The above assumption is that the estimated background noise power spectrum is λ b (f), the speech power spectrum is Y b (f), the full-band signal-to-noise ratio is g b , the normalized signal-to-noise ratio is In order to highlight the beam signal in the main speech direction, it is necessary tob Perform the following processing: Weighting factor W b Initialize to Update method is W b =α1*W b +(1-α1)*q b ,
[0062] In the hybrid method for real-time speech dereverberation disclosed in the present invention, for S4 of the above-mentioned embodiment, reference can be made to Figure 4 As shown in Figure 2, this figure is used as an example to illustrate the historical frame information required to estimate the mid- and late-stage reverberation components of the current frame. Assuming t p is the current frame, t p-1 , t p-2 ,…,t p-R+1 is the historical frame involved in the early reverberation, t p-R+2 ,…,t p-Q+1 The historical frames involved in the estimation of mid- and late-stage reverberation. In order to keep the speech full and rich while minimizing the impact on speech clarity, the reverberation components after 100ms are generally suppressed as mid- and late-stage reverberation components. Assuming that R frames are used to cover direct sound and early reverberation components, and Q frames are used to cover mid- and late-stage reverberation components, the reverberation suppression effect can be directly affected by adjusting the R and Q values.
[0063] Figure 5 The current frame is the Pth frame, and the P-1, P-2, ..., P-Q+1 frames are the historical frames required for reverberation estimation. i (f) and Y i P SD (f) are the spectrum and autopower spectrum of the i-th frame, i = P, ..., P-Q+1, Y i C PSD (f), i≠P is the cross power spectrum between the i-th frame and the P-th frame, Coh i (f), i≠P is the normalized spectral coherence coefficient between the i-th frame and the P-th frame. The above statistics are calculated as follows:
[0064] Y i P SD (f) = α2*Y i P SD (f)+(1-α2)*Y i (f)*conj(Y i (f)), i = P, ..., P - Q + 1
[0065] Yi C PSD (f) = α2*Y i C PSD (f)+(1-α2)*Y i (f)*conj(Y P (f)), i = P-1, ..., P-Q+1
[0066]
[0067] Coh i (f), i≠P measures the frequency domain correlation between the historical frame and the current frame to a certain extent. Based on the short-term correlation assumption of speech, it can be considered that the larger the value of the normalized correlation coefficient is, the greater the contribution to the reverberation component of the current frame is.
[0068] In a hybrid method for real-time speech dereverberation disclosed in the present invention, for S4 of the above embodiment, Figure 6 The figure is a schematic diagram of the power spectrum estimation of the mid- and late-stage reverberation components. The reverberation decay factor is abbreviated as df and is calculated using the following empirical formula: The correlation coefficient is the calculated Coh i (f), i≠P.
[0069] The mid- and late-stage reverberation power spectrum components of the current frame are calculated as follows:
[0070]
[0071] In the above formula, i = P-R+2, ..., P-Q+1, corresponding to Figure 4 The historical frames used to estimate the mid- and late-stage reverberation components.
[0072] R P (f) Apply a strong constraint so that the estimated reverberation power spectrum is no greater than the instantaneous autopower spectrum of the current frame, i.e.
[0073] YP(f)=real(Y P (f)*conj(Y P (f)))
[0074]
[0075] In a hybrid method for real-time speech dereverberation disclosed in the present invention, for S4 of the above embodiment, the a posteriori and a priori signal-reverberation ratios are calculated using the following formula: η(f)=α4*η(f)+(1-α4)*max(ξ(f)-1,0)), where δ is a constant factor to avoid "division by zero".
[0076] When dereverberation is performed, the dereverberation gain is The complex spectrum of the speech after dereverberation is Y dr (f) = G(f)*Y c (f).
[0077] Y dr (f) Perform inverse fast Fourier transform and combine overlap-addition method / overlap-preservation method to obtain the dedeverberated speech output y dr (n).
[0078] After estimating the power spectrum of the mid- and late-stage reverberation, many spectral subtraction denoising or Wiener filtering denoising techniques can be applied to calculate the dereverberation gain.
[0079] The present invention adopts a two-stage dereverberation measure for microphone array pickup devices, including a first-stage array beamforming dereverberation and a second-stage single-channel dereverberation. Among them, the multi-beam fusion output signal obtained by the first-stage dereverberation has a relatively strong suppression effect on mid- and high-frequency reverberation components, and a relatively weak suppression effect on low-frequency reverberation components. The second-stage single-channel dereverberation simply and effectively estimates the mid- and late-stage reverberation power spectrum components of the current frame by combining the normalized frequency domain coherence coefficient between the current frame and several previous frames and the empirical formula for reverberation attenuation, and suppresses them through Wiener filtering. Compared with other technical solutions, such as weighted prediction error solutions and inverse filtering solutions, it can significantly reduce the computational complexity and achieve relatively good reverberation suppression effects in different reverberation environments. Moreover, the computational complexity does not increase exponentially due to the increase in the reverberation level, so it is easy to implement in microphone array conference pickup devices with relatively scarce hardware computing resources. For a specific comparison, see Figure 8 and Fig. 9 .
[0080] Embodiment 2
[0081] A hybrid system for real-time speech dereverberation disclosed in Embodiment 2 of the present invention is introduced below. The hybrid system for real-time speech dereverberation described below and the hybrid method for real-time speech dereverberation described above can refer to each other.
[0082] See also Figure 7 As shown, Embodiment 2 of the present invention discloses a hybrid system for real-time speech dereverberation, including:
[0083] A data preprocessing module 10 is used to obtain original speech signals of multiple microphone channels in a real-time conference scene. The data preprocessing module 10 is used to perform windowing, framing and fast Fourier transform preprocessing on the original speech signals to obtain frequency domain speech signals of multiple microphone channels.
[0084] A fixed beam forming module 20, wherein the fixed beam forming module 20 is used to perform fixed beam forming in multiple beam directions for frequency domain speech signals of multiple microphone channels to obtain multiple beam output frequency domain signals;
[0085] A beam fusion module 30, wherein the beam fusion module 30 is used to perform beam fusion processing based on signal-to-noise ratio weighting on multiple beam output frequency domain signals to obtain a single-channel frequency domain speech signal after the first stage of dereverberation, and calculate a normalized frequency domain coherence coefficient of a current frame and a plurality of frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation;
[0086] A nonlinear dereverberation module 40 is used to estimate the mid- and late-stage reverberation power spectrum components of the current frame based on the normalized frequency domain coherence coefficient, suppress the mid- and late-stage reverberation power spectrum components of the current frame, obtain a speech spectrum after reverberation suppression, perform an inverse fast Fourier transform on the speech spectrum after reverberation suppression, and obtain a dereverberated time domain speech signal.
[0087] In a hybrid system for real-time speech dereverberation disclosed in the present invention, the data preprocessing module 10 includes:
[0088] The Fourier transform unit is used to perform fast Fourier transform on the original speech signal after windowing and framing processing, and convert the time domain signal into a frequency domain signal.
[0089] In a hybrid system for real-time speech dereverberation disclosed in the present invention, the beam fusion module 30 includes:
[0090] A normalized frequency domain coherence coefficient calculation unit is used to calculate the normalized frequency domain coherence coefficients of the current frame and several frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation.
[0091] In a hybrid system for real-time speech dereverberation disclosed in the present invention, the nonlinear dereverberation module 40 comprises:
[0092] A mid-late reverberation power spectrum calculation unit is used to estimate the mid-late reverberation power spectrum components of the current frame.
[0093] The hybrid system for real-time speech dereverberation of this embodiment is used to implement the aforementioned hybrid method for real-time speech dereverberation. Therefore, the specific implementation method of the system can be seen in the embodiment section of the hybrid method for real-time speech dereverberation in the previous text. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part, which will not be introduced in detail here.
[0094] In addition, since the hybrid system for real-time speech dereverberation of the present embodiment is used to implement the aforementioned hybrid method for real-time speech dereverberation, its function corresponds to that of the aforementioned method and will not be described in detail here.
[0095] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0099] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.
Claims
1. A hybrid approach to real-time speech dereverberation, It is characterized in that The following steps are involved: S1: Acquire original speech signals of multiple microphone channels in a real-time conference scene, perform windowing, framing and fast Fourier transform preprocessing on the original speech signals, and obtain frequency domain speech signals of multiple microphone channels; S2: performing fixed beamforming in multiple beam directions for frequency domain speech signals of multiple microphone channels to obtain multiple beam output frequency domain signals; S3: performing beam fusion processing based on signal-to-noise ratio weighting on multiple beam output frequency domain signals to obtain a single-channel frequency domain speech signal after the first stage of dereverberation, and calculating a normalized frequency domain coherence coefficient of a current frame and a plurality of frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation; S4: estimating the mid- and late-stage reverberation power spectrum components of the current frame based on the normalized frequency domain coherence coefficient, suppressing the mid- and late-stage reverberation power spectrum components of the current frame, obtaining a speech spectrum after reverberation suppression, and performing an inverse fast Fourier transform on the speech spectrum after reverberation suppression to obtain a dereverberated time-domain speech signal; In S2, fixed beamforming is performed on the frequency domain speech signals of multiple microphone channels in multiple beam directions, including: The frequency domain signals of multiple microphone channels are summed according to the beam direction and beamforming weight to obtain multiple beam output frequency domain signals ,in, represents the complex spectrum of each microphone, represents the complex beamforming weight, represents the output frequency domain signal of beamforming; In S3, the normalized frequency domain coherence coefficients of the current frame and several frame signals in the previous period are calculated based on the single-channel frequency domain speech signal after the first stage of dereverberation, including: The calculation formula of the normalized frequency domain coherence coefficient is as follows: , , , in, and Respectively The spectrum and autopower spectrum of the frame, , For the The cross power spectrum between the frame and the Pth frame, For the The normalized spectral coherence coefficient between frame and Pth frame; In S4, the mid-late reverberation power spectrum components of the current frame are estimated, including: The calculation formula of the mid-late reverberation power spectrum component of the current frame is: ,in, , which corresponds to the historical frames used to estimate the mid- and late-stage reverberation components, For the The normalized spectral coherence coefficient between frame and P frame.
2. The hybrid method for real-time speech dereverberation according to claim 1, Features: In S2, the division of the beam direction is pre-set. For a uniform circular array, the beam direction selects the direction of each mic. For a uniform linear array, the forward 180° horizontal azimuth angle is evenly divided into several beam ranges.
3. The hybrid method for real-time speech dereverberation according to claim 1, Features: In S4, after the mid-late reverberation power spectrum is calculated, a strong constraint is imposed on the mid-late reverberation power spectrum so that the estimated reverberation power spectrum is not greater than the instantaneous autopower spectrum of the current frame.
4. A hybrid system for real-time speech dereverberation, It is characterized in that The steps for implementing the hybrid method for real-time speech dereverberation as claimed in any one of claims 1 to 3 include: A data preprocessing module is used to obtain the original speech signals of multiple microphone channels in a real-time conference scene. The data preprocessing module is used to perform windowing, framing and fast Fourier transform preprocessing on the original speech signals to obtain frequency domain speech signals of multiple microphone channels. A fixed beam forming module, wherein the fixed beam forming module is used to perform fixed beam forming in multiple beam directions for frequency domain speech signals of multiple microphone channels to obtain multiple beam output frequency domain signals; A beam fusion module, wherein the beam fusion module is used to perform beam fusion processing based on signal-to-noise ratio weighting on multiple beam output frequency domain signals to obtain a single-channel frequency domain speech signal after the first stage of dereverberation, and calculate a normalized frequency domain coherence coefficient of a current frame and a plurality of frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation; A nonlinear dereverberation module, the nonlinear dereverberation module is used to estimate the mid- and late-stage reverberation power spectrum components of the current frame based on the normalized frequency domain coherence coefficient, suppress the mid- and late-stage reverberation power spectrum components of the current frame, obtain the speech spectrum after reverberation suppression, perform inverse fast Fourier transform on the speech spectrum after reverberation suppression, and obtain the dereverberated time domain speech signal.
5. A hybrid system for real-time speech dereverberation according to claim 4, Features: The data preprocessing module comprises: The Fourier transform unit is used to perform fast Fourier transform on the original speech signal after windowing and framing processing, and convert the time domain signal into a frequency domain signal.
6. A hybrid system for real-time speech dereverberation according to claim 4, Features: The beam fusion module comprises: A normalized frequency domain coherence coefficient calculation unit is used to calculate the normalized frequency domain coherence coefficients of the current frame and several frame signals in a previous period of time based on the single-channel frequency domain speech signal after the first stage of dereverberation.
7. The hybrid system for real-time speech dereverberation according to claim 4, Features: The nonlinear dereverberation module comprises: A mid-to-late reverberation power spectrum calculation unit is used to estimate the mid-to-late reverberation power spectrum components of the current frame.
