A multi-channel dereverberation method and apparatus
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YEALINK (XIAMEN) NETWORK TECHNOLOGY CO LTD
- Filing Date
- 2023-01-05
- Publication Date
- 2026-06-02
Smart Images
Figure CN116095566B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of dereverberation technology, and more particularly to a multi-channel dereverberation method and apparatus. Background Technology
[0002] Currently, reverberation and noise are two major factors that severely impact the audio quality and listening experience of users in teleconferences or remote video conferences. For example, in some medium to large conference rooms, the distance between the near-end speaker and the device (e.g., more than 6 meters) will significantly reduce the signal-to-noise ratio of the raw speech captured by the microphone. A spacious room will also result in significant reverberation. Interference from voices in other directions, environmental noise, and the booming sound of reverberation all greatly reduce the clarity of the near-end speaker's voice and severely damage the call experience of the far-end listener.
[0003] WPE (Weighted Prediction Error), RLS (Recursive Least Squares), and Kalman Array are three classic dreverberation algorithms based on MCLP (Multi-Channel Linear Prediction). WPE primarily estimates the variance of the dreverberation speech and the weighting coefficients for reverberation prediction using the maximum likelihood criterion. While WPE achieves good dreverberation results, its batch processing method limits its real-time engineering applications and lacks fast tracking capabilities. RLS supports iterative optimization of weighting coefficients by frame, but suffers from numerical instability and high computational complexity. Kalman Array is a generalized RLS algorithm with stronger time-varying tracking capabilities and better numerical stability, but it has the highest computational complexity.
[0004] Existing Kalman dereverberation algorithms employ a fully diagonalized approach, which addresses the issue of excessive computational complexity in Kalman dereverberation to meet real-time engineering requirements. However, they do not incorporate beamforming technology, resulting in a reduction in sound quality after simplification compared to the original Kalman dereverberation algorithm. Another approach uses amplitude spectrum inverse filtering, which effectively reduces computational complexity but inevitably causes some damage to sound quality. Summary of the Invention
[0005] This invention provides a multi-channel dereverberation method to solve the technical problem that existing dereverberation algorithms are prone to causing sound quality degradation.
[0006] To address the aforementioned technical problems, this invention provides a multi-channel dereverberation method, which includes inputting multi-channel audio data into a diffuse noise field model and calculating the super-directional beamout of the MVDR.
[0007] The super-pointing beam output is used as the desired signal of the Kalman algorithm, and the demixed output signal is predicted according to the Kalman algorithm.
[0008] The de-reverbearing output signal of the current frame is iteratively updated according to the convergence strategy until a preset condition is met, at which point the iterative update stops and the de-reverbearing signal is obtained; wherein, the convergence strategy includes a correction coefficient and a convergence count.
[0009] This invention uses the super-pointing beamform output as the desired signal of the Kalman algorithm, thereby predicting the demixing output signal based on the Kalman algorithm. Due to the directional interference suppression effect of the super-pointing beamform, the Kalman algorithm improves the mid-to-high frequency signal-to-noise ratio and suppresses low-frequency reverberation, making the multi-channel linearly predicted demixing output signal more accurate, thus improving the final demixing sound quality. Furthermore, by updating the demixing output signal through a convergence strategy, the Kalman algorithm has stronger dereverberation robustness. At the same time, it accelerates the convergence speed, overcomes the performance loss caused by complete diagonalization, and improves the demixing sound quality.
[0010] Furthermore, the step of inputting multi-channel audio data into the diffuse noise field model and calculating the super-directional beamout of the MVDR specifically involves:
[0011] The azimuth and elevation angles of the sound source direction are calculated using a sound source localization algorithm, and the steering vector is also calculated.
[0012] The multi-channel audio data is input into the diffuse noise field model, and the noise covariance matrix is calculated by combining the azimuth angle and the pitch angle.
[0013] The first weighting coefficient of MVDR is calculated based on the noise covariance matrix and the steering vector, and the super-directional beam output of each frequency point in the current frame is calculated.
[0014] This invention calculates the first weighting coefficient of MVDR using the noise covariance matrix and the steering vector, and calculates the super-directional beam output based on the coefficient, thereby serving as the desired signal for the Kalman algorithm and suppressing directional interference from the environment.
[0015] Furthermore, the step of using the super-pointing beam output as the desired signal for the Kalman algorithm and predicting the demixed output signal according to the Kalman algorithm specifically involves:
[0016] The number of early reflection blocks, the number of late reverberation blocks, and the number of second channels are preset; the frequency data of each channel are selected based on the number of early reflection blocks and the number of late reverberation blocks.
[0017] The reference signal and late reverberation weighting coefficients are calculated based on the number of late reverberation blocks and the number of second channels;
[0018] The super-pointing beam output is used as the desired signal in the Kalman algorithm, and the demixed output signal is calculated by combining the reference signal and the late reverberation weighting coefficients using the Kalman algorithm.
[0019] This invention controls the energy levels of early direct sound and early reverberation by setting a preset number of early reflection blocks, and adjusts the actual reverberation level by setting a late reverberation block number in order to predict the demixing output signal. Under the premise of ensuring appropriate computational complexity and meeting real-time requirements, the overall speech demixing effect is obvious and the listening experience is clear and natural.
[0020] Furthermore, the step of iteratively updating the demixed output signal of the current frame according to the convergence strategy until a preset condition is met specifically involves:
[0021] The Kalman gain and error covariance matrix are calculated based on the Kalman filtering algorithm and the demixed output signal; the Kalman gain and the error covariance matrix have the same matrix dimension.
[0022] The first correction coefficient and the second correction coefficient are set according to the convergence strategy, and the Kalman gain is updated according to the first correction coefficient and the error covariance matrix is updated according to the second correction coefficient.
[0023] The iteration continues until the number of updates reaches the convergence count, at which point the dereverberation signal is obtained.
[0024] This invention reduces the matrix dimension by using Kalman gains and error covariance matrices of the same dimension. This ensures that the Kalman gain and error covariance matrix at each frequency point only consider the autocorrelation of each channel and historical frame signal while ignoring their cross-correlation, thus significantly reducing computational complexity and ensuring the algorithm's real-time performance. Simultaneously, a multiple convergence strategy is introduced to overcome the performance loss caused by complete diagonalization.
[0025] Furthermore, before inputting the multi-channel audio data into the diffuse noise field model and calculating the super-directional beamout of the MVDR, the method further includes:
[0026] The multi-channel audio signal is subjected to frame division, windowing, and Fourier transform operations to obtain multi-channel audio data, which includes the number of first channels, sampling rate, and frame length.
[0027] Secondly, embodiments of the present invention provide a multi-channel dereverberation device, comprising: a super-directional beamforming calculation module, a dereverberation output signal calculation module, and an optimization module;
[0028] The super-pointing beam calculation module is used to input multi-channel audio data into the diffuse noise field model and calculate the super-pointing beam output of MVDR.
[0029] The demixing output signal calculation module is used to take the super-pointing beam output as the expected signal of the Kalman algorithm and predict the demixing output signal according to the Kalman algorithm.
[0030] The optimization module is used to iteratively update the de-reverbearing output signal of the current frame according to the convergence strategy until a preset condition is met, and then stop the iterative update to obtain the de-reverbearing signal; wherein, the convergence strategy includes a correction coefficient and a convergence count.
[0031] Furthermore, the super-directional beamforming calculation module is specifically used for:
[0032] The azimuth and elevation angles of the sound source direction are calculated using a sound source localization algorithm, and the steering vector is also calculated.
[0033] The multi-channel audio data is input into the diffuse noise field model, and the noise covariance matrix is calculated by combining the azimuth angle and the pitch angle.
[0034] The first weighting coefficient of MVDR is calculated based on the noise covariance matrix and the steering vector, and the super-directional beam output of each frequency point in the current frame is calculated.
[0035] Furthermore, the demixing output signal calculation module is specifically used for:
[0036] The number of early reflection blocks, the number of late reverberation blocks, and the number of second channels are preset; the frequency data of each channel are selected based on the number of early reflection blocks and the number of late reverberation blocks.
[0037] The reference signal and late reverberation weighting coefficients are calculated based on the number of late reverberation blocks and the number of second channels;
[0038] The super-pointing beam output is used as the desired signal in the Kalman algorithm, and the demixed output signal is calculated by combining the reference signal and the late reverberation weighting coefficients using the Kalman algorithm.
[0039] Furthermore, the optimization module is specifically used for:
[0040] The Kalman gain and error covariance matrix are calculated based on the Kalman filtering algorithm and the demixed output signal; the Kalman gain and the error covariance matrix have the same matrix dimension.
[0041] The first correction coefficient and the second correction coefficient are set according to the convergence strategy, and the Kalman gain is updated according to the first correction coefficient and the error covariance matrix is updated according to the second correction coefficient.
[0042] The iteration continues until the number of updates reaches the convergence count, at which point the dereverberation signal is obtained.
[0043] Furthermore, the multi-channel dereverberation device also includes a preprocessing module, specifically used for:
[0044] The multi-channel audio signal is subjected to frame division, windowing, and Fourier transform operations to obtain multi-channel audio data, which includes the number of first channels, sampling rate, and frame length. Attached Figure Description
[0045] Figure 1 This is a schematic flowchart of a multi-channel dereverberation method provided in an embodiment of the present invention;
[0046] Figure 2 This is a schematic diagram of a multi-channel dereverberation device provided in an embodiment of the present invention;
[0047] Figure 3 This is another flowchart illustrating the multi-channel dereverberation method provided in an embodiment of the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] Example 1
[0050] Please refer to Figure 1 , Figure 1 A flowchart illustrating a multi-channel dereverberation method provided in an embodiment of the present invention includes steps 101 to 103, as detailed below:
[0051] Step 101: Input the multi-channel audio data into the diffuse noise field model and calculate the super-directional beamout of the MVDR;
[0052] In this embodiment, the Kalman algorithm is used to dedeverage multi-channel audio data using a 16-element uniform circular array microphone array. This mainly involves calculating the super-directional beamout of the 16-channel MVDR and using the super-directional beamout as the desired signal to execute an 8-channel Kalman dedeverage algorithm, thereby obtaining the optimal output.
[0053] In this embodiment, before inputting the multi-channel audio data into the diffuse noise field model and calculating the super-directional beamout of the MVDR, the method further includes:
[0054] The multi-channel audio signal is subjected to frame division, windowing, and Fourier transform operations to obtain multi-channel audio data, which includes the number of first channels, sampling rate, and frame length.
[0055] In this embodiment, the multi-channel input audio signal is subjected to operations such as framing, windowing, and FFT transformation to the frequency domain. The first channel number M1 = 16, the sampling rate Fs = 16kHz, the frequency range is 0 to 8000Hz, the frame length Nfft = 512, and the frame shift L = 256 (i.e., 50% overlap retention). The window functions used for windowing operations include common windows such as Hanning window, Hamming window, or sine window.
[0056] In this embodiment, a sinusoidal window is used as the window function. The sinusoidal window exhibits superior robustness in super-pointing beamforming and de-reverberation algorithms, avoiding high-frequency hissing sounds, i.e., the inter-frame continuity problem. The specific window function is as follows:
[0057]
[0058] In this context, a single colon ":" indicates that all elements in a row or column are selected. For example, a0(:,k) represents all elements in the k-th column of matrix a0. 0:1:(Nfft-1) is an array representation method. It represents a one-dimensional array with 1 row and Nfft columns, starting at 0 and ending at Nfft-1, with adjacent numbers separated by a 1-digit interval (0, 1, 2, ...). Therefore, Wi n is also a one-dimensional array with 1 row and Nfft columns.
[0059] In this embodiment, the step of inputting multi-channel audio data into the diffuse noise field model and calculating the super-directional beamout of the MVDR specifically involves:
[0060] The azimuth and elevation angles of the sound source direction are calculated using a sound source localization algorithm, and the steering vector is also calculated.
[0061] The multi-channel audio data is input into the diffuse noise field model, and the noise covariance matrix is calculated by combining the azimuth angle and the pitch angle.
[0062] The first weighting coefficient of MVDR is calculated based on the noise covariance matrix and the steering vector, and the super-directional beam output of each frequency point in the current frame is calculated.
[0063] In this embodiment, the azimuth angle fai0 and the elevation angle theta0 of the near speaker's sound source direction are determined by the sound source localization algorithm. The preset range of the azimuth angle fai is [-pi,pi], and the preset range of the elevation angle theta is [0,pi / 2].
[0064] In this embodiment, the sound source localization algorithm is specifically the SRP-PHAT algorithm, which supports online real-time optimization and satisfies real-time performance and robustness.
[0065] In this embodiment, the guidance vector a0 can be calculated using the azimuth angle and the pitch angle, and the dimension of the guidance vector a0 is M1*(Nfft / 2+1). The first dimension of the guidance vector a0 represents the number of channels, and the second dimension represents the number of frequency points. Specifically, the guidance vector a0 is:
[0066]
[0067] Where c is the speed of sound, specifically c = 340 m / s; R is the radius of the circular array, specifically R = 0.08 m.
[0068] In this embodiment, f is the frequency, specifically... f consists of a one-dimensional array with 1 row and (Nfft / 2+1) columns, where f(k) represents the element in the k-th column of the array f. k can be understood as the sequence number of the frequency point. According to the Fourier transform property, the frequencies from 0 to 8 kHz can be evenly mapped to (Nfft / 2+1) frequency points to form the array f. The value range of k is [1, Nfft / 2+1].
[0069] In this embodiment, multi-channel audio data is input into a diffuse noise field model, and the noise covariance matrix is calculated by combining the azimuth angle and the elevation angle; the noise covariance matrix is specifically:
[0070]
[0071] Where Fvv represents a three-dimensional matrix M1*M1*(Nfft / 2+1); the number of array elements, i.e. the number of the first channel M1 = 16; the third dimension represents a total of Nfft / 2+1 frequency points; the first and second dimensions store the covariance matrix of M1*M1 corresponding to each frequency point; the spacing d represents the straight-line distance between adjacent array elements (microphones) in a 16-element uniform circular array.
[0072] In this embodiment, the straight-line distance between adjacent elements (microphones) in a 16-element uniform circular array is specifically as follows:
[0073]
[0074] In this embodiment, a robust super-directing beam is constructed using diagonal loading technology, and the first weighting coefficient W of MVDR can be calculated using the steering vector a0 and the super-directing beam, combined with the noise covariance matrix. SBF The first weighting coefficient W of the MVDR SBF Specifically:
[0075]
[0076] Rxx=inv(Fvv(:,:,k)+eye(M1)*0.1) (6)
[0077] Among them, W SBF The weight coefficient matrix has a dimension of M1*(Nfft / 2+1). The eye(M1)*0.1 on the right side of the Rxx equation indicates that a diagonal loading technique is used to construct a robust super-directive beam. 0.1 is a fixed diagonal loading factor, and the value of the diagonal loading factor ranges from 0.1 to 0.001.
[0078] In this embodiment, the first weighting coefficient W is used. SBF Calculate the super-pointing beamout at each frequency point in the current frame. The super-pointing beamout at all frequencies constitutes the super-pointing beamout of the MVDR. Specifically, the super-pointing beamout is as follows:
[0079] Y SBF (k)=X(:,k).′*conj(W SBF (:,k)) (7)
[0080] Among them, Y SBF The dimension is 1*(Nfft / 2+1), and X is the spectrum of the input signal of the current frame obtained during the preprocessing process, with a dimension of M1*(Nfft / 2+1).
[0081] In this embodiment, the first weighting coefficient of MVDR is calculated using the noise covariance matrix and the steering vector, and the super-directing beam output is calculated based on the coefficient, thus serving as the desired signal for the Kalman algorithm, while suppressing directional interference from the environment.
[0082] Step 102: Use the super-pointing beam output as the desired signal of the Kalman algorithm, and predict the demixed output signal according to the Kalman algorithm;
[0083] In this embodiment, the step of using the super-pointing beam output as the desired signal of the Kalman algorithm and predicting the demixing output signal according to the Kalman algorithm specifically involves:
[0084] The number of early reflection blocks, the number of late reverberation blocks, and the number of second channels are preset; the frequency data of each channel are selected based on the number of early reflection blocks and the number of late reverberation blocks.
[0085] The reference signal and late reverberation weighting coefficients are calculated based on the number of late reverberation blocks and the number of second channels;
[0086] The super-pointing beam output is used as the desired signal in the Kalman algorithm, and the demixed output signal is calculated by combining the reference signal and the late reverberation weighting coefficients using the Kalman algorithm.
[0087] In this embodiment, the number of early reflection blocks controls the energy level of the early direct sound and early reverberation, generally ranging from 2 to 6; the number of late reverberation blocks depends to some extent on the actual reverberation level, generally ranging from 8 to 30. For the number of early reflection blocks Ld, a value that is too small may result in insufficient early reverberation energy, making the speech sound dry and uncomfortable, while a value that is too large will result in a large amount of early reverberation, and the speech reverberation booming sensation will still be relatively large. For the number of late reverberation blocks Lc, a value that is too small may result in an inability to cover the reverberation tail length, and the reverberation sensation will still be obvious after elimination, while a value that is too large will increase the computational complexity and may cause the risk of filter erroneous convergence, resulting in speech damage.
[0088] In this embodiment, the dereverberation process is simulated through multiple simulation experiments, and preset values for the number of early reflection blocks and late reverberation blocks are obtained to ensure that the current configuration can make the computational complexity appropriate to meet real-time requirements, while improving the overall speech dereverberation effect and making the listening experience clear and natural.
[0089] In this embodiment, the second number of channels is the number of reference signal channels, and the value range is 2 to 16. In this embodiment, the second number of channels is set to 8, that is, the first 8 of the 16 array elements (microphones) are taken as reference signals for the de-reverberation operation.
[0090] In this embodiment, the number of early reflection blocks Ld is set to 3, that is, the early reverberation of (Ld-1)*L / Fs = 0.032s is retained; the number of late reverberation blocks Lc is set to 10, and the number of second channels for de-reverberation is set to 8. The reference signal u and the late reverberation weight coefficient W can be calculated by the number of early reflection blocks, the number of late reverberation blocks and the number of second channels.
[0091] In this embodiment, the dimension of the reference signal u is (Lc*M2)*(Nfft / 2+1), meaning u contains a total of Lc*M2 frames of data. Each channel selects Lc frames, with the selection rule being a total of Lc frames of data from the Ld-th frame to the (Ld+Lc-1)-th frame preceding the current frame. Each frame contains Nfft / 2+1 frequency points. Each channel selects Lc*(Nfft / 2+1) data, and the M2 channels are sequentially arranged to form a multi-channel reference signal u of (Lc*M2)*(Nfft / 2+1). The dimension of the late reverberation weight coefficient W is also (Lc*M2)*(Nfft / 2+1).
[0092] In this embodiment, the super-pointing beam output is used as the desired signal of the Kalman algorithm. The demixed output signal is calculated by combining the reference signal and the late reverberation weighting coefficients using the Kalman algorithm. The demixed output signal s of the multi-channel linear prediction is specifically as follows:
[0093] s=Y Q -sum(y,1) (8)
[0094] Y Q =Y SBF (9)
[0095] y = conj(W).*u (10)
[0096] Where s has a dimension of 1*(Nfft / 2+1), representing the demixed output signal; sum(y,1) represents summing Lc*M2 data points in each column (each frequency point), this step fully reflects that the estimation of late reverberation is the result of the joint superposition of multi-channel information; Y Q In existing techniques, the raw mic data from a specific channel is typically used to characterize the desired signal.
[0097] In this embodiment, the present invention controls the energy levels of early direct sound and early reverberation by setting the number of early reflection blocks, and adjusts the actual reverberation level by setting the number of late reverberation blocks to predict the demixing output signal. Under the premise of ensuring appropriate computational complexity and meeting real-time requirements, the overall speech demixing effect is obvious and the listening experience is clear and natural.
[0098] In this embodiment, the output of SBF is replaced to inherit the advantages of SBF—directional interference suppression, significant improvement in mid-to-high frequency signal-to-noise ratio, and further suppression of low-frequency reverberation, which further improves the sound quality after Kalman multi-channel dérothermization.
[0099] Step 103: Iteratively update the mixed output signal of the current frame according to the convergence strategy until a preset condition is reached, then stop the iterative update to obtain the dereverberation signal; wherein, the convergence strategy includes correction coefficients and convergence count.
[0100] In this embodiment, the step of iteratively updating the demixed output signal of the current frame according to the convergence strategy until a preset condition is met specifically involves:
[0101] The Kalman gain and error covariance matrix are calculated based on the Kalman filtering algorithm and the demixed output signal; the Kalman gain and the error covariance matrix have the same matrix dimension.
[0102] The first correction coefficient and the second correction coefficient are set according to the convergence strategy, and the Kalman gain is updated according to the first correction coefficient and the error covariance matrix is updated according to the second correction coefficient.
[0103] The iteration continues until the number of updates reaches the convergence count, at which point the dereverberation signal is obtained.
[0104] In this embodiment, the state variables of the current frame are iteratively updated using the Kalman filter formula. The state variables include updating the late reverberation weight coefficient W, the Kalman gain K, and the error covariance matrix P of the weight coefficient W together. K and P are two key state variables of the Kalman filter, which control the convergence speed and tracking speed of the Kalman filter.
[0105] In this embodiment, the Kalman gain and error covariance matrix are calculated based on the Kalman filtering algorithm and the demixed output signal, specifically as follows:
[0106] K=P.*u. / (gamma*S+Sum(P.*(u.*conj(u)),1)) (11)
[0107] P=(IK.*conj(u)).*P+beta*Sum(W.*conj(W),1) (12)
[0108] W = W + K.*conj(s); (13)
[0109] S = 0.5*S + 0.5*s.*conj(s) (14)
[0110] S = max(S, 1e-6) (15)
[0111] Where W represents the late reverberation weighting coefficient, K represents the Kalman gain, P represents the error covariance matrix of the weighting coefficient W, and I is an all-one matrix. All of these matrices have a dimension of (Lc*M²)*(Nfft / 2+1). The matrix dimension of K and P at each frequency point is (Lc*M²)*1. gamma is the first correction coefficient set according to the convergence strategy, and beta is the second correction coefficient set according to the convergence strategy.
[0112] In this embodiment, the first correction coefficient gamma and the second correction coefficient beta are specifically as follows:
[0113] gamma=2*Lc*M2 (16)
[0114] beta = 0.001 (17)
[0115] In this embodiment, gamma is a correction coefficient specifically introduced in this invention to control the algorithm's robustness in noisy fields. Too small a value will lead to excessive speech cancellation, while too large a value will result in excessive reverberation. The optimal value range is [Lc*M², 2*Lc*M²]. The beta parameter is used to adjust the tracking speed. Too small a value will result in insufficient tracking speed, while too large a value will lead to excessive speech cancellation. A suitable value of 0.001 ensures that the algorithm has good tracking speed and the filter is relatively stable.
[0116] In this embodiment, the first term in the denominator of the Kalman gain K update formula, gamma*S, is an adjustable observation noise term. S is recursively smoothed using the maximum likelihood principle. The update of S utilizes the demixed output signal s from multi-channel linear prediction. This step also demonstrates that the combination with SBF can obtain a more accurate demixed output signal s, thus leading to a more accurate Kalman gain K. Therefore, the entire algorithm moves towards a better weight coefficient W. Furthermore, S = max(S, 1e-6) further constrains the excessively small value of S to promote the stability of the Kalman filter.
[0117] In this embodiment, the second term of the update formula for the error covariance moment P, beta*sum(W.*conj(W), 1), is called the adjustable system noise term, which ensures that the algorithm can track the sound source switching direction and motion scene more quickly.
[0118] In this embodiment, the number of convergence iterations is set to two. The performance loss caused by complete diagonalization is overcome by using a two-step convergence strategy. Since the Kalman algorithm can be iterated multiple times to further improve the convergence speed, but considering the increase in computational complexity, two iterations are generally sufficient to achieve a considerable improvement in convergence speed and decongestion effect.
[0119] In this embodiment, after stopping the iterative update, the frequency domain de-mixed output signal s of the current frame is transformed back to the time domain through IFT, and the final de-mixed signal is obtained by using the overlap-preserving windowing operation.
[0120] This invention uses the super-pointing beamout as the desired signal of the Kalman algorithm to predict the demixed output signal. Due to the directional interference suppression effect of the super-pointing beamforming, the Kalman algorithm improves the mid-to-high frequency signal-to-noise ratio and suppresses low-frequency reverberation, making the multi-channel linearly predicted demixed output signal more accurate, thereby improving the final demixed sound quality. Furthermore, by updating the demixed output signal through a convergence strategy, the Kalman de-reverberation algorithm becomes more robust. At the same time, it accelerates the convergence speed, overcomes the performance loss caused by complete diagonalization, and improves the demixed sound quality.
[0121] Please refer to Figure 2 , Figure 2A schematic diagram of a multi-channel de-reverberation device provided in an embodiment of the present invention includes: a super-directional beamforming calculation module 201, a de-reverberation output signal calculation module 202, and an optimization module 203;
[0122] The super-pointing beam calculation module 201 is used to input multi-channel audio data into the diffuse noise field model and calculate the super-pointing beam output of MVDR.
[0123] The demixing output signal calculation module 202 is used to take the super-pointing beam output as the expected signal of the Kalman algorithm and predict the demixing output signal according to the Kalman algorithm.
[0124] The optimization module 203 is used to iteratively update the de-reverbly output signal of the current frame according to the convergence strategy until a preset condition is met, and then stop the iterative update to obtain the de-reverbly signal; wherein, the convergence strategy includes a correction coefficient and a convergence count.
[0125] In this embodiment, the super-pointing beam calculation module 201 is specifically used for:
[0126] The azimuth and elevation angles of the sound source direction are calculated using a sound source localization algorithm, and the steering vector is also calculated.
[0127] The multi-channel audio data is input into the diffuse noise field model, and the noise covariance matrix is calculated by combining the azimuth angle and the pitch angle.
[0128] The first weighting coefficient of MVDR is calculated based on the noise covariance matrix and the steering vector, and the super-directional beam output of each frequency point in the current frame is calculated.
[0129] In this embodiment, the demixing output signal calculation module 202 is specifically used for:
[0130] The number of early reflection blocks, the number of late reverberation blocks, and the number of second channels are preset; the frequency data of each channel are selected based on the number of early reflection blocks and the number of late reverberation blocks.
[0131] The reference signal and late reverberation weighting coefficients are calculated based on the number of late reverberation blocks and the number of second channels;
[0132] The super-pointing beam output is used as the desired signal in the Kalman algorithm, and the demixed output signal is calculated by combining the reference signal and the late reverberation weighting coefficients using the Kalman algorithm.
[0133] In this embodiment, the optimization module 203 is specifically used for:
[0134] The Kalman gain and error covariance matrix are calculated based on the Kalman filtering algorithm and the demixed output signal; the Kalman gain and the error covariance matrix have the same matrix dimension.
[0135] The first correction coefficient and the second correction coefficient are set according to the convergence strategy, and the Kalman gain is updated according to the first correction coefficient and the error covariance matrix is updated according to the second correction coefficient.
[0136] The iteration continues until the number of updates reaches the convergence count, at which point the dereverberation signal is obtained.
[0137] In this embodiment, the multi-channel dereverberation device further includes a preprocessing module, specifically used for:
[0138] The multi-channel audio signal is subjected to frame division, windowing, and Fourier transform operations to obtain multi-channel audio data, which includes the number of first channels, sampling rate, and frame length.
[0139] Please refer to Figure 3 , Figure 3 This is another flowchart illustrating the multi-channel dereverberation method provided in an embodiment of the present invention.
[0140] In this embodiment, the multi-channel audio input signal is first subjected to audio framing, windowing, and FFT transformation to the frequency domain; then, the azimuth and elevation angles are determined according to the sound source localization algorithm, and the steering vector a0 is calculated; next, the noise covariance matrix Fvv and the MVDR weighting coefficient W are calculated. SBF The process involves calculating the super-pointing beam output and using it as the desired signal for the Kalman algorithm, setting the convergence count t. The demixed output signal is then calculated based on the desired signal set by the Kalman algorithm. The Kalman gain, error covariance matrix, and late reverberation weight coefficients are updated according to the convergence strategy. When the convergence count reaches a preset value, the iterative update stops, and the demixed output signal is transformed back to the time domain using an IFT transform. Finally, the final output signal is obtained using an overlap-preserving windowing operation.
[0141] In this embodiment, the de-reverbearing output signal is predicted by combining super-pointing beamforming (SBF) technology with the Kalman algorithm. At the same time, the Kalman filter formula is updated according to the convergence strategy, which comprehensively improves the noise robustness, convergence speed and final de-reverbearing sound quality of the algorithm, improves the mid-to-high frequency signal-to-noise ratio, makes the low-frequency reverberation suppression cleaner, the sound signature clearer, and the overall listening experience more natural and comfortable.
[0142] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.
Claims
1. A multi-channel dereverberation method, characterized in that, include: The multi-channel audio data is input into the diffuse noise field model, and the super-directing beam output of MVDR is calculated. Specifically, the azimuth and elevation angles of the sound source direction are calculated using a sound source localization algorithm, and the steering vector is calculated. The multi-channel audio data is input into the diffuse noise field model, and the noise covariance matrix is calculated in combination with the azimuth and elevation angles. The first weighting coefficient of MVDR is calculated based on the noise covariance matrix and the steering vector, and the super-directional beam output of each frequency point in the current frame is calculated. The super-pointing beam output is used as the desired signal of the Kalman algorithm, and the demixed output signal is predicted according to the Kalman algorithm. The de-reverbearing output signal of the current frame is iteratively updated according to the convergence strategy until a preset condition is met, at which point the iterative update stops and the de-reverbearing signal is obtained; wherein, the convergence strategy includes a correction coefficient and a convergence count.
2. The multi-channel dereverberation method as described in claim 1, characterized in that, The step of using the super-pointing beam output as the desired signal for the Kalman algorithm and predicting the demixed output signal based on the Kalman algorithm specifically involves: The number of early reflection blocks, the number of late reverberation blocks, and the number of second channels are preset; the frequency data of each channel are selected based on the number of early reflection blocks and the number of late reverberation blocks. The reference signal and late reverberation weighting coefficients are calculated based on the number of late reverberation blocks and the number of second channels; The super-pointing beam output is used as the desired signal in the Kalman algorithm, and the demixed output signal is calculated by combining the reference signal and the late reverberation weighting coefficients using the Kalman algorithm.
3. The multi-channel dereverberation method as described in claim 1, characterized in that, The step of iteratively updating the demixed output signal of the current frame according to the convergence strategy until a preset condition is met is specifically as follows: The Kalman gain and error covariance matrix are calculated based on the Kalman filtering algorithm and the demixed output signal; the Kalman gain and the error covariance matrix have the same matrix dimension. The first correction coefficient and the second correction coefficient are set according to the convergence strategy, and the Kalman gain is updated according to the first correction coefficient and the error covariance matrix is updated according to the second correction coefficient. The iteration continues until the number of updates reaches the convergence count, at which point the dereverberation signal is obtained.
4. The multi-channel dereverberation method as described in claim 1, characterized in that, Before inputting multi-channel audio data into the diffuse noise field model and calculating the super-directional beamout of the MVDR, the process also includes: The multi-channel audio signal is subjected to frame division, windowing, and Fourier transform operations to obtain multi-channel audio data, which includes the number of first channels, sampling rate, and frame length.
5. A multi-channel dereverberation device, characterized in that, include: Super-directional beamforming calculation module, demixing output signal calculation module, and optimization module; The super-pointing beamforming calculation module is used to input multi-channel audio data into the diffuse noise field model and calculate the super-pointing beamform output of MVDR. Specifically, it calculates the azimuth and elevation angles of the sound source direction through a sound source localization algorithm and calculates the steering vector; it inputs multi-channel audio data into the diffuse noise field model and calculates the noise covariance matrix in combination with the azimuth and elevation angles. The first weighting coefficient of MVDR is calculated based on the noise covariance matrix and the steering vector, and the super-directional beam output of each frequency point in the current frame is calculated. The demixing output signal calculation module is used to take the super-pointing beam output as the expected signal of the Kalman algorithm and predict the demixing output signal according to the Kalman algorithm. The optimization module is used to iteratively update the de-reverbearing output signal of the current frame according to the convergence strategy until a preset condition is met, and then stop the iterative update to obtain the de-reverbearing signal; wherein, the convergence strategy includes a correction coefficient and a convergence count.
6. The multi-channel dereverberation device as described in claim 5, characterized in that, The demixing output signal calculation module is specifically used for: The number of early reflection blocks, the number of late reverberation blocks, and the number of second channels are preset; the frequency data of each channel are selected based on the number of early reflection blocks and the number of late reverberation blocks. The reference signal and late reverberation weighting coefficients are calculated based on the number of late reverberation blocks and the number of second channels; The super-pointing beam output is used as the desired signal in the Kalman algorithm, and the demixed output signal is calculated by combining the reference signal and the late reverberation weighting coefficients using the Kalman algorithm.
7. The multi-channel dereverberation device as described in claim 5, characterized in that, The optimization module is specifically used for: The Kalman gain and error covariance matrix are calculated based on the Kalman filtering algorithm and the demixed output signal; the Kalman gain and the error covariance matrix have the same matrix dimension. The first correction coefficient and the second correction coefficient are set according to the convergence strategy, and the Kalman gain is updated according to the first correction coefficient and the error covariance matrix is updated according to the second correction coefficient. The iteration continues until the number of updates reaches the convergence count, at which point the dereverberation signal is obtained.
8. The multi-channel dereverberation device as described in claim 5, characterized in that, The multi-channel dereverberation device also includes a preprocessing module, specifically used for: The multi-channel audio signal is subjected to frame division, windowing, and Fourier transform operations to obtain multi-channel audio data, which includes the number of first channels, sampling rate, and frame length.