Multi-objective neural network speech processing method and device based on microphone array

Abstract

The present invention discloses a multi-objective neural network speech processing method and device based on a microphone array. The method inputs multi-channel recording data and multi-channel echo data into an RLS filter for multi-channel linear echo cancellation, and then uses the frequency domain correlation between the filter's error signal and the estimated echo signal to calculate the residual echo energy of each frequency point; the acoustic features of the error signal and the estimated residual echo signal are input into a residual neural network, and multi-objective progressiveness is used to enable the residual neural network to converge quickly in a smaller network structure; the post-processing noise reduction, echo removal, and reverberation algorithm use a multi-channel Wiener filter with a mask for unified calculation, which has low complexity and avoids speech loss in the process of sequential processing of a single algorithm. Thus, the goals of speech noise reduction, echo removal, and reverberation are integrated, with the characteristics of low computational complexity, low latency, good real-time performance, and high speech quality, which meets actual usage requirements.

Description

Technical Field

[0001] The present invention relates to speech signal processing technology, and in particular to a multi-objective neural network speech processing method and device based on a microphone array. Background Art

[0002] Microphone array-based voice noise reduction, echo removal, and reverberation removal, that is, sound processing and modification, are widely used in various fields such as conferencing, education, and entertainment. Currently, how to obtain clear and easy-to-understand voice signals has become a top priority in conference communication scenarios.

[0003] Current speech noise reduction, echo removal, and dereverberation technologies fall into three general categories: The first utilizes multi-channel speech correlation and delay information acquired by microphone arrays, based on theoretical modeling of speech signals and statistical optimization theory, to calculate the optimal signal solution. In practice, speech signals vary with the scene, exhibiting complex variations, making it difficult to achieve effective processing results using hypothetical theoretical models. The second approach directly calculates the optimal speech waveform through end-to-end training using neural networks. This approach has a complex network structure, is computationally expensive in practice, and requires a large amount of speech training data. Furthermore, this approach is significantly limited by the quality of speech data and the degree of scene matching, resulting in unstable processing results and significant latency, making it unsuitable for real-time communication. The third approach combines neural networks with signal models, handing over some parameters that require tuning to the neural network for training. However, this approach fails to integrate noise reduction, echo removal, and dereverberation, often requiring separate optimization. These objectives conflict with each other, making it difficult to achieve effective results in real-world scenarios. Furthermore, the network structure is complex, computationally expensive, and latency is high, making it unsuitable for real-time communication. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-objective neural network speech processing method and device based on a microphone array, which can achieve optimal speech quality effects, have low computational complexity and meet real-time communication requirements.

[0005] The following is a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceivable aspects and is neither intended to identify key or critical elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that will be provided later.

[0006] According to one aspect of the present invention, a multi-objective neural network speech processing method based on a microphone array is provided, comprising:

[0007] Step 100: Use short-time Fourier transform to obtain frequency domain data of multi-channel recording and frequency domain data of multi-channel stereo re-collection;

[0008] Step 200: Feed the frequency domain data of the multi-channel recording and the frequency domain data of the multi-channel stereo recapture into an RLS filter to obtain an error signal and an estimated signal, calculate the frequency domain correlation between the two, smooth the frequency domain correlation of the historical frames using a specific learning rate, and then multiply the error signal by the error signal to obtain the estimated residual echo signal energy at each frequency point;

[0009] Step 300: Input the BFCC features of the error signal and the residual echo signal into a multi-objective neural network to obtain a fused speech energy mask for noise reduction, echo removal, and reverberation removal.

[0010] Step 400: Filter the signal using a beamformer and then perform an inverse transform to the time domain to obtain clean processed time domain audio.

[0011] In one embodiment, in step 200, the output of the RLS linear filtering is:

[0012] Lsty(n,f)=W H (n-1, f)X(n, f);

[0013] E(n, f)=Y(n, f)-Lsty(n, f);

[0014] W(n,f)=W(n-1,f)+k(n,f)E H (n, f);

[0015] k(n, f)=P(n-1, f)X(n, f) / (1+X H (n, f)P(n-1, f)X(n, f));

[0016]

[0017] Where X is the frequency domain vector of multi-channel resampled data; W is the RLS filter vector; E is the error signal; Lsty is the echo estimation signal; Y is the near-end speech signal; R is the covariance matrix at each resampled frequency point; P is the inverse matrix of R; k is the update coefficient of filter W; n is the number of frames; and f is the frequency point.

[0018] In one embodiment, in step 200, the residual echo signal energy is calculated as follows:

[0019] Res(n, f)=eta(n, f)*Yf(n, f);

[0020] Yf(n,f)=Lsty(n,f)*Lsty * (n, f);

[0021] Ef(n,f)=E(n,f)*E * (n, f);

[0022] EYf(n,f)=Lsty(n,f)*E * (n, f);

[0023] η(n,f)=abs(EYf(n,f)) 2 / (max(Yf(n, f), Ef(n, f))*Yf(n, f)+1e-9);

[0024] Where Res is the estimated energy of the echo leakage; η is the coefficient of the estimated echo leakage; Yf is the energy of the estimated echo signal; Ef is the energy of the error signal; EYf is the cross-correlation energy between the error signal and the leakage echo; E is the error signal; Lsty is the echo estimation signal; n is the number of frames; and f is the frequency point.

[0025] In one embodiment, the multi-objective neural network uses a GRU or LSTM network in conjunction with a fully connected layer to form a residual network with three different objectives. Each objective is output at a different layer and simultaneously serves as input to the next layer.

[0026] In one embodiment, the objectives of the multi-objective neural network are progressive, the objective of the first layer of the residual network is the noise mask, the objective of the second layer is the fusion mask of the echo and noise, and the objective of the third layer is the fusion mask of the echo, noise and reverberation, and these three layers of masks are numerically related.

[0027] In one embodiment, the beamforming filter in step 500 performs the following calculation:

[0028] H(f)=(R ss (f)+u*R nn (f)) -1 R ss (f)e1;

[0029] R ss (n, f) = λ*R ss (n-1, f)+(1-λ)*mask(n, f)X(n, f)X H (n, f)mask H (n, f);

[0030] Where H is the coefficient of the beamforming filter; R ss is the covariance matrix of the estimated target speech; R nnis the covariance matrix of the estimated non-target signal; u is the speech fidelity coefficient, ranging from (0-1); e1 is the vector (1; 0; ...; 0); λ is the covariance matrix update coefficient, ranging from (0-1); mask is the masking coefficient of the target speech estimated by the neural network; X is the original recorded data signal; n is the number of frames; f is the frequency point.

[0031] In one embodiment, the covariance matrix of each frequency point is calculated by continuously iterating the learning rate.

[0032] In one embodiment, the beamforming filter is a multi-channel Wiener filter.

[0033] According to another aspect of the present invention, a device is provided, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement the multi-objective neural network speech processing method based on a microphone array as described in the first aspect when executing the computer program.

[0034] The beneficial effects of the embodiments of the present invention are:

[0035] 1. This embodiment of the present invention uses a frequency-domain linear RLS filter for preprocessing multi-channel speech echo removal. The more commonly used LMS filter can achieve better linear filtering processing effects. It can also meet the needs of stereo echo cancellation. The more commonly used mono echo cancellation expands the application scenarios and optimizes the effect.

[0036] 2. This embodiment of the present invention uses the frequency-domain correlation between the error signal (Error) processed by the RLS filter and the echo estimation signal (Lsty) as the leakage coefficient of the echo signal. Different leakage coefficients are used at different frequency points to estimate the residual echo signal (Res). Speech features extracted from the error signal (Error) processed by the RLS filter and the estimated residual echo signal (Res) are then input into a neural network. This enables cleaner echo cancellation and ensures the integrity and clarity of the near-end signal during dual talk.

[0037] 3. The embodiment of the present invention uses the BFCC frequency band features of single-channel single-recall speech as the input of the neural network, so the information required to be carried by the neural network is greatly reduced, thereby creating a prerequisite for the miniaturization of the neural network.

[0038] 4. This embodiment of the present invention uses a multi-objective residual neural network for training and optimization, sequentially using the noise reduction gain, echo removal gain, and dereverberation gain as targets for different layers of the neural network and as inputs for the next layer. This allows the results of noise reduction, echo removal, and dereverberation to be integrated to achieve optimal speech processing results.

[0039] 5. The beamforming filter used in the embodiment of the present invention is a multi-channel Wiener filter (Speech Distortion Weighted Multichannel Wiener Filter). The covariance matrix of speech and noise is calculated by accumulating the data using real-time transformation learning rates, without directly using the information of the previous and next frame data. In addition, the pre-placed RLS filter and the input of the neural network do not use the expansion of the previous and next frames. Therefore, while ensuring the effect, the overall system latency is low, meeting the requirements of real-time communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0041] The above features and advantages of the present invention will be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or characteristics may have the same or similar reference numerals.

[0042] Figure 1 is a flow chart of an embodiment of the method of the present application;

[0043] Figure 2 1 is a schematic diagram of the general structure of the multi-objective neural network in the embodiment of the present application. DETAILED DESCRIPTION

[0044] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Note that the various aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention.

[0045] Please see the attached Figure 1 , an embodiment of the present application provides a multi-objective neural network speech processing method based on a microphone array, comprising:

[0046] Step 100: Use short-time Fourier transform to obtain frequency domain data of multi-channel recording and frequency domain data of multi-channel stereo re-collection;

[0047] Step 200: Feed the frequency domain data of the multi-channel recording and the frequency domain data of the multi-channel stereo recapture into an RLS filter to obtain an error signal and an echo estimation signal. Calculate the frequency domain correlation between the error signal and the echo estimation signal as the leakage coefficient of the echo signal, and then multiply it by the error signal to obtain the estimated residual echo signal at each frequency point.

[0048] Step 300: Extract BFCC features of the error signal and the residual echo signal and input them into a multi-objective neural network to obtain a fused speech energy mask for noise reduction, echo removal, and reverberation removal.

[0049] Step 400: After filtering the speech energy mask through a beamforming filter, inversely transform it into the time domain to obtain clean time-domain audio.

[0050] The embodiment of the present invention uses a frequency-domain linear RLS filter to perform preprocessing for multi-channel speech echo removal. The more commonly used LMS filter can achieve better linear filtering processing effects. At the same time, it can meet the needs of stereo echo cancellation. The more commonly used mono echo cancellation expands the application scenarios and optimizes the effect.

[0051] In addition, the embodiment of the present invention uses the frequency-domain correlation between the error signal (Error) processed by the RLS filter and the echo estimation signal (Lsty) as the leakage coefficient of the echo signal. In this way, different leakage coefficients are used at different frequency points to estimate the residual echo signal (Res). Subsequently, the speech features extracted from the error signal (Error) processed by the RLS filter and the estimated residual echo signal (Res) are input into a neural network, thereby achieving cleaner echo cancellation and ensuring the integrity and clarity of the near-end signal during dual speech.

[0052] At the same time, the embodiment of the present invention uses the single-channel single-recall speech BFCC frequency band characteristics as the input of the neural network, so the information required to be carried by the neural network is greatly reduced, thereby creating a prerequisite for the miniaturization of the neural network.

[0053] In a possible embodiment, in step 200, the output of the RLS linear filter is:

[0054] Lsty(n,f)=W H (n-1, f)X(n, f);

[0055] E(n, f)=Y(n, f)-Lsty(n, f);

[0056] W(n,f)=W(n-1,f)+k(n,f)E H (n, f);

[0057] k(n, f)=P(n-1, f)X(n, f) / (1+X H (n, f)P(n-1, f)X(n, f));

[0058]

[0059] Where X is the frequency domain vector of multi-channel sampled data; W is the RLS filter vector; E is the error signal; Lsty is the echo estimate signal; Y is the near-end speech signal; R is the covariance matrix at each sampled frequency point; P is the inverse matrix of R; k is the update coefficient of filter W; n is the number of frames; and f is the frequency point. The above formula updates the filter coefficients by estimating the cross-correlation between the near-end and far-end speech signals, as well as the autocorrelation of the near-end speech signal, ultimately filtering out the linear echo component in the near-end speech signal.

[0060] The calculation process of the residual echo signal Res is:

[0061] Res(n, f)=eta(n, f)*Yf(n, f);

[0062] Yf(n,f)=Lsty(n,f)*Lsty * (n, f);

[0063] Ef(n,f)=E(n,f)*E * (n, f);

[0064] EYf(n,f)=Lsty(n,f)*E * (n, f);

[0065] η(n,f)=abs(EYf(n,f)) 2 / (max(Yf(n,f),Ef(n,f))*Yf(n,f)+1e-9).

[0066] Where Res is the estimated echo leakage energy; η is the estimated echo leakage coefficient; Yf is the estimated echo signal energy; Ef is the error signal energy; EYf is the cross-correlation energy between the error signal and the leakage echo; E is the error signal; Lsty is the estimated echo signal; n is the number of frames; and f is the frequency bin. By estimating variations of the cross-correlation coefficient between the echo and error signal, we can better estimate the approximate energy of the nonlinear leakage echo and provide more accurate initialization parameters for model training.

[0067] Figure 2 The schematic diagram of the general structure of the multi-objective neural network in the embodiment of the present invention is as follows: Figure 2 As shown in the figure, the multi-target neural network uses a GRU or LSTM network with a fully connected layer to form a residual network with three different targets. Each target is output at a different layer and serves as the input of the next layer at the same time.

[0068] The objectives of the multi-objective neural network are progressive. The target of the first layer of the residual network is the noise mask, the target of the second layer is the fusion mask of echo and noise, and the target of the third layer is the fusion mask of echo, noise and reverberation. These three layers of masks are numerically related, so the results of noise reduction, echo removal and reverberation can be integrated to obtain the best speech processing effect.

[0069] In a possible embodiment, the beamforming filter in step 400 is a multi-channel Wiener filter (Speech Distortion Weighted Multichannel Wiener Filter), which can be calculated by the following formula:

[0070] H(f)=(R ss (f)+u*R nn (f)) -1 R ss (f)e1;

[0071] R ss (n, f) = λ*R ss (n-1, f)+(1-λ)*mask(n, f)X(n, f)X H (n, f)mask H (n, f);

[0072] Where H is the coefficient of the beamforming filter; R ss is the covariance matrix of the estimated target speech; R nn is the estimated covariance matrix of the non-target signal; u is the speech fidelity coefficient, ranging from (0-1); e1 is the vector (1; 0; ...; 0); λ is the covariance matrix update coefficient, ranging from (0-1); mask is the masking coefficient of the target speech estimated by the neural network; X is the original recorded data signal; n is the number of frames; and f is the frequency bin. This beamforming filter uses the target speech mask estimated by the neural network and specific update coefficients to update the target and non-target covariance matrix, thereby estimating the optimal filter coefficients.

[0073] The speech and noise covariance matrices for each frequency point are calculated using a continuous iterative learning rate approach, and the beamforming filters are updated in real time. Because the preceding and following frames are not directly used, the pre-processed RLS filter and the neural network inputs do not utilize previous and following frame expansion. This ensures low overall system latency while maintaining effective performance, meeting the requirements of real-time communication.

[0074] It is easy to understand that an embodiment of the present application also provides a multi-objective neural network speech processing device based on a microphone array, including a memory and a processor.

[0075] The memory can be used to store instructions, programs, codes, code sets, or instruction sets. The memory can include a program storage area and a data storage area. The program storage area can store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the above-mentioned multi-objective neural network speech processing method, etc. The data storage area can store data involved in the above-mentioned microphone array-based multi-objective neural network speech processing method, etc.

[0076] The processor may include one or more processing cores. The processor executes the various functions of the present application and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory, calling data stored in the memory. The processor may be at least one of a special application integrated circuit, a digital signal processor, a digital signal processing device, a programmable logic device, a field programmable gate array, a central processing unit, a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor functions can also be other, and the embodiments of the present application are not specifically limited.

[0077] If the above-mentioned multi-objective neural network speech processing device based on a microphone array in the embodiment of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read Only Memory), a magnetic disk or an optical disk. In this way, the embodiment of the present invention is not limited to any specific combination of hardware and software.

[0078] In summary, the present application provides a multi-target neural network speech processing method and device based on microphone array, which eliminates linear echoes in multi-channel frequency domain data through RLS filter, with better effect; the echo leakage coefficient is calculated by using the frequency domain correlation of the filter error signal and the estimated echo signal, so that the residual echo energy of each frequency point obtained is more accurate than the residual energy obtained by using a unified echo leakage coefficient for all frequency points; the neural network input feature references BFCC, and multi-target is progressive, the first layer of residual network target is noise mask, the second layer target is It is a fusion mask of echo and noise. The target of the third layer is the fusion mask of echo, noise and reverberation. The three layers of masks have a certain correlation in value. After being divided into multiple targets, they can converge quickly in a smaller network structure. The post-noise reduction, echo removal and reverberation algorithms use multi-channel Wiener filters in conjunction with masks for unified calculation. This has low complexity and avoids speech loss in the process of sequential processing of a single algorithm, thereby achieving the integration of speech noise reduction, echo removal and reverberation goals. It has the characteristics of low computational complexity, low latency, good real-time performance and high speech quality.

[0079] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0080] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0081] The above description is only a preferred example of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multi-objective neural network speech processing method based on microphone array, characterized in that: include: Step 100: Use short-time Fourier transform to obtain frequency domain data of multi-channel recording and frequency domain data of multi-channel stereo re-collection; Step 200: Feed the frequency domain data of the multi-channel recording and the frequency domain data of the multi-channel stereo recapture into an RLS filter to obtain an error signal and an echo estimation signal. Calculate the frequency domain correlation between the error signal and the echo estimation signal as the leakage coefficient of the echo signal, and then multiply it by the error signal to obtain the estimated residual echo signal at each frequency point. Step 300: Extract BFCC features of the error signal and the residual echo signal and input them into a multi-objective neural network to obtain a fused speech energy mask for noise reduction, echo removal, and reverberation removal. Step 400: After filtering the speech energy mask through a beamforming filter, inversely transforming it into the time domain to obtain clean time-domain audio, wherein the beamforming filter is a multi-channel Wiener filter; In step 200, the output of the RLS linear filter is: Lsty(n,f)=W H (n-1,f)X(n,f); E(n,f)=Y(n,f)-Lsty(n,f); W(n,f)=W(n-1,f)+k(n,f)E H (n,f); k(n,f)=P(n-1,f)X(n,f) / (1+X H (n,f)P(n-1,f)X(n,f)); Where X is the frequency domain vector of multi-channel resampled data; W is the RLS filter vector; E is the error signal; Lsty is the echo estimation signal; Y is the near-end speech signal; R is the covariance matrix at each resampled frequency point; P is the inverse matrix of R; k is the update coefficient of the filter W; n is the number of frames; f is the frequency point; The beamforming filter in step 400 is calculated using the following formula: H(f)=(R ss (f)+u*R nn (f)) -1 R ss (f)e1; R ss (n,f)=λ*R ss (n-1,f)+(1-λ)*mask(n,f)X(n,f)X H (n,f)mask H (n,f); Where H is the coefficient of the beamforming filter; R ss is the covariance matrix of the estimated target speech; R nn is the covariance matrix of the estimated non-target signal; u is the speech fidelity coefficient, ranging from (0-1); e1 is the vector (1; 0; ...; 0); λ is the covariance matrix update coefficient, ranging from (0-1); mask is the masking coefficient of the target speech estimated by the neural network; X is the original recording data signal; n is the number of frames; f is the frequency point; The speech and noise covariance matrix of each frequency point is calculated by continuously iterating the learning rate, and the beamforming filter is updated in real time.

2. The multi-objective neural network speech processing method based on microphone array according to claim 1, characterized in that: In step 200, the calculation process of the residual echo signal Res is: Res(n,f)=η(n,f)*Yf(n,f); Yf(n,f)=Lsty(n,f)*Lsty * (n,f); Ef(n,f)=E(n,f)*E * (n,f); EYf(n,f)Lsty(n,f)*E * (n,f) η(n,f)=abs(EYf(n,f)) 2 / (max(Yf(n,f),Ef(n,f))*Yf(n,f)+1e-9); Where Res is the estimated energy of the echo leakage; η is the coefficient of the estimated echo leakage; Yf is the energy of the estimated echo signal; Ef is the energy of the error signal; EYf is the cross-correlation energy between the error signal and the leakage echo; E is the error signal; Lsty is the echo estimation signal; n is the number of frames; and f is the frequency point.

3. The multi-objective neural network speech processing method based on microphone array according to claim 1, characterized in that: The multi-objective neural network uses a GRU or LSTM network in conjunction with a fully connected layer to form a residual network with three different objectives. Each objective is output at a different layer and simultaneously serves as input to the next layer.

4. The multi-objective neural network speech processing method based on microphone array according to claim 3 is characterized in that: The objectives of the multi-objective neural network are progressive. The objective of the first layer of residual network is the noise mask, the objective of the second layer is the fusion mask of echo and noise, and the objective of the third layer is the fusion mask of echo, noise and reverberation. These three layers of masks are numerically related.

5. A device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the microphone array-based multi-objective neural network speech processing method according to any one of claims 1 to 4 when executing the computer program.

Images