Multi-channel speech signal enhancement method and device, computer equipment and storage medium
By calculating the probability of prior speech existence using instantaneous estimator and frequency domain smoothing values in multi-channel speech signal enhancement, the problem of 'tailing effect' in the prior art is solved, and the update rate and noise reduction of the noise covariance matrix are improved Performance, achieving more accurate voice signal extraction.
Patent Information
- Application Number
- CN202210384863.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-13
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-04-13
AI Technical Summary
The existing multi-channel voice signal enhancement method cannot rapidly attenuate the probability of a prior voice after the speech is finished, resulting in a decrease in the update rate of the noise covariance matrix, affecting the noise reduction performance, and a ‘tailing effect’ occurs.
The probability of prior speech existence is estimated by instantaneous estimator and its frequency domain smoothing value, and the noise covariance matrix is calculated by probability weighting method, and spatial filtering is performed using an adaptive beamformer to extract the time-frequency domain speech signal, followed by short-term Four Leaf inverse transformation to obtain the time domain voice signal.
It effectively avoids the 'tailing effect in prior probability estimation, improves the update rate and noise reduction performance of the noise covariance matrix, and improves the voice signal enhancement effect.
Smart Images

Figure CN114882898B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech enhancement, and in particular to a multi-channel speech signal enhancement method and device, computer equipment and storage medium. Background Art
[0002] Multi-channel speech enhancement refers to the extraction of the desired speech signal using the multi-channel noisy signal collected by the microphone array. Compared with single-channel speech enhancement, multi-channel speech enhancement can simultaneously use the information in the time-frequency and spatial domains to extract the desired speech, and in theory it can ensure that the desired speech is distortion-free. Multi-channel speech enhancement plays an important role in conference systems, hearing aids and human-computer interaction systems.
[0003] The commonly used implementation method of multi-channel speech enhancement method is beamforming. According to whether the coefficients of the beamformer are adaptively adjusted according to the collected data, the beamformer can be divided into fixed beamforming and adaptive beamforming. Fixed beamformers generally assume that the noise field obeys some specific spatial distribution forms, and then design the optimal beamformer corresponding to the noise field. When the actual noise field meets the assumed spatial distribution form, the fixed beamformer works better. However, when the actual noise field does not meet the assumed distribution form (which is a common situation in practice), the effect of fixed beamforming on the noise effect becomes worse. Compared with the fixed beamformer, the adaptive beamformer automatically adjusts its coefficients according to the changes in the noise field in the environment, which can theoretically achieve better noise reduction effect. The design of many adaptive beamformers requires a relatively accurate estimation of the noise covariance matrix, and the estimation effect of the noise covariance matrix directly determines the residual noise size in the output signal and the degree of distortion of the desired speech.
[0004] At present, the estimation of noise covariance matrix mainly adopts the iterative smoothing method based on probability weighting, that is, the smoothing factor of noise covariance matrix estimation is adjusted in real time by the probability of speech existence, so as to realize the real-time update of noise covariance matrix. There are many ways to calculate the probability of speech existence, such as directly estimating the probability of speech existence by threshold mapping the interchannel level difference (ILD) or phase difference (IPD) between channels, and transforming the estimation problem of noise covariance matrix into the estimation problem of noise covariance of a single channel by using the characteristics of noise field (such as its spatial characteristics obey the distribution form of diffuse field noise). There are also many studies focusing on the calculation of speech existence probability under binary hypothesis model. This type of method assumes that there are only two possibilities for the composition of mixed signal at a certain moment. The first possibility is that the mixed signal contains only noise signal, and the second possibility is that the mixed signal contains both noise signal and speech signal. By assuming that the collected noise signal and speech signal obey certain specific probability distributions, the corresponding analytic expression of posterior speech existence probability can be obtained. However, this probability model requires the estimation of prior speech existence probability. Existing methods directly use the smoothed estimate to calculate the prior probability of speech presence, and the estimated result will have an estimated "tailing effect": that is, within a period of time after the end of the speech, the calculated value of the prior probability of speech presence cannot decay quickly to a smaller value. The estimated result of this method will reduce the update rate of the noise covariance matrix, thereby affecting the noise reduction performance of the beamformer. Summary of the invention
[0005] The purpose of the present invention is to overcome the problem that the smoothed estimate used in the existing speech signal enhancement method is used to calculate the prior probability of speech existence, which causes the calculated value of the prior probability of speech existence to fail to decay quickly to a smaller value within a period of time after the end of the speech, resulting in an estimated "tailing effect" and the inability to achieve the best noise reduction effect, thereby proposing a multi-channel speech signal enhancement method and device, a computer device and a computer-readable storage medium.
[0006] To achieve the above object, the present invention provides a multi-channel speech signal enhancement method, comprising:
[0007] Step 1) performing short-time Fourier transform on the time domain signals of multiple channels collected by the microphone array to obtain corresponding time-frequency domain signals;
[0008] Step 2) estimating the prior probability of speech existence and calculating the noise covariance matrix;
[0009] Step 3) constructing an adaptive beamformer using the calculated noise covariance matrix, and performing spatial filtering on the collected time-frequency domain multi-channel signals to obtain an estimated time-frequency domain speech signal;
[0010] Step 4) performing short-time inverse Fourier transform on the estimated time-frequency domain speech signal to obtain an estimated time-domain speech signal.
[0011] Further, the step 2) is specifically as follows: using the instantaneous estimator and its frequency domain smoothing value to estimate the prior probability of speech existence and calculate the noise covariance matrix, and using the probability weighting method to realize the noise covariance matrix of each time-frequency point Estimates.
[0012] The step 2) specifically includes:
[0013] Step 201) Calculate an estimated value of the instantaneous signal-to-noise ratio γ(l,k);
[0014] Step 202) using the estimated instantaneous signal-to-noise ratio γ(l,k) to estimate the prior probability of speech existence;
[0015] Step 203) estimating the prior probability of speech existence;
[0016] Step 204) using the estimated priori speech existence probability to calculate the posterior speech existence probability, and estimate the noise covariance matrix;
[0017] Step 205) Iterate and estimate to obtain a better estimate of the noise covariance matrix.
[0018] Furthermore, the method for calculating the estimated value of the instantaneous signal-to-noise ratio γ(l,k) in step 201) is:
[0019]
[0020] in and Respectively represent the instantaneous energy of speech and the noise power spectrum density estimation, and the corresponding calculation method is
[0021]
[0022]
[0023] Where l is the frame index in the time-frequency domain, k is the frequency index in the time-frequency domain, h(l,k)=[h1(l,k),...,h M (l,k)] T is the beamformer used at time l, is the noise smoothing factor at time l, is the estimated value of the noise covariance matrix at time l.
[0024] The step 202) uses the estimated instantaneous signal-to-noise ratio γ(l,k) to estimate the prior probability of speech existence, and the specific method is:
[0025] Perform three smoothing operations on γ(l,k) on the frequency axis range, and obtain smoothing results based on fewer adjacent frequency points, smoothing based on more adjacent frequency points, and smoothing based on all frequencies respectively:
[0026]
[0027]
[0028]
[0029] Where W(·) is the smoothing window, K loc and K glo Represents half the window length of the window function corresponding to local smoothing and broad smoothing.
[0030] The step 203) estimates the priori probability of speech existence, and the specific method is:
[0031] By performing threshold mapping on the three groups of signal-to-noise ratio smoothing results, three groups of prior speech existence probabilities can be obtained, where γ loc (l,k) and γ glo (l,k) uses the same mapping method:
[0032]
[0033]
[0034] Among them, the value of a is 316, and the value of b is 2.5;
[0035] γ fra The threshold mapping method corresponding to (l,k) is:
[0036]
[0037] Among them, & represents the logical AND operation, K1, K2 and K3, K4 are the artificially set cutoff ranges of low frequency and medium and high frequency respectively;
[0038] The prior probability of speech existence is calculated by the following formula:
[0039]
[0040] The step 204) uses the estimated prior speech existence probability to calculate the posterior speech existence probability, and estimates the noise covariance matrix. The specific calculation method is:
[0041] The formula for calculating the probability of the posterior speech existence is:
[0042]
[0043] Where y(l,k)=[Y1(l,k),...,Y M (l,k)] T , Noise covariance matrix It is estimated by the following iterative smoothing:
[0044]
[0045]
[0046] in, is the time-varying smoothing factor, α v is a fixed smoothing factor, the speech covariance matrix Obtained by the following calculation formula:
[0047]
[0048]
[0049] in, is the covariance matrix of the noisy signal, α y Estimate the corresponding fixed smoothing factor.
[0050] The step 205) iteratively estimates to obtain a better noise covariance matrix estimate, and the specific method is:
[0051] Repeat steps 201) to 204), and Uniformly replace with the noise covariance matrix estimated in the last iteration The initial beamformer h(0,k) can be set according to the direction information of the desired speech signal, for example, it can be set to a classic delay-sum beamformer. Noise power spectral density at the initial time The estimation can be performed directly based on the initial silent segment (ie, the portion without speech signal) in the collected data.
[0052] The step 3) specifically includes:
[0053] The adaptive beamformer is constructed using the estimated noise covariance matrix; the adaptive beamformer is expressed as:
[0054]
[0055] Among them, I M is the identity matrix of dimension M×M, and u is I MThe first column of ; α is a parameter for adjusting the amount of noise reduction of the beamformer, and its value range is 0 to 1;
[0056] The time-frequency domain estimate of the speech signal is:
[0057]
[0058] The present invention also provides a multi-channel speech signal enhancement device, comprising:
[0059] A short-time Fourier transform module is used to perform short-time Fourier transform on the time domain signals of multiple channels collected by the microphone array to obtain corresponding time-frequency domain signals;
[0060] A noise covariance matrix estimation module, used to estimate the prior probability of speech existence and calculate the noise covariance matrix;
[0061] The adaptive beamforming module constructs an adaptive beamformer using the calculated noise covariance matrix and performs spatial filtering on the collected time-frequency domain multi-channel signals to obtain an estimated time-frequency domain speech signal;
[0062] The short-time inverse Fourier transform module performs short-time inverse Fourier transform on the estimated time-frequency domain speech signal to obtain an estimated time-domain speech signal.
[0063] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method as claimed in any one of claims 1 to 9 when executing the computer program.
[0064] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method according to any one of claims 1 to 9.
[0065] The multi-channel speech signal enhancement method and device, computer equipment and computer-readable storage medium provided by the present invention have the following advantages:
[0066] 1. The method of the present invention adopts a method based on improving the calculation of the prior probability of speech existence to estimate the noise covariance matrix. The estimation simultaneously utilizes the instantaneous estimator and the smoothing estimator, which can effectively avoid the "tailing effect" that occurs in the prior probability estimation.
[0067] 2. The noise covariance matrix estimation method based on the calculation of the prior probability of speech existence adopted by the method of the present invention can estimate the noise covariance matrix more quickly and accurately, thereby improving the noise reduction performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1 The figure shows a schematic diagram of using a microphone array to collect audio signals in a real environment;
[0069] Figure 2 Shown is a flow chart of a multi-channel speech signal enhancement method;
[0070] FIG3( a ) is a diagram showing the calculation result of the prior speech existence probability estimated using the existing method;
[0071] FIG3( b ) is a diagram showing the calculation result of the posterior speech existence probability estimated using the existing method;
[0072] FIG. 4( a ) is a diagram showing the calculation result of the priori speech existence probability estimated by the method of the present invention;
[0073] FIG4( b ) is a diagram showing the calculation result of the posterior speech existence probability estimated by the method of the present invention;
[0074] Figure 5 Shown is a schematic diagram of the multi-channel speech signal enhancement system module. DETAILED DESCRIPTION
[0075] The technical solution provided by the present invention is further illustrated below in conjunction with embodiments.
[0076] The present invention provides a multi-channel speech signal enhancement method and system, wherein the method comprises:
[0077] Performing short-time Fourier transform on the time domain signals of multiple channels collected by the microphone array to obtain corresponding time-frequency domain signals; estimating the probability of the existence of prior speech and calculating the noise covariance matrix; constructing an adaptive beamformer using the calculated noise covariance matrix, and performing spatial filtering on the collected time-frequency domain multi-channel signals to obtain estimated time-frequency domain speech signals; performing short-time inverse Fourier transform on the estimated time-frequency domain speech signals to obtain estimated time domain speech signals.
[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0079] like Figure 1As shown, when using a microphone array to collect audio signals in an actual environment, in addition to collecting the signal of the desired speaker, the reverberation signal and noise signal of the speaker's voice will inevitably be collected. The adaptive beamformer system extracts the desired speech signal by linearly filtering the collected multiple channel signals. When designing an adaptive beamformer, it is necessary to provide a more accurate estimate of the noise covariance matrix. The existing estimation method will have a "tailing effect" when estimating the prior speech probability, which makes the residual noise in the beamformer output larger and affects the enhanced speech quality. The main reason for this phenomenon is that the existing method directly relies on the smoothed estimate to calculate the prior speech existence probability.
[0080] The present invention provides a multi-channel speech signal enhancement method, such as Figure 2 As shown, including:
[0081] 101: Step 1) Short-time Fourier transform: Perform short-time Fourier transform on the time domain signals of multiple channels collected by the microphone array to obtain corresponding time-frequency domain signals.
[0082] 102: Step 2) Noise covariance matrix estimation: Estimate the prior probability of speech existence and calculate the noise covariance matrix, and use the probability weighting method to achieve the estimation of the noise covariance matrix at each time-frequency point.
[0083] 103: Step 3) Adaptive beamforming: An adaptive beamformer is constructed using the estimated noise covariance matrix, and spatial filtering is performed on the collected time-frequency domain multi-channel signal to obtain an estimated time-frequency domain speech signal.
[0084] 104: Step 4) Inverse short-time Fourier transform: Perform inverse short-time Fourier transform on the estimated time-frequency domain speech signal to obtain an estimated time-domain speech signal.
[0085] Among them, the specific method of step 101 ) short-time Fourier transform includes:
[0086] Perform short-time Fourier transform on the time domain signals of the M channels collected by the microphone to obtain the corresponding time-frequency domain signals of the M channels. Let the signal collected by the mth channel at time n be y m (n), the corresponding signal in the time-frequency domain is Y m (l,k), where l is the frame index in the time-frequency domain, k is the frequency index in the time-frequency domain, and 1≤k≤K, 1≤l≤L. K corresponds to the number of points in the short-time Fourier transform, and L corresponds to the number of frames after the short-time Fourier transform. Assuming the sampling rate is 16000Hz, the number of Fourier transform points is 512, the length of the acquired signal is 1s, and the inter-frame overlap rate of the short-time Fourier transform is 75%, then K=512, L=(16000Hz*1s-512) / (512*(1-0.75))+1=122.
[0087] Because of y m (n) is a real signal, and the time-frequency domain information obtained by its short-time Fourier transform is redundant in the frequency axis range. When processing, only half of the frequency index value is taken, that is, in The operator represents rounding down. When performing short-time Fourier transform, the signal length of each frame needs to be selected. Based on experience, the signal frame length is generally between 32ms and 64ms.
[0088] 102 Step 2) The specific method of estimating the noise covariance matrix includes:
[0089] Step 2) Estimate the prior probability of speech existence and calculate the noise covariance matrix, and use the probability weighting method to estimate the noise covariance matrix at each time-frequency point
[0090] Step 201) Calculate the estimated value of the instantaneous signal-to-noise ratio γ(l,k), that is,
[0091]
[0092] in and Respectively represent the instantaneous energy of speech and the noise power spectrum density estimation, and the corresponding calculation method is
[0093]
[0094]
[0095] where h(l,k)=[h1(l,k),...,h M (l,k)] T is the beamformer used at time l, is the noise smoothing factor at time l, is the estimated value of the noise covariance matrix at time l. For the specific calculation methods of the three, please refer to the subsequent description of step 301), step 204) and step 205).
[0096] Step 202) The estimated instantaneous signal-to-noise ratio γ(l,k) is used to calculate the priori speech existence probability. Specifically, the instantaneous signal-to-noise ratio is firstly smoothed on three frequency axis ranges to obtain smoothing results based on fewer adjacent frequency points, smoothing results based on more adjacent frequency points, and smoothing results based on all frequencies. The purpose of smoothing is to use the correlation of the time-frequency domain signal in different frequency ranges to achieve a more accurate estimation of the priori speech existence probability. The calculation methods of these three smoothings are:
[0097]
[0098]
[0099]
[0100] Where W(·) is a smoothing window, which can be a normalized Hamming window or Kaiser window. loc and K glo K represents half the window length of the window function corresponding to local smoothing and broad smoothing. loc Generally, 1 is taken, K glo Generally, it is a constant greater than 3.
[0101] Step 203) Estimate the priori speech existence probability. By threshold mapping the three groups of signal-to-noise ratio smoothing results, three groups of prior speech existence probabilities can be obtained, where γ loc (l,k) and γ glo (l,k) uses the same mapping method, that is,
[0102]
[0103]
[0104] Among them, a can be taken as 316 in practice, and b can be taken as 2.5.
[0105] γ fra The threshold mapping method corresponding to (l,k) is:
[0106]
[0107] Among them, & represents the logical AND operation, K1, K2 and K3, K4 are the artificially set low-frequency and medium-high frequency cutoff ranges. When the sampling rate of the signal is 16000Hz, the frequency range of the low-frequency cutoff can be set to 500Hz~2000Hz, and the frequency range of the high-frequency cutoff can be set to 4000Hz~8000Hz. Th1 and Th2 correspond to the thresholds within these two frequency bands, which are set to 2 and 4 respectively. The prior probability of speech existence can be calculated as:
[0108]
[0109] Step 204) Calculate the a posteriori speech existence probability using the estimated prior speech existence probability, and estimate the noise covariance matrix. Generally, assuming that the collected multi-channel speech signal and noise signal obey independent multivariate Gaussian distributions, calculating the a posteriori speech existence probability has a good effect. The corresponding a posteriori speech existence probability can be expressed as:
[0110]
[0111] where y(l,k)=[Y1(l,k),...,Y M (l,k)] T , Get the probability p of the posterior speech existence corresponding to a certain time-frequency point (l, k) x (l,k), the noise covariance matrix can be estimated by the following iterative smoothing:
[0112]
[0113]
[0114] in, is the time-varying smoothing factor, and α v is a fixed smoothing factor that determines the update rate of the noise covariance matrix when the expected speech does not exist. v The general value range is between 0.9 and 1.
[0115] Covariance matrix of noisy signal The calculation is done using the following formula:
[0116]
[0117] Among them, α y is a fixed smoothing factor, which generally ranges from 0.9 to 1. The corresponding covariance matrix of the speech signal is It can be expressed as
[0118] Step 205) Iterate and estimate to obtain a better noise covariance matrix estimate. Repeat steps 201) to 204), but now the Uniformly replace with the noise covariance matrix estimated in the previous iteration The initial beamformer h(0,k) can be set according to the direction information of the desired speech signal, for example, it can be set to a classic delay-sum beamformer. Noise power spectral density at the initial time The estimation can be performed directly based on the first silent segment (i.e., the part without speech signal) in the collected data. Theoretically, the iterative calculation process can be repeated many times to improve the accuracy of the noise covariance matrix estimation. However, actual calculations show that the noise covariance matrix can be estimated more accurately with one iteration.
[0119] 103 Step 3) The specific method of adaptive beam forming includes:
[0120] First, the estimated noise covariance matrix is used to construct an adaptive beamformer. Commonly used adaptive beamformers include multichannel Wiener filtering (MWF) and minimum variance distortionless response (MVDR) beamformer. They can be uniformly expressed as:
[0121]
[0122] Among them I M is the identity matrix of dimension M×M, and u is I M In the first column, α is a weighting factor that determines the noise reduction performance of the beamformer. When α=0, it corresponds to the MVDR beamformer, when α=1, it corresponds to the standard MWF, and when α>1, it corresponds to the MWF with a stronger noise reduction effect. The value of α can be selected according to the actual needs of the noise reduction amount and the speech distortion amount: if you want to expect less speech distortion, set α=0, and if you want more noise reduction, set α to a value greater than 1.
[0123] After solving the beamformer using the above formula, the time-frequency domain estimation value of the speech signal can be expressed as:
[0124]
[0125] 104 Step 4) The specific method of short-time inverse Fourier transform includes:
[0126] For the time-frequency domain speech signal obtained in step 3) Perform inverse short-time Fourier transform to obtain the time domain signal of the desired speech.
[0127] Considering that the short-time Fourier transform of real signal has the characteristics of conjugate symmetry, we first use Restore the time-frequency domain speech signal in the entire frequency range, and then perform inverse Fourier transform and windowing synthesis operations on it to obtain the corresponding time-domain speech signal estimate
[0128] Figures 3(a) and 3(b) respectively show a priori speech probability and a posteriori speech probability estimated using existing methods. It can be clearly seen from Figures 3(a) and 3(b) that this type of method has a "tailing effect". Compared with the existing prior speech probability calculation method, the calculation method proposed in the present invention uses the estimation of the instantaneous signal-to-noise ratio after smoothing in the frequency domain, which can avoid the influence of using only the smoothed estimate on the update speed of the noise covariance matrix. Figures 4(a) and 4(b) show the calculation results of the prior speech probability and a posteriori speech probability proposed by the present invention. It can be found that the method disclosed in the present invention significantly improves the "tailing effect".
[0129] Finally, we further explain why the multi-channel speech enhancement method based on improved prior speech existence probability calculation proposed by the present invention can achieve better enhancement effect. The existing noise covariance matrix estimation method often relies only on smoothed statistics when calculating the prior probability, which will cause the estimated prior speech existence probability to fail to decay to a smaller value quickly after the speech ends, thereby affecting the update rate of the noise covariance matrix. In response to this problem, the present invention adopts instantaneous estimators and their frequency domain smoothing values to achieve a better estimation of the prior speech existence probability. Since the estimated instantaneous signal-to-noise ratio is generally lower after the speech ends, the estimation method proposed by the present invention can effectively eliminate the "tailing effect" of frequency estimation encountered in traditional methods, thereby ensuring the update rate of the noise covariance matrix.
[0130] like Figure 5 As shown, the present invention also provides a multi-channel speech signal enhancement system, the system comprising:
[0131] A short-time Fourier transform module 301, which is used to transform the collected multi-channel time domain signal into the time-frequency domain, including framing, windowing and Fourier transform;
[0132] A noise covariance matrix estimation module 302 estimates the noise covariance matrix by using an improved priori speech existence probability;
[0133] The adaptive beamforming module 303 constructs an adaptive beamformer using the estimated noise covariance matrix and filters the collected signal in the time-frequency domain to obtain an estimated speech signal in the time-frequency domain;
[0134] And a short-time inverse Fourier transform module 304 is used to transform the estimated time-frequency domain speech signal into the time domain, including inverse Fourier transform, windowing and synthesis.
[0135] The present invention also provides a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.
[0136] The user interface may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).
[0137] It is understood that the memory in the disclosed embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.
[0138] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.
[0139] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.
[0140] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in the application program, to:
[0141] Execute the steps of the multi-channel speech signal enhancement method.
[0142] The multi-channel speech signal enhancement method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the present invention can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0143] It is understood that the embodiments described in the present invention can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.
[0144] For software implementation, the technology of the present invention can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present invention. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.
[0145] The present invention also provides a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method can be implemented.
[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.
Claims
1. A multi-channel speech signal enhancement method, comprising: Step 1) performing short-time Fourier transform on the time domain signals of multiple channels collected by the microphone array to obtain corresponding time-frequency domain signals; Step 2) estimating the prior probability of speech existence and calculating the noise covariance matrix; Step 3) constructing an adaptive beamformer using the calculated noise covariance matrix, and performing spatial filtering on the collected time-frequency domain multi-channel signals to obtain an estimated time-frequency domain speech signal; Step 4) performing short-time inverse Fourier transform on the estimated time-frequency domain speech signal to obtain an estimated time-domain speech signal; The step 2) specifically includes: Step 201) Calculate an estimated value of the instantaneous signal-to-noise ratio γ(l,k); Step 202) using the estimated instantaneous signal-to-noise ratio γ(l,k) to estimate the prior probability of speech existence; Step 203) estimating the prior probability of speech existence; Step 204) using the estimated priori speech existence probability to calculate the posterior speech existence probability, and estimate the noise covariance matrix; Step 205) repeats steps 201) to 204) to iteratively estimate and obtain a better noise covariance matrix estimate; The step 3) specifically includes: The adaptive beamformer is constructed using the estimated noise covariance matrix; the adaptive beamformer is expressed as: Among them I M is the identity matrix of dimension M×M, and u is I M The first column of ; α is a parameter for adjusting the amount of noise reduction of the beamformer, and its value range is 0 to 1; Represents the estimate of the noise covariance matrix at each time-frequency point; The time-frequency domain estimate of the speech signal is:
2. The multi-channel speech signal enhancement method according to claim 1, characterized in that: The step 2) uses the instantaneous estimator and its frequency domain smoothing value to estimate the priori speech existence probability and calculate the noise covariance matrix, and uses the probability weighting method to realize the noise covariance matrix of each time-frequency point Estimates.
3. The multi-channel speech signal enhancement method according to claim 1, characterized in that: The method for calculating the estimated value of the instantaneous signal-to-noise ratio γ(l,k) in step 201) is: in, and They represent the instantaneous energy of speech and the noise power spectrum density estimation respectively, and the corresponding calculation method is: Among them, the superscript H represents the conjugate transpose operation of the vector, that is, the imaginary part of each complex element of the vector is changed to the opposite of the original number, and then the row vector after the element transformation is converted into a column vector; l is the frame index in the time-frequency domain; k is the frequency index in the time-frequency domain; h(l,k)=[h1(l,k),...,h M (l,k)] T is the beamformer used at time l; superscript T Represents the vector transpose operation, which converts a row vector into a column vector; is the noise smoothing factor at time l; is the estimated value of the noise covariance matrix at time l.
4. The multi-channel speech signal enhancement method according to claim 1, characterized in that: The step 202) uses the estimated instantaneous signal-to-noise ratio γ(l,k) to estimate the prior probability of speech existence, and the specific method is: Perform three smoothing operations on γ(l,k) on the frequency axis range, and obtain smoothing results based on fewer adjacent frequency points, smoothing based on more adjacent frequency points, and smoothing based on all frequencies respectively: Where W(·) is the smoothing window, K loc and K glo Represents half the window length of the window function corresponding to local smoothing and broad smoothing.
5. The multi-channel speech signal enhancement method according to claim 1, characterized in that: The step 203) estimates the priori probability of speech existence, and the specific method is: First, three sets of prior speech existence probabilities can be obtained by threshold mapping the three sets of signal-to-noise ratio smoothing results, where γ loc (l,k) and γ glo (l,k) uses the same mapping method: Among them, the value of a is 316, and the value of b is 2.5; γ fra The threshold mapping method corresponding to (l,k) is: Among them, & represents the logical AND operation, K1, K2 and K3, K4 are the artificially set cutoff ranges of low frequency and medium and high frequency respectively; Then, the prior probability of speech existence is calculated by the following formula:
6. The multi-channel speech signal enhancement method according to claim 1, characterized in that: The step 204) uses the estimated prior speech existence probability to calculate the posterior speech existence probability, and estimates the noise covariance matrix. The specific calculation method is: The formula for calculating the probability of the posterior speech existence is: in, Noise covariance matrix It is estimated by the following iterative smoothing: in, is the time-varying smoothing factor, α v is a fixed smoothing factor, the speech covariance matrix Obtained by the following calculation formula: in, is the covariance matrix of the noisy signal, α y Estimate the corresponding fixed smoothing factor.
7. The multi-channel speech signal enhancement method according to claim 1, characterized in that: The step 205) iteratively estimates to obtain a better noise covariance matrix estimate, and the specific method is: Repeat steps 201) to 204), and Uniformly replace with the noise covariance matrix estimated in the last iteration The beamformer h(0,k) at the initial moment is set according to the direction information of the desired speech signal; the noise power spectrum density at the initial moment The estimation is made directly from the initial silence segment in the acquired data.
8. A multi-channel speech signal enhancement device, comprising: A short-time Fourier transform module is used to perform short-time Fourier transform on the time domain signals of multiple channels collected by the microphone array to obtain corresponding time-frequency domain signals; A noise covariance matrix estimation module, used to estimate the prior probability of speech existence and calculate the noise covariance matrix; The adaptive beamforming module constructs an adaptive beamformer using the calculated noise covariance matrix and performs spatial filtering on the collected time-frequency domain multi-channel signals to obtain an estimated time-frequency domain speech signal; A short-time inverse Fourier transform module performs a short-time inverse Fourier transform on the estimated time-frequency domain speech signal to obtain an estimated time-domain speech signal; The execution process of the noise covariance matrix estimation module includes: Step 201) Calculate an estimated value of the instantaneous signal-to-noise ratio γ(l,k); Step 202) using the estimated instantaneous signal-to-noise ratio γ(l,k) to estimate the prior probability of speech existence; Step 203) estimating the prior probability of speech existence; Step 204) using the estimated priori speech existence probability to calculate the posterior speech existence probability, and estimate the noise covariance matrix; Step 205) repeats steps 201) to 204) to iteratively estimate and obtain a better noise covariance matrix estimate; The execution process of the short-time inverse Fourier transform module includes: The adaptive beamformer is constructed using the estimated noise covariance matrix; the adaptive beamformer is expressed as: Among them I M is the identity matrix of dimension M×M, and u is I M The first column of ; α is a parameter for adjusting the amount of noise reduction of the beamformer, and its value range is 0 to 1; Represents the estimate of the noise covariance matrix at each time-frequency point; The time-frequency domain estimate of the speech signal is:
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Voice beam forming method based on channel associated time-frequency mask
CN111508516A
Voice signal enhancement method and system based on voice signal prior probability characteristics
CN117373473A
Time-frequency masking and deep neural network-based sound source direction estimation method
WO2020042708A1
