A voice dereverberation method, device, apparatus and medium

Abstract

The application provides a speech dereverberation method, device, equipment and medium, and relates to the technical field of speech signal processing. The method comprises the following steps: obtaining a reverberation signal through a microphone array, performing short-time Fourier transform on the reverberation signal to obtain a time-frequency domain signal; using a plurality of high-directivity beamformers to process the time-frequency domain signal respectively to obtain a plurality of time-frequency signals; designing a dereverberation filter based on a weighted linear prediction algorithm according to the plurality of time-frequency signals; performing dereverberation filtering on the time-frequency signal output by one of the high-directivity beamformers by using the dereverberation filter to obtain an expected time-frequency domain signal after dereverberation; and performing inverse short-time Fourier transform on the expected time-frequency domain signal to obtain a dereverberation time domain signal. The application can improve the performance of the existing WPE algorithm, reduce the computational complexity, improve the dereverberation performance of the WPE algorithm in a noisy environment, remove speech reverberation and improve speech quality.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, specifically to a speech de-reverberation method, apparatus, device, and medium. Background Technology

[0002] Speech processing technology has been widely applied in various military and civilian systems. For example, in video conferencing, voice communication technology enables people to communicate seamlessly over long distances; in the field of humanoid robots, intelligent voice technology enables people to interact with robots. In these voice communication and intelligent voice interaction applications, one of the most critical tasks is to pick up the target sound source signal through a microphone. However, in real acoustic environments, the signal received by the microphone includes not only the direct signal but also reverberation. Reverberation has a significant adverse effect on acoustic systems, such as reducing the speech quality of communication systems, affecting the speech intelligibility of hearing aids, and impacting the speech recognition rate of speech recognition systems. Therefore, speech dereverberation has always been a hot research topic in the field of speech processing. Reverberation severely degrades speech quality and intelligibility, posing a significant challenge to voice communication systems and intelligent voice interaction applications.

[0003] In dereverberation problems, the Weighted Prediction Error (WPE) algorithm has demonstrated effectiveness in practical applications. The WPE algorithm divides reverberation into early and late reverberation, calculating the late reverberation from the array observation signal using multi-channel delay linear prediction technology and subtracting it to obtain the desired signal. However, the WPE algorithm's signal model only considers reverberation and not noise, resulting in poor robustness in noisy environments. In real-world sound pickup environments, noise is almost unavoidable, thus hindering the WPE algorithm's effectiveness in practical applications. Furthermore, the WPE algorithm requires a large dereverberation filter, involving numerous matrix inversions and complex multiplications, leading to high computational complexity and preventing direct application to real-time systems. Therefore, improving the performance of existing WPE algorithms to reduce computational complexity while enhancing their dereverberation performance in noisy environments is essential. Summary of the Invention

[0004] The purpose of this invention is to provide a speech dereverberation method, apparatus, device, and medium to address the problems of poor performance and high computational complexity of the WPE algorithm in practical dereverberation applications. This invention can improve the performance of existing WPE algorithms, reducing computational complexity while enhancing the dereverberation performance of the WPE algorithm in noisy environments.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a speech de-reverberation method, comprising:

[0006] Step 1: Acquire the reverberation signal through a microphone array, and perform a short-time Fourier transform on the reverberation signal to obtain the time-frequency domain signal;

[0007] Step 2: Use multiple highly directional beamformers to process the time-frequency domain signal to obtain the time-frequency signal output by the multiple highly directional beamformers;

[0008] Step 3: Based on the time-frequency signals output by multiple highly directional beamformers, design a dereverberation filter using a weighted linear prediction algorithm;

[0009] Step 4: Use a dereverberation filter to dereverberate the time-frequency signal output from one of the high-directivity beamformers to obtain the desired time-frequency domain signal after dereverberation.

[0010] Step 5: Perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain the dereverberated time-domain signal.

[0011] According to the speech de-reverberation method provided by the present invention, the microphone array includes M microphones, and the M microphones correspond to M channels; in step 1, the first... Frame, First The time-frequency domain signals of M channels at each frequency point are:

[0012]

[0013] In the formula, It is the first Frame, First Time-frequency domain signal of M channels at a given frequency point. This is the received signal from the Mth microphone. It is the first Frame, First Reverberation signals of M channels at a frequency point It is the first Frame, First Noise signals from M channels at a frequency point.

[0014] According to the speech de-reverberation method provided by the present invention, the highly directional beamformer is a superdirectional beamformer, a first-order supercardioid differential beamformer, or a first-order hypercardioid differential beamformer.

[0015] According to a speech de-reverberation method provided by the present invention, in step 2, the time-frequency domain signal is processed by Q highly directional beamformers, wherein the first... q The time-frequency signal output by the highly directional beamformer is:

[0016]

[0017] In the formula, For the first q A highly directional beamformer, This indicates the conjugate transpose operation.

[0018] According to the speech de-reverberation method provided by the present invention, step 3 specifically includes:

[0019] Step 31: Obtain the desired signal from the signal model based on the weighted linear prediction algorithm;

[0020] Step 32: Model the desired signal as a complex Gaussian process with time-varying variance, and solve for the dereverberation filter using the maximum likelihood function method.

[0021] According to the speech de-eruption method provided by the present invention, the desired signal is:

[0022]

[0023] in,

[0024]

[0025]

[0026] In the formula, It is the expected signal estimated at the current time and frequency point. For length is The A de-reverberation filter for each frequency point; The stack length is The time-frequency signal output by the highly directional beamformer; The preset delay number; For the stacked first frame Q The time-frequency signal output by a highly directional beamformer.

[0027] According to a speech de-reverberation method provided by the present invention, in step 32, the cost function is obtained by the maximum likelihood function method as follows:

[0028]

[0029] In the formula, is the time-varying variance estimate of the desired signal; N is the total number of frames.

[0030] By minimizing the cost function, the length is obtained. The The de-reverberation filter for each frequency point is:

[0031]

[0032] in,

[0033]

[0034] In the formula, For size is The weighted covariance matrix, The length is The weighted covariance vector, This indicates the conjugate operation.

[0035] In a second aspect, the present invention provides a speech de-reverberation device, comprising:

[0036] The transformation unit is used to acquire the reverberation signal through the microphone array, perform a short-time Fourier transform on the reverberation signal, and obtain a time-frequency domain signal.

[0037] The processing unit is used to process the time-frequency domain signal using multiple highly directional beamformers to obtain the time-frequency signal output by the multiple highly directional beamformers.

[0038] Design unit for designing dereverberation filters based on a weighted linear prediction algorithm using time-frequency signals output from multiple highly directional beamformers;

[0039] The dereverberation unit is used to perform dereverberation filtering on the time-frequency signal output by one of the high directivity beamformers to obtain the desired time-frequency domain signal after dereverberation.

[0040] The inverse transform unit is used to perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a dereverberated time-domain signal.

[0041] Thirdly, the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech de-reverberation method of the first aspect.

[0042] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the speech de-reverberation method of the first aspect.

[0043] The technical solution of the present invention has at least the following technical effects:

[0044] This invention provides a speech dereverberation method, apparatus, device, and medium. It achieves speech dereverberation by jointly optimizing highly directional beamforming and the WPE algorithm. First, multiple highly directional beamformers are used to preprocess the reverberation signal (observation signal) acquired by a microphone array. Then, dereverberation is performed on the output of the highly directional beamformers. A highly directional beamformer is a device used for fixed beamforming, including highly directional types of superdirectional beamformers and differential beamformers, such as supercardioid differential beamformers and hypercardioid differential beamformers. After preprocessing by the highly directional beamformers, some reverberation in the observation signal is removed, and most noise is also removed. However, highly directional beamformers cannot completely suppress reverberation. Therefore, further removing residual reverberation from the output of the highly directional beamformers can further improve speech quality. Unlike simply combining beamforming with the WPE algorithm, the joint optimization process of this invention involves predicting late reverberation from the outputs of multiple highly directional beamformers and generating a dereverberation reference signal from the outputs of one of the highly directional beamformers, i.e., the time-frequency signal output by one of the highly directional beamformers. Furthermore, since the number of highly directional beamformers is less than the number of microphones, the dereverberation filter is shorter than in traditional methods, significantly reducing the computational complexity of the algorithm. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] In the attached diagram:

[0047] Figure 1 This is a flowchart of the speech de-reverberation method of the present invention;

[0048] Figure 2 This is the beam pattern of the highly directive beamformer used in this invention;

[0049] Figure 3 This is a graph showing the computational complexity of different dereverberation methods used in this invention with different numbers of microphones.

[0050] Figure 4 This is a structural block diagram of the electronic device of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] The following detailed description of some embodiments of the present invention will be provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0053] This invention, starting from practical application scenarios, considers the impact of reverberation on microphone pickup signals in voice communication and intelligent voice interaction applications such as video conferencing and humanoid robots. It proposes a dereverberation method that jointly optimizes highly directional microphone array beamforming with the WPE algorithm, reducing the computational complexity of the WPE algorithm and improving its robustness. This invention first uses highly directional beamforming technology to process the array observation signal. Highly directional beamforming is a fixed beamforming technique; when the array structure is fixed, the filter coefficients are correspondingly determined, resulting in very low computational cost. Simultaneously, its high directivity can suppress some reverberation and remove most of the noise. Based on this, jointly optimizing the WPE dereverberation technique further removes residual reverberation and improves voice quality. Furthermore, the multi-channel observation signal is transformed into a single-channel signal after processing by a highly directional beamformer, significantly shortening the length of the dereverberation filter and reducing its computational complexity. The method provided by this invention can be implemented using computer software technology.

[0054] Example 1

[0055] Please see Figure 1 This embodiment provides a speech de-reverberation method, including:

[0056] Step 1: Acquire the reverberation signal through a microphone array, and perform a short-time Fourier transform (STFT) on the reverberation signal to obtain a time-frequency domain signal;

[0057] Specifically, in step 1, the reverberation signal is the multi-channel time-domain signal acquired through the microphone array. A short-time Fourier transform is performed on the multi-channel time-domain signal to obtain the multi-channel time-frequency domain signal. The short-time Fourier transform is then used to transform the reverberation signal from... The reverberation signal acquired by a microphone array consisting of M microphones is converted to the time-frequency domain, where M microphones correspond to M channels. Among them, the... Frame, First The time-frequency domain signals of M channels at each frequency point are:

[0058]

[0059] In the formula, It is the first Frame, First Time-frequency domain signal of M channels at a given frequency point. This is the received signal from the Mth microphone. and Definition and Similarly, these are reverberation signal and noise signal, respectively.

[0060] Step 2: Use multiple highly directional beamformers to process the time-frequency domain signal to obtain the time-frequency signal output by the multiple highly directional beamformers;

[0061] Specifically, firstly provide Q A highly directional beamformer, which can be a super-directional beamformer, a first-order supercardioid differential beamformer, or a first-order hypercardioid differential beamformer, etc., beamformers with high directivity. Figure 2 The images show the beammaps of the highly directional beamformer used in this invention, where, from left to right, they are the beammaps of a superdirectional beamformer, a first-order high-cardioid differential beamformer, and a first-order supercardioid differential beamformer. Utilizing... Q A highly directional beamformer processes multi-channel time-frequency domain signals, wherein the first... q The time-frequency signal output by the highly directional beamformer is:

[0062]

[0063] In the formula, For the first q A highly directional beamformer, This indicates the conjugate transpose operation.

[0064] Step 3: Based on the time-frequency signals output by multiple highly directional beamformers, design a dereverberation filter using a weighted linear prediction algorithm;

[0065] Specifically, the design method for dereverberation filters uses a weighted linear prediction algorithm. The design process for dereverberation filters involves two steps:

[0066] Step 31: Derive the desired signal from the signal model based on the WPE algorithm. In the WPE signal model, the desired signal is defined as the direct sound plus early reflections, equal to the time-frequency signal output by the high-directivity beamformer minus the late reverberation. The late reverberation is obtained by filtering the signals processed by multiple high-directivity beamformers using a dereverberation filter. The desired signal is:

[0067]

[0068] in,

[0069]

[0070]

[0071] In the formula, It is the expected signal estimated at the current time and frequency point. For length is The A de-reverberation filter for each frequency point; The stack length is The time-frequency signal output by the highly directional beamformer; The preset delay number; For the stacked first frame Q The time-frequency signal output by a highly directional beamformer.

[0072] Step 32: Next, model the desired signal as a complex Gaussian process with time-varying variance, and solve for the dereverberation filter using the maximum likelihood function method.

[0073] The cost function obtained through the maximum likelihood function method is:

[0074]

[0075] In the formula, Let N be the time-varying variance estimate of the desired signal; N is the total number of frames. By minimizing the cost function, the length of the calculated value can be obtained. The The de-reverberation filter for each frequency point is:

[0076]

[0077] in:

[0078]

[0079] In the formula, For size is The weighted covariance matrix, The length is The weighted covariance vector, superscript This represents the conjugate operation. After performing the above operation on each frequency point, a full-band linear prediction-based dereverberation filter can be obtained and stored for subsequent dereverberation operations.

[0080] Step 4: Use a dereverberation filter The time-frequency signal output from one of the highly directional beamformers is subjected to dérake filtering to obtain the desired dérake-free time-frequency domain signal. Specifically, using...

[0081]

[0082] Can be used for the first Frame, First The desired signal at each frequency point is de-reverberated. This operation is then applied to the desired signals of all frames and frequency points to obtain the de-reverberated desired time-frequency domain signal.

[0083] Step 5: Perform an inverse short-time Fourier transform (iSTFT) on the desired time-frequency domain signal after déresonance to convert it to the time domain, and obtain the final déresonance time-domain signal.

[0084] It should be noted that in the linear prediction-based dereverberation process, late reverberation is estimated by using past L-frame observations with delays, and then subtracted from the current observed signal. This is achieved by selecting the output of one of the highly directional beamformers as the reference signal. A special case considers a single highly directional beamformer, in which case only a beamformer of length [missing information] is needed. L A proper dereverberation filter can significantly reduce the computational complexity of dereverberation. In some cases, using multiple highly directional beamformers may enhance dereverberation performance, but at the cost of increased computational complexity. However, as long as... Q The value is less than The computational complexity can still be kept lower than that of the traditional WPE algorithm.

[0085] Figure 3 This is a graph showing the computational complexity of different dereverberation methods used in this embodiment with different numbers of microphones. WPE represents the dereverberation method using the traditional WPE algorithm; DMA-WPE represents the dereverberation method using the output of a differential beamformer as the reference signal for the WPE algorithm; MWPE-DMA represents the dereverberation method using a MIMO WPE algorithm coupled with a differential beamformer; JDMA-WPE represents the dereverberation method provided by this invention, which combines the WPE algorithm and highly directional beamforming. Q represents the number of differential beamformers used. It can be seen that as the number of microphones increases, the computational complexity of WPE, DMA-WPE, and MWPE-DMA increases quadratically, while the computational complexity of the speech dereverberation method provided by this invention hardly increases. Therefore, this invention is more suitable for situations with multiple microphones.

[0086] Example 2

[0087] Based on the same inventive concept, this embodiment provides a speech de-reverberation device, which corresponds to the method of Embodiment 1, and the device includes:

[0088] The transformation unit is used to acquire the reverberation signal through the microphone array, perform a short-time Fourier transform on the reverberation signal, and obtain a time-frequency domain signal.

[0089] The processing unit is used to process the time-frequency domain signal using multiple highly directional beamformers to obtain the time-frequency signal output by the multiple highly directional beamformers.

[0090] Design unit for designing dereverberation filters based on a weighted linear prediction algorithm using time-frequency signals output from multiple highly directional beamformers;

[0091] The dereverberation unit is used to perform dereverberation filtering on the time-frequency signal output by one of the high directivity beamformers to obtain the desired time-frequency domain signal after dereverberation.

[0092] The inverse transform unit is used to perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a dereverberated time-domain signal.

[0093] Example 3

[0094] Figure 4 An example is a schematic diagram of the structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The memory 830 stores a computer program that can run on the processor 810, and when the processor 810 executes the computer program, it implements the speech de-reverberation method of Embodiment 1.

[0095] Example 4

[0096] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech de-reverberation method of Embodiment 1.

[0097] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0098] In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this embodiment, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable program. This propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0099] The computer-readable storage medium described above can be used to write computer programs for executing this embodiment in one or more programming languages ​​or combinations thereof. These programming languages ​​include object-oriented programming languages—such as Java, Python, and C++—and conventional procedural programming languages—such as C or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] In summary, the speech dereverberation method, apparatus, device, and medium provided by this invention employ a strategy combining highly directional microphone array beamforming with the WPE algorithm. The output of the highly directional beamformer is used as the reference signal for dereverberation in the WPE algorithm, and late reverberation is also predicted from the output of the highly directional beamformer. This invention first designs multiple highly directional beamformers to preprocess the observed signal, and then predicts late reverberation from the output of the multiple highly directional beamformers, thereby improving dereverberation performance in noisy environments and significantly reducing computational complexity by shortening the length of the dereverberation filter.

[0101] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that the invention is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A speech dereverberation method, characterized in that, include: Step 1: Acquire the reverberation signal through a microphone array, and perform a short-time Fourier transform on the reverberation signal to obtain a time-frequency domain signal; Step 2: Process the time-frequency domain signal using multiple highly directional beamformers to obtain the time-frequency signals output by the multiple highly directional beamformers; Step 3: Based on the time-frequency signals output by multiple highly directional beamformers, design a dereverberation filter using a weighted linear prediction algorithm; Step 4: Using the dereverberation filter, perform dereverberation filtering on the time-frequency signal output by one of the high directivity beamformers to obtain the desired time-frequency domain signal after dereverberation. Step 5: Perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a refracted time-domain signal; The microphone array includes M microphones, and the M microphones correspond to M channels; In step 1, the first Frame, First The time-frequency domain signals of M channels at each frequency point are: In the formula, It is the first Frame, First Time-frequency domain signal of M channels at a given frequency point. This is the received signal from the Mth microphone. It is the first Frame, First Reverberation signals of M channels at a frequency point It is the first Frame, First Noise signals from M channels at a given frequency point; In step 2, the time-frequency domain signal is processed by Q highly directional beamformers, wherein the first... q The time-frequency signal output by the highly directional beamformer is: In the formula, For the first q A highly directional beamformer, This represents the conjugate transpose operation; Where Q < M; Step 3 specifically includes: Step 31: Obtain the desired signal from the signal model based on the weighted linear prediction algorithm; Step 32: Model the desired signal as a complex Gaussian process with time-varying variance, and solve the dereverberation filter using the maximum likelihood function method; The desired signal is: in, In the formula, It is the expected signal estimated at the current time and frequency point. For length is The A de-reverberation filter for each frequency point; The stack length is The time-frequency signal output by the highly directional beamformer; The preset delay number; For the stacked first frame Q The time-frequency signal output by a highly directional beamformer.

2. The speech de-reverberation method according to claim 1, characterized in that, The highly directional beamformer is a superdirectional beamformer, a first-order supercardioid differential beamformer, or a first-order hypercardioid differential beamformer.

3. The speech de-reverberation method according to claim 1, characterized in that, In step 32, the cost function obtained through the maximum likelihood function method is: In the formula, This is the time-varying variance estimate of the desired signal; N is the total number of frames; By minimizing the cost function, the length is obtained. The The de-reverberation filter for each frequency point is: in, In the formula, For size is The weighted covariance matrix, The length is The weighted covariance vector, This indicates the conjugate operation.

4. A speech de-reverberation device, characterized in that, The apparatus for implementing the speech de-eraseurization method as described in any one of claims 1 to 3 comprises: The transformation unit is used to acquire the reverberation signal through the microphone array, and perform a short-time Fourier transform on the reverberation signal to obtain a time-frequency domain signal; The processing unit is used to process the time-frequency domain signal using multiple highly directional beamformers to obtain the time-frequency signal output by the multiple highly directional beamformers. Design unit for designing dereverberation filters based on a weighted linear prediction algorithm using time-frequency signals output from multiple highly directional beamformers; The dereverberation unit is used to perform dereverberation filtering on the time-frequency signal output by one of the high directivity beamformers using the dereverberation filter to obtain the desired time-frequency domain signal after dereverberation. The inverse transform unit is used to perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a dereverberated time-domain signal.

5. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the speech de-reverberation method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech de-reverberation method as described in any one of claims 1 to 3.

Images