Voice de-reverberation method, device, equipment and medium

Abstract

The invention provides a voice de-reverberation method, device and equipment and a medium, and relates to the technical field of voice signal processing, and the method comprises the steps: obtaining a reverberation signal through a microphone array, and carrying out the short-time Fourier transform of the reverberation signal, and obtaining a time-frequency domain signal; respectively processing the time-frequency domain signal by using a plurality of high-directivity beam formers to obtain a time-frequency signal; designing a dereverberation filter based on a weighted linear prediction algorithm according to the plurality of time-frequency signals; performing dereverberation filtering on the time-frequency signal output by one of the high-directivity beam formers by using a dereverberation filter to obtain an expected time-frequency domain signal after dereverberation; and performing inverse short-time Fourier transform on the expected time-frequency domain signal to obtain a reverberation-removed time-domain signal. According to the method, the performance of an existing WPE algorithm can be improved, the reverberation removing performance of the WPE algorithm in a noise environment is improved while the calculation complexity is reduced, voice reverberation can be removed, and the voice quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and in particular to a speech dereverberation method, device, equipment and medium. Background Art

[0002] Speech processing technology has been widely used in various military and civilian systems. For example, in video conferencing, voice communication technology enables people to communicate without obstacles over long distances; in the field of humanoid robots, intelligent voice technology enables people to interact with robots. In these voice communication and intelligent voice interaction applications, the most critical task is to pick up the target sound source signal through the microphone. However, in the actual acoustic environment, the signal received by the microphone contains reverberation in addition to the direct signal. Reverberation can have a significant adverse effect on the acoustic system, such as reducing the voice quality of the communication system, affecting the voice intelligibility of the hearing aid device, and affecting the voice recognition rate of the speech recognition system. Therefore, speech dereverberation has always been a hot research issue in the field of speech processing. Reverberation can seriously reduce the quality and intelligibility of speech, bringing major challenges to voice communication systems and intelligent voice interaction applications.

[0003] In the problem of dereverberation, the Weighted Prediction Error (WPE) algorithm has shown its effectiveness in practical applications. The WPE algorithm divides reverberation into early reverberation and late reverberation, and calculates the late reverberation from the array observation signal through multi-channel delayed linear prediction technology and subtracts it from it to obtain the expected signal. However, the signal model of the WPE algorithm only considers reverberation and does not consider noise, so its robustness in noisy environments is very poor. But in the actual sound pickup environment, noise is almost unavoidable, so the effect of the WPE algorithm in practical applications is not good. Moreover, the length of the dereverberation filter that the WPE algorithm needs to design is relatively large, involving a large number of matrix inversion and complex multiplication operations, which makes its computational complexity very high and cannot be directly applied to real-time systems. In view of the above problems, it is very necessary to improve the performance of the existing WPE algorithm and improve the dereverberation performance of the WPE algorithm in noisy environments while reducing the computational complexity. Summary of the invention

[0004] The purpose of the present invention is to provide a speech dereverberation method, device, equipment and medium, which are used to solve the problems that the WPE algorithm is not effective in actual dereverberation applications and has high computational complexity. The present invention can improve the performance of the existing WPE algorithm, and improve the dereverberation performance of the WPE algorithm in a noisy environment while reducing the computational complexity.

[0005] In order to achieve the above object, in a first aspect, the present invention provides a speech dereverberation method, comprising: Step 1: Obtain a reverberation signal through a microphone array, and perform short-time Fourier transform on the reverberation signal to obtain a time-frequency domain signal; Step 2: Use multiple high-directivity beamformers to process the time-frequency domain signals respectively to obtain time-frequency signals output by the multiple high-directivity beamformers; Step 3: designing a dereverberation filter based on a weighted linear prediction algorithm according to the time-frequency signals output by the multiple high-directivity beamformers; Step 4: Using a dereverberation filter, dereverberation filtering is performed on the time-frequency signal output by one of the high-directivity beamformers to obtain a desired time-frequency domain signal after dereverberation; Step 5: Perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a dereverberated time-domain signal.

[0006] According to a speech dereverberation method provided by the present invention, the microphone array includes M microphones, and the M microphones correspond to M channels; in step 1, Frame, The time-frequency domain signal of M channels with frequency points is:

[0007] In the formula, It is Frame, The time-frequency domain signal of M channels with frequency points, is the received signal of the Mth microphone, It is Frame, The reverberation signal of M channels with frequency points, It is Frame, The noise signal of M channels with frequency points.

[0008] According to a speech dereverberation method provided by the present invention, the high-directivity beamformer is a super-directivity beamformer, a first-order supercardioid differential beamformer or a first-order high-cardioid differential beamformer.

[0009] According to a speech dereverberation method provided by the present invention, in step 2, the time-frequency domain signals are processed respectively by Q high-directivity beam formers, wherein the first q The time-frequency signal output by a high-directivity beamformer is:

[0010] In the formula, For the q A highly directive beamformer, Represents the conjugate transpose operation.

[0011] According to a speech dereverberation method provided by the present invention, step 3 specifically includes: Step 31, deriving an expected signal based on a signal model of a weighted linear prediction algorithm; Step 32: Model the expected signal as a complex Gaussian process with time-varying variance, and solve the dereverberation filter by using the maximum likelihood function method.

[0012] According to a speech dereverberation method provided by the present invention, the expected signal is:

[0013] in,

[0014]

[0015] In the formula, is the expected signal estimated at the current time-frequency point, The length is No. De-reverberation filter with frequency points; The length of the stack is The time-frequency signal output by the high-directivity beamformer; is the preset delay number; For stacking frame Q The time-frequency signal output by a highly directional beamformer.

[0016] According to a speech dereverberation method provided by the present invention, in step 32, the cost function is obtained by the maximum likelihood function method as follows:

[0017] In the formula, is the time-varying variance estimate of the desired signal; N is the total number of frames.

[0018] By minimizing the cost function, the length is obtained No. The dereverberation filter for each frequency point is:

[0019] in,

[0020] In the formula, For size is The weighted covariance matrix of The length is The weighted covariance vector of Represents the conjugate operation.

[0021] In a second aspect, the present invention provides a speech dereverberation device, comprising: A transform unit, used to obtain a reverberation signal through a microphone array, and perform a short-time Fourier transform on the reverberation signal to obtain a time-frequency domain signal; A processing unit, used to use a plurality of high-directivity beamformers to process the time-frequency domain signals respectively, and obtain the time-frequency signals output by the plurality of high-directivity beamformers; A design unit, used for designing a dereverberation filter based on a weighted linear prediction algorithm according to time-frequency signals output by a plurality of high-directivity beamformers; A de-reverberation unit, used to perform de-reverberation filtering on the time-frequency signal output by one of the high-directivity beamformers using a de-reverberation filter to obtain a desired time-frequency domain signal after de-reverberation; The inverse transform unit is used to perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a dereverberation time-domain signal.

[0022] In a third aspect, the present invention provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech dereverberation method of the first aspect when executing the computer program.

[0023] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech dereverberation method of the first aspect.

[0024] The technical solution of the present invention has at least the following technical effects: The present invention provides a speech dereverberation method, device, equipment and medium, which realize speech dereverberation by jointly optimizing high-directivity beamforming and WPE algorithm. First, multiple high-directivity beamformers are used to pre-process the reverberation signal obtained by the microphone array, that is, the observation signal, and then the dereverberation operation is performed on the output of the high-directivity beamformer. A high-directivity beamformer is a device for fixed beamforming, including super-directivity beamformers and differential beamformers with higher directivity, such as super-cardioid differential beamformers, high-cardioid differential beamformers, etc. After preprocessing by the high-directivity beamformer, the reverberation in the observation signal is partially removed, and most of the noise is also removed, but the high-directivity beamformer cannot completely suppress the reverberation. Therefore, further removing the residual reverberation on the output result of the high-directivity beamformer can further improve the speech quality. Different from simply combining beamforming with the WPE algorithm, the joint optimization process of the present invention involves predicting late reverberation from the outputs of multiple high-directivity beamformers, and generating a dereverberation reference signal from the output of the high-directivity beamformers, that is, the time-frequency signal output by one of the high-directivity beamformers. At the same time, the number of high-directivity beamformers is less than the number of microphones, so the dereverberation filter is shorter than the traditional method, which greatly reduces the computational complexity of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] In the attached picture: Figure 1 It is a flow chart of the speech dereverberation method of the present invention; Figure 2 is the beam pattern of the high-directivity beamformer used in the present invention; Figure 3 A graph showing the computational complexity of different dereverberation methods using different numbers of microphones in the present invention; Figure 4 It is a structural block diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Some embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] Starting from the actual application scenario, the present invention considers the influence of reverberation on the microphone pickup signal in voice communication and intelligent voice interaction applications such as video conferencing and humanoid robots, and proposes a dereverberation method that jointly optimizes the high-directional microphone array beamforming and the WPE algorithm, which reduces the computational complexity of the WPE algorithm and improves its robustness. The present invention first uses the high-directional beamforming technology to process the array observation signal. High-directional beamforming is a fixed beamforming technology. When the array structure is certain, the filter coefficient is determined accordingly, and the amount of calculation is very small. At the same time, its high directivity can suppress part of the reverberation and remove most of the noise. On this basis, the WPE dereverberation technology is jointly optimized to further remove residual reverberation and improve voice quality. And the multi-channel observation signal is converted into a single-channel signal after being processed by a high-directional beamformer, which greatly shortens the length of the dereverberation filter and reduces its computational complexity. The method provided by the present invention can implement the process using computer software technology.

[0030] Example 1 See also Figure 1 This embodiment provides a speech dereverberation method, including: Step 1: Acquire a reverberation signal through a microphone array, and perform short-time Fourier transform (STFT) on the reverberation signal to obtain a time-frequency domain signal; Specifically, in step 1, the reverberation signal is a multi-channel time domain signal obtained by the microphone array, and the multi-channel time domain signal is subjected to short-time Fourier transform to obtain a multi-channel time-frequency domain signal. The reverberation signal obtained by the microphone array composed of microphones is converted into the time-frequency domain, and M microphones correspond to M channels. Frame, The time-frequency domain signal of M channels with frequency points is:

[0031] In the formula, It is Frame, The time-frequency domain signal of M channels with frequency points, is the received signal of the Mth microphone, and Definition and Similarly, they are reverberation signal and noise signal respectively.

[0032] Step 2: Use multiple high-directivity beamformers to process the time-frequency domain signals respectively to obtain time-frequency signals output by the multiple high-directivity beamformers; Specifically, first provide Q A high-directivity beamformer is provided. These high-directivity beamformers can be beamformers with higher directivity such as a super-directivity beamformer, a first-order super-cardioid differential beamformer and a first-order high-cardioid differential beamformer. Figure 2 : The beam pattern of the high-directivity beamformer used in the present invention, wherein from left to right are the beam pattern of the super-directivity beamformer, the beam pattern of the first-order high-cardioid differential beamformer, and the beam pattern of the first-order supercardioid differential beamformer. Q A highly directional beamformer processes multi-channel time-frequency domain signals. q The time-frequency signal output by a high-directivity beamformer is:

[0033] In the formula, For the q A highly directive beamformer, Represents the conjugate transpose operation.

[0034] Step 3: designing a dereverberation filter based on a weighted linear prediction algorithm according to the time-frequency signals output by the multiple high-directivity beamformers; Specifically, the design method of the dereverberation filter uses a weighted linear prediction algorithm. The process of the design method of the dereverberation filter includes two steps: Step 31: Obtain the expected signal based on the signal model of the WPE algorithm. In the signal model of the WPE, the expected signal is defined as the direct sound plus the early reflection, which is equal to the time-frequency signal output by the high-directivity beamformer minus the late reverberation, where the late reverberation is obtained by filtering the signal processed by multiple high-directivity beamformers through a dereverberation filter. The expected signal is:

[0035] in,

[0036]

[0037] In the formula, is the expected signal estimated at the current time-frequency point, The length is No. De-reverberation filter with frequency points; The length of the stack is The time-frequency signal output by the high-directivity beamformer; is the preset delay number; For stacking frame Q The time-frequency signal output by a highly directional beamformer.

[0038] Step 32: Model the expected signal as a complex Gaussian process with time-varying variance, and solve the dereverberation filter by the maximum likelihood function method.

[0039] Among them, the cost function obtained by the maximum likelihood function method is:

[0040] In the formula, is the time-varying variance estimate of the desired signal; N is the total number of frames. By minimizing the cost function, we can solve the problem of length No. The dereverberation filter for each frequency point is:

[0041] in:

[0042] In the formula, For size is The weighted covariance matrix of The length is The weighted covariance vector of After performing the above operation on each frequency point, a full-band de-reverberation filter based on linear prediction can be obtained and stored for subsequent de-reverberation operations.

[0043] Step 4: Use a dereverberation filter , perform dereverberation filtering on the time-frequency signal output by one of the high-directional beamformers to obtain the desired time-frequency domain signal after dereverberation.

[0044] Can be Frame, The desired signal of each frequency point is subjected to dereverberation calculation, and this operation is performed on the desired signal of all frames and frequencies to obtain the desired time-frequency domain signal after dereverberation.

[0045] Step 5: Perform an inverse short-time Fourier transform (iSTFT) on the desired time-frequency domain signal after dereverberation, transform it into the time domain, and obtain the final dereverberation time domain signal.

[0046] It should be noted that in the dereverberation process based on linear prediction, the late reverberation is estimated by the observation value of the past delayed L frames, and then subtracted from the current observation signal. By selecting the output of one of the highly directive beamformers as the reference signal. A special case is to consider a single highly directive beamformer, then only a length of L The computational complexity of dereverberation can be significantly reduced by using a dereverberation filter. In some cases, using multiple highly directional beamformers may enhance dereverberation performance, but at the expense of increased computational complexity. However, as long as Q The value is less than , the computational complexity can still be kept lower than that of the traditional WPE algorithm.

[0047] Figure 3 The graph is a graph of the computational complexity of different dereverberation methods used in this embodiment with different numbers of microphones. Among them, WPE represents a dereverberation method using a traditional WPE algorithm; DMA-WPE represents a dereverberation method using the output of a differential beamformer as a reference signal for the WPE algorithm; MWPE-DMA represents a dereverberation method using a multi-input multi-output WPE algorithm coupled with a differential beamformer; JDMA-WPE represents a dereverberation method combining a WPE algorithm and high-directional beamforming provided by the present invention, and Q represents the number of differential beamformers used. It can be seen that as the number of microphones increases, the computational complexity of WPE, DMA-WPE, and MWPE-DMA rises in a quadratic curve, while the computational complexity of the speech dereverberation method provided by the present invention hardly increases. In the case of multiple microphones, the present invention is more applicable.

[0048] Example 2 Based on the same inventive concept, this embodiment provides a speech dereverberation device, which corresponds to the method of embodiment 1, and includes: A transform unit, used to obtain a reverberation signal through a microphone array, and perform a short-time Fourier transform on the reverberation signal to obtain a time-frequency domain signal; A processing unit, used to use a plurality of high-directivity beamformers to process the time-frequency domain signals respectively, and obtain the time-frequency signals output by the plurality of high-directivity beamformers; A design unit, used for designing a dereverberation filter based on a weighted linear prediction algorithm according to time-frequency signals output by a plurality of high-directivity beamformers; A de-reverberation unit, used to perform de-reverberation filtering on the time-frequency signal output by one of the high-directivity beamformers using a de-reverberation filter to obtain a desired time-frequency domain signal after de-reverberation; The inverse transform unit is used to perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a dereverberation time-domain signal.

[0049] Example 3 Figure 4 An example of a structural diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The memory 830 stores a computer program that can be run on the processor 810, and when the processor 810 executes the computer program, the speech dereverberation method of embodiment 1 is implemented.

[0050] Example 4 This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech dereverberation method of embodiment 1 is implemented.

[0051] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0052] In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. In this embodiment, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0053] The computer readable storage medium may be written in one or more programming languages ​​or a combination thereof to execute the computer program of the present embodiment, and the programming language includes an object-oriented programming language, such as Java, Python, C++, and a conventional procedural programming language, such as C or a similar programming language. The program may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0054] In summary, the speech dereverberation method, device, equipment and medium provided by the present invention adopt a strategy of combining high-directional microphone array beamforming with the WPE algorithm, the output of the high-directional beamformer is used as a reference signal for dereverberation of the WPE algorithm, and the late reverberation is also predicted from the output of the high-directional beamformer. The present invention first designs multiple high-directional beamformers to pre-process the observed signal, and then predicts the late reverberation from the output of the multiple high-directional beamformers, thereby improving the dereverberation performance in a noisy environment, and significantly reducing the computational complexity by shortening the length of the dereverberation filter.

[0055] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the embodiments disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A speech dereverberation method, characterized in that: include: Step 1: Acquire a reverberation signal through a microphone array, and perform short-time Fourier transform on the reverberation signal to obtain a time-frequency domain signal; Step 2: using a plurality of high-directivity beamformers to process the time-frequency domain signals respectively, to obtain time-frequency signals output by the plurality of high-directivity beamformers; Step 3: designing a dereverberation filter based on a weighted linear prediction algorithm according to the time-frequency signals output by the multiple high-directivity beamformers; Step 4: using the dereverberation filter, dereverberation filtering is performed on the time-frequency signal output by one of the high-directivity beamformers to obtain a desired time-frequency domain signal after dereverberation; Step 5: Perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a dereverberation time-domain signal.

2. The speech dereverberation method according to claim 1, characterized in that: The microphone array includes M microphones, and the M microphones correspond to M channels; In step 1, Frame, The time-frequency domain signal of M channels with frequency points is: In the formula, It is Frame, The time-frequency domain signal of M channels with frequency points, is the received signal of the Mth microphone, It is Frame, The reverberation signal of M channels with frequency points, It is Frame, The noise signals of M channels with different frequencies.

3. The speech dereverberation method according to claim 2, characterized in that: The high-directivity beamformer is a super-directivity beamformer, a first-order supercardioid differential beamformer or a first-order high-cardioid differential beamformer.

4. The speech dereverberation method according to claim 2, characterized in that: In step 2, the time-frequency domain signals are processed respectively by Q highly directional beamformers, wherein the first q The time-frequency signal output by a high-directivity beamformer is: In the formula, For the q A highly directive beamformer, Represents the conjugate transpose operation.

5. The speech dereverberation method according to claim 4, characterized in that: The step 3 specifically includes: Step 31, deriving an expected signal based on a signal model of a weighted linear prediction algorithm; Step 32: Model the desired signal as a complex Gaussian process with time-varying variance, and solve the dereverberation filter by using a maximum likelihood function method.

6. The speech dereverberation method according to claim 5, characterized in that: The expected signal is: in, In the formula, is the expected signal estimated at the current time-frequency point, The length is No. De-reverberation filter with frequency points; The length of the stack is The time-frequency signal output by the high-directivity beamformer; is the preset delay number; For stacking frame Q The time-frequency signal output by a highly directional beamformer.

7. The speech dereverberation method according to claim 6, characterized in that: In step 32, the cost function is obtained by the maximum likelihood function method: In the formula, is the time-varying variance estimate of the desired signal; N is the total number of frames. By minimizing the cost function, the length is obtained No. The dereverberation filter for each frequency point is: in, In the formula, For size is The weighted covariance matrix of The length is The weighted covariance vector of Represents the conjugate operation.

8. A speech dereverberation device, characterized in that: include: A transform unit, used to obtain a reverberation signal through a microphone array, and perform a short-time Fourier transform on the reverberation signal to obtain a time-frequency domain signal; A processing unit, configured to use a plurality of high-directivity beamformers to process the time-frequency domain signals respectively, to obtain time-frequency signals output by the plurality of high-directivity beamformers; A design unit, used for designing a dereverberation filter based on a weighted linear prediction algorithm according to time-frequency signals output by a plurality of high-directivity beamformers; A de-reverberation unit, configured to perform de-reverberation filtering on the time-frequency signal output by one of the high-directivity beamformers using the de-reverberation filter to obtain a desired time-frequency domain signal after de-reverberation; The inverse transform unit is used to perform an inverse short-time Fourier transform on the desired time-frequency domain signal to obtain a dereverberation time-domain signal.

9. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the speech dereverberation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech dereverberation method according to any one of claims 1 to 7 is implemented.

Images