Speech dereverberation method, device and equipment based on progressive reverberation perception network

Abstract

The present invention discloses a speech dereverberation method, device and equipment based on progressive reverberation perception. The speech dereverberation method comprises generating a reverberation filter group by using a reverberation filter generator based on a graph method; constructing a reverberation data set, passing a pure speech through a reverberation filter to generate a reverberation speech and three progressive target speech; performing a short-time Fourier transform on the reverberation speech to extract its real part and imaginary part, and using the real part and the imaginary part as two channels of input features; constructing a progressive reverberation perception network; using the three progressive target speech as the first, second and third supervisory information, using the reverberation time of the reverberation speech as auxiliary supervisory information, and training the progressive reverberation perception network; performing a short-time Fourier transform on the reverberation speech to be enhanced in the test set and sending it to the trained progressive reverberation perception network, and performing a short-time Fourier inverse transform on the frequency domain result of the enhanced speech output by the network to obtain the enhanced speech after the reverberation is removed.

Description

Technical Field

[0001] The present invention belongs to the field of speech signal processing and deep learning, and relates to a speech dereverberation method, device and computer equipment based on a progressive reverberation perception network. Background Art

[0002] Speech dereverberation technology is one of the key technologies in the field of speech enhancement. The purpose of reverberation elimination is to convert reverberant speech into reverberation-free speech. Reverberation is mainly divided into early reverberation and late reverberation (divided by 50ms reverberation time). Studies have shown that early reverberation can make speech sound more natural, while the main reason for the decline in speech quality and speech intelligibility is attributed to the late reverberation in the speech signal.

[0003] In the field of speech enhancement, speech dereverberation technology can effectively improve the overall perceived quality and intelligibility of speech signals in application scenarios such as voice calls, telephone conferences, scene recording, military eavesdropping, and hearing assistance.

[0004] Speech dereverberation can be divided into three categories. 1) The reverberation elimination method uses a convolutional distortion model to model the signal. It assumes that the reverberant speech is generated by a linear system. First, the parameters of the acoustic system are estimated, and then the estimation of the reverberation-free signal is obtained; 2) The reverberation suppression method uses an additive distortion model to model the signal. It assumes that the reverberant speech is generated by an additive process and the reverberation is independent of the speech; 3) The speech dereverberation method based on deep learning is different from the reverberation elimination method and the reverberation suppression method. This method is unknown to the reverberant acoustic system and uses a large amount of data for training to establish a nonlinear mapping from reverberant speech to pure speech. It is combined with other machine learning methods according to the characteristics of the reverberant speech to remove noise.

[0005] In recent years, deep learning-based speech dereverberation methods have made good progress in deep learning speech dereverberation tasks and have received widespread attention in the field of speech enhancement.

[0006] At present, there are three main challenges in the speech dereverberation task based on deep learning:

[0007] (1) Speech dereverberation problem in low signal-to-noise ratio environments. In a -5dB environment, the speech power is less than 1 / 3 of the noise power, and the speech amplitude is often only half of the noise amplitude. After short-time Fourier transform, the amplitude spectrum is dominated by noise, which makes some masking-based models lose their advantages.

[0008] (2) Simply taking pure speech as supervisory information or taking a single reverberated speech as supervisory information ignores the problem that the reverberated speech with gradually increasing reverberation time makes it difficult for the model to gradually learn the difference between the direct wave of speech and the reverberation.

[0009] (3) Multi-scale feature information of deep networks is lost. The model generally extracts single-scale features at the same depth in the feature extraction path. Due to the single-scale feature uniformity of the single-scale feature, the model has a limited receptive field and is not easy to combine multi-scale features to obtain enhanced speech that conforms to the multi-scale previous and next frame information. Summary of the invention

[0010] In view of this, an object of the present invention is to provide a speech dereverberation method, apparatus and computer equipment based on a progressive reverberation perception network.

[0011] The invention provides a speech dereverberation method, device and computer equipment of a progressive reverberation perception network, and in particular relates to a speech dereverberation method of progressive supervision based on a multi-scale reverberation perception network. The multi-scale feature extraction path is used to extract multi-scale feature information. Since the multi-scale structure contains more scales of previous and next frame information, and the previous and next frames of speech have a strong correlation, a speech dereverberation result that conforms to the previous and next frame information at the frame level can be obtained; the multi-branch decoding path sets three parallel decoding branches, and uses reverberation speech with different reverberation times as supervision, so that the network gradually learns the difference between the direct wave of speech and the reverberation speech from easy to difficult; in addition, the introduction of reverberation time as auxiliary supervision information of the fourth decoding branch can make it easier for the network to locate the end time of reverberation, thereby eliminating reverberation more accurately.

[0012] The present invention provides a speech dereverberation method based on a progressive reverberation perception network, the method comprising the following steps:

[0013] S1, constructing a reverberation filter generator, and generating a reverberation filter bank by configuring the required parameters of the generator;

[0014] S2, constructing a reverberation data set, passing the clean speech in the speech data set through a reverberation filter to generate a reverberation speech of the reverberation data set, and inputting the clean speech into the reverberation filter with the same parameters after time truncation to generate the first, second, and third target speech of the reverberation speech;

[0015] S3, performing short-time Fourier transform on the reverberant speech to extract its real part and imaginary part, and using the real part and imaginary part as two channels of input features;

[0016] S4, constructing a progressive reverberation perception network including a multi-scale feature extraction path and a multi-branch decoding path;

[0017] S5, taking the three target voices as the first, second, and third target voices of the progressive reverberation perception network, using the scale-invariant signal-to-noise ratio loss function to minimize the difference between the enhanced voice and the three target voices, taking the reverberation time of the reverberation voice as auxiliary supervision information, using the mean square error loss function to minimize the difference between the reverberation time prediction value and the reverberation time, and training the progressive reverberation perception network;

[0018] S6. Obtain the reverberant speech to be enhanced, and send the reverberant speech to be enhanced into the trained progressive reverberation perception network after short-time Fourier transform, and use the output of the first coding branch as the frequency domain result of the enhanced speech, and then obtain the enhanced speech after removing the reverberation through inverse short-time Fourier transform.

[0019] Furthermore, to construct a reverberation filter generator, it is necessary to configure the parameters required by the generator, including: room size, sound source location, reverberation time, and microphone location.

[0020] Furthermore, the reverberated speech is obtained by adding a reverberation of a specified reverberation time to the pure speech, and the reverberation time is 500ms, 650ms, and 700ms.

[0021] Furthermore, the time truncation operation can convert the system function into a system function of a specified time length, and then output the first, second, and third target voices of the reverberation speech; wherein the reverberation time length of the first target voice is 50ms, and the reverberation time length of the second and third target voices can be 150ms, 200ms, 300ms, etc.

[0022] Further, the short-time Fourier transform is expressed as:

[0023]

[0024] Among them, l and k represent the time domain and frequency domain respectively, L represents the frame length of speech, n represents discrete time, ω(n) is the window function and the window length is N, e represents a natural constant, j represents an imaginary unit, x() represents reverberant speech, and x′() represents the short-time Fourier transform result of the reverberant speech.

[0025] Furthermore, the two channels of using the real part and the imaginary part as input features in step S3 include: stacking the real part and the imaginary part of the reverberation speech frame after short-time Fourier transform along the feature channel direction, so that the real part and the imaginary part information of the reverberation speech are simultaneously input into the progressive reverberation perception network, thereby retaining the phase information of the reverberation speech.

[0026] Furthermore, the progressive reverberation perception network includes a multi-scale feature extraction path and a multi-branch decoding path, wherein the multi-scale feature extraction path includes 4 repeated encoding units and 1 multi-scale feature pyramid unit; each encoding unit includes 2 convolutional layers and 1 average pooling layer; the multi-scale feature pyramid unit includes 4 parallel feature extraction branches and 1 convolutional layer, each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer, thereby mapping feature maps of the same size into feature maps of 4 scales, thereby achieving the purpose of multi-scale feature extraction; the multi-branch decoding path has a total of 4 decoding branches, each decoding branch includes 4 decoding units with the same feature depth as the encoding unit in the multi-scale feature extraction path, and each unit includes 2 convolutional layers and 1 transposed convolutional layer.

[0027] Furthermore, the feature maps of the four scales may be 1×1, 2×2, 3×3, 6×6, or 1×1, 2×2, 4×4, 8×8 or 1×1, 2×2, 4×4, 16×16, etc. The feature map size of 1×1 is retained to introduce global features, and the other three feature map sizes can be selected as needed.

[0028] Furthermore, the step S5 of using the three target speech as the first, second, and third supervisory information of the reverberation perception network includes using the first, second, and third target speech as the supervisory information of the first, second, and third decoding branches of the progressive reverberation perception network, respectively; and using the reverberation time of the reverberation speech as auxiliary supervisory information includes using the 500ms reverberation time as the supervisory information of the fourth branch of the progressive reverberation perception network.

[0029] Further, the scale-invariant signal-to-noise ratio loss function described in step S5 is used to minimize the difference between the enhanced speech and the three target speech of the first, second and third decoding branches. The loss function includes but is not limited to the scale-invariant signal-to-noise ratio loss function. Other derivable speech quality assessment indicators can also be used as the loss function in the present invention; the calculation formula of the scale-invariant signal-to-noise ratio loss function is:

[0030]

[0031] Where, X T and X N Respectively expressed as:

[0032]

[0033] X N =f(x)-X T

[0034] Where x represents reverberant speech, f(x) represents enhanced speech, y represents pure speech, and ||y|| 2、||X T || 2 and ||X N || 2 is the 2-norm result.

[0035] Furthermore, the mean square error loss function is used to minimize the difference between the reverberation time prediction value and the reverberation time, wherein the reverberation time is the reverberation time set in the reverberation filter generator corresponding to the reverberation speech, and the mean square error loss function calculation formula is expressed as:

[0036] l MSE =(t(x)-T) 2

[0037] Where x represents the reverberant speech, t(x) represents the predicted value of the reverberation time, and T represents the true value of the reverberation time.

[0038] Further, the short-time inverse Fourier transform is expressed as:

[0039]

[0040] Among them, φ(n) represents the comprehensive window function, l and k represent the time domain and frequency domain respectively, L represents the frame length of the speech, n represents the discrete time, i represents the time sequence number, ω(n) is the window function and the window length is N, e represents the natural constant, j represents the imaginary unit, x() represents the reverberant speech, x′() represents the short-time Fourier transform result of the reverberant speech, and the relationship between the window function and the comprehensive window function is defined as:

[0041]

[0042] Among them, commonly used window functions ω(n) include rectangular window, Hamming window and Gaussian window.

[0043] The present invention also proposes a speech dereverberation device based on a progressive reverberation perception network, the device comprising:

[0044] A clean speech acquisition module is used to obtain clean speech in the reverberation data set;

[0045] A reverberation speech acquisition module, used to acquire the reverberation speech in the reverberation data set and the reverberation speech to be enhanced;

[0046] A target speech acquisition module is used to acquire the target speech in the reverberation data set, including a target speech with a reverberation time of 50 ms, a target speech with a reverberation time of 200 ms, and a target speech with a reverberation time of 300 ms;

[0047] A short-time Fourier transform module is used to transform the time-domain speech into a frequency-domain speech frame before the reverberant speech is input into the progressive reverberation perception network;

[0048] A network construction module, used to construct a progressive reverberation perception network including a multi-scale feature extraction path and a multi-branch decoding path;

[0049] A short-time inverse Fourier transform module is used to transform the frequency-domain enhanced speech frame output by the progressive reverberation perception network into time-domain speech;

[0050] The speech enhancement module is used to pass the reverberated speech to be enhanced through a progressive reverberation perception network to obtain the enhanced speech after the reverberation is removed.

[0051] Furthermore, the reverberation speech acquisition module includes:

[0052] A graph method unit, generating a reverberation filter generator based on a graph method;

[0053] A parameter configuration unit, which configures the parameters required by the reverberation filter generator to construct a reverberation filter with variable parameters;

[0054] The reverberation speech generating unit is used to obtain the reverberation speech by passing the clean speech through a 500ms reverberation time filter whose reverberation time parameter is configured as 500ms.

[0055] Furthermore, the target speech acquisition module includes:

[0056] A reverberation filter time truncation unit intercepts the system function of a 500ms reverberation time filter, thereby obtaining 50ms, 200ms and 300ms reverberation time system functions;

[0057] The target speech acquisition unit is used to pass the clean speech through the 50ms, 200ms and 300ms reverberation time system function, so as to obtain the 50ms, 200ms and 300ms reverberation time target speech.

[0058] Furthermore, the short-time Fourier transform module includes:

[0059] A framing unit, used for dividing the reverberant speech into a series of short segments to obtain short-term stable reverberant speech frames, where there are overlaps between the short segments;

[0060] A windowing unit, used to window the reverberation speech frame to prevent spectrum leakage;

[0061] The Fourier transform unit is used to perform Fourier transform on the windowed reverberation speech frame to obtain the frequency domain reverberation speech frame, and then extract the real part and imaginary part information.

[0062] Furthermore, the network construction module includes:

[0063] A progressive reverberation perception network, comprising a multi-scale feature extraction path layer and a multi-branch decoding path layer; wherein the multi-scale feature extraction path layer comprises 4 repeated encoding units and 1 multi-scale feature pyramid unit; each encoding unit comprises 2 convolutional layers and 1 average pooling layer; the multi-scale feature pyramid unit comprises 4 parallel feature extraction branches and 1 convolutional layer, and each feature extraction branch comprises 1 average pooling layer and 1 bilinear interpolation layer;

[0064] The multi-branch decoding path layer includes four decoding branches, each decoding branch includes four decoding units having the same feature depth as the encoding unit in the multi-scale feature extraction path layer, and each unit includes two convolutional layers and one transposed convolutional layer;

[0065] A first supervision unit, configured to use a target speech with a reverberation time of 50 ms as first supervision information;

[0066] A first loss function unit is used to train a progressive reverberation perception network according to a target speech with a reverberation time of 50 ms and a predicted enhanced speech;

[0067] A second supervision unit, used for taking the target speech with a reverberation time of 200 ms as the second supervision information;

[0068] The second loss function unit is used to train a progressive reverberation perception network according to the target speech with a reverberation time of 200 ms and the predicted enhanced speech;

[0069] A third supervision unit, used for taking the 300 ms reverberation time target speech as the third supervision information;

[0070] A third loss function unit is used to train a progressive reverberation perception network according to the target speech with a reverberation time of 300 ms and the predicted enhanced speech;

[0071] An auxiliary supervision unit, used to use the 500ms reverberation time as auxiliary supervision information;

[0072] Auxiliary loss function unit, used to train the progressive reverberation perception network based on the 500ms reverberation time and the predicted reverberation time.

[0073] Furthermore, the short-time inverse Fourier transform module includes:

[0074] An inverse Fourier transform unit, used to transform the frequency domain result of the enhanced frame output by the network into a time domain result;

[0075] The time domain superposition unit is used to overlap and add the enhanced frames after the inverse Fourier transform to perfectly restore the final enhanced speech.

[0076] The present invention also proposes a computer device, comprising at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute a speech dereverberation method based on a progressive reverberation perception network as proposed in the above-mentioned embodiment of the present invention.

[0077] Beneficial effects of the present invention:

[0078] 1) The present invention proposes a speech dereverberation method based on a progressive reverberation perception network, which can improve the accuracy of speech dereverberation and enhance the auditory effect of speech under low signal-to-noise ratio conditions.

[0079] 2) The present invention deeply extracts multi-scale front and back frame information, can analyze the difference between reverberant speech and pure speech from different feature scales, improve the representation ability of the model from the feature level, and thus more accurately remove the reverberation in the speech.

[0080] 3) The present invention utilizes reverberated speech with different reverberation times of the same pure speech and the reverberation time as supervision. The multi-supervision strategy enables the network to gradually learn the difference between the direct wave of speech and the reverberated speech from easy to difficult, and effectively guides the convergence direction of the model from the speech level. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0082] Figure 2 A schematic diagram of a data generation process of the present invention;

[0083] Figure 3 This is a schematic diagram of the structure of the progressive reverberation perception network used in the present invention;

[0084] Figure 4 It is a data flow diagram of the present invention;

[0085] Figure 5 It is a schematic diagram of the reverberation removal effect of the present invention. DETAILED DESCRIPTION

[0086] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0087] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0088] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0089] like Figure 1 As shown, a speech dereverberation method based on a progressive reverberation perception network of the present method may specifically include the following steps:

[0090] S1, constructing a reverberation filter generator, and generating a reverberation filter bank by configuring the required parameters of the generator;

[0091] S2, constructing a reverberation data set, passing the clean speech in the speech data set through a reverberation filter to generate a reverberation speech of the reverberation data set, and inputting the clean speech into the reverberation filter with the same parameters after time truncation to generate the first, second, and third target speech of the reverberation speech;

[0092] In one embodiment, Figure 2 The following is a flow chart of data generation. First, a reverberation filter is constructed using the reverberation filter generator configured by S1, and the reverberation time is set to 500ms. Secondly, the system function of the reverberation filter is intercepted to generate reverberation filters with reverberation times of 50ms, 200ms, and 300ms. Next, the reverberation speech is input into the above four reverberation filters respectively. Finally, one reverberation speech and three target speeches are generated.

[0093] S3, performing short-time Fourier transform on the reverberant speech to extract its real part and imaginary part, and using the real part and imaginary part as two channels of input features;

[0094] S4, constructing a progressive reverberation perception network including a multi-scale feature extraction path and a multi-branch decoding path;

[0095] Short-time Fourier transform is a common operation in the field of audio signal processing. It uses the short-time stability of speech to regard long-time non-stationary speech as a superposition of a series of short-time stable speech. First, the reverberant speech is divided into a series of short segments to obtain short-time stable reverberant speech frames with a certain overlap. Secondly, the reverberant speech frame is windowed to prevent spectrum leakage. Finally, the reverberant speech frame after windowing is Fourier transformed to obtain the frequency domain reverberant speech frame, and then the real and imaginary part information is extracted. The definition of short-time Fourier transform is shown in formula (1):

[0096]

[0097] Among them, l and k represent the time domain and frequency domain respectively, L represents the frame length of speech, n represents discrete time, ω(n) is the window function and the window length is N, e represents a natural constant, j represents an imaginary unit, x() represents reverberant speech, and x′() represents the short-time Fourier transform result of the reverberant speech.

[0098] In one embodiment, the progressive reverberation perception network may refer to Figure 3 , this embodiment provides a multi-supervised dereverberation network with a multi-scale feature pyramid structure. The network consists of a multi-scale feature extraction path and a multi-branch decoding path. The multi-scale feature extraction path reduces the model scale while extracting audio features in speech, and the multi-branch decoding path is used to restore the length of speech. The multi-scale feature extraction path includes 4 repeated encoding units and 1 multi-scale feature pyramid unit. Each encoding unit includes 2 convolutional layers and 1 average pooling layer with a size of 2×2 and a step size of 2. The multi-scale feature pyramid unit includes 4 parallel feature extraction branches and 1 convolutional layer, and each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer. The size and step size of the average pooling layer inside the feature extraction branch are determined by the speech length of the input network, which enables the common audio feature map output by the encoding unit 4 to be mapped into a series of feature maps of different sizes, thereby achieving the purpose of multi-scale feature extraction. Subsequently, the feature extraction branch resamples the feature maps of 4 different sizes to the size of the common audio feature map through bidirectional interpolation. Finally, the results of the four feature extraction branches are concatenated and then convolved to fuse the feature maps with different receptive fields after the pooling operation.

[0099] The multi-branch decoding path has a total of 4 decoding branches, each of which includes 4 decoding units with the same feature depth as the encoding unit in the multi-scale feature extraction path. Each unit includes 2 convolutional layers and 1 transposed convolutional layer with a size of 3×3, a step size of 1, and 2 zeros filled. The function of the transposed convolutional layer is to make the speech with reduced size after convolution to the size of the original speech. The network finally uses the Sigmoid activation function to limit the output range.

[0100] Among them, the size of all convolution kernels in the network is 3×3 and the stride is 1.

[0101] The concatenation operation is to stack the feature maps along the channel direction.

[0102] S5, taking the three target voices as the first, second, and third target voices of the progressive reverberation perception network, using the scale-invariant signal-to-noise ratio loss function to minimize the difference between the enhanced voice and the three target voices, taking the reverberation time of the reverberation voice as auxiliary supervision information, using the mean square error loss function to minimize the difference between the reverberation time prediction value and the reverberation time, and training the progressive reverberation perception network;

[0103] In one embodiment, Figure 3 As shown, the first, second and third target speech are used as the supervision information of decoding branch 1, decoding branch 2 and decoding branch 3 respectively. The scale-invariant signal-to-noise ratio function is used as the loss function, and the differences between the first, second and third target speech and the enhanced speech after short-time Fourier inverse transform are minimized respectively. In addition, the reverberation time parameter required to configure the reverberation filter generator, that is, the reverberation time of the reverberated speech, is used as the supervision information of decoding branch 4, and the mean square error function is used as the loss function. Among them, the definition of the scale-invariant signal-to-noise ratio loss function is as shown in formula (2):

[0104]

[0105] Where, X T and X N The definition of is as follows:

[0106]

[0107] X N =f(x)-X T (4)

[0108] Where x represents reverberant speech, f(x) represents enhanced speech, y represents pure speech, and ||y|| 2 、||X T || 2 and ||X N || 2 is the 2-norm result. The mean square error loss function is calculated as shown in formula (5):

[0109] l MSE =(t(x)-T) 2 (5)

[0110] Where x represents the reverberant speech, t(x) represents the predicted value of the reverberation time, and T represents the true value of the reverberation time.

[0111] The definition of inverse short-time Fourier transform is shown in formula (6):

[0112]

[0113] Wherein, φ(n) represents the comprehensive window function, l and k represent the time domain and frequency domain respectively, L represents the frame length of speech, n represents discrete time, i represents the time sequence number, ω(n) is the window function and the window length is N. Commonly used window functions include rectangular window, Hamming window and Gaussian window, etc., e represents a natural constant, j represents an imaginary unit, x() represents the reverberant speech, x′() represents the short-time Fourier transform result of the reverberant speech, and the relationship between the window function and the comprehensive window function is defined as shown in formula (7):

[0114]

[0115] S6, such as Figure 5 As shown in the upper part of the figure, the reverberant speech to be enhanced is obtained, and the reverberant speech to be enhanced is sent to the trained progressive reverberation perception network after short-time Fourier transform, and the output of the first coding branch is used as the frequency domain result of the enhanced speech, and then the enhanced speech after removing the reverberation is obtained through short-time Fourier inverse transform. Figure 5 shown in the lower half of .

[0116] like Figure 4 As shown, in one embodiment, this process belongs to the testing stage, which also needs to perform a short-time Fourier transform operation on the speech to be enhanced to convert the time domain speech into the frequency domain and input it into the progressive reverberation perception network. In this stage, only the short-time inverse Fourier transform result of decoding branch 1 is used as the final enhanced speech result, and decoding branch 2, decoding branch 3 and decoding branch 4 do not participate in this stage.

[0117] The present invention also proposes a speech dereverberation device based on a progressive reverberation perception network, the device comprising:

[0118] A clean speech acquisition module is used to obtain clean speech in the reverberation data set;

[0119] A reverberation speech acquisition module, used to acquire the reverberation speech in the reverberation data set and the reverberation speech to be enhanced;

[0120] A target speech acquisition module is used to acquire the target speech in the reverberation data set, including a target speech with a reverberation time of 50 ms, a target speech with a reverberation time of 200 ms, and a target speech with a reverberation time of 300 ms;

[0121] A short-time Fourier transform module is used to transform the time-domain speech into a frequency-domain speech frame before the reverberant speech is input into the progressive reverberation perception network;

[0122] A network construction module, used to construct a progressive reverberation perception network including a multi-scale feature extraction path and a multi-branch decoding path;

[0123] A short-time inverse Fourier transform module is used to transform the frequency-domain enhanced speech frame output by the progressive reverberation perception network into time-domain speech;

[0124] The speech enhancement module is used to pass the reverberated speech to be enhanced through a progressive reverberation perception network to obtain the enhanced speech after the reverberation is removed.

[0125] Furthermore, the reverberation speech acquisition module includes:

[0126] A graph method unit, generating a reverberation filter generator based on a graph method;

[0127] A parameter configuration unit, which configures the parameters required by the reverberation filter generator to construct a reverberation filter with variable parameters;

[0128] The reverberation speech generating unit is used to obtain the reverberation speech by passing the clean speech through a 500ms reverberation time filter whose reverberation time parameter is configured as 500ms.

[0129] Furthermore, the target speech acquisition module includes:

[0130] A reverberation filter time truncation unit intercepts the system function of a 500ms reverberation time filter, thereby obtaining 50ms, 200ms and 300ms reverberation time system functions;

[0131] The target speech acquisition unit is used to pass the clean speech through the 50ms, 200ms and 300ms reverberation time system function, so as to obtain the 50ms, 200ms and 300ms reverberation time target speech.

[0132] Furthermore, the short-time Fourier transform module includes:

[0133] A framing unit, used for dividing the reverberant speech into a series of short segments to obtain short-term stable reverberant speech frames, where there are overlaps between the short segments;

[0134] A windowing unit, used to window the reverberation speech frame to prevent spectrum leakage;

[0135] The Fourier transform unit is used to perform Fourier transform on the windowed reverberation speech frame to obtain the frequency domain reverberation speech frame, and then extract the real part and imaginary part information.

[0136] Furthermore, the network construction module includes:

[0137] A progressive reverberation perception network, including a multi-scale feature extraction path layer and a multi-branch decoding path layer;

[0138] The multi-scale feature extraction path layer includes 4 repeated encoding units and 1 multi-scale feature pyramid unit. Each encoding unit includes 2 convolutional layers and 1 average pooling layer. The multi-scale feature pyramid unit includes 4 parallel feature extraction branches and 1 convolutional layer, and each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer.

[0139] The multi-branch decoding path layer includes four decoding branches, each decoding branch includes four decoding units having the same feature depth as the encoding unit in the multi-scale feature extraction path layer, and each unit includes two convolutional layers and one transposed convolutional layer;

[0140] A first supervision unit, configured to use a target speech with a reverberation time of 50 ms as first supervision information;

[0141] A first loss function unit is used to train a progressive reverberation perception network according to a target speech with a reverberation time of 50 ms and a predicted enhanced speech;

[0142] A second supervision unit, used for taking the target speech with a reverberation time of 200 ms as the second supervision information;

[0143] The second loss function unit is used to train a progressive reverberation perception network according to the target speech with a reverberation time of 200 ms and the predicted enhanced speech;

[0144] A third supervision unit, used for taking the 300 ms reverberation time target speech as the third supervision information;

[0145] A third loss function unit is used to train a progressive reverberation perception network according to the target speech with a reverberation time of 300 ms and the predicted enhanced speech;

[0146] An auxiliary supervision unit, used to use the 500ms reverberation time as auxiliary supervision information;

[0147] Auxiliary loss function unit, used to train the progressive reverberation perception network based on the 500ms reverberation time and the predicted reverberation time.

[0148] Furthermore, the short-time inverse Fourier transform module includes:

[0149] An inverse Fourier transform unit, used to transform the frequency domain result of the enhanced frame output by the network into a time domain result;

[0150] The time domain superposition unit is used to overlap and add the enhanced frames after the inverse Fourier transform to perfectly restore the final enhanced speech.

[0151] The present invention also proposes a computer device, comprising at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute a speech dereverberation method based on a progressive reverberation perception network proposed by the present invention.

[0152] Of course, it is understandable that some features of the method, apparatus, and computer device in the present invention can be referenced to each other, and the present invention will not list them one by one to save space.

[0153] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0154] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A speech dereverberation method based on a progressive reverberation perception network, characterized in that: The method comprises the following steps: S1, constructing a reverberation filter generator, and generating a reverberation filter bank by configuring the required parameters of the generator; S2, constructing a reverberation data set, passing the clean speech in the speech data set through a reverberation filter to generate a reverberation speech of the reverberation data set, and inputting the clean speech into the reverberation filter with the same parameters after time truncation to generate the first, second, and third target speech of the reverberation speech; S3, performing short-time Fourier transform on the reverberant speech to extract its real part and imaginary part, and using the real part and imaginary part as two channels of input features; S4, constructing a progressive reverberation perception network including a multi-scale feature extraction path and a multi-branch decoding path; S5, taking the three target voices as the first, second, and third target voices of the progressive reverberation perception network, using the scale-invariant signal-to-noise ratio loss function to minimize the difference between the enhanced voice and the three target voices, taking the reverberation time of the reverberation voice as auxiliary supervision information, using the mean square error loss function to minimize the difference between the reverberation time prediction value and the reverberation time, and training the progressive reverberation perception network; S6, obtaining the reverberated speech to be enhanced, and sending the reverberated speech to be enhanced into the trained progressive reverberation perception network after short-time Fourier transform, and taking the decoding branch supervised by the first target speech as the frequency domain result of the enhanced speech, and then obtaining the enhanced speech after removing the reverberation through inverse short-time Fourier transform; The time truncation in step S2 includes time-truncation of the reverberation filter system function to obtain three system functions, where the truncation time periods are 0-50ms, 0-200ms, and 0-300ms; inputting the clean speech into the truncation system function to generate the first, second, and third target speech respectively; The scale-invariant signal-to-noise ratio loss function in step S5 is expressed as: Where, X T and X N Respectively expressed as: X N =f(x)-X T Where x represents reverberant speech, f(x) represents enhanced speech, y represents pure speech, and ||y|| 2 、||X T || 2 and ||X N || 2 is the 2-norm result; The mean square error loss function in step S5 is expressed as: l MSE =(t(x)-T) 2 Where x represents the reverberant speech, t(x) represents the predicted value of the reverberation time, and T represents the true value of the reverberation time.

2. A speech dereverberation method based on a progressive reverberation perception network according to claim 1, characterized in that: The construction of the reverberation filter generator described in step S1 includes: constructing the reverberation filter generator using a graph method, and generating multiple reverberation filter generator groups by configuring the parameters required by the generator; the required parameters include: room size, sound source position, reverberation time and microphone position.

3. The method for speech dereverberation based on a progressive reverberation perception network according to claim 1, characterized in that: Generating the reverberated speech of the reverberated data set in step S2 includes passing the clean speech through a reverberation filter with a reverberation time of 500 ms, thereby generating the reverberated speech in the reverberated data set.

4. The method for speech dereverberation based on a progressive reverberation perception network according to claim 1, characterized in that: The short-time Fourier transform in step S3 is expressed as: Among them, l and k represent the time domain and frequency domain respectively, L represents the frame length of speech, n represents discrete time, ω(n) is the window function and the window length is N, e represents a natural constant, j represents an imaginary unit, x() represents reverberant speech, and x′() represents the short-time Fourier transform result of the reverberant speech.

5. The method for speech dereverberation based on a progressive reverberation perception network according to claim 1, characterized in that: The method of using the real part and the imaginary part as two channels of input features in step S3 includes: stacking the real part and the imaginary part of the reverberation speech frame after short-time Fourier transform along the feature channel direction, so as to input the real part and the imaginary part information of the reverberation speech into the progressive reverberation perception network at the same time, thereby retaining the phase information of the reverberation speech.

6. A speech dereverberation method based on a progressive reverberation perception network according to claim 1, characterized in that: The step S4 constructs a progressive reverberation perception network including a multi-scale feature extraction path and a multi-branch decoding path, wherein the multi-scale feature extraction path includes 4 repeated encoding units and 1 multi-scale feature pyramid unit; each encoding unit includes 2 convolutional layers and 1 average pooling layer; the multi-scale feature pyramid unit includes 4 parallel feature extraction branches and 1 convolutional layer, and each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer, thereby achieving the purpose of multi-scale feature extraction.

7. The method for speech dereverberation based on a progressive reverberation perception network according to claim 1, characterized in that: The step S4 constructs a progressive reverberation perception network comprising a multi-scale feature extraction path and a multi-branch decoding path; wherein the multi-branch decoding path has a total of 4 decoding branches, each decoding branch includes 4 decoding units having the same feature depth as the encoding unit in the multi-scale feature extraction path, and each unit includes 2 convolutional layers and 1 transposed convolutional layer.

8. The method for speech dereverberation based on a progressive reverberation perception network according to claim 1, characterized in that: In step S5, the three target speech is used as the first, second and third supervisory information of the reverberation perception network, including using the first, second and third target speech as the supervisory information of the first, second and third decoding branches of the progressive reverberation perception network respectively; the reverberation time of the reverberation speech is used as auxiliary supervisory information, including using the 500ms reverberation time as the supervisory information of the fourth branch of the progressive reverberation perception network.

9. The method for speech dereverberation based on a progressive reverberation perception network according to claim 1, characterized in that: The short-time inverse Fourier transform in step S6 is expressed as: Among them, φ(n) represents the comprehensive window function, l and k represent the time domain and frequency domain respectively, L represents the frame length of the speech, n represents the discrete time, i represents the time sequence number, ω(n) is the window function and the window length is N, e represents the natural constant, j represents the imaginary unit, x() represents the reverberant speech, and x′() represents the short-time Fourier transform result of the reverberant speech. The relationship between the window function and the comprehensive window function is defined as: ω(n) is a window function and the window length is N.

10. A reverberation elimination device based on a progressive reverberation perception network, characterized in that: The device comprises: A clean speech acquisition module is used to obtain clean speech in the reverberation data set; A reverberation speech acquisition module, used to acquire the reverberation speech in the reverberation data set and the reverberation speech to be enhanced; A target speech acquisition module is used to acquire the target speech in the reverberation data set, including a target speech with a reverberation time of 50 ms, a target speech with a reverberation time of 200 ms, and a target speech with a reverberation time of 300 ms; A short-time Fourier transform module is used to transform the time-domain speech into a frequency-domain speech frame before the reverberant speech is input into the progressive reverberation perception network; A network construction module, used to construct a progressive reverberation perception network including a multi-scale feature extraction path and a multi-branch decoding path; A short-time inverse Fourier transform module is used to transform the frequency-domain enhanced speech frame output by the progressive reverberation perception network into time-domain speech; The speech enhancement module is used to pass the reverberated speech to be enhanced through a progressive reverberation perception network to obtain the enhanced speech after the reverberation is removed; Wherein, the network construction module includes: A progressive reverberation perception network, comprising a multi-scale feature extraction path layer and a multi-branch decoding path layer; wherein the multi-scale feature extraction path layer comprises 4 repeated encoding units and 1 multi-scale feature pyramid unit; each encoding unit comprises 2 convolutional layers and 1 average pooling layer; the multi-scale feature pyramid unit comprises 4 parallel feature extraction branches and 1 convolutional layer, and each feature extraction branch comprises 1 average pooling layer and 1 bilinear interpolation layer; The multi-branch decoding path layer includes four decoding branches, each decoding branch includes four decoding units having the same feature depth as the encoding unit in the multi-scale feature extraction path layer, and each unit includes two convolutional layers and one transposed convolutional layer; A first supervision unit, configured to use a target speech with a reverberation time of 50 ms as first supervision information; A first loss function unit is used to train a progressive reverberation perception network according to a target speech with a reverberation time of 50 ms and a predicted enhanced speech; A second supervision unit, used for taking the target speech with a reverberation time of 200 ms as the second supervision information; The second loss function unit is used to train a progressive reverberation perception network according to the target speech with a reverberation time of 200 ms and the predicted enhanced speech; A third supervision unit, used for taking the 300 ms reverberation time target speech as the third supervision information; A third loss function unit is used to train a progressive reverberation perception network according to the target speech with a reverberation time of 300 ms and the predicted enhanced speech; An auxiliary supervision unit, used to use the 500ms reverberation time as auxiliary supervision information; Auxiliary loss function unit, used to train the progressive reverberation perception network based on the 500ms reverberation time and the predicted reverberation time.

11. The reverberation elimination device based on the progressive reverberation perception network according to claim 10, characterized in that: The reverberation speech acquisition module comprises: A graph method unit, generating a reverberation filter generator based on a graph method; A parameter configuration unit, which configures the parameters required by the reverberation filter generator to construct a reverberation filter with variable parameters; The reverberation speech generating unit is used to obtain the reverberation speech by passing the clean speech through a 500ms reverberation time filter whose reverberation time parameter is configured as 500ms.

12. The reverberation elimination device based on the progressive reverberation perception network according to claim 10, characterized in that: The target speech acquisition module comprises: A reverberation filter time truncation unit, which intercepts the system function of the 500 ms reverberation time filter, thereby obtaining 50 ms, 200 ms and 300 ms reverberation time system functions; The target speech acquisition unit is used to pass the clean speech through the 50ms, 200ms and 300ms reverberation time system function, so as to obtain the 50ms, 200ms and 300ms reverberation time target speech.

13. The reverberation elimination device based on the progressive reverberation perception network according to claim 10, characterized in that: The short-time Fourier transform module comprises: A framing unit, used for dividing the reverberant speech into a series of short segments to obtain short-term stable reverberant speech frames, where there are overlaps between the short segments; A windowing unit, used to window the reverberation speech frame to prevent spectrum leakage; The Fourier transform unit is used to perform Fourier transform on the windowed reverberation speech frame to obtain the frequency domain reverberation speech frame, and then extract the real part and imaginary part information.

14. The reverberation elimination device based on the progressive reverberation perception network according to claim 10, characterized in that: The short-time inverse Fourier transform module comprises: An inverse Fourier transform unit, used to transform the frequency domain result of the enhanced frame output by the network into a time domain result; The time domain superposition unit is used to overlap and add the enhanced frames after the inverse Fourier transform to perfectly restore the final enhanced speech.

15. A computer device comprising at least one processor; and at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 9 by calling the program instructions.

Images