Speech enhancement device, speech enhancement method, and program

Abstract

To provide a speech enhancement device, a speech enhancement method, and a program that can improve robustness of speed enhancement.SOLUTION: A speech enhancement device comprises: a preprocessing section which generates an amplitude spectrogram of a speech signal; a mask generation section which uses two or more methods for estimating time-frequency masks for the obtained amplitude spectrogram, and generates time-frequency masks by the respective methods; an integration section which integrates at least two of the plurality of generated time-frequency masks to generate an integrated mask; and a speech enhancement section which performs enhancement processing on a speech signal obtained using the generated integrated mask.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a voice enhancement device, a voice enhancement method, and a program. [Background technology]

[0002] One of the problems in speech recognition is that in a noisy environment, the number of false positives of words and speech parts increases, resulting in a deterioration in recognition accuracy. To address this problem, speech enhancement technologies have been researched (see, for example, Patent Document 1). Among them, a speech enhancement method that combines time-frequency masking and beamforming has been reported to have high enhancement performance (see, for example, Non-Patent Document 1).

[0003] FIG. 11 is a diagram showing an overview of the flow of the conventional speech enhancement process. As shown in FIG. 11, in the conventional speech enhancement method, time-frequency mask estimation is performed using a time-frequency mask g902 of each sound source for a spectrogram g901 of an observed signal. The time-frequency mask passes components originating from each sound source included in the amplitude spectrogram of the observed speech signal and blocks other components. Next, in the conventional speech enhancement method, beamforming processing is performed to obtain a spectrogram g903 of the target speech. Note that beamforming can emphasize sounds arriving from a target direction and suppress sounds from other directions. In the conventional method, the mask is estimated using a neural network or some other model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-88404 A [Non-patent literature]

[0005] [Non-Patent Document 1] J. Heymann et al. “Neural network based spectral mask estimation for acoustic beamforming” In Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2016. Summary of the Invention [Problem to be solved by the invention]

[0006] However, since conventional speech enhancement methods estimate a time-frequency mask from a single model or network, they may not be able to fully utilize speech features and may not be able to enhance speech accurately. As a result, conventional speech enhancement methods may not be robust enough.

[0007] The present invention has been made in consideration of the above problems, and has an object to provide a voice emphasizing device, a voice emphasizing method, and a program that can improve the robustness of voice emphasizing. [Means for solving the problem]

[0008] (1) In order to achieve the above object, a voice enhancement device according to one embodiment of the present invention includes a preprocessing unit that generates an amplitude spectrogram of a voice signal, a mask creation unit that uses a method for estimating two or more time-frequency masks for the obtained amplitude spectrogram to create the time-frequency masks using each of the methods, an integration unit that creates an integrated mask by integrating at least two of the created multiple time-frequency masks, and a voice enhancement unit that performs an enhancement process of the acquired voice signal using the created integrated mask.

[0009] (2) In the voice emphasizing device according to an aspect of the present invention, the integration unit may integrate two or more of the plurality of time-frequency masks created by multiplying them by a weighting coefficient.

[0010] (3) In addition, in a speech enhancement device according to one embodiment of the present invention, the method for estimating the two or more time-frequency masks may be two or more of a bidirectional LSTM (BLSTM (Bidirectional Long Short-Term Memory)) method, an MNMFDP method that combines multi-channel non-negative matrix factorization (MNMF) and a DNN (Deep Neural Network), two types of LSTM (DTLN (Dual-signal Transformation LSTM)) methods, and a SEGAN (Speech Enhancement GAN (Generative Adversarial Network)) method that uses a generative adversarial network for speech enhancement.

[0011] (4) In the voice emphasizing device according to an aspect of the present invention, the integration unit may perform a simple average or a weighted average on two or more of the two or more time-frequency masks.

[0012] (5) In addition, in the voice enhancement device according to one aspect of the present invention, the voice enhancement unit may be configured to enhance and extract the target voice by performing Generalized EigenValue (GEV) beamforming using the integrated mask.

[0013] (6) In addition, in a voice enhancement device according to one aspect of the present invention, the weighting coefficients may be calculated by solving an optimization problem that minimizes a mean square error between an ideal mask and the time-frequency mask of a target voice or noise that is quantized by a method for estimating the time-frequency mask.

[0014] (7) In order to achieve the above-mentioned object, a speech enhancement method according to one embodiment of the present invention includes a preprocessing unit that generates an amplitude spectrogram of a speech signal, a mask creation unit that creates two or more time-frequency masks using respective methods for estimating the time-frequency masks for the obtained amplitude spectrogram, an integration unit that integrates at least two of the multiple time-frequency masks that have been created to create an integrated mask, and a speech enhancement unit that performs enhancement processing of the acquired speech signal using the created integrated mask.

[0015] (8) In order to achieve the above-mentioned object, a program according to one embodiment of the present invention causes a computer to generate an amplitude spectrogram of an audio signal, create two or more time-frequency masks using each of a method for estimating the time-frequency masks from the obtained amplitude spectrogram, integrate at least two of the multiple time-frequency masks created to create an integrated mask, and perform enhancement processing of the acquired audio signal using the created integrated mask. Effect of the Invention

[0016] According to the above (1) to (8), a time-frequency mask is set by combining a plurality of mask estimation methods, so that the robustness of speech enhancement can be improved. According to the above (2) and (6), weighting is applied to the methods for estimating time-frequency masks based on a plurality of different design concepts, so that optimization can be performed according to the environment, voice, and application. According to the above-mentioned (3), the mask is estimated by a comprehensive method based on a plurality of different design concepts, so that the speech enhancement performance can be improved. According to (4) above, it is possible to optimize the sound depending on the environment, voice, and application. [Brief description of the drawings]

[0017] [Figure 1] FIG. 2 is a diagram showing an outline of a flow of audio processing according to the embodiment. [Diagram 2] 1 is a block diagram showing an example of the configuration of a voice emphasizing device according to an embodiment; [Diagram 3] FIG. 2 is a conceptual diagram of an ideal mask I according to the embodiment. [Figure 4] FIG. 13 is a conceptual diagram of a time-frequency mask Mn according to a method n according to an embodiment. [Diagram 5] FIG. 1 is a diagram showing an overview of the process of BLSTM, which is a first mask creation method. [Figure 6] FIG. 13 is a diagram showing an overview of the process of MNMFDP, which is a second mask creation method. [Figure 7] FIG. 13 is a diagram showing an overview of the process of DTLN, which is a third mask creation method. [Figure 8] FIG. 13 is a diagram showing an overview of the processing of SEGAN, which is a fourth mask creation method. [Figure 9] 1 is a flowchart of an example of a processing procedure of a voice emphasizing device according to an embodiment. [Figure 10] FIG. 13 is a diagram showing an evaluation result according to the embodiment. [Figure 11] FIG. 1 is a diagram showing an outline of the flow of a voice enhancement process according to the prior art; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] [overview] First, an outline of the flow of audio processing in this embodiment will be described. Fig. 1 is a diagram showing an overview of the flow of speech processing according to this embodiment. As shown in Fig. 1, in this embodiment, speech enhancement is performed on a spectrogram g11 of an observed signal using a time-frequency mask g21 and beamforming g22 to obtain a spectrogram g13 of a target speech. Then, in this embodiment, multiple time-frequency masks are created using multiple speech enhancement methods, and these are ensembled to create a new mask, an ensemble time-frequency mask g12.

[0020] [Example of voice enhancement device configuration] Next, a configuration example of the voice emphasizing device according to the present embodiment will be described. Fig. 2 is a block diagram showing an example of the configuration of the voice enhancement device 1 according to this embodiment. As shown in Fig. 1, the voice enhancement device 1 includes a voice signal acquisition unit 11 (pre-processing unit), a short-time Fourier transform unit 12 (pre-processing unit), a mask creation unit 13, an integration unit 14, a voice enhancement unit 15, an inverse short-time Fourier transform unit 17, and an output unit 18.

[0021] Mask creation unit 13 includes, for example, first mask creation unit 131, second mask creation unit 132, third mask creation unit 133, and fourth mask creation unit 134. Note that, although an example in which mask creation unit 13 includes four Nth mask creation units has been described in Fig. 1, it is sufficient for mask creation unit 13 to include two or more Nth mask creation units. The voice enhancing unit 15 includes, for example, a beamforming unit 16 .

[0022] The audio signal acquisition unit 11 acquires an audio signal and outputs the acquired audio signal to the short-time Fourier transform unit 12. The audio signal acquisition unit 11 may be a microphone or a microphone array including a plurality of microphones. Alternatively, the audio signal acquisition unit 11 may acquire, for example, a pre-recorded audio signal.

[0023] The short-time Fourier transform unit 12 performs a short-time Fourier transform on the acquired voice signal to convert it into an amplitude spectrogram. The short-time Fourier transform unit 12 outputs the amplitude spectrogram to the mask creation unit 13.

[0024] The mask creation unit 13 applies N (N is an integer equal to or greater than 2, for example, 4) speech enhancement techniques to the amplitude spectrogram to estimate N time-frequency masks. The first mask generation unit 131 performs time-frequency mask estimation by a first technique. The second mask generating unit 132 performs time-frequency mask estimation using a second technique different from the first technique. The third mask generating unit 133 performs time-frequency mask estimation using a third technique that is different from the first and second techniques. The fourth mask creation unit 134 performs time-frequency mask estimation using a fourth technique that is different from the first to third techniques. Examples of each technique will be described later.

[0025] The integrating unit 14 creates an ensemble time-frequency mask by integrating (ensembling) the N time-frequency masks estimated by the mask creating unit 13. The integrating unit 14 estimates a spatial correlation matrix from the created ensemble time-frequency mask, and outputs the estimated spatial correlation matrix to the beamforming unit 16. Note that in this embodiment, for example, simple averaging and weighted averaging are used as the integrating method. The simple averaging and weighted averaging will be described later.

[0026] The beamforming unit 16 extracts a spectrogram of the target voice by, for example, a Generalized EigenValue (GEV) beamforming method.

[0027] The inverse short-time Fourier transform unit 17 performs a short-time Fourier transform on the spectrogram of the extracted target voice to convert it into a signal in the time domain. The inverse short-time Fourier transform unit 17 outputs the converted signal to the output unit 18 as a target voice signal.

[0028] The output unit 18 outputs the emphasized and extracted target voice signal. The output unit 18 is, for example, a speaker.

[0029] [Time-frequency mask integration method] Next, an example of a method for integrating the time-frequency masks will be described. In this embodiment, for example, a simple average or a weighted average is used as a method for integrating the time-frequency masks. In the simple average ensemble, two or more methods are taken from all enhancement methods, and a simple average is taken over all cases to create a mask.

[0030] In the weighted average, an optimization problem is solved in advance using learning data to determine the weight for the mask of each technique. In this embodiment, when N enhancement techniques are used, the mask weights α=[α1, α2, ..., α] that minimize the mean square error between the ideal mask I and the ensemble time-frequency mask are determined as shown in the following formula (1). N ] T We defined it as an optimization problem to find the following:

[0031]

number

[0032] In equation (1), I(∈R) D×F×T ) is the ideal mask, and M n (∈R(double line character) D×F×T ) is the time-frequency mask of the target speech or noise generated by enhancement method n (n = 1, 2, ..., N). Also, Σ n=1 N α n M ndft The term is the ensemble time-frequency mask.

[0033] α n represents the weight for each mask. N is the number of methods used for integration. As shown in Figs. 3 and 4, D is the number of data, F is the frequency, and T is the number of frames. d, f, and t indicate the corresponding indexes. Fig. 4 is an image diagram of an ideal mask I according to this embodiment. Fig. 5 is an image diagram of a time-frequency mask M of method n according to this embodiment. n This is an image diagram.

[0034] In this embodiment, the following equation (2) is imposed as a constraint. The integration unit 14 determines the weight α n In this manner, in this embodiment, weighting is performed on the methods for estimating time-frequency masks based on a plurality of different design concepts, so that optimization can be performed according to the environment, voice, and application.

[0035]

number

[0036] The integration unit 14 may calculate the weights using, for example, a trained model. The integration unit 14 may also perform integration for each combination of a plurality of mask creation methods. The integration unit 14 may update the trained model based on, for example, a result of a person listening to and judging the result of the voice enhancement, or may learn a combination of mask creation methods for each voice, environment, and purpose.

[0037] [Mask creation method example] Next, an example of a method for creating a mask will be described with reference to FIGS. FIG. 5 is a diagram showing an outline of the process of BLSTM, which is the first mask creation method. The first mask creation method is, for example, a bidirectional long short-term memory (BLSTM) method (see Non-Patent Document 1).

[0038] As shown in Figure 5, in BLSTM, multi-channel observed signals are input and converted to absolute values ​​(201), then the time-frequency masks of the target speech and noise are estimated by a neural network (202), followed by smoothing by median filters (203a, 203b), and the PSD (Power Spectral Density) matrices of the speech and noise are estimated by PSD (204a, 204b), after which beamforming processing (205) is performed. In addition, there are two types of networks for estimating the time-frequency mask, and bidirectional LSTM is used.

[0039] FIG. 6 is a diagram showing an outline of the process of MNMFDP, which is the second mask creation method. The second mask creation method is, for example, the MNMFDP method (see Reference 1), which combines MNMF (Multichannel Non-negative Matrix Factorization) and DNN (Deep Neural Network).

[0040] As shown in Figure 6, MNMFDP models speech using a DNN-based network (301) and noise using multi-channel NMF (302). In addition, the estimated mask is used to calculate the SCM (Spatial Covariance Matrices) of the cost function of speech and noise. Using these, MNMFDP obtains the spectrum (304) of the observed multi-channel signal with the target speech emphasized. In this way, MNMFDP can model the sound source more accurately than multi-channel NMF by combining multi-channel NMF and deep neural network.

[0041] Reference 1: Sekiguchi et.al., “Semi-Supervised Multichannel Speech Enhancement With a Deep Speech Prior”, IEEE / ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 27, NO. 12, DECEMBER 2019

[0042] FIG. 7 is a diagram showing an outline of the process of DTLN, which is the third mask creation method. A third mask creation method is, for example, the dual-signal transformation (DTLN) method (see Reference 2).

[0043] As shown in FIG. 7, in DTLN, for example, a target voice is emphasized by transforming a signal using an LSTM network (401) for time-frequency masking and an LSTM network (402) for signal feature representation. The time-frequency masking LSTM network (401) includes, for example, an STFT (short-time Fourier transform) layer, an absolute value layer, a multi-layer LSTM, an FC (fully connected) layer, a sigmoid function layer, an iFFT (inverse fast Fourier transform) layer, and the like. The LSTM network (402) of signal feature representation includes, for example, a 1D-Conv layer that extracts feature representations, an iLN (instant layer normalization) layer, a multi-layer LSTM, an FC layer, a sigmoid function layer, and an Overlap-Add layer that reconstructs signals by a procedure of overlap and addition. In this embodiment, a time-frequency mask is created from the pre-enhancement and post-enhancement signals and used for integration.

[0044] Reference 2: Westhausen et.al., “Dual-Signal Transformation LSTM Network for Real-Time Noise Suppression”, arXiv preprint, 2020

[0045] FIG. 8 is a diagram showing an outline of the process of SEGAN, which is the fourth mask creation method. The third mask creation method is, for example, the Speech Enhancement GAN (SEGAN) method (see Reference 3), which uses a generative adversarial network for speech enhancement, and performs time-frequency mask estimation from the results of the GAN.

[0046] SEGAN is a speech enhancement method that uses a generative adversarial network (GAN). As shown in Figure 8, in SEGAN, for example, when a single-channel observed signal is input to a generator, it outputs enhanced speech. In SEGAN, noise-free speech data is treated as true data, and data output by the generator to imitate the true data is treated as false data. Next, in SEGAN, the true data and false data are input to a classifier along with the correct answer label and trained. Finally, in SEGAN, the generator is trained until the classifier makes a misclassification. In SEGAN, enhanced speech is obtained by inputting the observed signal into a trained generator. In this embodiment, similar to DTLN, a time-frequency mask is created from the pre- and post-enhancement signals and used for integration.

[0047] Reference 3: Pascual et.al., “SEGAN: Speech Enhancement Generative Adversarial Network”, arXiv preprint,2020

[0048] Note that the four mask creation methods (speech enhancement methods) described with reference to Figs. 5 to 9 are merely examples, and the method used by the mask creation unit 13 is not limited to these. The method used by the mask creation unit 13 may be another method depending on the environment, the speech signal, or the purpose. Furthermore, according to this embodiment, the mask is estimated by a comprehensive method based on a plurality of concepts with different designs, so that the final speech enhancement performance can be improved.

[0049] [Processing Procedure] Next, an example of a processing procedure of the voice emphasizing device 1 will be described. FIG. 9 is a flowchart of an example of a processing procedure of the voice emphasizing device 1 according to the present embodiment.

[0050] (Step S1) The voice signal acquisition unit 11 acquires a voice signal.

[0051] (Step S2) The short-time Fourier transform unit 12 performs a short-time Fourier transform on the acquired voice signal to convert it into an amplitude spectrogram.

[0052] (Step S3) Mask creation unit 13 applies N methods to the amplitude spectrogram to estimate four time-frequency masks.

[0053] (Step S4) The integrating unit 14 integrates the N time-frequency masks estimated by the mask creating unit 13 to create an ensemble time-frequency mask.

[0054] (Step S5) The beam forming unit 16 performs a voice enhancement process by extracting a spectrogram of the target voice, for example, by the GEV beam forming method.

[0055] (Step S6) The inverse short-time Fourier transform unit 17 performs a short-time Fourier transform on the extracted spectrogram of the target voice to convert it into a signal in the time domain.

[0056] (Step S7) The output unit 18 outputs the emphasized and extracted target voice signal.

[0057] [Evaluation Results] Next, an example of the results of evaluating the method of this embodiment will be described. In order to evaluate the effectiveness of the method of this embodiment, the quality of the enhanced voice was evaluated using STOI (Short-Time Objective Intelligibility measure) (see Reference 4) and PESQ (Perceptual Evaluation of Speech Quality) (see Reference 5), which are correlated with human hearing. STOI is calculated as a real number from 0 to 1, and PESQ is calculated as a real number from 1 to 5, and the higher the value, the easier the voice is to hear.

[0058] Reference 4; Cees H. Taal; Richard C. Hendriks et al, “An Algorithm for Intelligibility Prediction of Time-Frequency Weighted Noisy Speech”, IEEE, p2125-2136, 2011 Reference 5; ITU-T Recommendation. Perceptual evaluation of speech quality (pesq): “An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs”, International Telecommunication Union Telecommunication Standardization Sector Recommendation P. 862, 2001

[0059] The data used for the evaluation was the CHiME3 data (see Reference 6), which is widely used as a benchmark for speech enhancement. The CHiME-3 dataset was created by combining the WSJ0 corpus with environmental noise, and is composed of real and simulated recordings. For the evaluation, we used a total of 1,320 pieces of real data, created by recording 330 utterances in each of four noisy environments: "on the bus," "cafe," "pedestrian area," and "street," at 6 channels and 16 kHz.

[0060] Reference 6; Jon Barker et al. “The third' chime 'speech separation and recognition challenge: Dataset, task and baselines”, In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 504-511. IEEE, 2015.

[0061] Four types of methods were used for the integration. The methods used for integration were the above-mentioned BLSTM, MNMFDP, DTLN, and SEGAN. In BLSTM, the time-frequency mask was estimated using a bidirectional LSTM network (BLSTM). In MNMFDP, the mask was estimated by combining multi-channel non-negative matrix factorization (MNMF) and DNN. In DTLN, speech was enhanced using dual-signal transformation LSTM network (DTLN) with two types of LSTM networks. As four speech enhancement methods were used for integration, a total of 11 types of simple average integration were performed.

[0062] Before conducting the evaluation experiments, we estimated the weights α for the masks generated by the four methods using a weighted average integration weight determination algorithm. For weight estimation, we used training recordings from the CHiME-3 dataset, which consists of a total of 7,138 samples created from 84 speeches and four types of noise. The weights α for the time-frequency masks of the target speech and noise generated by each enhancement method (BLSTM, MNMF, DTLN, SEGAN) were N , α X are expressed as the following equations (3) and (4). Note that in equations (3) and (4), the superscript T represents transposition.

[0063]

number

[0064]

number

[0065] Fig. 10 is a diagram showing the evaluation results according to this embodiment. Fig. 10 shows a graph of the averages of unemphasized speech, speech emphasized by integrating simple averages, and speech emphasized by integrating weighted averages, evaluated by STOI and PESQ, and a combination table. Graph g51 is a graph of the averages evaluated by STOI. Graph g52 is a graph of the averages evaluated by PESQ. The horizontal axis of graphs g51 and g52 is the item, and the vertical axis is the evaluation value.

[0066] Table g53 shows the combination of the voices enhanced by which method. For example, the column in the dashed box g55 shows the simple average of the two, since BLSTM and MNMFDP are marked with a circle.

[0067] As shown in Figure 10, the simple average of BLSTM and DTLN obtained the best evaluation value for both STOI and PESQ. In addition, the weighted average results were equivalent to the simple average of BLSTM and DTLN.

[0068] Based on this result, the integration unit 14 may perform integration of simple averages or integration of weighted averages using the optimization of the objective function. Alternatively, the integration unit 14 may perform integration of simple averages and integration of weighted averages using the optimization of the objective function, and output both results.

[0069] As described above, in the present embodiment, a time-frequency mask is set by combining a plurality of mask estimation methods. In addition, the voice enhancement device 1 may perform simple average integration or weighted average integration using optimization of an objective function depending on, for example, the environment, the voice signal, or the purpose.

[0070] Conventional techniques estimate time-frequency masks from a single model or network, and therefore do not fully utilize the key features contained in the input signal that are essential for speech enhancement. In contrast, in the present embodiment, a plurality of time-frequency masks estimated from a plurality of different mask creation methods are integrated, so that key features for speech enhancement contained in the input signal can be fully utilized, and noise can be suppressed by enhancing the target speech, thereby improving the speech enhancement performance and the robustness of the speech enhancement.

[0071] The above-mentioned voice emphasizing device 1 may be provided in, for example, a robot, a vehicle, a voice recognition device, a smartphone, a tablet terminal, a toy equipped with a voice recognition device, a hearing aid, an artificial ear, etc. Also, the application may be provided via the cloud.

[0072] In addition, a program for realizing all or part of the functions of the voice emphasizing device 1 of the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processing performed by the voice emphasizing device 1. In addition, the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" also refers to a storage device that holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0073] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0074] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0075] 1...Speech enhancement device, 11...Speech signal acquisition unit, 12...Short-time Fourier transform unit, 13...Mask creation unit, 14...Synthesis unit, 15...Speech enhancement unit, 16...Beamforming unit, 17...Inverse short-time Fourier transform unit, 18...Output unit, 131...First mask creation unit, 132...Second mask creation unit, 133...Third mask creation unit, 134...Fourth mask creation unit

Claims

1. a pre-processing unit for generating an amplitude spectrogram of the audio signal; a mask creation unit that uses two or more time-frequency mask estimation methods for the obtained amplitude spectrogram to create the time-frequency masks by each method; an integration unit that integrates at least two of the multiple time-frequency masks to generate an integrated mask; a speech enhancement unit that performs enhancement processing of the acquired speech signal using the created integrated mask, The integration unit integrates two or more of the created plurality of time-frequency masks by multiplying them by a weighting coefficient. Speech enhancement device.

2. The method for estimating the two or more time-frequency masks is two or more of a bidirectional LSTM (BLSTM (Bidirectional Long Short-Term Memory) method), an MNMFDP method that combines multi-channel nonnegative matrix factorization (MNMF) and DNN (Deep Neural Network), two types of LSTM (DTLN (Dual-signal Transformation LSTM) methods, and a SEGAN (Speech Enhancement GAN (Generative Adversarial Network) method that uses a generative adversarial network for speech enhancement. The voice enhancement device according to claim 1 .

3. The integration unit performs a weighted average on two or more of the two or more time-frequency masks.

3. The voice emphasizing device according to claim 1.

4. The speech enhancement unit performs generalized equal value (GEV) beamforming using the integrated mask to enhance and extract a target speech. The voice emphasizing device according to any one of claims 1 to 3.

5. The weighting coefficients are obtained by solving an optimization problem that minimizes the mean square error between an ideal mask and the time-frequency mask of the target speech or noise quantized by a method for estimating the time-frequency mask. The voice enhancement device according to claim 1 .

6. a pre-processing unit for generating an amplitude spectrogram of the audio signal; a mask creation unit that uses two or more time-frequency mask estimation methods for the obtained amplitude spectrogram to create the time-frequency masks by each method; an integration unit that integrates at least two of the multiple time-frequency masks to generate an integrated mask; a speech enhancement unit that performs enhancement processing of the acquired speech signal using the created integrated mask, The method for estimating the two or more time-frequency masks is two or more of a bidirectional LSTM (BLSTM (Bidirectional Long Short-Term Memory) method), an MNMFDP method that combines multi-channel nonnegative matrix factorization (MNMF) and DNN (Deep Neural Network), two types of LSTM (DTLN (Dual-signal Transformation LSTM) methods, and a SEGAN (Speech Enhancement GAN (Generative Adversarial Network) method that uses a generative adversarial network for speech enhancement. Speech enhancement device.

7. A speech enhancement method executed by a computer having a preprocessing unit, a mask creation unit, an integration unit, and a speech enhancement unit, comprising: The pre-processing unit generates an amplitude spectrogram of the audio signal; the mask creation unit creates the time-frequency masks by using two or more time-frequency mask estimation methods for the obtained amplitude spectrogram, the integration unit integrates at least two of the multiple time-frequency masks to generate an integrated mask; The speech enhancement unit performs enhancement processing of the acquired speech signal using the created integrated mask.

1. A method for speech enhancement, comprising: The integration unit integrates two or more of the created plurality of time-frequency masks by multiplying them by a weighting coefficient. Speech enhancement methods.

8. A speech enhancement method executed by a computer having a preprocessing unit, a mask creation unit, an integration unit, and a speech enhancement unit, comprising: The pre-processing unit generates an amplitude spectrogram of the audio signal; the mask creation unit creates the time-frequency masks by using two or more time-frequency mask estimation methods for the obtained amplitude spectrogram, the integration unit integrates at least two of the multiple time-frequency masks to generate an integrated mask; The speech enhancement unit performs enhancement processing of the acquired speech signal using the created integrated mask, The method for estimating the two or more time-frequency masks is two or more of a bidirectional LSTM (BLSTM (Bidirectional Long Short-Term Memory) method), an MNMFDP method that combines multi-channel nonnegative matrix factorization (MNMF) and DNN (Deep Neural Network), two types of LSTM (DTLN (Dual-signal Transformation LSTM) methods, and a SEGAN (Speech Enhancement GAN (Generative Adversarial Network) method that uses a generative adversarial network for speech enhancement. Speech enhancement methods.

9. On the computer, generating an amplitude spectrogram of the audio signal; using two or more time-frequency mask estimation techniques for the obtained amplitude spectrogram to generate the time-frequency masks using each of the techniques; aggregating at least two of the plurality of time-frequency masks to generate an aggregate mask; performing enhancement processing on the acquired audio signal using the created integrated mask; A program, Creating the integrated mask includes multiplying two or more of the multiple time-frequency masks created by a weighting coefficient and integrating them. program.

10. On the computer, generating an amplitude spectrogram of the audio signal; using two or more time-frequency mask estimation techniques for the obtained amplitude spectrogram to generate the time-frequency masks using each of the techniques; aggregating at least two of the plurality of time-frequency masks to generate an aggregate mask; performing enhancement processing on the acquired audio signal using the created integrated mask; A program, The method for estimating the two or more time-frequency masks is two or more of a bidirectional LSTM (BLSTM (Bidirectional Long Short-Term Memory) method), an MNMFDP method that combines multi-channel nonnegative matrix factorization (MNMF) and DNN (Deep Neural Network), two types of LSTM (DTLN (Dual-signal Transformation LSTM) methods, and a SEGAN (Speech Enhancement GAN (Generative Adversarial Network) method that uses a generative adversarial network for speech enhancement. program.

Images