Signal processing device, signal processing method and signal processing program

Abstract

To estimate a signal of a virtually disposed microphone without making an explicit assumption with respect to a signal.SOLUTION: An estimation device 10 comprises an estimation section for receiving input of a time domain signal corresponding to amplitude and phase components of an observation signal of a real microphone and outputting a time domain signal corresponding to amplitude and phase components of an observation signal of a virtual microphone that is virtually disposed. The estimation section includes one neural network 11. The one neural network 11 is a deep learning model, estimates the time domain signal corresponding to the amplitude and phase components of the observation signal of the virtual microphone that is virtually disposed from the inputted time domain signal corresponding to the amplitude and phase components of the observation signal of the real microphone, and outputs an estimation result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a signal processing device, a signal processing method, and a signal processing program. [Background technology]

[0002] Array signal processing techniques using microphone arrays (multiple microphones) are widely used in various applications such as speech enhancement, sound source separation, and sound source direction estimation.

[0003] The performance of array signal processing basically depends on the number of microphones, but in actual operation, many devices have restrictions and it is often difficult to increase the number of microphones. For this reason, it is desirable to improve the performance of microphone array technology when the number of microphones is small.

[0004] In response to this, methods have been studied that estimate the signal of a virtual microphone virtually placed at a position where no microphone is actually placed, thereby making it possible to virtually increase the number of observation microphones. For example, there is a method of estimating the phase component of a virtual microphone signal based on a physical model. The physical model is a model that assumes a plane wave, sparseness of the voice, a microphone array with sufficiently narrow intervals, etc. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Hiroki Katahira, “Nonlinear speech enhancement by virtual increase of channels and maximum SNR beamformer”, [online], [Retrieved January 25, 2021], Internet <URL:https: / / asp-eurasipjournals.springeropen.com / track / pdf / 10.1186 / s13634-015-0301-3.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] In previous research, virtual microphone signals were estimated based on physical models, but these physical models did not always hold true, making it difficult to estimate the virtual microphone signals (especially the phase).

[0007] The present invention has been made in consideration of the above, and aims to provide a signal processing device, a signal processing method, and a signal processing program that can estimate signals from virtually placed microphones without making explicit assumptions about the signals. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the object, a signal processing device according to the present invention is a signal processing device that processes an acoustic signal, and has an estimation unit that accepts an input of a time domain signal corresponding to the amplitude and phase components of an observation signal of a real microphone, and outputs a time domain signal corresponding to the amplitude and phase components of an observation signal of a virtually placed virtual microphone, wherein the estimation unit has one neural network, and the one neural network is a deep learning model, and estimates a time domain signal corresponding to the amplitude and phase components of the observation signal of the virtually placed virtual microphone from the time domain signal corresponding to the amplitude and phase components of the observation signal of the real microphone that is input, and outputs the estimation result.

[0009] Moreover, a signal processing method according to the present invention is a signal processing method executed by a signal processing device, and includes a step of receiving an input of a time domain signal corresponding to the amplitude and phase components of an observation signal of a real microphone, and outputting a time domain signal corresponding to the amplitude and phase components of an observation signal of a virtually placed virtual microphone, wherein the outputting step uses one neural network, and the one neural network is a deep learning model, and estimates a time domain signal corresponding to the amplitude and phase components of the observation signal of the virtually placed virtual microphone from the time domain signal corresponding to the amplitude and phase components of the observation signal of the real microphone that has been input, and outputs the estimation result. Effect of the Invention

[0010] According to the present invention, signals from virtually placed microphones can be estimated without making any explicit assumptions about the signals. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of an estimation device according to the first embodiment. [Diagram 2] FIG. 2 is a flowchart illustrating a processing procedure of the estimation process according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a learning device according to the second embodiment. [Figure 4] FIG. 4 is a flowchart illustrating a processing procedure of the learning process according to the second embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a signal processing device according to the third embodiment. In FIG. [Figure 6] FIG. 6 shows the microphone array arrangement for the CHiME-4 corpus. [Figure 7] FIG. 7 is a diagram illustrating an example of a computer that realizes an estimation device, a learning device, and a signal processing device by executing a program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are indicated by the same reference numerals. Note that, in the following, when "^A" is written for A, which is a vector, matrix, or scalar, it is equivalent to "a symbol with "^" written immediately above "A".

[0013] [Embodiment 1] In the first embodiment, an estimation device that estimates signals from virtually arranged virtual microphones for array signal processing using a microphone array will be described.

[0014] The estimation device according to the first embodiment estimates a signal of a virtually placed microphone (virtual microphone) without making any explicit assumptions about the signal. Fig. 1 is a diagram illustrating an example of the estimation device according to the first embodiment.

[0015] The estimation device 10 (estimation unit) is realized, for example, by loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and executing the predetermined program by the CPU. The estimation device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a wired connection or a network, etc.

[0016] As shown in Fig. 1, an estimation device 10 according to the first embodiment includes a neural network (NN) 11. For the sake of simplicity, Fig. 1 illustrates an example in which two channels corresponding to actual microphones that are actually observed are received and one channel corresponding to a virtual microphone is generated.

[0017] NN11 estimates the observed signal (amplitude and phase components) of a virtually placed virtual microphone from the observed signal observed by the input real microphone. The real microphone is a microphone that is actually installed (microphones 1 and 3 in Figure 1). The observed signal r of the real microphone is a mixed acoustic signal observed at the real microphone (solid circle 1 and 3 in Figure 1). The virtual microphone is a microphone (microphone 2 in Figure 1) that is virtually placed at a position different from the position of the real microphone. NN11 estimates and outputs the observed signal ^v of the virtual microphone (dashed circle 2 in Figure 1).

[0018] The NN11 is, for example, a time-domain deep learning model having high phase estimation performance. The NN11 is a NN that operates directly in the time domain without being based on physical assumptions, and can accurately estimate a time-domain signal. The estimation device 10 uses the NN11 to estimate a time-domain signal that is an observation signal of a virtual microphone from a time-domain signal that is an observation signal of an input real microphone. Hereinafter, in the first embodiment, a NN-based virtual microphone signal estimation (NN-VME: Neural Network-based Virtual Microphone Estimator) is proposed, which is a method of estimating an observation signal of a virtual microphone directly from the time domain. Note that the NN11 does not necessarily need to be a time-domain model, and may be realized by a frequency-domain model. The NN11 has an encoder 111, a convolution block 112, and a decoder 113.

[0019] The encoder 111 is a neural network that maps an audio signal to a predetermined feature space, i.e., converts the audio signal into a feature vector. The convolution block 112 is a set of layers for performing one-dimensional convolution, etc. The decoder 113 is a neural network that maps features in a predetermined feature space to the space of audio signals, i.e., converts feature vectors into audio signals. The NN 11 outputs the observed signal converted by the decoder 113 as an estimated signal ^v of the virtual microphone.

[0020] The configurations of the convolution block, the encoder, and the decoder may be the same as those described in Reference 1 (Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation”, IEEE / ACM Trans. ASLP, vol. 27, no. 8, pp. 1256-1266, 2019.). The time-domain acoustic signal may be obtained by the method described in Reference 1. In addition, each feature in the following description is represented by a vector.

[0021] [Estimation process] Next, a case where the NN 11 estimates one or more virtual microphones simultaneously will be described. First, c is the T long-term domain waveform of the cth real microphone, and ^v c´ denotes the estimated signal of the c'th virtual microphone. c=1 ,…,r c=Cr} as input, the NN-VME module NN11 generates a virtual microphone signal ^v={^v c´=1 ,…,^v c´=Cv} is estimated as shown in equation (1).

[0022]

number

[0023] Here, C r denotes the number of observation channels (i.e., real microphones), and C v denotes the number of virtual estimation channels (i.e., virtual microphones), and NN-VME(·) is the neural network.

[0024] [Estimation process procedure] 2 is a flowchart showing a processing procedure of the estimation process according to the embodiment 1. When an observed signal r of a real microphone is input, the estimation device 10 converts the input observed signal r of the real microphone in the time domain into a feature quantity (step S1). The convolution block 112 performs one-dimensional convolution (step S2).

[0025] The decoder 113 converts the feature amount into an observation signal at the position of the virtual microphone (step S3). The NN 11 outputs the observation signal converted by the decoder 113 as an estimated signal ^v of the virtual microphone (step S4).

[0026] [Advantages of the first embodiment] In this way, the estimation device 10 uses a time domain / deep learning model with high phase estimation performance to directly estimate the observation signal of the virtual microphone from the observation signal observed by the input real microphone. In the tenth embodiment, such a data doblind framework makes it possible to directly estimate the signal (amplitude and phase components) of the virtual microphone without making any explicit assumptions about the signal (e.g., a physical model). In addition, the estimation device 10 realizes estimation of both the amplitude and phase of the virtual microphone signal by using a time domain / deep learning model with high phase estimation performance.

[0027] Therefore, according to the first embodiment, it is possible to virtually increase the number of observation microphones, and even when the number of microphones is small, it is possible to improve the performance of the microphone array technology.

[0028] [Embodiment 2] Next, a second embodiment will be described. In the second embodiment, a learning device that performs learning of the NN 11 in the estimation device 10 will be described. In order to make the NN 11, which is an NN-VNE module, estimate the signal of the virtual microphone, the learning device 20 employs supervised learning, and uses, as learning data, the observed signal of the real microphone at the position of the virtual microphone in addition to the observed signal of the real microphone that is actually placed during operation.

[0029] Fig. 3 is a diagram showing a schematic diagram of an example of a learning device according to embodiment 2. Note that the same components as those in embodiment 1 are given the same reference numerals and will not be described. For the sake of simplicity, Fig. 3 shows an example in which the learning device 20 receives two channels corresponding to real microphones and performs learning on an NN 11 that generates one channel corresponding to a virtual microphone.

[0030] 3 is realized by, for example, loading a predetermined program into a computer or the like including a ROM, a RAM, a CPU, etc., and having the CPU execute the predetermined program. The learning device 20 also has a communication interface for transmitting and receiving various information to and from other devices connected via a wired connection or a network, etc. The learning device 20 has an NN 11, an input unit 21, and a parameter update unit 22.

[0031] The input unit 21 receives, as learning data, an observation signal (indicated by a solid circle 1, 3 in FIG. 3) of a real microphone (microphone 1, 3) installed during operation and an observation signal (indicated by a solid circle 2 in FIG. 3) actually observed at the position of a virtually placed virtual microphone (microphone 2) that is the estimation target. The input unit 21 inputs, to the NN, a time domain observation signal r (indicated by a solid circle 1, 3 in FIG. 3) of a real microphone installed during operation. The input unit 21 inputs, to the parameter update unit 22, an observation signal t (indicated by a solid circle 2 in FIG. 1) actually observed at the position of the virtual microphone.

[0032] NN11 (estimation unit) estimates the observed signal ^v (in Figure 3, indicated by the dashed circle 2) of a virtually placed virtual microphone (microphone 2) from the observed signal r observed by the input real microphones (microphones 1 and 3).

[0033] The parameter update unit 22 updates the parameters of the NN 11 so that the observed signal ^v of the virtual microphone estimated by the NN 11 approaches the observed signal t actually observed at the position of the virtual microphone.

[0034] [Learning process] Next, the learning process will be described. The learning device 20 employs supervised learning to make the NN 11, which is an NN-VME module, estimate the virtual microphone signal. Therefore, during learning, the observed signal of the real microphone at the position of the virtual microphone is used as the learning target, along with the observed signal of the real microphone.

[0035] So, assume that a set of input and target signals {r,t} is available, where t={t c´=1 ,...,t c´=Cv}, and t c´ denotes the target signal for the c'th virtual microphone. In Fig. 3, a subset of the microphones (e.g., channels 1 and 3) are assigned as the network input values ​​r, and another subset (e.g., channel 2) is used as the network target values ​​t.

[0036] The NN 11 is trained based on the time-domain loss between the estimated signal and the real signal at the virtual microphone position. The parameter update unit 22 adopts a scale-dependent signal-to-noise ratio (SNR) as the loss, for example, as shown in Equation (2).

[0037]

number

[0038] Here, as explained in equation (1), ̂v = NN-VME(r).

[0039] [Learning process procedure] Next, a description will be given of the learning process according to the embodiment 2. Fig. 4 is a flowchart showing the processing procedure of the learning process according to the embodiment 2.

[0040] 4, as learning data, an observed signal of a real microphone installed during operation and an observed signal actually observed at the position of a virtually placed virtual microphone, which is an estimation target, are input (step S11). The input unit 21 inputs a time domain observed signal r of the real microphone installed during operation to the NN 11 (step S12).

[0041] The NN11 performs the same processing as steps S1 to S4 shown in FIG. 2 to estimate the observed signal ^v of the virtually placed virtual microphone from the input observed signal r observed by the real microphone (steps S13 to S16).

[0042] The parameter update unit 22 updates the parameters of the NN 11 so that the observed signal ^v of the virtual microphone estimated by the NN 11 approaches the observed signal t actually observed at the position of the virtual microphone (step S17). The parameter update unit 22 updates the parameters of the NN 11 so that the loss calculated by equation (2) is optimized.

[0043] Then, the parameter update unit 22 determines whether or not a termination condition has been reached (step S18). If the termination condition has been reached (step S18: Yes), the learning device 20 ends the process, and if the termination condition has not been reached (step S18: No), the process returns to step S12. The termination condition is, for example, that the parameter updates for the NN 11 have reached a predetermined number of times, that the loss value used in the parameter update has become equal to or less than a predetermined threshold, or that the amount of parameter update (such as the derivative value of the loss function value) has become equal to or less than a predetermined threshold.

[0044] [Effects of the second embodiment] Thus, unlike the learning of the speech enhancement method, the learning device 20 according to the second embodiment does not require a pair of noisy and clean signals, but only requires the observed signals of multiple real microphones as the learning data. In other words, since the learning device 20 requires only the observed signals (mixed acoustic signals) containing multi-channel noise as the learning data, there is no restriction on the shape of the device, and the mixed acoustic signals of multiple channels can be used as the learning data. In other words, the learning device 20 can use actual recordings recorded with multiple microphones as they are, rather than simulated recordings, as the learning data.

[0045] Therefore, training data can be easily prepared at low cost in the training device 20. Furthermore, by using a large amount of training data, the training device 20 can build a powerful NN 11, which enables precise modeling of real recordings.

[0046] [Embodiment 3] Since the estimation device 10 enables generation of virtual microphone signals, the signals can be used for various array processing. Therefore, in the third embodiment, a configuration in which the estimation device 10 is combined with a frequency domain beamformer will be described as an example.

[0047] [Signal processing device] Fig. 5 is a diagram illustrating an example of a signal processing device according to the third embodiment. The signal processing device 100 illustrated in Fig. 5 is realized, for example, by loading a predetermined program into a computer or the like including a ROM, a RAM, a CPU, and the like, and causing the CPU to execute the predetermined program. The signal processing device 100 also has a communication interface for transmitting and receiving various information to and from other devices connected via a wired connection or a network, etc. The signal processing device 100 has an estimation device 10, a microphone signal processing unit 30, and an application unit 40 (signal processing unit).

[0048] The microphone signal processing unit 30 generates a speech enhancement signal from which noise components have been removed, based on the observed signal of the real microphone and the observed signal of the virtual microphone estimated by the estimation device 10. Note that the microphone signal processing unit 30 may also include sound source separation processing, sound source localization processing, and the like.

[0049] The application unit 40 performs another task-dependent process using the voice enhancement signal. The application unit 40 performs, for example, voice recognition. Note that the processing order of the signal processing device 100 is an example, and there are cases where voice recognition is performed after sound source separation, and cases where voice enhancement or sound source separation is performed after sound source localization.

[0050] [Speech enhancement processing] [Basic Procedure] First, the estimation device 10 is used to estimate the real microphone signal r ∈ R as described in equation (1). T×Cr Let, ^v∈R be the virtual microphone signal. T×Cv and estimate the extended microphone signal y=[r,^v]∈R T×C (C=C r +C v ) is obtained. Next, the microphone signal processor 30 uses a frequency domain beamformer in addition to the enhanced microphone signal in a frequency domain representation (i.e., a Short-Time Fourier Transform (STFT)) to obtain the enhanced speech signal. Finally, the inverse STFT is used to recover the enhanced time domain waveform.

[0051] STFT area^X t,f The enhanced speech signal in ∈C is ^X t,f =w H f Y t,f where Y t,f ∈C C is a vector containing the C-channel STFT coefficients of the extended microphone signal at time-frequency bin (t,f), and w f ∈C C is a vector containing the beamforming filter coefficients, H represents the conjugate transpose.

[0052] [MVDR format] The microphone signal processing unit 30 uses, for example, Minimum Variance Distortionless Response (MVDR) beamforming (Reference 2: Mehrez Souden, Jacob Benesty, and Sofiene Affes, “On optimal frequency-domain multichannel linear filtering for noise reduction”, IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 2, pp. 260-276, 2009.) to obtain a time-invariant filter coefficient w f is calculated as shown in equation (3).

[0053]

number

[0054] Here, Φ S f ∈C C×C and Φ N f ∈C C×C are the spatial covariance (SC) matrices of the speech signal and the noise signal, respectively. C is a one-hot vector representing the reference microphone.

[0055] Then, the SC matrix is ​​estimated using the time-frequency mask as shown in equation (4) (Reference 3: Jahn Heymann, Lukas Drude, and Reinhold Haeb-Umbach, “Neural network based spectral mask estimation for acoustic beamforming”, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 196-200.).

[0056]

number

[0057] where ν∈{S,N}. m S t,f ∈[0,1] and m N t,f ∈[0,1] are the time-frequency masks of speech and noise, respectively.

[0058] [Virtual microphone loading] Experiments described below reveal that the use of virtual microphones in beamforming, while effective at increasing signal-to-distortion ratios (SDR), does not necessarily improve Automatic Speech Recognition (ASR) performance because the virtual microphone estimation introduces processing artifacts.

[0059] To reduce the effect of this artifact, we use the virtual microphone loading term Z∈R in Eq. (5). C Let us use the SC matrix Φ N f That is, the microphone signal processing unit 30 adds a loading term that reduces the weight of the virtual microphone channel to the spatial covariance matrix of the audio signal and the noise signal.

[0060]

number

[0061] Here, Z={z c,c´} C,C c=1,c´=1 is a matrix with zeros except for the diagonal elements corresponding to the virtual microphones. That is, z cv,cv = 1, and c vdenotes the channel index corresponding to the virtual microphone, and ε is a loading hyperparameter that controls the contribution of the virtual microphone in forming the beamformer. For example, setting a large value for ε means that the virtual microphone is contaminated with large noise that is uncorrelated with other microphones. Therefore, the estimated beamformer can expect to improve the ASR performance by reducing the channel weight of the virtual microphone.

[0062] [Effects of the third embodiment] The virtual microphone signals estimated by the estimation device 10 with the NN-VME module also allow for improved performance of speech enhancement and signal processing enhanced by the NN-VME.

[0063] [experiment] To evaluate NN-VME, we conducted the following two evaluations: Experiment 1, which evaluates the virtual microphone estimation performance by NN-VME, and Experiment 2, which evaluates the enhancement performance by a beamformer using an estimated virtual microphone. Note that in the experiments, we reported the results of estimating one virtual microphone, but it can naturally be expanded to estimate multiple virtual microphones.

[0064] Figure 6 shows the microphone array layout for the CHiME-4 corpus. All microphones in Figure 6 face forward except for microphone 2.

[0065] [Experimental conditions] We evaluated NN-VME on the CHiME-4 corpus (Jon Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe, “The third CHiME speech separation and recognition challenge: Dataset, task and baselines”, in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2015, pp. 504-511. The CHiME-4 corpus contains speech recorded using a tablet device equipped with a 6-channel rectangular microphone array, as shown in Figure 6. This corpus contains not only simulated data but also real recordings in noisy public environments.

[0066] The training set consisted of 3 hours of real speech data from 4 speakers and 15 hours of simulated speech data from 83 speakers. The evaluation set included 1,320 utterances from the real speech data and the simulated speech data containing noise from the 4 speakers. Of these utterances, we used the evaluation set consisting of 1,149 utterances after removing utterances due to microphone malfunctions.

[0067] The evaluation indices used were the SDR and word error rate (WER) of BSSEval (Reference 5: Emmanuel Vincent, Remi Gribonval, and Cedric Fevotte, “Performance measurement in blind audio source separation”, IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 4, pp. 1462-1469, 2006.). To evaluate the virtual microphone estimation performance, we calculated the SDR between the estimated virtual microphone signal in the channel corresponding to the virtual microphone and the observed real microphone signal.

[0068] To evaluate the enhancement performance of the beamformer, we used the clean reverberant signal in the fourth channel as a reference signal. Since access to the clean signal is required, this evaluation is only performed on simulated data.

[0069] To evaluate the ASR performance, we used Kaldi’s CHiME-4 recipe (Reference 6: Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, PetrMotlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, GeorgStemmer, and Karel Vesely, “The Kaldi speech recognition toolkit”, in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2011. Reference 7: [online], [Retrieved January 25, 2021], Internet<https: / / github.com / kaldi-asr / kaldi / tree / master / egs / chime4 / s5_6ch> ) was used.This is based on a deep neural network hidden Markov model hybrid acoustic model (Herve Bourlard and Nelson Morgan, Connectionist speech recognition: A hybrid approach,1994, Ref. 10: Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdelrahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, and Brian Kings bury, “Purely sequence-trained neural networks for ASR based on lattice-free MMI”) trained with the lattice-free maximum mutual information criterion (Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur, “Purely sequence-trained neural networks for ASR based on lattice-free MMI”, in Interspeech, 2016, pp.2751-2755.) “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups”, IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 8297, 2012.) A trigram language model was used for decoding.

[0070] [Experimental configuration] The network configuration of NN-VME uses a Conv-TasNet-based network architecture. Following the description in Reference 1, the hyperparameters were set as N = 256, L = 20, B = 256, H = 512, P = 3, X = 8, and R = 4.

[0071] We trained the NN-VME by employing the Adam algorithm with gradient clipping (Reference 11: Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization”, in International Conference on Learning Representations (ICLR), 2015.). The initial learning rate was set to 0.0001. Training was terminated after 200 epochs.

[0072] The MVDR beamformer is available from the GitHub repository used in Kaldi's CHiME-4 recipe (Reference 12: [online], [searched on January 25, 2021], Internet<URL:https: / / github.com / fgnt / nn-gev,> We used the pretrained mask estimation model provided by the IEEE International Conference on Mask Estimation (ISCM) (see ref. 3). For the STFT computation, we used a Blackman window with a set of length and shift of 64 ms and 16 ms, respectively. For the ASR experiments, the loading hyperparameter ε in Eq. (5) was set to 0.05.

[0073] [Experimental Results] [Evaluation of virtual microphone estimation performance] Table 1 shows the SDR [dB] of the virtual microphone estimation using the noisy observed signal as the reference signal.

[0074] [Table 1]

[0075] In Table 1, RM represents the real microphone, and VM represents the virtual microphone estimated by NN-VME (NN11). Here, the reference signal for calculating the SDR is not a clean signal, but a noisy observed signal of the channel corresponding to the virtual microphone. Therefore, the virtual microphone estimation performance can be evaluated for real recordings as well.

[0076] In Table 1, the first column "eval ch" indicates the channel index of the virtual or real microphone signal used as an estimated signal in the calculation of the SDR. The second column "ref ch" indicates the channel index of the real microphone signal used as a reference signal. Here, the notation "5(4,6)" indicates that the virtual microphone signal in channel 5 was estimated using the real microphone signals in channels 4 and 6. As a reference, the score is compared with the SDR obtained with the closest real microphone (i.e., with the highest SDR). These results are shown in the first row (eval ch4, ref ch5) and the fourth row (eval ch5, ref ch6) of Table 1.

[0077] Table 1 shows that the signal estimated by the NN-VME module (e.g., “5(4,6)”) has a higher SDR score than the observed signal recorded by a nearby microphone (e.g., “4”). These results show that even in real recordings, the NN-VME (NN11) can estimate a virtual microphone signal that is not actually observed at the microphone by utilizing spatial information inferred from the few observed real microphone signals.

[0078] Table 1 shows the results of interpolation (i.e., virtual microphones located between real microphones) (e.g., "5(4,6)") and extrapolation in the horizontal direction (e.g., "6(4,5)"). In both cases, the NN-VME (NN11) can predict virtual microphone signals with small distortion of the time waveform with an SDR of about 12 dB or more.

[0079] [Evaluation of beamformer enhancement performance] Table 2 shows the SDR [dB] of the beamformer using the clean signal as the reference signal. Note that the higher the SDR value, the better, and the lower the WER [%] value, the better the performance.

[0080] [Table 2]

[0081] In Table 2, VM BF indicates a beamformer with an estimated virtual microphone (the output of NN11), and RM BF indicates a beamformer with only real microphones. In Table 2, the columns "real" and "virtual" in "used ch" indicate the channel indexes corresponding to the real and virtual microphones used to form the beamformer, respectively. For example, "VM BF" in row (4) is formed using two real microphone signals (i.e., channels 4 and 6) and one virtual microphone signal (i.e., channel 5).

[0082] Table 2 shows that the VM BF proposed in the first embodiment (e.g., row (4)) has a higher SDR score than the RM BF formed by the same real microphone signal (e.g., row (2)). Here, another RM BF (e.g., row (3)) corresponds to the upper limit performance of the VM BF.

[0083] To evaluate the performance of the beamformer on real recordings, we performed an ASR evaluation in addition to the SDR-based evaluation above. Table 2 also shows the WERs of the RM BF and VM BF evaluated on real data.

[0084] It can also be seen from the table that in real recordings, the VM BF proposed in the first embodiment (e.g., row (4)) reduced the WER by 0.9% compared to the corresponding RM BF (e.g., row (2)). A similar trend was observed when more microphones were used (rows (5)-(7)).

[0085] These results demonstrate that the estimated virtual microphone signals improve the enhancement performance when combined with a beamformer.

[0086] Furthermore, Table 2 shows the results of VM BF with virtual microphone loading. The WER score of VM BF without loading is 15.1% under the same condition as row (4) and 13.4% under the same condition as row (7), which indicates that virtual microphone loading is effective in improving the ASR performance of VM BF.

[0087] Thus, it was shown that the virtual microphone signal estimated by NN-VME (NN11) provides improved performance for speech enhancement and signal processing enhanced by NN-VME.

[0088] [System configuration of the embodiment] Each component of the estimation device 10, the learning device 20, and the signal processing device 100 is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the estimation device 10, the learning device 20, and the signal processing device 100 is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0089] Further, all or any part of the processes performed in the estimation device 10, the learning device 20, and the signal processing device 100 may be realized by a CPU, a GPU (Graphics Processing Unit), and a program analyzed and executed by the CPU and the GPU. Further, each process performed in the estimation device 10, the learning device 20, and the signal processing device 100 may be realized as hardware using wired logic.

[0090] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the process procedures, control procedures, specific names, and information including various data and parameters described above and shown in the drawings can be changed as appropriate unless otherwise specified.

[0091] [program] 7 is a diagram showing an example of a computer in which the estimation device 10, the learning device 20, and the signal processing device 100 are realized by executing a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0092] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0093] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes of the estimation device 10, the learning device 20, and the signal processing device 100 are implemented as program modules 1093 in which codes executable by the computer 1000 are written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to the functional configurations in the estimation device 10, the learning device 20, and the signal processing device 100 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0094] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes the program.

[0095] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0096] Although the embodiment of the invention made by the present inventor has been described above, the present invention is not limited by the description and drawings that form a part of the disclosure of the present invention according to the present embodiment. In other words, other embodiments, examples, operation techniques, etc. made by those skilled in the art based on the present embodiment are all included in the scope of the present invention. [Explanation of symbols]

[0097] 10 Estimation device 11 Neural Networks (NN) 111 Encoder 112 Convolution Blocks 113 Decoder 20 Learning Device 21 Input section 22 Parameter Update Section 30 Microphone signal processing section 40 Application Section 100 Signal processing section

Claims

1. A signal processing device for processing an acoustic signal, An estimation unit that receives an input of a time domain signal corresponding to the amplitude and phase components of an observation signal of a real microphone and outputs a time domain signal corresponding to the amplitude and phase components of an observation signal of a virtually placed virtual microphone. having The estimation unit includes a neural network. The signal processing device is characterized in that one of the neural networks is a deep learning model, and estimates a time domain signal corresponding to the amplitude and phase components of the observation signal of the virtually placed virtual microphone from a time domain signal corresponding to the amplitude and phase components of the observation signal of the real microphone input, and outputs the estimation result.

2. 2. The signal processing device according to claim 1, wherein parameters of the one neural network are updated based on a time-domain signal of an observation signal of a real microphone, which is learning data, and a time-domain signal of an observation signal actually observed at the position of the virtually placed virtual microphone, so that the time-domain signal of the observation signal of the virtual microphone estimated by the neural network from the time-domain signal of the observation signal of the real microphone approaches the time-domain signal of the observation signal actually observed at the position of the virtual microphone.

3. A signal processing method executed by a signal processing device, comprising: receiving an input of a time domain signal corresponding to the amplitude and phase components of an observation signal of a real microphone, and outputting a time domain signal corresponding to the amplitude and phase components of an observation signal of a virtually arranged virtual microphone; Including, The outputting step uses one neural network, The signal processing method is characterized in that one of the neural networks is a deep learning model, and estimates a time domain signal corresponding to the amplitude and phase components of the observation signal of the virtually placed virtual microphone from a time domain signal corresponding to the amplitude and phase components of the observation signal of the real microphone input, and outputs the estimation result.

4. A signal processing program for causing a computer to function as the signal processing device according to claim 1 or 2.

Images