Signal processing device and signal processing method

Abstract

This signal processing device comprises a virtual microphone signal estimation unit, an array processing unit, and an update unit. The virtual microphone signal estimation unit estimates, by using a neural network, a second acoustic signal observed by a virtual microphone on the basis of a first acoustic signal observed by a real microphone. The array processing unit performs estimation processing on the basis of the first acoustic signal and the second acoustic signal. The update unit updates a parameter of the neural network such that the loss function of the second acoustic signal with respect to a reference signal and the loss function of the estimation processing are reduced. Furthermore, the virtual microphone signal estimation unit estimates, on the basis of a third acoustic signal observed by the real microphone, a fourth acoustic signal observed by the virtual microphone, by using the neural network in which the parameter has already been updated.

Description

Signal processing device and signal processing method

[0001] The present invention relates to a signal processing device and a signal processing method.

[0002] Conventionally, a neural network-based virtual microphone estimation framework (NN-VME) is known, which is a technique for generating virtual microphones based on supervised learning at the virtual microphone output level (see, for example, Non-Patent Document 1). In addition, array processing may be performed as a post-stage process of NN-VME.

[0003] Array processing performs voice enhancement, which emphasizes the voice of a specific speaker, and sound source separation, which separates mixed sounds into their respective signal sources, based on acoustic signals recorded by multiple microphones (hereinafter referred to as "mics") installed in a space.

[0004] NN-VME can estimate virtual microphone signals independent of subsequent array processing. Furthermore, unlike conventional tasks such as speech enhancement, the virtual microphone generation task in NN-VME allows model training based solely on observed signals from real microphones.

[0005] Here, the performance of array processing depends on the number of microphones, and with NN-VME, the performance of array processing can be improved by virtually increasing the number of microphones.

[0006] T. Ochiai, et al., “Neural network-based virtual microphone estimator,” in Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 6114-6118. K. Yamaoka, et al., “CNN-based virtual microphone signal estimation for MPDR beamforming in underdetermined situations,” in Proc. European Signal Processing Conference (EUSIPCO), 2019, pp. 1-5.

[0007] However, conventional NN-VMEs have the problem that their performance is not necessarily optimal for a specific array processing stage. For example, in conventional NN-VMEs, the NN is trained independently of the array processing stage. Therefore, the estimation results from the NN-VME do not necessarily contribute to improving the accuracy of the array processing.

[0008] In order to solve the above-mentioned problems and achieve the object, a signal processing device estimates a second acoustic signal observed by a virtual microphone based on a first acoustic signal observed by a real microphone using a neural network, performs estimation processing based on the first acoustic signal and the second acoustic signal, and estimates a fourth acoustic signal observed by a virtual microphone based on a third acoustic signal observed by the real microphone using the neural network that has been trained by processing to update parameters of the neural network so that a weighted sum of a loss function of the second acoustic signal with respect to a reference signal and a loss function of the estimation processing becomes small.

[0009] According to the present invention, the performance of the NN-VME can be improved.

[0010] FIG. 1 is a diagram showing an example of the configuration of a signal processing device according to a first embodiment. FIG. 2 is a flowchart showing the flow of processing by the signal processing device according to the first embodiment. FIG. 3 is a diagram showing experimental results. FIG. 4 is a diagram showing experimental results. FIG. 5 is a diagram showing an example of the configuration of a signal processing device according to a second embodiment. FIG. 6 is a flowchart showing the flow of processing by the signal processing device according to the second embodiment. FIG. 7 is a diagram showing an example of a computer that executes a signal processing program.

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A signal processing device and a signal processing method according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0012] [First embodiment] [Configuration of first embodiment] First, the configuration of a signal processing device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a signal processing device according to the first embodiment.

[0013] The signal processing device 10 performs a learning process for a neural network (NN). That is, as shown in Fig. 1, the signal processing device 10 receives an input of an observed signal r recorded by a real microphone and updates the parameters of the NN. For example, the signal processing device 10 can perform NN learning using a deep learning technique.

[0014] The signal processing device 10 can perform not only learning processing but also inference processing. In the inference processing, the signal processing device 10 calculates an estimated value of the acoustic signal recorded at the position of the virtual microphone (^ just above v) or an estimated value of the acoustic signal (sound source signal) generated from the sound source (x ｉ ＢＦ Output a ^) directly above the x.

[0015] The observed signal (or observed signal) is an acoustic signal recorded by a real microphone (hereinafter referred to as a real microphone observed signal), or an acoustic signal estimated to be recorded by a virtual microphone (hereinafter referred to as a virtual microphone observed signal).

[0016] A real microphone is a microphone that is actually installed. A virtual microphone is a microphone that is virtually installed at a specific location. A virtual microphone observation signal is an acoustic signal that is estimated to be recorded if a microphone were installed at a specific location.

[0017] The following description will be given assuming that the signal processing device 10 performs NN learning processing. Here, I sound source signals are recorded by C microphones. The sound source signals are sound signals emitted from sound sources such as a speaker and a speaker. The C microphones include both real microphones and virtual microphones.

[0018] A microphone is a device that records sound as a signal. ｉ As shown in equation (1), σ is a T-dimensional real vector (i=1, ..., I) representing the time-domain sound source signal of the i-th sound source, where T is the length of the waveform representing the sound (e.g., the number of samples).

[0019]

[0020] The observed signal y of the cth microphone ｃ is a T-dimensional vector as shown in equation (2) and is modeled as shown in equation (3).

[0021]

[0022]

[0023] x ｃ，ｉ is the observed signal (e.g., spatial image) containing reverberation of the i-th sound source in the c-th channel (c=1,...,C). The c-th channel means that it is recorded by the c-th microphone. ｃ is a T-dimensional vector as shown in equation (4) and is an additive noise signal.

[0024]

[0025] 1, the signal processing device 10 has a virtual microphone signal estimation unit 11, an array processing unit 13, and an update unit 14. The virtual microphone signal estimation unit 11, the array processing unit 13, and the update unit 14 are realized by a processor such as a CPU (Central Processing Unit) executing a program. The signal processing device 10 also stores NN-VME parameters 12. The NN-VME parameters 12 are parameters of the NN and include, for example, weights and biases.

[0026] The virtual microphone signal estimation unit 11 estimates the virtual microphone observation signal based on the real microphone observation signal using a NN constructed from the NN-VME parameters 12. The array processing unit 13 estimates the sound source signal by array processing based on the real microphone observation signal and the virtual microphone observation signal. The update unit 14 updates the NN-VME parameters 12 so that both the loss function based on the estimated virtual microphone observation signal and the loss function based on the estimated sound source signal are optimized (minimized).

[0027] The virtual microphone signal estimation unit 11 inputs the real microphone observation signal to a NN constructed based on the NN-VME parameters 12, and estimates the virtual microphone observation signal. For example, the virtual microphone signal estimation unit 11 estimates a time-domain waveform as the virtual microphone observation signal. The virtual microphone signal estimation unit 11 may also estimate the amplitude and phase at each time as the virtual microphone observation signal.

[0028] The actual microphone observation signal is as shown in equations (5) and (6). ｒ is the number of real microphones. ｃ is the observed signal recorded by the cth real microphone.

[0029]

[0030]

[0031] The virtual microphone observation signals are as shown in equations (7) and (8). ｖ is the number of virtual microphones. ｃ´ (The left side of equation (8)) is the observed signal (estimated value) recorded by the c'th virtual microphone.

[0032]

[0033]

[0034] If NN is a function NNVME(·), the virtual microphone signal estimation unit 11 is expressed as in equation (9).

[0035]

[0036] The array processing unit 13 combines the virtual microphone observation signals and the real microphone observation signals to obtain an extended observation signal (y with an overline), as shown in equation (10). Also, C is the number of real microphones, C, as shown in equation (11). ｒ and the number of virtual microphones C ｖ is the sum of

[0037]

[0038]

[0039] The array processing unit 13 performs array processing on the extended observation signals. In this case, the number of observation signals increases compared to when array processing is performed only on the real microphone observation signals, and it is therefore expected that the performance of the array processing by the array processing unit 13 will improve.

[0040] Examples of array processing include speaker enhancement, sound source separation, and sound source localization. Thus, the array processing may be signal prediction or may be something other than signal prediction. In this embodiment, the array processing unit 13 performs mask-based frequency-domain beamforming (BF) as the array processing. Specifically, the array processing unit 13 estimates a specific sound source signal that has been enhanced using a BF designed based on a time-frequency mask, based on the observed signal. The array processing unit 13 can use the BF method described in Reference 1 or 2.

[0041] Reference 1: J. Heymann, L. Drude, and R. Haeb-Umbach, “Neural network based spectral mask estimation for acoustic beamforming,” in Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 196-200.

[0042] Reference 2: H. Erdogan, J. R. Hershey, S. Watanabe, M. I. Mandel, and J. Le Roux, “Improved MVDR beamforming using single-channel mask prediction networks.” in Proc. Interspeech, 2016, pp. 1981-1985.

[0043] The array processing unit 13 estimates the sound source signal of the i-th sound source as shown in equation (12). MaskBF(·) is a function that calculates the sound source signal of the i-th sound source using the mask-based BF.

[0044]

[0045] Next, the update unit 14 calculates the virtual microphone level loss function and the array processing level loss function.

[0046] The update unit 14 calculates a virtual microphone level loss function based on the estimated virtual microphone observation signal and the reference observation signal, and also calculates an array processing level loss function based on the estimated sound source signal and the reference sound source signal.

[0047] The update unit 14 calculates each loss function using an SNR loss that applies the signal-to-noise ratio (SNR). Generally, a reference signal (teaching data) is expressed as z ｒｅｆ (equation (13)), and the estimated signal is z ｅｓｔIf (equation (14)), the SNR loss is calculated by the function SNR_Loss(·) of equation (15). Note that the updating unit 14 is not limited to the SNR loss, but may also calculate, for example, z ｒｅｆ and Z ｅｓｔ Alternatively, the L1 distance and the L2 distance (Euclidean distance) between the two may be calculated as the loss function.

[0048]

[0049]

[0050]

[0051] [Virtual microphone level loss function] A method for calculating the virtual microphone level loss function will be described. Here, the reference signal in this embodiment may be an acoustic signal recorded by a microphone (hereinafter referred to as a reference microphone) that is actually placed at a position where the virtual microphone is assumed to be installed.

[0052] The number of reference microphones is the same as the number of virtual microphones, C ｖ The number of actual microphones is C ｒ In this case, it follows from equation (11) that C microphones actually exist, and C acoustic signals are observed as shown in equation (16).

[0053]

[0054] Of the C observed signals, C recorded by the reference microphone ｖ The acoustic signals v (Equation (17)) are extracted as reference observation signals. Of the C observation signals, C ｒ The observed signals correspond to the observed signal r input to the signal processing device 10 .

[0055]

[0056] The update unit 14 calculates the virtual microphone level loss function L as shown in Equation (18) based on the virtual microphone observation signal (Equation (8)) and the reference observation signal. ＶＭ SNR_Loss(·) is as explained in equation (15).

[0057]

[0058] [Array Processing Level Loss Function] Next, a method for calculating the array processing level loss function will be described. The array processing level loss function becomes smaller as the estimated sound source signal of the i-th sound source (Equation (12)) approaches the reference sound source signal.

[0059] It is assumed that a combination of an input observed sound source signal y and a reference sound source signal x is prepared in advance as training data. The reference sound source signal x is expressed as in equation (19).

[0060]

[0061] For example, the reference sound source signal x is ｒｅｆ are observed signals including reverberation corresponding to each sound source (speaker) (i=1, . . . , I) in

[0062] The update unit 14 calculates the array processing level loss function L as shown in equation (20) based on the estimated sound source signal (equation (12)) and the reference sound source signal. ＢＦ SNR_Loss(·) is as explained in equation (15).

[0063]

[0064] Array processing level loss function L ＢＦ In order to handle multiple sound sources, the PIT scheme (see Reference 3) is applied because the sound source identified by i in the estimated sound source signal is not necessarily the same as the sound source identified by i in the reference sound source signal.

[0065] Reference 3: M. Kolbeak, D. Yu, Z.-H. Tan, and J. Jensen, “Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 10, pp. 1901-1913, 2017.

[0066] perm(I) includes all possible permutations of I elements. For example, if I=3, perm(I) includes six possible permutations: (1,2,3), (1,3,2), (2,1,3), (2,3,1), (3,1,2), and (3,2,1). p ｉ is the i-th element in each permutation. For example, if the permutation is (3, 1, 2), then p １ = 3, p ２ = 1, p ３ =2.

[0067] The array processing level loss function L of equation (20) ＢＦ is the smallest sum of SNR_Loss for each permutation included in perm(I). Note that SNR_Loss(·) is as explained in equation (15).

[0068] Since the sound source signal is reconstructed by array processing, the array processing level loss function L ＢＦ can be referred to as the reconstruction error. By applying such a reconstruction error to all sound sources, the update unit 14 can impose a constraint that the virtual observation signal estimated by the NN-VME retains the spectral and spatial information of all sound sources.

[0069] Furthermore, the update unit 14 calculates a multitasking loss function as shown in equation (21) using the virtual microphone level loss function and the array processing level loss function.

[0070]

[0071] α is a hyperparameter satisfying 0≦α≦1, and adjusts the trade-off relationship between the virtual microphone-level loss function and the array processing-level loss function. The multitasking loss function can be considered as a weighted sum of the virtual microphone-level loss function and the array processing-level loss function.

[0072] The update unit 14 updates the NN-VME parameters 12, i.e., the parameters of the NN used in the virtual microphone signal estimation unit 11, so that the multitasking loss function is optimized (minimized). The update unit 14 can update the parameters by a known method such as backpropagation.

[0073] [Processing of First Embodiment] Fig. 2 is a flowchart showing the flow of processing of the signal processing device according to the first embodiment. As shown in Fig. 2, first, the signal processing device 10 receives input of a real microphone observation signal (step S101).

[0074] Next, the signal processing device 10 constructs a neural network based on the parameters (step S102).The signal processing device 10 then estimates the virtual microphone observation signal using the neural network based on the real microphone observation signal (equation (9)) (step S103).

[0075] Next, the signal processing device 10 calculates the virtual microphone level loss function (Equation (18)) based on the virtual microphone virtual signal and the reference observed signal (step S104).

[0076] Furthermore, the signal processing device 10 estimates the sound source signal based on the real microphone observation signal and the virtual microphone observation signal (Equation (12)) (step S105).

[0077] Here, the signal processing device 10 calculates the array processing level loss function (Equation (20)) based on the estimated sound source signal and the reference sound source signal (step S106).

[0078] Then, the signal processing device 10 updates the parameters so that the multitasking loss function (Equation (21)) that combines the virtual microphone level loss function and the array processing level loss function is optimized (step S107).

[0079] If the termination condition is not satisfied (step S108, No), the signal processing device 10 returns to step S102 and repeats the process. On the other hand, if the termination condition is satisfied (step S108, Yes), the signal processing device 10 proceeds to step S109. The termination condition is that the parameter updates have been performed for a specified number of epochs, the parameter update amounts have converged, etc.

[0080] The process from step S109 onwards is an inference process. The inference process may be performed at a different timing from the learning process before step S108. First, the signal processing device 10 receives input of an observed signal for inference (step S109).

[0081] Next, the signal processing device 10 estimates the virtual microphone observation signals using a neural network based on the observation signals for inference (step S110). The parameters of the neural network used here have been updated at least once in step S107. The signal processing device 10 then outputs the estimated virtual microphone observation signals (step S111).

[0082] [Effects of the First Embodiment] As described above, the virtual microphone signal estimation unit 11 estimates a second acoustic signal observed by a virtual microphone using a neural network based on a first acoustic signal observed by a real microphone (actual microphone) (Equation (9)). The array processing unit 13 performs array processing (e.g., estimation processing to estimate an acoustic signal generated from a sound source) based on the first acoustic signal and the second acoustic signal (Equation (12)). The update unit 14 updates the parameters of the neural network so that the loss function (Equation (18)) for the reference signal of the second acoustic signal and the loss function (Equation (20)) of the array processing by the array processing unit 13 are reduced. For example, the update unit 14 updates the parameters of the neural network so that the weighted sum (Equation (21)) of the loss function for the reference signal of the second acoustic signal and the loss function of the array processing is reduced.

[0083] Furthermore, the virtual microphone signal estimation unit 11 estimates a fourth acoustic signal observed by a virtual microphone based on a third acoustic signal observed by a real microphone (actual microphone) using a neural network that has been trained (with updated parameters) by the above-mentioned processing (Equation (9)). Furthermore, the third acoustic signal is an acoustic signal for inference, and the corresponding sound source signal may be unknown.

[0084] Furthermore, the inference process of the signal processing device 10 may be rephrased as follows: That is, the signal processing device 10 estimates a second acoustic signal observed by a virtual microphone based on a first acoustic signal observed by a real microphone using a neural network, performs array processing based on the first acoustic signal and the second acoustic signal, and estimates a fourth acoustic signal observed by the virtual microphone based on a third acoustic signal observed by the real microphone using a trained neural network that has been trained by processing to update the parameters of the neural network so as to reduce the weighted sum of a loss function for the reference signal of the second acoustic signal and a loss function for the array processing.

[0085] In this way, by using the multitasking loss function, the signal processing device 10 can prevent overlearning compared to when only the array processing level loss function is used. Furthermore, by using the multitasking loss function, the signal processing device 10 can improve the performance of array processing compared to when only the virtual microphone level loss function is used. In other words, by using the multitasking loss function, the signal processing device 10 can improve the performance of the NN-VME so as to adapt to the expected array processing.

[0086] On the other hand, even if the signal processing device 10 uses a multitasking loss function, the function of the NN-VME alone, that is, the estimation of virtual microphone observation signals, is not degraded.

[0087] Furthermore, by using a weighted sum, the signal processing device 10 can optimize the balance of the influence of each loss function.

[0088] In array processing, the array processing unit 13 can estimate the acoustic signal generated from the sound source based on the first acoustic signal and the second acoustic signal using a mask-based frequency domain BF. The following experiment has shown that the use of the BF has the effect of improving the performance of the NN-VME.

[0089] [Experiments on the First Embodiment] An experiment conducted by the inventor to confirm the effects of the first embodiment will be described. For training and evaluation of the NN in the experiment, mixed speech signals of three speakers including reverberation and background noise were used (I=3).

[0090] The mixed speech signal used was the source signal included in the Wall Street Journal corpus (see Reference 4), and the noise signal used was the noise signal included in the CHiME-3 corpus (see Reference 5).

[0091] Reference 4: D. B. Paul and J. Baker, “The design for the Wall Street Journalbased CSR corpus,” in Speech and Natural Language: Proceedings of a Workshop, 1992, pp. 357-362.

[0092] Reference 5: J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third 'CHiME' speech separation and recognition challenge: Dataset, task and baselines,” in Proc. Automatic Speech Recognition and Understanding Workshop (ASRU), 2015, pp. 504-511.

[0093] The number of microphones (number of channels) constituting the array is three (C=3). One of the three microphones is a reference microphone that is actually placed at the position where the virtual microphone is assumed to be installed. The configuration of the NN-VME is the same as that described in Non-Patent Document 1.

[0094] In the experiment, performance evaluation was performed for a number of methods including the first embodiment. The SDR (signal-to-distortion ratio) defined by equation (22) was used as an evaluation index for the estimation accuracy of the virtual microphone observation signal and the estimation accuracy of the sound source signal by array processing. ｅｓｔ is the estimated signal. ｔｇｔ is the reference signal z ｒｅｆ Z ｅｓｔ In addition, z ｔｇｔ , z ｒｅｆ , z ｅｓｔ are all T-dimensional vectors.

[0095]

[0096] The evaluation index for the estimation accuracy of the virtual microphone observation signal is as shown in equation (23). The evaluation index for the estimation accuracy of the sound source signal by array processing is as shown in equation (24). Note that the better the performance (estimation accuracy), the larger the SDR.

[0097]

[0098]

[0099] Furthermore, the word error rate (WER) when the estimated sound source signal was input to the speech recognition model was used as an evaluation index. Note that a smaller WER indicates that the sound source signal obtained by array processing is such that the accuracy of the speech recognition model is higher.

[0100] 3 and 4 are diagrams showing the results of the experiment. In FIG. 3, the SDR is changed when the value of 1-α is changed in the first embodiment. ＶＭ , S.D. ＲＢF and WER are shown. Note that, from equation (21), 1-α being 0 is equivalent to performing learning using only the array processing level loss function. Also, 1-α being 1 (α = 0) is equivalent to performing learning using only the virtual microphone level loss function.

[0101] 3, when 1-α is 0 or 1, the overall performance is lower than when 1-α is 0.3, for example. This means that the performance of array processing is improved by using a multitasking loss function as in the first embodiment.

[0102] FIG. 4 also shows the evaluation index values ​​for each method and for each value of 1-α. (1) in FIG. 4 shows the results when the real microphone observation signal is input directly to the speech recognition model. (2) and (3) in FIG. 4 show the results when two real microphones and three real microphones are used, respectively, and no virtual microphone is used. (4), (5), (6), and (7) in FIG. 4 all show the results when two real microphones and one virtual microphone are used in the first embodiment, but the values ​​of 1-α are different for each.

[0103] (2) can be said to be a method in which the virtual microphone is removed from the first embodiment. Also, (3) can be said to be a method in which the virtual microphone of the first embodiment is replaced with a real microphone. Therefore, the performance of the first embodiment ((4), (5), (6), and (7)) is considered to be higher than that of (2) but lower than that of (3).

[0104] As shown in FIG. 4, performance is improved when the value of 1-α is 0.1 or 0.3 ((6) or (7)) compared to when the value of 1-α is 0 or 1 ((4) or (5)).

[0105] [Second Embodiment] [Configuration of Second Embodiment] In the first embodiment, a multitasking loss function is used that combines a virtual microphone level loss function and an array processing level loss function corresponding to one array process. On the other hand, the multitasking loss function may be a combination of a virtual microphone level loss function and an array processing level loss function corresponding to each of a plurality of array processes.

[0106] In the second embodiment, the signal processing device updates the parameters of the NN-VME using a multitasking loss function that combines a virtual microphone level loss function and an array processing level loss function corresponding to each of a plurality of array processes.

[0107] Furthermore, array processing is processing for estimating some estimated values, such as a sound source signal and a sound source direction, based on a real microphone observation signal and an estimated virtual microphone observation signal. The array processing may be signal prediction such as mask-based frequency domain BF, or processing other than signal prediction. For example, the array processing is speech enhancement using a neural network, blind source separation, or sound source localization.

[0108] The following mainly describes the differences between the first and second embodiments. Fig. 5 is a diagram showing an example of the configuration of a signal processing device according to the second embodiment. As shown in Fig. 5, the signal processing device 10a has multiple array processing units (a first array processing unit 13_1, a second array processing unit 13_2, and a third array processing unit 13_3).

[0109] The first array processing unit 13_1, the second array processing unit 13_2, and the third array processing unit 13_3 perform array processing using different methods. Any of the first array processing unit 13_1, the second array processing unit 13_2, and the third array processing unit 13_3 may perform mask-based frequency domain BF.

[0110] The first array processing unit 13_1 performs array processing based on the input real microphone observation signals and the virtual microphone observation signals estimated by the virtual microphone signal estimation unit 11. The update unit 14a updates the array processing level loss function L based on the estimated values ​​of the array processing by the first array processing unit 13_1 and the reference information. ＳＣ１ Calculate.

[0111] The reference information is information that serves as training data for the estimated value of the array processing. The reference information may be the reference sound source signal described in the first embodiment, or may be information other than the reference sound source signal. In addition, in FIG. 5, the loss function L ＳＣ１ , L ＳＣ２ , L ＳＣ２Although the reference information for calculating each of the loss functions is shown as being common, the reference signals for calculating each loss function may be different from each other.

[0112] The second array processing unit 13_2 performs array processing based on the input real microphone observation signals and the virtual microphone observation signals estimated by the virtual microphone signal estimation unit 11. Furthermore, the update unit 14a updates the array processing level loss function L based on the estimated values ​​of the array processing by the second array processing unit 13_2 and the reference information. ＳＣ２ Calculate.

[0113] The third array processing unit 13_3 performs array processing based on the input real microphone observation signals and the virtual microphone observation signals estimated by the virtual microphone signal estimation unit 11. Furthermore, the update unit 14a updates the array processing level loss function L based on the estimated values ​​of the array processing by the third array processing unit 13_3 and the reference information. ＳＣ３ Calculate.

[0114] The array processing level loss function may be the SNR loss described in the first embodiment, or may be something other than SNR loss (for example, L1 distance and L2 distance (Euclidean distance)). Furthermore, the number of array processing units is not limited to that described here. The number of array processing units may be two, or may be four or more.

[0115] The update unit 14a calculates the multitasking loss function as shown in equation (25) based on the virtual microphone level loss function and the array processing level loss function.

[0116]

[0117] However, the second embodiment differs from the first embodiment in the method of setting hyperparameters that adjust the trade-off relationship between the virtual microphone level loss function and the array processing level loss function. 0≦α, β, γ≦1 and α+β+γ≦1 are satisfied. For example, α=β=γ=0.2 (1−α−β−γ=0.4). The multitasking loss function can be considered as a weighted sum of the virtual microphone level loss function and K array processing level loss functions.

[0118] [Processing of Second Embodiment] Fig. 6 is a flowchart showing the flow of processing of a signal processing device according to Embodiment 2. As shown in Fig. 6, first, the signal processing device 10a receives input of a real microphone observation signal (step S201).

[0119] Next, the signal processing device 10a constructs a neural network based on the parameters (step S202).The signal processing device 10a then estimates a virtual microphone observation signal using the neural network based on the real microphone observation signal (equation (9)) (step S203).

[0120] Next, the signal processing device 10a calculates a virtual microphone level loss function (Equation (18)) based on the virtual microphone virtual signal and the reference observed signal (step S204).

[0121] Here, the signal processing device 10a sets k to 1 (step S205), where k is a number for distinguishing K array processes (k=1, . . . , K). In the example of FIG. 2, K=3.

[0122] The signal processing device 10a estimates the sound source signal by the k-th array processing based on the real microphone observation signal and the virtual microphone observation signal (Equation (12)) (step S206).

[0123] The signal processing device 10a calculates the array processing level loss function of the k-th array processing based on the estimated sound source signal and the reference sound source signal (step S207).

[0124] If k is less than K (step S208, No), the signal processing device 10a increments k by 1 (step S209), returns to step S206, and repeats the process. On the other hand, if k is equal to or greater than K (step S208, Yes), the signal processing device 10a proceeds to step S210.

[0125] Then, the signal processing device 10a updates the parameters so that the multitasking loss function (Equation (25)) that combines the virtual microphone level loss function and the K array processing level loss functions is optimized (step S210).

[0126] If the termination condition is not satisfied (step S211, No), the signal processing device 10a returns to step S202 and repeats the process. On the other hand, if the termination condition is satisfied (step S211, Yes), the signal processing device 10a proceeds to step S212. The termination condition is that the parameter update has been performed for a specified number of epochs, the parameter update amount has converged, etc.

[0127] The process from step S212 onwards is an inference process. The inference process may be performed at a timing different from the learning process before step S211. First, the signal processing device 10a receives input of an observed signal for inference (step S212).

[0128] Next, the signal processing device 10a estimates the virtual microphone observation signals using a neural network (NN) based on the observation signals for inference (step S213). The parameters of the NN used here have been updated at least once in step S209. The signal processing device 10a then outputs the estimated virtual microphone observation signals (step S214).

[0129] [Effects of the Second Embodiment] The first array processing unit 13_1, the second array processing unit 13_2, and the third array processing unit 13_3 are collectively referred to as the array processing unit 13a. The array processing unit 13a performs array processing using each of a plurality of methods based on the first acoustic signal and the second acoustic signal. In the parameter updating process, the update unit 14a updates the neural network parameters so that the loss function for the reference signal of the second acoustic signal and the loss functions of the plurality of array processing operations are each reduced. For example, the update unit 14a updates the neural network parameters so that the weighted sum (Equation (25)) of the loss function for the reference signal of the second acoustic signal and the loss functions of the plurality of array processing operations is reduced.

[0130] In the second embodiment, it is possible to prevent the NN-VME from estimating virtual microphone observation signals that depend on a single array process. As a result, the versatility of the NN-VME is improved in the second embodiment compared to the first embodiment.

[0131] [System Configuration, etc.] The components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. The program may be executed not only by the CPU but also by other processors such as a GPU.

[0132] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0133] [Program] In one embodiment, the signal processing device can be implemented by installing a program that executes the above-described processing as package software or online software on a desired computer. For example, by executing the above-described program on an information processing device, the information processing device can function as the signal processing device of the embodiment. The information processing device referred to here includes desktop and notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

[0134] The signal processing device may also be implemented as a server device that provides services related to the above processing to a client terminal device used by a user. For example, the server device may be implemented as a server device that receives an observed signal from a real microphone as input and outputs trained neural network parameters. In this case, the server device may be implemented as a web server or as a cloud that provides services related to the above processing through outsourcing.

[0135] 7 is a diagram showing an example of a computer that executes a signal processing program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0136] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0137] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the signal processing device of the embodiment is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, the program module 1093 for executing processes similar to those of the functional configuration of the signal processing device is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0138] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0139] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0140] 10, 10a Signal processing device 11 Virtual microphone signal estimation unit 12 NN-VME parameters 13 Array processing unit 13_1 First array processing unit 13_2 Second array processing unit 13_3 Third array processing unit 14, 14a Update unit

Claims

1. A signal processing device comprising: an estimation unit that estimates a second acoustic signal observed by a virtual microphone based on a first acoustic signal observed by a real microphone using a neural network; performing an estimation process based on the first acoustic signal and the second acoustic signal; and estimating a fourth acoustic signal observed by a virtual microphone based on a third acoustic signal observed by a real microphone using the neural network that has been trained by a process of updating parameters of the neural network so that a weighted sum of a loss function of the second acoustic signal with respect to a reference signal and a loss function of the estimation process is small.

2. The signal processing device according to claim 1, characterized in that the estimation process estimates the acoustic signal generated from the sound source based on the first acoustic signal and the second acoustic signal using each of a plurality of methods, and the parameter updating process updates the parameters of the neural network so as to reduce a weighted sum of a loss function for a reference signal of the second acoustic signal and a loss function for each reference signal of the plurality of acoustic signals estimated by the estimation process.

3. The signal processing device according to claim 1 or 2, characterized in that the estimation process estimates the acoustic signal generated from the sound source based on the first acoustic signal and the second acoustic signal using a mask-based frequency domain BF.

4. A signal processing method executed by a signal processing device, comprising: estimating a second acoustic signal observed by a virtual microphone based on a first acoustic signal observed by a real microphone using a neural network; performing an estimation process based on the first acoustic signal and the second acoustic signal; and estimating a fourth acoustic signal observed by a virtual microphone based on a third acoustic signal observed by a real microphone using the neural network that has been trained by a process of updating parameters of the neural network so that a weighted sum of a loss function of the second acoustic signal with respect to a reference signal and a loss function of the estimation process is small.

Images