Signal analysis device, signal analysis method, and program
The EM-based optimization algorithm for the GMVAE method in sound source separation reduces computational complexity, enabling faster and efficient sound source separation under underdetermined conditions, matching or exceeding the performance of existing MM-based methods.
Patent Information
- Application Number
- JP2022028027
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-02-25
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2042-02-25
AI Technical Summary
The existing GMVAE method for sound source separation under underdetermined conditions requires significant computational resources, limiting its efficiency in processing speed.
A signal analysis device and method utilizing an EM-based parameter optimization algorithm for the GMVAE method, which reduces computational complexity by employing an E-step and M-step iterative process to maximize the conditional expectation value of the log-likelihood function, combined with a multi-channel Wiener filter for sound source separation.
The proposed method significantly reduces the computational load for sound source separation, achieving comparable performance to MM-based methods while enhancing processing speed, particularly when using deep neural networks like ACVAE models.
Smart Images

Figure 0007583290000038 
Figure 0007583290000039 
Figure 0007583290000040
Abstract
Description
[Technical field]
[0001] The present invention relates to a signal analysis device, a signal analysis method, and a program. [Background technology]
[0002] Source separation is a technique for separating individual source signals contained in a mixed signal observed by a microphone array. Source separation contributes greatly to improving the performance of other tasks such as speech recognition and speaker diarization. As one of the source separation techniques, the generalized multichannel variational autoencoder (GMVAE) method has been proposed (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] S. Seki, H. Kameoka, L. Li, T. Toda and K. Takeda, "Underdetermined Source Separation Based on Generalized Multichannel Variational Autoencoder", IEEE Access, vol. 7, pp. 168104-168115, 2019. Summary of the Invention [Problem to be solved by the invention]
[0004] The GMVAE method described above is capable of source separation even under underdetermined conditions, where the number of sound sources is greater than the number of microphones. In order to perform source separation more quickly, it is necessary to reduce the amount of calculations.
[0005] In view of the above circumstances, an object of the present invention is to provide a signal analysis device, a signal analysis method, and a program that can reduce the amount of calculation required for sound source separation. [Means for solving the problem]
[0006] One aspect of the present invention is a signal analysis device that estimates a multi-channel Wiener filter for separating sound source signals representing the respective sounds of a plurality of sound sources from an observation signal representing multiple sounds observed at each of a plurality of positions, the signal analysis device estimating a multi-channel Wiener filter for separating sound source signals representing the respective sounds of a plurality of sound sources from an observation signal representing multiple sounds observed at each of a plurality of positions, the signal analysis device comprising: an encoder that inputs a feature amount of a sound source signal and sound source label information that identifies a sound source, and outputs an average and variance of a latent representation in which the feature amount of the sound source signal is reduced in dimension; and a decoder that inputs a latent representation based on the average and the variance output by the encoder and the sound source label information, and outputs a variance representing the feature amount of the sound source signal input to the encoder, and a conditional expectation value L of a logarithmic likelihood function for complete data is calculated based on an output obtained by performing a process of inputting a latent representation of an initial value of the sound source signal and sound source label information to the decoder of a sound source model for each sound source of multiple sounds represented by an observation signal for learning. EM and a step E of calculating the conditional expectation L EM and a learning unit that performs an M step of updating the latent representation and the sound source label information of each of the sound sources using a filter calculation process to calculate the multi-channel Wiener filter for each of the sound sources based on the output from the decoder obtained in the E step, and repeats the E step, the M step, and the filter calculation process using the updated latent representation and the sound source label information as input to the decoder.
[0007] One aspect of the present invention is a signal analysis method for estimating a multi-channel Wiener filter for separating sound source signals representing the respective sounds of a plurality of sound sources from an observation signal representing multiple sounds observed at each of a plurality of positions, the method comprising: an encoder for inputting a feature amount of a sound source signal and sound source label information for identifying a sound source, and outputting an average and variance of a latent representation in which the feature amount of the sound source signal is reduced in dimension; and a decoder for inputting a latent representation based on the average and the variance output by the encoder and the sound source label information, and outputting a variance representing the feature amount of the sound source signal input to the encoder; and a signal analysis method for estimating a conditional expectation value L of a logarithmic likelihood function for complete data based on an output obtained by performing a process of inputting a latent representation of an initial value of the sound source signal and sound source label information to the decoder of a sound source model for each sound source of multiple sounds represented by an observation signal for learning. EM and a step E of calculating the conditional expectation L EM an M step of updating the latent representation and the sound source label information of each of the sound sources using the latent representation and the sound source label information of each of the sound sources, an updating process of performing a filter calculation process of calculating the multi-channel Wiener filter of each of the sound sources based on the output from the decoder obtained in the E step, and a repetitive process of repeating the E step, the M step, and the filter calculation process using the updated latent representation and the sound source label information as an input to the decoder.
[0008] One aspect of the present invention is a program for causing a computer to function as the signal analysis device described above. Effect of the Invention
[0009] According to the present invention, it is possible to reduce the amount of calculation required for sound source separation. [Brief description of the drawings]
[0010] [Figure 1] 1 is a functional block diagram showing a configuration of a signal analyzing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a signal analyzing device according to the embodiment. [Diagram 3] FIG. 2 is a diagram showing a GMVAE model according to the embodiment. [Figure 4] FIG. 13 is a diagram showing a DNN used in the encoder of the GMVAE model according to the embodiment. [Diagram 5] FIG. 13 is a diagram showing a DNN used in a decoder of the GMVAE model according to the embodiment. [Figure 6] FIG. 2 is a diagram showing an ACVAE model according to the embodiment. [Figure 7] FIG. 1 is a diagram showing a DNN used in an encoder of an ACVAE model according to the embodiment. [Figure 8] FIG. 1 is a diagram showing a DNN used in a decoder of the ACVAE model according to the embodiment. [Figure 9] FIG. 2 is a diagram showing a DNN used in a classifier of an ACVAE model according to the embodiment. [Figure 10] FIG. 4 is a flowchart showing a learning process of the signal analyzing device according to the embodiment. [Figure 11] FIG. 4 is a flowchart showing a sound source separation process of the signal analysis device according to the embodiment. [Figure 12] 11A to 11C are diagrams illustrating experimental results using the signal analysis device according to the embodiment. [Figure 13] 11A to 11C are diagrams illustrating experimental results using the signal analysis device according to the embodiment. [Figure 14] 11A to 11C are diagrams illustrating experimental results using the signal analysis device according to the embodiment. [Figure 15] 11A to 11C are diagrams illustrating experimental results using the signal analysis device according to the embodiment. [Figure 16] 11A to 11C are diagrams illustrating experimental results using the signal analysis device according to the embodiment. [Figure 17] 11A to 11C are diagrams illustrating experimental results using the signal analysis device according to the embodiment. [Figure 18] 11A to 11C are diagrams illustrating experimental results using the signal analysis device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The present embodiment relates to a signal processing technique for decomposing an acoustic signal.
[0012] [1. Overview] In sound source separation, the problem is generally formulated using a frequency domain representation. In this case, various modeling methods are used for the sound source signal and the spatial transfer characteristics between the sound source and the microphone. Independent component analysis (ICA) has been proposed as one of the effective approaches to modeling. In ICA, a separation process called a separation system in which the sound source signal is obtained from the observation signal is assumed. On the other hand, there is a method that assumes a generation process called a mixture system in which the observation signal is obtained from the sound source signal, which is the opposite of the separation system. One of these methods is full-rank spatial covariance analysis (FCA) (see, for example, Reference 1). By assuming a mixture system with more relaxed constraints compared to the separation system, FCA is capable of source separation under underdetermined conditions where the number of sound sources is greater than the number of microphones. In both ICA and FCA, an independent sound source separation problem is solved for each frequency. Therefore, it is necessary to solve permutation matching, which associates the separated signals obtained at each frequency with which sound source they belong.
[0013] For the permutation matching problem, a method has been proposed that incorporates the tendency that the magnitude of frequency components originating from the same sound source tends to change synchronously over time into the modeling of the sound source. To date, a more generalized modeling method of sound sources called non-negative matrix factorization (NMF) has been used as this method (see, for example, Reference 2). NMF is a method that has been shown to be effective in application to automatic music transcription tasks. In NMF, the power spectrum of a mixed signal observed at each time is approximated as a cone combination of a small number of spectral templates. The NMF framework has been applied to modeling each sound source power spectrogram in ICA and FCA, and methods called independent low-rank matrix analysis (ILRMA) and multichannel NMF (MNMF), respectively, have been proposed (see, for example, References 3 and 4). By using spectral templates as acoustic cues, ILRMA and MNMF can simultaneously solve the sound source separation problem for each frequency and the permutation matching problem. On the other hand, there is a problem that separation performance is limited for sound source signals whose power spectrograms are difficult to model using NMF.
[0014] In recent years, generative approaches have been proposed that use deep neural networks (DNNs) to model sound sources more flexibly than NMF. One of these approaches is a method using a variational autoencoder (VAE), which has attracted attention. Assuming a separation system similar to ICA and ILRMA, a method using a conditional VAE (CVAE) called a multichannel variational autoencoder (MVAE) method as a sound source model has been proposed for the sound source separation problem under overdetermined conditions in which the number of sound sources is equal to the number of microphones (see, for example, Reference 5). In the MVAE method, a sound source model using CVAE has a higher expressive ability than a sound source model using NMF, and each sound source can be discriminated more accurately, so that high separation performance has been demonstrated. To date, a generalized MVAE (GMVAE) method has also been proposed that extends the MVAE method to sound source separation under underdetermined conditions similar to FCA and MNMF (see, for example, Non-Patent Document 1).
[0015] In the GMVAE method described in Non-Patent Document 1, a parameter optimization algorithm with guaranteed convergence is derived using an iterative algorithm called the majorization-minimization / minorization-maximization (MM) method, similar to the MVAE method. On the other hand, in FCA and MNMF, it is known that an optimization algorithm with guaranteed convergence, similar to the MM method, can be designed by using an iterative algorithm called the expectation-maximization (EM) method in addition to the MM method. Up to now, in FCA and MNMF, a comparison of optimization algorithms using the MM method and the EM method has been reported in terms of computational complexity and separation performance, but the GMVAE method has not yet been studied.
[0016] The signal analysis device of this embodiment has the following two features. The first feature is the use of a parameter optimization algorithm based on the EM method. The signal analysis device of this embodiment performs a parameter optimization algorithm based on the EM method in the GMVAE method. This algorithm, like the parameter optimization algorithm based on the MM method, enables parameter estimation that ensures convergence to a stationary point.
[0017] The second feature is the use of an auxiliary classifier VAE (ACVAE source model (see, for example, Reference 6)). In the GMVAE method, the CVAE source model is used. However, there are cases where the power spectrogram output from the CVAE is a sample that ignores speaker information. The signal analysis device of this embodiment extends the CVAE source model and can also use the source model of the ACVAE method, which has been shown to be effective in other tasks.
[0018] [2. Device configuration] FIG. 1 is a diagram showing a configuration of a signal analysis device 1 of this embodiment. The signal analysis device 1 includes an input unit 11, a storage unit 12, a learning unit 13, and an estimation unit 14. Microphones 6-1 to 6-I (I is an integer of 2 or more) installed at different positions each collect a sound that is a mixture of sounds from sound sources 7-1 to 7-J (J is an integer of 2 or more). Each microphone 6-1 to 6-I outputs an observation signal indicating the collected sound. The observation signal is a mixed signal in which the sounds of each sound source are mixed. Hereinafter, the microphone 6-i (i is an integer of 1 to I) is also referred to as microphone i, and the sound source 7-j (j is an integer of 1 to J) is also referred to as sound source j.
[0019] The input unit 11 inputs various data such as learning data and data to be separated from sound sources. The learning data includes observation signals collected by the microphones 6-1 to 6-I and sound source signals representing the sounds of the sound sources 7-1 to 7-J. The data to be separated from the sound sources includes observation signals of sounds collected by the microphones 6-1 to 6-I. The input unit 11 may receive the input data from the microphones 6-1 to 6-I or other devices, or may read the data from a recording medium. The storage unit 12 stores various data used in the processing of the learning unit 13 and the estimation unit 14.
[0020] The learning unit 13 learns a sound source separation model using learning data. The sound source separation model is a model used to estimate the sound source signals of the sound sources 7-1 to 7-J from the observation signals obtained by the microphones 6-1 to 6-I. The sound source separation model represents the correspondence between the feature amounts of the observation signals and the feature amounts of the sound source signals. The learning unit 13 uses a sound source model of the GMVAE method or a sound source model of the ACVAE method to learn the sound source separation model. The learning unit 13 also uses a parameter optimization algorithm based on the EM method to learn these sound source separation models. The estimation unit 14 obtains estimated sound source signals of the sound sources 7-1 to 7-J using the sound source separation model learned by the learning unit 13 and the observation signals of the microphones 6-1 to 6-I.
[0021] The signal analyzing device 1 is realized, for example, by a computer device. The signal analyzing device 1 may be realized by a plurality of computer devices connected to a network. In this case, it is possible to arbitrarily determine which of the plurality of computer devices each functional unit of the signal analyzing device 1 is realized by. For example, the learning unit 13 and the estimation unit 14 may be realized by different computer devices. Also, the same functional unit of the signal analyzing device 1 may be realized by a plurality of computer devices. For example, the learning unit 13 may be realized in a distributed manner by a plurality of computer devices.
[0022] FIG. 2 is a device configuration diagram showing an example of a hardware configuration of the signal analysis device 1. The signal analysis device 1 includes a processor 51, a storage unit 52, a communication interface 53, and a user interface 54. The processor 51 is a central processing unit that performs calculations and control. The processor 51 is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The storage unit 52 realizes the storage unit 12. The processor 51 also realizes the functions of the learning unit 13 and the estimation unit 14 described above by reading and executing a program from the storage unit 52. The storage unit 52 further has a work area and the like when the processor 51 executes various programs. The communication interface 53 is connected to other devices so as to be able to communicate with them. The user interface 54 is an input device such as a keyboard, a pointing device (mouse, tablet, etc.), a button, a touch panel, etc., or a display device such as a display. An artificial operation is input through the user interface 54.
[0023] In addition, all or part of the functions of the memory unit 12, the learning unit 13, and the estimation unit 14 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).
[0024] Below, the algorithm used in the signal analyzing device 1 will be explained, and then the operation and effects of the signal analyzing device 1 will be described.
[0025] 3. Problem formulation Let the number of sound sources be J and the number of microphones be I. Let s be the complex spectrogram obtained by short-time Fourier transform (STFT) of the sound source signal of sound source j (j is an integer between 1 and J). j Let (f,n) be the complex spectrogram obtained by STFTing the observed signal of microphone i (i is an integer between 1 and I) and denote it as x iLet (f,n), where f represents a frequency index and n represents a time frame index. The frequency of index f is also referred to as frequency f, and the time frame of index n is also referred to as time frame n.
[0026] As shown in the following equation (1), s j Let s(f,n) be a vector with elements (f,n), and let x i Let x(f,n) be a vector with elements (f,n).
[0027]
number
[0028]
number
[0029] where ( ) T is the transpose, and C represents the set of complex numbers.
[0030] The complex spectrogram s of the sound source j j For (f, n), a complex Gaussian distribution shown in the following equation (3), called local Gaussian modeling (LGM), is assumed.
[0031]
number
[0032] ~N C (α1,α2) represents a complex Gaussian distribution with mean α1 and covariance matrix α2. That is, the complex spectrogram s j (f,n) has mean 0 and variance v j It is assumed that the complex Gaussian distribution of (f,n) with variance v j (f, n) is expressed by the following equation (4).
[0033]
number
[0034] E represents the calculation of the expected value. According to the above, the variance v j (f,n) represents the power spectral density (PSD) of the sound source j. j ≠ j', and the complex spectrogram s j (f,n) and the complex spectrogram s of the sound source j' j’ When (f,n) and (f,n) are statistically independent, the vector s(f,n) follows a complex Gaussian distribution with a mean of 0 and a variance of V(f,n), as shown in the following equation (5).
[0035]
number
[0036] V(f,n) has diagonal elements v1(f,n), v2(f,n),...,v J The diagonal matrix is given by (f,n). A mixing system that can also take into account a situation where the number of sound sources J is greater than the number of microphones I is given by the following equation (6).
[0037]
number
[0038] Here, A(f) represents the mixing matrix shown in the following equation (7).
[0039]
number
[0040] From equations (5) and (6), the vector x(f,n) also has a mean of 0 and a variance of A(f)V(f,n)A H It follows a complex Gaussian distribution on (f,n), where (·) H represents the conjugate transpose. Also, the steering vector a of the sound source j included in the mixing matrix jFor the direct product of (f), the full rank spatial covariance matrix (SCM) model R is given by the following equation (8): j Assume (f).
[0041]
number
[0042] The above R j Assuming (f), the vector x(f,n) is given as a complex Gaussian distribution as shown in equation (9) below.
[0043]
number
[0044] Therefore, the log-likelihood function L is expressed by the following equation (10).
[0045]
number
[0046] tr[ ] denotes the trace of a matrix, det[ ] denotes the determinant, and the following denotes the equality signs for the parameters:
[0047]
number
[0048] Vector X(f,n) is expressed by the following equation (12), and vector ^X(f,n) is expressed by the following equation (13).
[0049]
number
[0050]
number
[0051] {v j (f,n)} j,f,n and {R j (f) j,f By estimating, the separated signal of the sound source j is filtered by the multi-channel Wiener filter M j (f,n) and apply the inverse STFT. j (f,n) is the complex spectrogram of the observed signal of the estimated sound source j. That is, the multi-channel Wiener filter M j (f, n) is the sound source separation model trained by the training unit 13 of the signal analyzing device 1.
[0052]
number
[0053] [4. GMVAE Law] The GMVAE method is explained.
[0054] [4.1 CVAE sound source model] FIG. 3 is a diagram showing a GMVAE sound source model. The encoder is a neural network (NN) that compresses the input data ^S into a low-dimensional latent variable Z. The encoder and decoder are NNs that restore the input data ^S from the latent variable Z. FIG. 4 is a diagram showing an example of a DNN used in the encoder shown in FIG. 3, and FIG. 5 is a diagram showing an example of a DNN used in the decoder shown in FIG. 3. In FIG. 4 and FIG. 5, Conv indicates a convolution layer, BN indicates a batch normalization layer, GLU indicates a gated linear unit layer, and Deconv indicates a deconvolution layer. In addition, [α1;α2] indicates an input channel α1 and an input frame length α2, and (β1,β2) indicates a kernel size β1 and a stride size β2 along the frame. The channel and frame at the input of the encoder correspond to a frequency f and a time frame n, respectively. F indicates the number of frequency indexes, and T indicates the number of time frames.
[0055] In the GMVAE method, a CVAE decoder pre-trained on a labeled dataset is used as a power spectrogram model of the sound source. ~ Given S and one-hot encoded source label c, the distribution of the CVAE encoder output q φ (Z| ~ S, c) is expressed by the Gaussian distribution shown in the following equation (15), and the decoder output distribution p θ ( ~ Assume that S|Z,c) follows the complex Gaussian distribution expressed by the following equation (16). The sound source label c is a vector in which only the element corresponding to the sound source j is 1 and the other elements are 0.
[0056]
number
[0057]
number
[0058] μ φ ( ~ S, c) and σ φ 2 ( ~ S,c) is the output of the encoder, and σ θ 2 (Z, c) is the output of the decoder. Note that diagα represents a diagonal matrix with α as the diagonal element.
[0059] In CVAE, the network parameters φ and θ of the encoder and decoder are trained to maximize the training criterion I(φ,θ) shown in the following equation (17).
[0060]
number
[0061] Here, the following represents the sample mean from the training sample:
[0062]
number
[0063] Additionally, the following represents the Kullback-Leivler (KL) divergence:
[0064]
number
[0065] Moreover, the above p(Z) is the standard Gaussian distribution as follows:
[0066]
number
[0067] Since the decoder distribution is isomorphic to the LGM, by using the trained decoder, equation (3) is transformed into the following equation (21).
[0068]
number
[0069] Here, σ θ 2 (f,n;Z,c) is the decoder output σ for element (f,n). θ 2 (Z,c). Also, g j is a variable that represents the scale of the sound source j.
[0070] [4.2 Parameter optimization algorithm based on MM method] The MM method is an iterative algorithm that designs a surrogate function called a minorizer that represents a lower bound for the learning criterion to be maximized, and searches for a stationary point for the desired learning criterion by iteratively maximizing it.
[0071] By applying the MM method to the log-likelihood function given by equation (10), the minorizerL shown in the following equation (22) is obtained. MM (≦L) is obtained. Note that tr represents the sum of the diagonal elements of a square matrix.
[0072]
number
[0073] Here, P j (f,n) and K(f,n) are auxiliary variables. j (f,n) is defined by the following formula (23), and K(f,n) is defined by the following formula (24). Note that ← indicates updating the value on the left side to the value on the right side.
[0074]
number
[0075]
number
[0076] The parameter optimization algorithm based on the MM method uses auxiliary variables {P j (f,n)} j,f,n , {K(f,n)} f,n Update and parameter {g j} j , {Z j} j , {c j} j , {R j (f) j,f minorizerL MM The decoder input {Z j} j and {c j} j is updated by the backpropagation algorithm as shown in the following equation (25).
[0077]
number
[0078] Here, η represents the learning rate. Label information c j In estimating, we need to consider the constraint that the sum is 1. This is because j This can be achieved by preparing a softmax layer whose output is {g j} j and {R j (f) j,f The update equations shown in the following equations (26) and (27) can be derived for
[0079]
number
[0080]
number
[0081] where # denotes the geometric mean between two positive semidefinite matrices. j (f) is the following equation (28), and Ω j (f) is the following equation (29).
[0082]
number
[0083]
number
[0084] [5. Algorithm in this embodiment] [5.1 Parameter optimization algorithm based on EM method] The EM algorithm maximizes the conditional expectation of the log-likelihood function for complete data, called the Q function, by repeating two steps (E step and M step). f,n are the observed data, and the source signal {s(f,n)} f,n Assuming that is unobserved data, the Q function L shown in the following equation (30) is EM (≦L) is obtained.
[0085]
number
[0086] Here, Λ j (f,n) is the conditional expectation of the direct product of s(f,n) and corresponds to the E-step. j (f, n) is expressed by the following equation (31).
[0087]
number
[0088] The M step is the parameter {g j} j , {Z j} j , {c j} j and {R j (f) j,f As with the parameter optimization algorithm based on the MM method, the decoder output {Z j} j and {c j} j is estimated by the backpropagation algorithm as shown in the following equation (32).
[0089]
number
[0090] {g j} j The following update equation (33) can be derived for {R j (f) j,f The following update equation (34) can be derived:
[0091]
number
[0092]
number
[0093] We focus on the calculation of matrix inversion and matrix multiplication in the parameter optimization algorithms based on the MM method and the EM method. In each iteration of the algorithm, the MM method uses the auxiliary variable P j (f,n) and K j (f,n) needs to be updated, and {R j (f) j,fIn addition, matrix inversion and matrix multiplication are required when updating . Ultimately, (3N+2J)F matrix inversions and (5N+2)JF matrix multiplications are required. On the other hand, in the EM algorithm, the auxiliary variable Λ is updated at each iteration. j (f,n) is updated once, and {R j (f) j,f is updated by a weighted sum. Therefore, only (N+J)F matrix inversions and 2NJF matrix multiplications are required. Therefore, the amount of calculations required for the parameter optimization algorithm based on the EM method is small, and it is expected that the processing time will be reduced.
[0094] [5.2 ACVAE Sound Source Model] FIG. 6 is a diagram showing an ACVAE sound source model. ACVAE is an extended model of CVAE (see, for example, Reference 6). ACVAE is a configuration in which a discriminator is added to the GMVAE sound source model shown in FIG. 3. FIG. 7 is a diagram showing an example of a DNN used in the encoder shown in FIG. 6, FIG. 8 is a diagram showing an example of a DNN used in the decoder shown in FIG. 6, and FIG. 9 is a diagram showing an example of an NN used in the discriminator shown in FIG. 6. In FIGS. 7 to 9, "w" represents the width, "c" represents the channel number, and "k" represents the kernel size. In addition, in FIGS. 7 to 9, Conv represents a one-dimensional convolution layer, BN represents a batch normalization layer, GLU represents a gated linear unit layer, and Deconv represents a one-dimensional deconvolution layer.
[0095] ACVAE is the output of the decoder. ~ S~p θ ( ~ S|Z,c) and the source label c are as correlated as possible, so that the mutual information I(c; ~ S|Z) is used as the learning criterion. Since it is difficult to directly use mutual information as a learning criterion, ACVAE uses the variational lower bound J(φ,θ,ψ) shown in the following equation (35) instead.
[0096]
number
[0097] where r Ψ (c| ~ S) represents the distribution of the output of the classifier with the network parameter ψ. In ACVAE, the cross-entropy K(ψ) shown in the following equation (36) is also used as a learning criterion.
[0098]
number
[0099] Therefore, the final learning criterion for ACVAE is given by I(φ, θ) shown in the following equation (37).
[0100]
number
[0101] Here, λ J ≧0 and λ K ≧0 is a weight parameter.
[0102] [6. Operation of the signal analyzer] Next, the operation of the signal analyzing device 1 will be described.
[0103] 10 is a flow diagram showing the learning process in the signal analyzing device 1. First, the signal analyzing device 1 learns a CVAE decoder for each sound source j using a labeled data set (step S11). Specifically, the input unit 11 receives a label c j The learning unit 13 inputs a plurality of sound source signals to which the label c is added. The learning unit 13 performs STFT on the input sound source signals to generate a complex spectrogram. The learning unit 13 normalizes the complex spectrogram and obtains a label c j With ~ S j ={s(f,n)} f,n The input unit 11 generates the label c j With ~ S j={s(f,n)} f,n The learning unit 13 may input the label c j Normalized power spectrogram annotated with ~ S j We train the decoder using
[0104] When the CVAE sound source model is used, the learning unit 13 performs the following on the encoder shown in FIG. ~ S j and sound source label c j Input the source label c j teeth, ~ S j The label c assigned to j is a vector that is one-hot encoded so that an element corresponding to the sound source j represented by is set to 1. The learning unit 13 calculates the average μ φ ( ~ S j ,c j ) and variance σ φ 2 ( ~ S j ,c j ) based on the latent representation Z j The learning unit 13 generates the latent representation Z j and sound source label c j and input to the decoder to obtain the variance σ θ 2 (Z j ,c j ) is obtained. The learning unit 13 learns the encoder parameter φ and the decoder parameter σ of the sound source j so as to maximize the learning criterion of equation (17).
[0105] When the ACVAE excitation model is used, the learning unit 13 obtains the decoder output in the same manner as when the CVAE excitation model is used. That is, the learning unit 13 adds the following to the encoder shown in FIG. ~ S j and sound source label c j Input the average μ φ ( ~ S j ,c j ) and variance σφ 2 ( ~ S j ,c j ) based on the latent representation Z j The learning unit 13 generates the latent representation Z j and sound source label c j is input to the decoder, and the variance σ θ 2 (Z j ,c j ) as an output. The learning unit 13 further obtains the variance σ θ 2 (Z j ,c j ) is input to the discriminator and the output p j The learning unit 13 learns the encoder parameter φ, the decoder parameter σ, and the classifier parameter ψ of the sound source j so as to maximize the learning criterion of equation (37).
[0106] Next, the learning unit 13 uses the decoder learned in step S11 to calculate the multi-channel Wiener filter M j (f, n) is learned. The input unit 11 inputs the observation signals of the microphones 6-1 to 6-I and the sound source signals of the sound sources 7-1 to 7-J as learning data. The learning unit 13 performs STFT on the input observation signals and sound source signals to obtain complex spectrograms s i (f,n) and the complex spectrogram of each source signal x i (f, n) (step S12). Alternatively, the input unit 11 calculates the complex spectrogram s i (f,n) and the complex spectrogram of each source signal x i You may also enter (f,n).
[0107] The learning unit 13 learns the latent representation Z j To calculate ~ S j and sound source label c j is input to the encoder of the sound source j learned in step S11. ~ Sj For example, a complex spectrogram of the sound source signal of the sound source j separated from the observed signal using MNMF or ConvTasNet can be used as the sound source j, but is not limited to these. φ ( ~ S j ,c j ) and σ φ 2 ( ~ S j ,c j ) based on the initial latent representation Z j is generated (step S13).
[0108] The learning unit 13 learns a latent representation Z j and sound source label c j is input to the decoder of the trained sound source j in step S11, and σ θ 2 (Z j ,c j ) (step S14). The learning unit 13 executes the E step. That is, the learning unit 13 obtains the current multi-channel Wiener filter M j (f,n), the vector x(f,n) obtained from the complex spectrogram of the observed signal, and the current spatial covariance model R j (f) and the variance of the complex spectrogram of the source signal v j (f,n) and the updated Λ j (f, n). The learning unit 13 calculates the variance v j (f,n) is expressed as the current scale g of the sound source j as shown in (21). j and σ obtained from the output of the decoder for sound source j θ 2 (f,n;Z j ,c j ) for each sound source j. The learning unit 13 further calculates Λ for combinations of all frequencies f, all time frames n, and all sound sources j. j (f,n) and the variance v j (f,n) and the spatial covariance model R j Using (f) and (30), the Q function LEM is calculated (step S15).
[0109] Next, the learning unit 13 executes the M step. That is, the learning unit 13 performs the M step on the Q function L calculated in the E step of step S15 for each sound source j. EM and {Z j , c j} j Using the gradient of and the updated {Z j , c j} j (Step S16). Furthermore, the learning unit 13 calculates, for each excitation j, σ θ 2 (f,n;Z j ,c j ) and the spatial covariance model R of the current sound source j j (f) and Λ of the sound source j calculated in step S15 j Using (f, n), the updated scale g j The learning unit 13 calculates σ θ 2 (f,n;Z j ,c j ) and the scale g of the current sound source j j and Λ of the sound source j calculated in step S15. j (f, n) and the updated R j (f) (step S17). The learning unit 13 calculates the updated g for each sound source j as shown in equation (21). j and σ obtained from the decoder output 2 θ (f,n;Z j ,c j ) to get the updated variance v j (f, n). The learning unit 13 calculates the updated spatial covariance model R j (f) and the updated variance v j Using (f, n), the updated multi-channel Wiener filter M j (f, n) is calculated (step S18).
[0110] The learning unit 13 determines whether or not a termination condition is satisfied (step S19). The termination condition may be, for example, that the number of repetitions reaches a predetermined value. If the learning unit 13 determines that the termination condition is not satisfied, the signal analyzing device 1 repeats the processes from step S14. If the learning unit 13 determines that the termination condition is satisfied, the signal analyzing device 1 repeats the processes from step S13 to step S19. Note that in step S13, the learning unit 13 updates the updated {Z j , c j} j is input to the decoder of the sound source j. When it is determined that the termination condition is satisfied, the learning unit 13 inputs the multi-channel Wiener filter M j (f, n) is written to the storage unit 12, and the process of FIG. 10 ends.
[0111] 11 is a flow diagram showing a sound source separation process in the signal analyzing device 1. The input unit 11 of the signal analyzing device 1 inputs the observed signals of the microphones 6-1 to 6-I. The estimation unit 14 performs STFT on each observed signal to calculate a complex spectrogram, and generates a vector x(f,n) for each combination of frequency f and time frame n (step S21). The estimation unit 14 receives from the storage unit 12 the multi-channel Wiener filter M j The estimation unit 14 estimates the vector x(f,n) by applying the multi-channel Wiener filter M j Multiplying by (f,n), we obtain the estimated complex spectrogram ^s for each sound source j. j (f, n) for each sound source j in the order of frequency f and time frame n (step S22). j After arranging (f, n), the sound source signal of the sound source j is estimated by performing inverse STFT (step S23).
[0112] Conventionally, the GMVAE method used the MM method as an iterative algorithm for deriving a parameter optimization algorithm with guaranteed convergence. However, multiple inverse matrix operations and matrix multiplication operations were required each time parameters were updated in the iteration for optimization, which caused problems with the calculation speed. In this embodiment, when performing sound source separation by the GMVAE method using a deep neural network, the conditional expectation value of the logarithmic likelihood function for complete data, called a Q function, is maximized by repeating two steps, the E step and the M step. As a result, the optimal solution is converged using an algorithm that maximizes the logarithmic likelihood function for the target incomplete data. Therefore, sound source separation with reduced calculation amount is possible.
[0113] [7. Effects of the Present Invention] In order to investigate the effectiveness of this embodiment, a sound source separation experiment was carried out assuming an underdetermined condition with three sound sources and two microphones.
[0114] A total of 400 utterances from two males and two females included in the VCC2018 dataset were used as training data, and a total of 40 utterances from another two males and two females were used as test data. Speech signals were randomly extracted from the test data to create a total of 40 samples of mixed signals from three speakers in a reverberation environment with T60=78ms. The sampling frequency was 16kHz, and the STFT analysis conditions were frame length and shift length of 128ms and 64ms, respectively.
[0115] As baselines to be compared for separating sound source signals from observed signals by the signal analysis device 1 of this embodiment, the EM-based MNMF, the MM-based MNMF, and ConvTasNet were used. These methods were also used as the initial values of the GMVAE method in the signal analysis device 1 of this embodiment. The number of bases in NMF was set to 10 for each speaker, and the Itakura-Saito NMF (IS-NMF) with 1000 iterations was applied to the training data to learn the spectral templates. During testing, the number of iterations of MNMF was set to 300, and the separated signal at 200 iterations was used to initialize the GMVAE method. ConvTasNet was trained using Libri3Mix, a three-speaker monaural mixed signal dataset created from LibriSpeech data, using a training script provided by the open source asteroid. During testing, for multi-channel observed signals, the signals of each channel were considered as monaural signals and input to the network, and the output separated signals were combined to obtain a multi-channel separated signal. In addition, the separated signals obtained by ConvTasNet were also used to initialize the GMVAE method.
[0116] As the ACVAE sound source model used in the GMVAE method in the signal analysis device 1 of this embodiment, a three-layer convolutional network, a three-layer deconvolutional network, and a four-layer convolutional network with a gate mechanism are used for the encoder, the decoder, and the classifier, respectively. The weight parameter λ included in the learning criterion shown in equation (37) is J and λ K was set to 1, and the number of learning epochs was set to 1000. The Adam algorithm with learning rates of 0.0001 and 0.01, respectively, was used for the optimization algorithm in the network learning and the error backpropagation method of the GMVAE method.
[0117] The evaluation measures used were signal-to-distortion ratio (SDR), source image-to-spatial distortion ratio (ISR), signal-to-inference ratio (SIR), and signal-to-artifact ratio (SAR), which indicate separation performance, perceptual evaluation of speech quality (PESQ), which indicates speech quality, and short-time objective intelligibility (STOI), which indicates intelligibility.
[0118] FIG. 12 shows a table of the comparison results of the processing time by each method. FIG. 12 shows the processing time [s] required for separating a voice signal of 6.89 s (seconds). Here, the signal analysis device 1 of this embodiment is compared with the case where the MM method and the EM method are used for the MNMF, which has the same learning standard as this embodiment, and the case where the MM method is used for the GMVAE method. The numbers in parentheses indicate the number of iterations by the backpropagation method. In addition, in FIG. 12, the method with the faster processing time under the same conditions is shown in bold. As the CPU and GPU, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz and NVIDIA Tesla K80 were used. The upper row shows the processing time of the MNMF of the EM method and the GMVAE method of the EM method when using the CPU, the GMVAE method of the MM method, and the GMVAE method of the EM method according to this embodiment. The lower row shows the processing time of the GMVAE method of the MM method when using the GPU, and the GMVAE method of the EM method according to this embodiment. As shown in Fig. 12, unlike MNMF, it was confirmed that the parameter optimization algorithm of the EM method according to this embodiment is consistently faster than the MM method in the GMVAE method. Also, in the GMVAE method of the EM method according to this embodiment, the number of iterations of the backpropagation method was less than 10 [s] when using a GPU, and was as fast as the MNMF of the MM method even when using a CPU with an iteration number of 1.
[0119] 13 to 18 show the experimental results using the EM MNMF, the MM MNMF, and ConvTasNet as the initialization method. That is, in step S13, ~ S was input to the encoder to calculate the initial latent representation Z. Figure 13 shows the separation performance using SDR, Figure 14 shows ISR, Figure 15 shows SIR, Figure 16 shows SAR, Figure 17 shows PESQ, and Figure 18 shows STOI. Error bars indicate 95% confidence intervals. The numbers in parentheses indicate the number of iterations in the backpropagation method.
[0120] In the GMVAE method, performance improvement was observed as the number of updates in the backpropagation method increased, and the same separation performance was achieved in both the EM and MM parameter optimization algorithms. It was also shown that significant performance improvement is possible by using a method with higher separation performance, such as ConvTasNet, as the initialization.
[0121] From the above, it was confirmed that the GMVAE method based on the EM method of this embodiment (1) achieves separation performance as fast as the MNMF based on the MM method and higher, and (2) can achieve separation performance as fast as the GMVAE method based on the MM method.
[0122] Although the embodiment of the present invention has been described above with reference to the drawings, it is clear that the above embodiment is merely an example of the present invention and the present invention is not limited to the above embodiment. Therefore, addition, omission, substitution, and other modifications of components may be made without departing from the technical spirit and scope of the present invention.
[0123] (Reference 1) N. Q. Duong, E. Vincent, and R. Gribonval, "Under-Determined Reverberant Audio Source Separation Using a Full-Rank Spatial Covariance Model," IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 7, pp. 1830-1840, 2010.
[0124] (Reference 2) D. D. Lee and H. S. Seung, "Learning the parts of objects by non-negative matrix factorization," Nature, vol. 401, no. 6755, pp. 788-791, 1999.
[0125] (Reference 3) S. Arberet, A. Ozerov, N. Q.K. Duong, E. Vincent, R. Gribonval, F. Bimbot, and P. Vandergheynst, "Nonnegative matrix factorization and spatial covariance model for under-determined reverberant audio source separation," in Proc. International Conference on Information Science, Signal Processing and their Applications(ISSPA), pp. 1-4, 2010.
[0126] (Reference 4) H. Sawada, H. Kameoka, S. Araki, and N. Ueda, "Multichannel Extensions of Non-Negative Matrix Factorization With Complex-Valued Data," IEEE Transactions on Audio, Speech, and Language Processing, vol. 21, no. 5, pp. 971-982, 2013.
[0127] (Reference 5) DP Kingma, DJ Rezende, S. Mohamed, and M. Welling, "Semi-Supervised Learning with Deep Generative Models," in Proc. Neural Information Processing Systems (NIPS), 2014.
[0128] (Reference 6) H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, "ACVAE-VC: Non-Parallel Voice Conversion With Auxiliary Classifier Variational Autoencoder," IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 9, pp. 1432-1443, 2019. [Explanation of symbols]
[0129] 1 ... signal analysis device, 6-1 to 6-I ... microphones, 7-1 to 7-J ... sound source, 11 ... input unit, 12 ... memory unit, 13 ... learning unit, 14 ... estimation unit, 51 ... processor, 52 ... memory unit, 53 ... communication interface, 54 ... user interface
Claims
1. A signal analysis device that estimates a multi-channel Wiener filter for separating sound source signals representing the respective sounds of a plurality of sound sources from an observation signal representing multiple sounds observed at each of a plurality of positions, the signal analysis device comprising: A sound source model includes an encoder that receives a feature amount of a sound source signal and sound source label information for identifying a sound source, and outputs an average and variance of a latent representation obtained by reducing the dimension of the feature amount of the sound source signal, and a decoder that receives a latent representation based on the average and the variance output by the encoder and the sound source label information, and outputs a variance representing the feature amount of the sound source signal input to the encoder. The decoder receives a process of inputting a latent representation of an initial value of the sound source signal and sound source label information into the decoder, and calculates a conditional expected value L of a logarithmic likelihood function for complete data based on an output obtained by performing a process for each sound source of multiple sounds represented by an observation signal for learning. EM a step E of calculating the conditional expected value L EM a learning unit that performs an M step of updating the latent representation and the sound source label information of each of the sound sources using the above-mentioned, and a filter calculation process of calculating the multi-channel Wiener filter of each of the sound sources based on the output from the decoder obtained in the E step, and repeats the E step, the M step, and the filter calculation process using the updated latent representation and the sound source label information as input to the decoder; A signal analysis device comprising:
2. The sound source model is a neural network used in a generalized multichannel variational autoencoder (GMVAE) method or an auxiliary classifier variational autoencoder (ACVAE) method. The signal analysis device according to claim 1 .
3. The latent representation of the initial value corresponding to each of the sound sources is generated based on the mean and variance obtained by inputting the sound source signal of each of the sound sources obtained by separating the observation signal for learning by MNMF (multichannel non-negative matrix factorization) or ConvTasNet into the encoder of the sound source model corresponding to the sound source. The signal analyzing device according to claim 1 or 2.
4. the feature amount is a complex spectrogram, In the filter calculation process, the learning unit calculates a variance v of a complex spectrogram in a frequency f of the sound source j and a time frame n based on an output from the decoder corresponding to the j-th sound source j among the number of sound sources J. j (f, n) and the spatial covariance model R of frequency f j (f) is calculated for all sound sources j of the sound source number J, j (f, n) and the spatial covariance model R j Using (f), a multi-channel Wiener filter M j Calculate (f, n); The signal analyzing device according to any one of claims 1 to 3.
5. Variance v of sound source j j (f, n) is a variable g representing a frequency f and an element corresponding to time frame n in the output from the decoder corresponding to the sound source j and a scale of the sound source j. j It is calculated by multiplying In the E step, the learning unit calculates a conditional expectation Λ of the sound source j for a direct product of vectors in which a complex spectrogram of a frequency f of the observation signal for learning and a time frame n is arranged based on an output from the decoder corresponding to the sound source j. j (f, n) and calculate the conditional expectation Λ of all sound sources j out of the number of sound sources J. j (f, n) and the spatial covariance model R j (f) is used to calculate the conditional expectation L EM In the M step, the variable g j and the spatial covariance model R j The conditional expectation value Λ of the sound source j j Update using (f, n), 5. The signal analysis device according to claim 4.
6. The apparatus further includes an estimation unit that estimates a sound source signal of each of a plurality of sound sources based on the multi-channel Wiener filter updated by the learning unit and an observation signal observed at each of the plurality of positions. The signal analyzing device according to any one of claims 1 to 5.
7. A signal analysis method for estimating a multi-channel Wiener filter for separating sound source signals representing each of a plurality of sound sources from an observation signal representing multiple sounds observed at each of a plurality of positions, the method comprising: A sound source model includes an encoder that receives a feature amount of a sound source signal and sound source label information for identifying a sound source, and outputs an average and variance of a latent representation obtained by reducing the dimension of the feature amount of the sound source signal, and a decoder that receives a latent representation based on the average and the variance output by the encoder and the sound source label information, and outputs a variance representing the feature amount of the sound source signal input to the encoder. The decoder receives a process of inputting a latent representation of an initial value of the sound source signal and sound source label information into the decoder, and calculates a conditional expected value L of a logarithmic likelihood function for complete data based on an output obtained by performing a process for each sound source of multiple sounds represented by an observation signal for learning. EM a step E of calculating the conditional expected value L EM An update process of performing an M step of updating the latent representation and the sound source label information of each of the sound sources using the above, and a filter calculation process of calculating the multi-channel Wiener filter of each of the sound sources based on the output from the decoder obtained in the E step; a repeating step of repeating the E step, the M step and the filter calculation process using the updated latent representation and the sound source label information as an input to the decoder; A signal analysis method comprising:
8. Computer, A program for causing the signal analyzer according to any one of claims 1 to 6 to function as such.
Citation Information
Patent Citations
Musical sound emphasis device, convolution auto encoder learning device, musical sound emphasis method, and program
JP2019078864A
Signal analysis device, method, and program
JP2020034870A
