Signal analysis device, signal analysis method, and signal analysis program

Abstract

To precisely separate each of constituent tones from a mixture signal in which the constituent tones are mixed while reducing the calculation cost.SOLUTION: A learning unit 32 learns an encoder that estimates a latent vector series in response to an input of a spectrogram of a tone on the basis of a spectrogram of each constituent tone and an attribute class thereof, an identifier that identifies an attribution class which indicates the attribute of the tone in response to an input of the spectrogram of the tone, and a decoder that generates distribution of the spectrogram of the tone in response to an input of the latent vector series and the attribute class. A parameter estimation unit 36 estimates a separate matrix and a scale parameter to optimize an objective function in response to an input of an observation signal in which the constituent tones are mixed.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The disclosed technology relates to a signal analysis device, a signal analysis method, and a signal analysis program. [Background technology]

[0002] Blind Source Separation (BSS) is a technology that separates and extracts individual source signals from only the observed mixed signals without using prior information such as information about the sound sources or transfer functions between the sound sources and microphones. Under overdetermined conditions where the number of microphones is equal to or greater than the number of sound sources, it is known that Independent Component Analysis (ICA), which aims to estimate a separation filter so as to maximize the independence between source signals, is effective, and many methods that extend the principle of ICA have been proposed. Among them, methods formulated in the time-frequency domain have the advantage of being able to effectively utilize various assumptions that hold in the time-frequency domain regarding sound sources and assumptions regarding the frequency response of microphone arrays. For example, Independent Low-Rank Matrix Analysis (ILRMA) described in Non-Patent Document 1 is based on the assumption that the power spectrogram of each sound source signal can be modeled by the product of two non-negative matrices (low-rank non-negative matrix). However, the separation performance of this method is inevitably limited for sound sources that do not follow this assumption.

[0003] In recent years, attempts have been made to improve separation accuracy by introducing deep learning (Deep Neural Network: DNN) into methods based on signal processing such as ICA. The Multichannel Variational Autoencoder (MVAE) method described in Non-Patent Document 2 is a method in which a generative model of a sound source spectrogram represented by a Conditional VAE (CVAE) is pre-trained, and the decoder input of the CVAE is estimated together with the separation matrix during separation, and has achieved particularly high separation accuracy among methods using DNN. In this method, parameters are updated so that the likelihood function increases in each iteration, so the convergence of the likelihood function to a stationary point is guaranteed, but there is a problem in that a high calculation cost is required because the error backpropagation method is used to update the decoder input value.

[0004] The FastMVAE method described in Non-Patent Document 3 is a method proposed for the purpose of reducing the calculation cost of the MVAE method, and uses a classifier-equipped VAE (Auxiliary Classifier VAE: ACVAE) to learn a classifier distribution and an encoder distribution that approximate the distribution of the sound source classes and the posterior distribution of the latent variables together with a decoder, which is a generation model of the sound source spectrogram, and predicts the decoder input value that maximizes the posterior distribution using the classifier and encoder obtained by learning. While this method can speed up the sound source separation algorithm compared to the MVAE method, separation performance tends to decrease when the conditions during testing do not match those during learning, such as in the case of unknown speakers or long reverberation. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka, and Hiroshi Saruwatari, “Determined blind source separation unifying independent vector analysis and nonnegative matrix factorization,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no.9, pp. 1626-1641, Sep. 2016. [Non-Patent Document 2] Hirokazu Kameoka, Li Li, Shota Inoue, and Shoji Makino, “Supervised Determined Source Separation with Multichannel Variational Autoencoder,” Neural Computation, vol. 31, no. 9, pp.1891-1914, Sep. 2019. [Non-Patent Document 3] Li Li, Hirokazu Kameoka, Shota Inoue, and Shoji Makino, “FastMVAE: A Fast Optimization Algorithm for the Multichannel Variational Autoencoder Method,” IEEE Access, vol. 8, pp. 228740-228753, Dec. 2020. [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] In the MVAE method, parameters are updated so that the log-likelihood increases with each iteration, which has the advantage of guaranteeing that the log-likelihood will converge to a stationary point. However, there is a problem in that updating the parameters of the speech production model using the backpropagation method requires a huge computational cost.

[0007] In contrast, the FastMVAE method in Non-Patent Document 3 has achieved a significant increase in the speed of the sound source separation algorithm by predicting the update value of the parameter using an encoder and a classifier that have been trained in advance together with the decoder. However, since the output values ​​of the encoder and classifier in the FastMVAE method are merely approximations of the update value in the direction of the steepest increase in the log-likelihood for the parameter, it has been experimentally confirmed that the FastMVAE method is inferior to the MVAE method in terms of sound source separation accuracy.

[0008] The disclosed technology has been made in consideration of the above-mentioned points, and aims to provide a signal analysis device, method, and program that can accurately separate each component sound from a mixed signal in which the component sounds are mixed, while reducing calculation costs. [Means for solving the problem]

[0009] A first aspect of the present disclosure is a signal analysis device, comprising: a learning unit that learns an encoder that estimates a latent vector sequence by taking a spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound; a classifier that receives the spectrogram of the sound as an input and identifies an attribute class indicating an attribute of the sound; and a decoder that receives the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound; and a learning unit that learns the latent vector sequence estimated for each constituent sound separated by the separation matrix by the trained encoder by taking an observation signal in which the constituent sounds are mixed as an input; The present invention includes a parameter estimation unit that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by the attribute class identified for each constituent sound separated by the separation matrix by the trained classifier, a spectrogram of the constituent sound calculated from the variance of the spectrogram of the constituent sound and a scale parameter generated by the trained decoder for each constituent sound, the scale parameter of the spectrogram of each constituent sound, a separation matrix for separating a mixed sound in which the constituent sounds are mixed in the time-frequency domain into each constituent sound, and a signal obtained by separating the observation signal into each constituent sound.

[0010] A second aspect of the present disclosure is a signal analysis method, in which a learning unit learns an encoder that estimates a latent vector sequence by taking a spectrogram of a sound as an input based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound, a classifier that receives the spectrogram of the sound as an input and identifies an attribute class indicating an attribute of the sound, and a decoder that receives the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound, and a parameter estimation unit learns a parameter estimation unit that receives an observation signal in which the constituent sounds are mixed, and estimates a variance of each constituent sound separated by the separation matrix by the trained encoder. The separation matrix and the scale parameter are estimated so as to optimize an objective function represented by using the latent vector sequence obtained by the trained classifier, the attribute class identified for each constituent sound separated by the separation matrix by the trained classifier, a spectrogram of the constituent sounds calculated from the variance of the spectrogram of the constituent sound and a scale parameter generated by the trained decoder for each constituent sound, the scale parameter of the spectrogram of each constituent sound, a separation matrix for separating a mixed sound in which each constituent sound is mixed in the time-frequency domain into each constituent sound, and a signal obtained by separating the observed signal into each constituent sound.

[0011] A third aspect of the present disclosure is a program for causing a computer to function as the signal analysis device of the first aspect. Effect of the Invention

[0012] According to the disclosed technique, it is possible to obtain an effect that it is possible to reduce calculation costs and to accurately separate each constituent sound from a mixed signal in which each constituent sound is mixed. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a conceptual diagram for explaining the configurations of a teacher encoder and a teacher decoder according to the present embodiment. [Diagram 2] FIG. 2 is a conceptual diagram for explaining the configurations of an encoder, a discriminator, and a decoder according to the present embodiment. [Diagram 3] 3A to 3C are diagrams illustrating examples of the configuration of an encoder and a discriminator, and an example of the configuration of a decoder according to the present embodiment. [Figure 4] FIG. 2 is a schematic block diagram of an example of a computer that functions as a signal analysis device according to the present embodiment. [Diagram 5] 1 is a block diagram showing a configuration of a signal analysis device according to an embodiment of the present invention; [Figure 6] 4 is a flowchart showing a learning processing routine in the signal analyzing device of the present embodiment. [Figure 7] 4 is a flowchart showing a signal analysis processing routine in the signal analyzing device of the present embodiment. [Figure 8] FIG. 1 is a diagram showing the arrangement of microphones and sound sources in an experimental example. [Figure 9] FIG. 11 is a diagram showing the calculation time in each iteration according to the method of this embodiment and the conventional method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0015] <Outline of this embodiment> First, an overview of this embodiment will be described.

[0016] In this embodiment, the encoder and discriminator in the FastMVAE method are integrated into a single multitask neural network (NN) to achieve even faster speeds. In addition, knowledge distillation (KD) is performed on the multitask NN and decoder so that their output distributions are as close as possible to the output distributions of the encoder and decoder acquired by pre-learning in the MVAE method. This makes each NN behave similarly to the parameter update of the speech generation model in the MVAE method, and brings it closer to the high separation accuracy of the MVAE method. These ideas enable faster and more accurate source separation for unknown speakers than the conventional FastMVAE method.

[0017] <Principle of this embodiment> <Formulation of multi-channel sound source separation problem under overdetermined conditions> Consider the case where signals coming from J sound sources are observed with I microphones. The complex spectrograms of the observed signal from microphone i and the signal from sound source j are denoted by x i (f,n), s j Let (f,n) be the elements of the vector. JPEG0007658212000001.jpg992 (1) JPEG0007658212000002.jpg1094 (2) However, here we consider the dominant decision condition for I=J. Here, ( ) T denotes the transposition, and f and n are the frequency and time indices, respectively.

[0018] Under the condition of I=J, the instantaneous separation system is expressed as the relation between the complex spectrogram vector s(f,n) of the sound source signal and the vector x(f,n) of the observed signal. JPEG0007658212000003.jpg1279 (3) JPEG0007658212000004.jpg882 (4) We can assume that WH (f) represents the separation matrix, and ( ) H is the Hermitian transpose.

[0019] Under the above assumption of an instantaneous mixture system, the complex spectrogram s of the sound source j is j (f,n) has mean 0 and variance JPEG0007658212000005.jpg1279 Complex normal distribution of JPEG0007658212000006.jpg11116 (5) If we treat each sound source signal s as a random variable that follows j (f,n) and s j’ When (f,n), j ≠ j' are statistically independent, the source signal s(f,n) is JPEG0007658212000007.jpg797 (6) Follow.

[0020] where V(f,n) is v1(f,n), ,v I It is a diagonal matrix with elements (f,n). From equations (3) and (6), the observed signal x is JPEG0007658212000008.jpg9105 (7) Follow.

[0021] Therefore, the observed signal X={x(f,n)} f,n Given the separation matrix W={W(f)} f and the power spectrogram of each sound source V={v j (f,n)} j,f,n The log-likelihood function of JPEG0007658212000009.jpg17152 (8) Here, = c represents an equality only for terms that depend on parameters. jIf there are no constraints on (f,n), equation (8) is decomposed into terms for each frequency f, so the index of the separated signal obtained by W based on equation (8) has the arbitrariness of permutation. j If (f,n) has structural constraints in the frequency domain, it is possible to derive an approach that simultaneously solves permutation matching and source separation by utilizing those constraints. Examples of this are the ILRMA and MVAE methods.

[0022] <Conventional technology 1: MVAE method> In the MVAE method, the decoder distribution of the CVAE with the sound source class label as an auxiliary input is used as a generation model for the complex spectrogram of each sound source. The complex spectrogram of a certain sound source signal is defined as S={s(f,n)} f,n Let the corresponding source class label be a one-hot vector c. Figure 1 shows the concept of CVAE. CVAE is based on the encoder distribution q * φ (z|S,c) and the decoder distribution p * θ So that (S|z,c) is consistent, i.e., q * φ (z|S,c) and p * θ The posterior distribution p derived from (S|z,c) * θ (z|S,c) ∝ p * θ The NN parameters φ and θ of the encoder and decoder are trained so that (S|z,c)p(z) matches as closely as possible. Here, the decoder distribution of the CVAE is a probability model with the same shape as the local Gaussian source model in Eq. (5). JPEG0007658212000010.jpg1692 (9) JPEG0007658212000011.jpg1483 (10) where variance σ * θ 2 (f,n;z,c) is the output of the decoder network, and g is a variable that represents the scale of the power spectrogram.

[0023] On the other hand, the encoder distribution q * φ (z|S,c) is the standard normal distribution, just like the usual CVAE. JPEG0007658212000012.jpg992 (11) Assume that.

[0024] Here, μ * φ (S,c), σ * φ 2 (S,c) is the output of the encoder. The CVAE parameters θ and φ are the training samples of the complex spectrograms of the sound source signals of various classes {S m , c m} M m=1 Using JPEG0007658212000013.jpg16104 (12) is trained to maximize

[0025] JPEG0007658212000014.jpg1771 represents the sample mean of the training sample, and KL[·||·] is the Kullback-Leibler (KL) divergence. The decoder distribution p * θ (S|z,c,g) is called the CVAE source model. The CVAE source model is a generative model that can represent the complex spectrograms of various classes of sources contained in the training sample, where c can be considered as a variable that adjusts the categorical features of the source class, and z can be considered as a variable that adjusts the variation within the class.

[0026] The complex spectrogram S of the sound source j j ={s j (f,n)} f,n The generative model of z j , c j , g jBy expressing the likelihood function of the sound source model parameters as a decoder distribution with the input, the likelihood function of the sound source model parameters can be reduced to a likelihood function of the same form as equation (8). Therefore, the separation matrix W and the CVAE sound source model parameters Ψ={z j ,c j} j , scale parameter G={g j} j By iteratively updating, we can search for a stationary point of equation (8). To update W to raise equation (8), we use the Iterative Projection (IP) method, similar to ILRMA. JPEG0007658212000015.jpg1481 (13) JPEG0007658212000016.jpg2079 (14) can be used.

[0027] however, JPEG0007658212000017.jpg1793 and e j is the j-th column vector of the I × I unit matrix. The update of Ψ, which raises equation (8), is done by the backpropagation method, and the update of G is done by JPEG0007658212000018.jpg30106 (15) Here, equation (15) is the update equation that maximizes equation (8) when W and Ψ are fixed. From the above, the inference process of MVAE can be summarized as follows.

[0028] 1. Learn θ and φ using equation (12) as the criterion. 2. Initialize W to be the identity matrix and initialize Ψ. 3. Repeat steps a to c below for each j. (Step a) Using equations (13) and (14), j (f) j,f Update. (Step b) Ψ by backpropagation j ={z j,c j} to update. (Step c) Using equation (15), g j Update.

[0029] <Conventional technology 2: FastMVAE method> In the MVAE method, parameters are updated so that the log-likelihood increases in each iteration, so there is an advantage in that the log-likelihood is guaranteed to converge to a stationary point. θ (z j ,c j |S j ) parameter z that maximizes j , c j The problem with the FastMVAE method in Non-Patent Document 3 is that it requires a large amount of computational cost to update the posterior distribution p θ (z,c|S) is p θ (z|S,c)p θ (c|S) into a product of two conditional distributions, and then we approximate each distribution by using distribution q * φ (z|S,c), r * ψ (c|S) is represented by a NN and pre-trained. This allows the parameter search by the backpropagation method in the MVAE method to be replaced by forward calculation of each NN, enabling high-speed inference. However, the encoder q * φ (z|S,c) and the discriminator r * ψ Since the output value of (c|S) is merely an approximation of the update value of the log-likelihood for the parameter in question in the direction of the steepest increase, it has been experimentally confirmed that the FastMVAE method is inferior to the MVAE method in terms of sound source separation accuracy.

[0030] <Method of the Present Embodiment> In the FastMVAE2 method used in this embodiment, it is first assumed that the latent variable z and the attribute class c of the sound source are conditionally independent. This is equivalent to assuming that, given a given spectrogram S, the speaker information c and the information z about the speech content are independent. In other words, the posterior probability p θ(z,c|S) is p θ (z|S)p θ It is different from the prior art in that it is assumed that (c|S) can be expressed. If the approximate distributions of these two conditional distributions are obtained, parameter search can be performed quickly in the forward calculation of the NN, similar to the FastMVAE method.

[0031] <Chimera ACVAE Sound Source Model> ACVAE is an extended version of CVAE originally proposed for the purpose of applying to voice conversion. In order to emphasize the influence of the input class label c on the decoder output, the encoder and decoder are learned in a way that uses the mutual information I(c,S|z) between the decoder output and the class label c as a regularization term. It is not easy to directly optimize the criterion including I(c,S|z), but by introducing a variational lower bound similar to the learning of CVAE and increasing the combined criterion of the variational lower bound and J(φ,θ), the original criterion can be indirectly increased. I(c,S|z) is given as the sum of the expected value of log p(c|S) and a constant, but the replacement of p(c|S) with an appropriate auxiliary distribution r(c|S) is a lower bound of I(c,S|z). By modeling this auxiliary distribution r(c|S) with a neural network of parameters ψ ψ can be learned together with φ and θ using the above lower bound as a criterion. The auxiliary distribution represented by the neural network of parameter ψ is denoted as r ψ (c|S) and is called a discriminator.

[0032] On the other hand, the "Chimera ACVAE" of this embodiment is a model in which the encoder and discriminator of ACVAE are represented as an integrated multi-task neural network. That is, it is a model that simultaneously infers the distributions q + φ (z|S), r + ψ (c|S) from the spectrogram S. Fig. 2 shows a conceptual diagram of Chimera ACVAE.

[0033] ChimeraACVAE is structured to extract the latent variable z only from the input spectrogram, which makes it possible to avoid inference errors of z caused by estimation errors of the class label c. In addition, since it can be described with a more compact network structure than the conventional ACVAE model, it is expected to enable faster inference.

[0034] The criterion for training ChimeraACVAE, i.e., the objective function to be maximized with respect to the NN parameters θ, φ, and ψ, is the CVAE training criterion. JPEG0007658212000019.jpg11148 (16) and mutual information JPEG0007658212000020.jpg12132 (17) In addition, the labeled training sample {S m ,c m} M m can also be used for training, so the training data S m and the corresponding class label c m Negative cross entropy of JPEG0007658212000021.jpg13102 (18) can also be included in the criteria for learning. Up to this point, it is the same as the conventional ACVAE except for the model structure.

[0035] However, while the ACVAE trained according to the above criteria was capable of highly accurate inference when the test conditions and training conditions matched, when they did not match, the estimated latent variables tended to deviate from the assumed distribution, and the model's generalization ability was insufficient.

[0036] Therefore, in order to improve the generalization ability of the model, the following criteria and knowledge distillation are used in addition to the above criteria when training ChimeraACVAE. With ChimeraACVAE, the spectrogram S can be reconstructed using the estimated class information. Since this process is also used during inference, training a model using a criterion that evaluates the accuracy of the spectrogram S reconstructed by the same process is thought to lead to improved accuracy during inference. Therefore, the reconstruction criterion of equation (19) to be maximized and the class discrimination criterion of equation (20) are also included in the criteria for training. JPEG0007658212000022.jpg9104 (19) JPEG0007658212000023.jpg7122 (20)

[0037] Alternatively, instead of these criteria, approximations may be used to simplify implementation. JPEG0007658212000024.jpg994 (twenty one) JPEG0007658212000025.jpg10114 (twenty two) However, JPEG0007658212000026.jpg1476 It is.

[0038] Knowledge distillation (KD) is a methodology for using a large NN trained with a large amount of data as a teacher model and transferring that knowledge to a student model with a lightweight or different NN structure. It is known that this method can obtain a student model with high generalization ability. Here, a CVAE model that can achieve high separation accuracy even for unknown speakers is used as the teacher model, and the distribution q of the latent variables trained by CVAE is used as the teacher model. * φ (z|S,c) and the spectrogram generation model p * θWe consider inheriting the knowledge of (S|z,c) to the student model ChimeraACVAE. Specifically, the distribution of the latent variables inferred by CVAE, q * φ (z|S,c) and the variance σ * φ 2 Normal distribution N(0, diag(σ * θ 2 (z,c))) are the output distributions of the student models q + φ (z|S) and the decoder output σ + φ 2 The prior distribution is a normal distribution using the above, and the student model is trained so that its output approaches the prior distribution. The degree of divergence between the teacher model and the student model distribution is measured using the KL divergence, and is used as the knowledge distillation criterion, as shown in equations (23) to (25).

[0039] JPEG0007658212000027.jpg9117 (twenty three) JPEG0007658212000028.jpg17151 (twenty four) JPEG0007658212000029.jpg17150 (twenty five)

[0040] From the above, the criterion to be maximized when training ChimeraACVAE is JPEG0007658212000030.jpg15152 (26) Here, λ is a non-negative value and is the weighting coefficient for each criterion. Figure 2 shows the conceptual diagram of learning for ChimeraACVAE using knowledge distillation.

[0041] Fig. 3 shows an example of the network structure of ChimeraACVAE. Each layer of the encoder and the discriminator is composed of a convolutional layer, Layer Normalization (LN), and Sigmoid Linear Unit (SiLU), and each layer of the decoder is composed of a transposed convolutional layer, LN, and SiLU. Here, by using LN, it is possible to avoid the inconsistency in the calculation method of normalization during learning and inference. SiLU is a data-driven activation function that controls the information passed between layers by a gate, similar to the Gated Linear Unit (GLU) used in the CVA sound source model, and the number of parameters of GLU can be halved.

[0042] <FastMVAE2 Method: Fast Inference Algorithm> By using the encoder and discriminator learned by ChimeraACVAE, the maximization step of p θ (z j , c j |S j ) in the conventional MVAE method can be replaced with the forward calculations of q + φ (z j |S j ) and r + ψ (c j |S j ). Therefore, the following algorithm is obtained. This is called the FastMVAE2 method.

[0043] 1. Learn θ, φ, and ψ using Equation (26) as the criterion for learning. 2. Initialize W as the identity matrix. 3. Repeat the following steps a to c for each j. (Step a) Update {w j (f)} j,f according to Equations (13) and (14). (Step b) Using the spectrogram separated by W as the input, update z j and c j with the mean of the Gaussian distribution output from the encoder and the output value (continuous value vector) of the discriminator, respectively. (Step c) Using equation (15), g j Update.

[0044] <Configuration of the signal analysis device according to this embodiment> FIG. 4 is a block diagram showing a hardware configuration of the signal analyzing device 100 of this embodiment.

[0045] 4, the signal analyzing device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0046] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a learning program for executing a learning process, and a signal analysis program for executing a signal analysis process. The learning program and the signal analysis program may be one program, or may be a group of programs composed of multiple programs or modules.

[0047] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.

[0048] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0049] The input unit 15 receives, as learning data, time series data of the signal of each of a plurality of constituent sounds and an attribute class indicating the attribute of the signal of the constituent sound. The input unit 15 also receives, as data to be analyzed, time series data of a mixed signal (hereinafter, observed signal) in which a plurality of constituent sounds are mixed. Note that the attribute class indicating the attribute of the signal of the constituent sound may be given manually. In addition, the attribute of the signal of the constituent sound may be, for example, gender, adult / child, speaker ID, etc.

[0050] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0051] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0052] Next, a description will be given of the functional configuration of the signal analyzing device 100. FIG.

[0053] As shown in FIG. 5 , the signal analysis device 100 functionally includes a time-frequency expansion unit 24, a teacher learning unit 30, a learning unit 32, a sound source signal model storage unit 34, a parameter estimation unit 36, and an output unit 38.

[0054] The time-frequency expansion unit 24 calculates a power spectrogram representing the spectrum at each time for each component sound based on the time series data of the signal of the component sound. The time-frequency expansion unit 24 also calculates a power spectrogram representing the spectrum at each time based on the time series data of the observed signal. In this embodiment, the time-frequency expansion is performed using a short-time Fourier transform, a wavelet transform, or the like.

[0055] The teacher learning unit 30 learns a teacher encoder that estimates a latent vector sequence using the spectrogram and attribute class of the sound as input, based on the spectrogram and attribute class of each constituent sound input as learning data, and a teacher decoder that generates the variance of the sound spectrogram using the latent vector sequence and attribute class as input, and stores the trained data in the sound source signal model storage unit 34.

[0056] Specifically, the teacher learning unit 30 learns the teacher encoder and the teacher decoder so as to maximize the value of the objective function of the above formula (12), which is expressed using the error between the power spectrogram generated by the teacher decoder and the power spectrogram in the signal of the original constituent sound, and the distance between the latent vector sequence estimated by the teacher encoder and the latent vector sequence in the signal of the original constituent sound, for each constituent sound, and stores the trained results in the sound source signal model storage unit 34. Here, each of the teacher encoder and the teacher decoder is configured using a convolutional network or a recurrent network.

[0057] The learning unit 32 learns an encoder that estimates a latent vector sequence using the spectrogram of the sound as input, based on the spectrogram and attribute class for each constituent sound input as learning data, a classifier that identifies the attribute class using the spectrogram of the sound as input, and a decoder that generates the variance of the sound spectrogram using the latent vector sequence and the attribute class as input.

[0058] Specifically, the learning unit 32 learns the encoder, the classifier, and the decoder so as to maximize the criteria of the above formula (26), which includes a learning criterion for evaluating the output of the encoder and the output of the decoder, the mutual information of the decoder output and the attribute class, a reconstruction criterion for evaluating a spectrogram generated using the output of the decoder to which the output of the encoder and the output of the classifier are input, a class classification criterion for evaluating the output of the classifier to which the spectrogram generated using the output of the decoder to which the output of the encoder and the output of the classifier are input, and a knowledge distillation criterion for matching the output of the encoder and the output of the trained teacher encoder and for matching the output of the decoder and the output of the trained teacher decoder, and stores the learned model in the sound source signal model storage unit 34. Here, the encoder and the classifier are an integrated neural network, and the encoder and the classifier share some layers. Each of the encoder, the classifier, and the decoder is configured using a convolutional network or a recurrent network.

[0059] The parameter estimation unit 36 ​​estimates a separation matrix and a scale parameter based on the power spectrogram of the observation signal, using an observation signal in which each constituent sound is mixed as an input, so as to optimize the objective function of the above-mentioned equation (8) represented using the latent vector sequence estimated for each constituent sound separated by a separation matrix by the trained encoder, the attribute class identified for each constituent sound separated by a separation matrix by the trained classifier, the spectrogram of the constituent sound calculated from the variance of the spectrogram of the constituent sound and the scale parameter generated by the trained decoder for each constituent sound, the scale parameter of the spectrogram of each constituent sound, the separation matrix for separating a mixed sound in which each constituent sound is mixed in the time-frequency domain into each constituent sound, and the signal obtained by separating the observation signal into each constituent sound.

[0060] Specifically, the parameter estimation unit 36 ​​includes an initial value setting unit 40, a separating matrix updating unit 42, a latent variable class updating unit 44, a scale parameter updating unit 46, and a convergence determination unit 48.

[0061] The initial value setting unit 40 sets initial values ​​for the separation matrix, the latent vector sequence of each constituent tone, the attribute class of each constituent tone, and the scale parameter of each constituent tone.

[0062] The separation matrix update unit 42 updates the separation matrix in accordance with the above formulas (13) and (14) based on the power spectrogram of the observed signal, and the previously updated or initialized latent vector series of each constituent sound, the attribute class of each constituent sound, the scale parameter of each constituent sound, and the separation matrix, so as to increase the objective function shown in the above formula (8).

[0063] The latent variable class update unit 44 updates the latent vector series of each constituent sound obtained by using the average of the Gaussian distribution of the output of the encoder to which the power spectrogram of each constituent sound obtained by using the observation signal and the separation matrix is ​​input, and updates the attribute class of each constituent sound by using the output of a classifier to which the power spectrogram of each constituent sound obtained by using the observation signal and the separation matrix is ​​input.

[0064] The scale parameter update unit 46 updates the scale parameters according to the above formula (15) based on the power spectrogram of the observed signal, the updated latent vector series of each constituent sound, the attribute class of each constituent sound, the scale parameter of each constituent sound, and the separation matrix, so as to increase the objective function shown in the above formula (8).

[0065] The convergence determination unit 48 determines whether or not the convergence condition is satisfied, and causes the update process in the separating matrix update unit 42, the update process in the latent variable class update unit 44, and the update process in the scale parameter update unit 46 to be repeated until the convergence condition is satisfied.

[0066] The convergence condition may be, for example, that the number of iterations reaches an upper limit, or that the difference between the value of the objective function in the above formula (8) and the value of the previous objective function is equal to or less than a predetermined threshold.

[0067] The output unit 38 obtains a power spectrogram of each constituent sound generated using a decoder based on the latent vector series of each constituent sound, the attribute class of each constituent sound, and the scale parameter of each constituent sound acquired in the parameter estimation unit 36, and generates and outputs a signal of each constituent sound from the power spectrogram of each constituent sound.

[0068] <Function of the signal analysis device according to the present embodiment> Next, the operation of the signal analyzing device 100 according to this embodiment will be described.

[0069] 6 is a flowchart showing the flow of the learning process by the signal analyzing device 100. The learning process is performed by the CPU 11 reading out a learning program from the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it. In addition, for each of a plurality of constituent tones, time-series data of the signal of the constituent tone and an attribute class indicating the attribute of the signal of the constituent tone are input to the signal analyzing device 100 as learning data.

[0070] First, in step S100, the CPU 11, functioning as the time-frequency expansion section 24, calculates, for each constituent sound, a power spectrogram that represents a spectrum at each time based on the time-series data of the signal of that constituent sound.

[0071] In the next step S102, the CPU 11, as the teacher learning unit 30, learns a teacher encoder that estimates a latent vector sequence using the spectrogram and attribute class of the sound as input, based on the spectrogram and attribute class for each constituent sound input as learning data, and a teacher decoder that generates the variance of the sound spectrogram using the latent vector sequence and attribute class as input.

[0072] In step S104, the CPU 11, as the learning unit 32, learns an encoder that estimates a latent vector sequence by taking the spectrogram of the sound as input, based on the spectrogram and attribute class for each constituent sound input as learning data, a classifier that identifies the attribute class by taking the spectrogram of the sound as input, and a decoder that generates the variance of the sound spectrogram by taking the latent vector sequence and the attribute class as input, and stores parameters of the trained encoder, classifier, and decoder in the sound source signal model storage unit 34.

[0073] 7 is a flowchart showing the flow of signal analysis processing by the signal analyzing device 100. The signal analysis processing is performed by the CPU 11 reading out a signal analysis program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. In addition, time-series data of an observed signal in which each component sound is mixed is input to the signal analyzing device 100.

[0074] First, in step S120, the CPU 11, functioning as the time-frequency expanding unit 24, calculates a power spectrogram representing the spectrum at each time based on the time-series data of the observed signal.

[0075] In step S122, the CPU 11, functioning as the initial value setting unit 40, sets initial values ​​to the separation matrix, the latent vector sequence of each constituent tone, the attribute class of each constituent tone, and the scale parameter of each constituent tone.

[0076] In step S124, the CPU 11, as the separation matrix update unit 42, updates the separation matrix in accordance with the above equations (13) and (14) based on the power spectrogram of the observed signal calculated in the above step S120, and the previously updated or initialized latent vector series of each constituent sound, the attribute class of each constituent sound, the scale parameter of each constituent sound, and the separation matrix, so as to increase the objective function shown in the above equation (8).

[0077] In step S126, the CPU 11, as the latent variable class update unit 44, updates the latent vector series of each constituent sound to a latent vector series of each constituent sound obtained by using the average of the Gaussian distribution of the output of an encoder to which is input the power spectrogram of each constituent sound obtained by using the observation signal and the separation matrix, and updates the attribute class of each constituent sound by using the output of a classifier to which is input the power spectrogram of each constituent sound obtained by using the observation signal and the separation matrix.

[0078] In step S128, the CPU 11, as the scale parameter update unit 46, updates the scale parameters according to the above formula (15) based on the power spectrogram of the observed signal calculated in the above step S120, the updated latent vector series of each constituent sound, the attribute class of each constituent sound, the scale parameter of each constituent sound, and the separation matrix, so as to increase the objective function shown in the above formula (8).

[0079] Next, in step S130, it is determined whether or not the convergence condition is satisfied. If the convergence condition is satisfied, the process proceeds to step S132, and if the convergence condition is not satisfied, the process proceeds to step S124, and the processes of steps S124 to S128 are repeated.

[0080] In step S132, a power spectrogram of each constituent sound is generated using a decoder based on the latent vector series of each constituent sound, the attribute class of each constituent sound, and the scale parameter of each constituent sound, which were finally updated in steps S124 to S128 above. A signal of each constituent sound is generated from the power spectrogram of each constituent sound and output from output unit 38, thereby completing the signal analysis process.

[0081] <Experimental Results> In order to verify the sound source separation performance by the method of this embodiment, a speaker-dependent separation experiment using the Voice Conversion Challenge (VCC) 2018 speech database and an arbitrary speaker separation experiment using the WSJ0 speech database were performed. The comparison targets were the ILRMA described in Non-Patent Document 1, the MVAE method described in Non-Patent Document 2, and the FastMVAE method described in Non-Patent Document 3, and the source-to-distortions ratio (SDR), source-to-interferences ratio (SIR), and sources-to-artifacts ratio (SAR) were used as evaluation criteria. In all methods, the separation matrix W(f) was initialized to a unit matrix and updated 60 times.

[0082] The basis number of ILRMA was set to 2. Table 1 shows the number of parameters for each model.

[0083] [Table 1]

[0084] ChimeraACVAE was able to reduce the number of parameters by up to 40% compared to ACVAE. Table 2 shows the experimental results.

[0085] [Table 2]

[0086] Under all conditions, the method of this embodiment (FastMVAE2 method) showed higher separation performance than the ILRMA and FastMVAE methods, significantly narrowing the difference with the MVAE method.

[0087] To evaluate the separation performance and calculation time of each method when the number of sound sources is more than two, a mixed signal with the number of sound sources {2,3,6,9} was created using the speech of 18 speakers from the WSJ0 speech database. The impulse response was created using the mirror method, and the wall reflection coefficient was set to 0.2. Figure 8 shows the arrangement of the microphones and sound sources. A mixed signal was created for 10 sentences for each condition. All processing was performed using an Intel(R) Xeon(R) Gold 6130 CPU@2.10GHz and a Tesla V100 GPU. Table 3 shows the average SDR for each condition.

[0088] [Table 3]

[0089] In addition, the calculation time for each iteration of each method is shown in Fig. 9. Performance improvement was confirmed in the method of this embodiment (FastMVAE2, FastMVAE2_CPU, FastMVAE2_GPU). It was also confirmed that the method of this embodiment can achieve separation in the same calculation time as ILRMA when there are three or fewer sound sources, and can achieve separation in a shorter calculation time than ILRMA when there are three or more sound sources.

[0090] As described above, the signal analysis device according to the present embodiment learns an encoder that estimates a latent vector sequence by taking a spectrogram of a sound as an input based on the spectrogram and attribute class of each constituent sound, a classifier that identifies an attribute class indicating an attribute of the sound by taking the spectrogram of the sound as an input, and a decoder that generates a variance of the spectrogram of the sound by taking the latent vector sequence and the attribute class as an input.The signal analysis device then learns a separation matrix and a scale parameter so as to optimize an objective function expressed by using an observation signal in which each constituent sound is mixed, the latent vector sequence estimated for each constituent sound separated by a separation matrix by the trained encoder, the attribute class identified for each constituent sound separated by a separation matrix by the trained classifier, the variance of the spectrogram of each constituent sound generated by the trained decoder for each constituent sound, the scale parameter of the spectrogram of each constituent sound, the separation matrix for separating a mixed sound in which each constituent sound is mixed in the time-frequency domain into each constituent sound, and the signal obtained by separating the observation signal into each constituent sound. This makes it possible to reduce calculation costs and accurately separate each of the constituent sounds from a mixed signal in which the constituent sounds are mixed.

[0091] <Modification> The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0092] For example, the case where the power spectrogram of the observed signal and the power spectrogram of the constituent sounds are calculated has been described as an example, but the present invention is not limited to this, and the amplitude spectrogram of the observed signal and the amplitude spectrogram of the constituent sounds may be calculated. In this case, the learning unit 32 learns an encoder that estimates a latent vector sequence by inputting the amplitude spectrogram of the sound based on the amplitude spectrogram and attribute class of each constituent sound, a discriminator that identifies the attribute class by inputting the amplitude spectrogram of the sound, and a decoder that generates the variance of the amplitude spectrogram of the sound by inputting the latent vector sequence and the attribute class. In addition, the parameter estimation unit 36 ​​estimates the separation matrix and the scale parameter so as to optimize an objective function represented by an input of the observation signal, the latent vector sequence estimated by the trained encoder, the attribute class identified by the trained classifier, the amplitude spectrogram of the constituent sounds calculated from the variance of the amplitude spectrogram of the constituent sounds generated by the trained decoder for each constituent sound and the scale parameter, the scale parameter of the amplitude spectrogram of each constituent sound, a separation matrix for separating a mixed sound in which the constituent sounds are mixed in the time-frequency domain into each constituent sound, and a signal obtained by separating the observation signal into each constituent sound.

[0093] Furthermore, the order in which the parameters are updated is arbitrary and is not limited to the order in the above embodiment.

[0094] In addition, various types of processing that the CPU reads and executes software (programs) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and dedicated electric circuits that are processors having a circuit configuration designed exclusively for executing specific processing, such as ASICs (Application Specific Integrated Circuits). Furthermore, the learning process and the signal analysis process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). Moreover, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0095] In addition, in each of the above embodiments, the learning program and the signal analysis program are described as being pre-stored (installed) in the storage 14, but the present invention is not limited to this. The programs may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network.

[0096] The following supplementary notes are further disclosed regarding the above embodiment.

[0097] (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, An encoder that estimates a latent vector sequence by taking a spectrogram of a sound as an input based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound, a classifier that identifies an attribute class indicating an attribute of the sound by taking the spectrogram of the sound as an input, and a decoder that generates a variance of the spectrogram of the sound by taking the latent vector sequence and the attribute class as input, The separation matrix and the scale parameter are estimated so as to optimize an objective function represented by using an observation signal in which each constituent sound is mixed, the latent vector sequence estimated for each constituent sound separated by the separation matrix by the trained encoder, the attribute class identified for each constituent sound separated by the separation matrix by the trained classifier, a spectrogram of the constituent sound calculated from the variance of the spectrogram of the constituent sound and a scale parameter generated by the trained decoder for each constituent sound, a scale parameter of the spectrogram of each constituent sound, a separation matrix for separating a mixed sound in which each constituent sound is mixed into each constituent sound in the time-frequency domain, and a signal obtained by separating the observation signal into each constituent sound. A signal analysis device configured as follows.

[0098] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform a signal analysis process, The signal analysis process includes: An encoder that estimates a latent vector sequence by taking a spectrogram of a sound as an input based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound, a classifier that identifies an attribute class indicating an attribute of the sound by taking the spectrogram of the sound as an input, and a decoder that generates a variance of the spectrogram of the sound by taking the latent vector sequence and the attribute class as input, The separation matrix and the scale parameter are estimated so as to optimize an objective function represented by using an observation signal in which each constituent sound is mixed, the latent vector sequence estimated for each constituent sound separated by the separation matrix by the trained encoder, the attribute class identified for each constituent sound separated by the separation matrix by the trained classifier, a spectrogram of the constituent sound calculated from the variance of the spectrogram of the constituent sound and a scale parameter generated by the trained decoder for each constituent sound, a scale parameter of the spectrogram of each constituent sound, a separation matrix for separating a mixed sound in which each constituent sound is mixed into each constituent sound in the time-frequency domain, and a signal obtained by separating the observation signal into each constituent sound. Non-transitory storage medium. [Explanation of symbols]

[0099] 11 CPU 14. Storage 15 Input section 16 Display 24 Time-frequency expansion section 30 Teacher Learning Department 32 Learning Department 34 Sound source signal model storage unit 36 Parameter Estimation Section 38 Output section 40 Initial value setting section 42 Separation matrix update unit 44 Latent variable class update section 46 Scale parameter update section 48 Convergence judgment section 100 Signal analysis equipment

Claims

1. a learning unit that learns an encoder that estimates a latent vector sequence by using only a spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class that indicates an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input and identifies an attribute class that indicates an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound; a parameter estimation unit that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from the variance of the spectrogram of the constituent sounds and the scale parameter of the spectrogram of each constituent sound, which are generated by the learned decoder for each constituent sound, and a signal obtained by separating the observation signal into each constituent sound; and A signal analysis device comprising: The system further includes a teacher learning unit that learns a teacher encoder that estimates a latent vector sequence using the spectrogram and the attribute class of a sound as input, based on the spectrogram and the attribute class of each component sound, and a teacher decoder that generates a variance of the spectrogram of the sound using the latent vector sequence and the attribute class as input, The learning unit is a signal analysis device that learns the encoder and the decoder so that the output of the encoder corresponds to the output of the learned teacher encoder, and so that the output of the decoder corresponds to the output of the learned teacher decoder.

2. a learning unit that learns an encoder that estimates a latent vector sequence by using only a spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class that indicates an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input and identifies an attribute class that indicates an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound; a parameter estimation unit that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from the variance of the spectrogram of the constituent sounds and the scale parameter of the spectrogram of each constituent sound, which are generated by the learned decoder for each constituent sound, and a signal obtained by separating the observation signal into each constituent sound; and A signal analysis device comprising: The signal analysis device, wherein the encoder and the classifier are an integrated neural network, and the encoder and the classifier share some layers.

3. a learning unit that learns an encoder that estimates a latent vector sequence by using only a spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class that indicates an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input and identifies an attribute class that indicates an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound; a parameter estimation unit that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from the variance of the spectrogram of the constituent sounds and the scale parameter of the spectrogram of each constituent sound, which are generated by the learned decoder for each constituent sound, and a signal obtained by separating the observation signal into each constituent sound; and A signal analysis device comprising: The learning unit is a learning criterion for evaluating the output of the encoder and the output of the decoder; Mutual information of the decoder output and the attribute classes; a reconstruction criterion for evaluating the spectrogram generated using the output of the decoder to which the output of the encoder and the output of the classifier are input; a class identification criterion for evaluating an output of the classifier using as input the spectrogram generated using the output of the decoder using as input the output of the encoder and the output of the classifier; The signal analysis apparatus trains the encoder, the classifier, and the decoder so as to optimize a criterion including:

4. The parameter estimation unit an initial value setting unit that sets initial values ​​to the separating matrix, the latent vector sequence, the attribute class, and the scale parameter; a separation matrix update unit that updates the separation matrix so as to increase the objective function based on a spectrogram of the observed signal, the latent vector sequence, the attribute class, the scale parameter, and the separation matrix that have been previously updated or have initial values ​​set; a latent variable class update unit that updates the latent vector sequence to the latent vector sequence obtained by using an output of the encoder to which a spectrogram of each component sound obtained by using the observation signal and the separation matrix is ​​input, and updates the attribute class by using an output of a classifier to which a spectrogram of each component sound obtained by using the observation signal and the separation matrix is ​​input; a scale parameter update unit that updates the scale parameter so as to increase the objective function based on a spectrogram of the observed signal and the updated latent vector sequence, the attribute class, the scale parameter, and the separating matrix; a convergence determination unit that determines whether a predetermined convergence condition is satisfied and repeats the update by the separating matrix update unit, the update by the latent variable class update unit, and the update by the scale parameter update unit until the convergence condition is satisfied; The signal analysis device according to any one of claims 1 to 3, comprising:

5. a learning unit learns an encoder that estimates a latent vector sequence by taking only the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input to identify an attribute class indicating an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input to generate a variance of the spectrogram of the sound; a parameter estimation unit receives as input an observation signal in which the constituent sounds are mixed, and receives as input the latent vector sequence estimated for each constituent sound separated by a separation matrix for separating a mixed sound in which the constituent sounds are mixed in a time-frequency domain into each constituent sound by the trained encoder, and the attribute class identified by the trained classifier for each constituent sound separated by the separation matrix, and estimates the separation matrix and the scale parameter so as to optimize an objective function expressed using a spectrogram of the constituent sound, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from a variance of the spectrogram of the constituent sound and a scale parameter of the spectrogram of each constituent sound, which are generated by the trained decoder for each constituent sound.

1. A signal analysis method, comprising: The teacher learning unit further includes learning a teacher encoder that estimates a latent vector sequence using the spectrogram and the attribute class of a sound as input based on the spectrogram and the attribute class for each component sound, and a teacher decoder that generates a variance of the spectrogram of the sound using the latent vector sequence and the attribute class as input, A signal analysis method in which the learning unit learns the encoder and the decoder so that the output of the encoder corresponds to the output of the trained teacher encoder, and so that the output of the decoder corresponds to the output of the trained teacher decoder.

6. a learning unit learns an encoder that estimates a latent vector sequence by taking only the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input to identify an attribute class indicating an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input to generate a variance of the spectrogram of the sound; a parameter estimation unit receives as input an observation signal in which the constituent sounds are mixed, and receives as input the latent vector sequence estimated for each constituent sound separated by a separation matrix for separating a mixed sound in which the constituent sounds are mixed in a time-frequency domain into each constituent sound by the trained encoder, and the attribute class identified by the trained classifier for each constituent sound separated by the separation matrix, and estimates the separation matrix and the scale parameter so as to optimize an objective function expressed using a spectrogram of the constituent sound, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from a variance of the spectrogram of the constituent sound and a scale parameter of the spectrogram of each constituent sound, which are generated by the trained decoder for each constituent sound.

1. A signal analysis method, comprising: A signal analysis method in which the encoder and the classifier are an integrated neural network, and the encoder and the classifier share some layers.

7. a learning unit learns an encoder that estimates a latent vector sequence by taking only the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input to identify an attribute class indicating an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input to generate a variance of the spectrogram of the sound; a parameter estimation unit receives as input an observation signal in which the constituent sounds are mixed, and receives as input the latent vector sequence estimated for each constituent sound separated by a separation matrix for separating a mixed sound in which the constituent sounds are mixed in a time-frequency domain into each constituent sound by the trained encoder, and the attribute class identified by the trained classifier for each constituent sound separated by the separation matrix, and estimates the separation matrix and the scale parameter so as to optimize an objective function expressed using a spectrogram of the constituent sound, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from a variance of the spectrogram of the constituent sound and a scale parameter of the spectrogram of each constituent sound, which are generated by the trained decoder for each constituent sound.

1. A signal analysis method, comprising: The learning unit is a learning criterion for evaluating the output of the encoder and the output of the decoder; Mutual information of the decoder output and the attribute classes; a reconstruction criterion for evaluating the spectrogram generated using the output of the decoder to which the output of the encoder and the output of the classifier are input; a class identification criterion for evaluating an output of the classifier using as input the spectrogram generated using the output of the decoder using as input the output of the encoder and the output of the classifier; The signal analysis method further comprises training the encoder, the classifier, and the decoder to optimize a criterion including:

8. A signal analysis program for causing a computer to function as each unit of the signal analysis device according to any one of claims 1 to 4.

Images