Audio signal enhancement device, method and program

Abstract

To provide acoustic signal enhancement technology that optimizes an overall process.SOLUTION: An acoustic signal enhancement device consists of: a reverberation suppression unit 2 that receives, at each time t, an observed signal vector at the time t and an inverse matrix of a time-space covariance matrix obtained at time t-1, and generates a reverberation suppression signal vector corresponding to the observed signal vector; a sound source separation unit 3 that receives, at each time t, the generated reverberation suppression signal vector, and obtains an enhanced sound of sound source n and power of the sound source n; and a time-space parameter updating unit 4 that receives, at each time t, the power of the sound source n and the observed signal vector, and obtains the inverse matrix of the time-space covariance matrix corresponding to the sound source n.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an audio signal enhancement technology that suppresses reverberation and separates individual sounds from an audio signal that is a mixture of multiple sounds and their reverberations and is collected by multiple microphones, without any prior information about each component sound. [Background technology]

[0002] Conventionally, online dereverberation methods have been devised that suppress the reverberation related to all component sounds based on online processing in a situation where there is no prior information related to each component sound (see, for example, Non-Patent Document 1).

[0003] In addition, an online sound source separation method has been devised that separates a mixed sound that does not contain reverberation into individual sounds (for example, Non-Patent Document 2).

[0004] Therefore, by connecting these steps as a dereverberation step and a sound source separation step, as shown in Fig. 6, it has been possible to construct an acoustic signal enhancement method that suppresses the reverberation of all component sounds and separates them into individual sounds through online processing. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] J. Caroselli, et al. “Adaptive multichannel dereverberation for automatic speech recog-nition.”, inProc. Interspeech, 2017, pp. 3877-3881. [Non-Patent Document 2] T. Taniguchi, et al. “An anauxiliary-function approach to online independent vector anal-ysis for real-time blind source separation”, inProc. HSCMA, 2014, pp. 107-111. Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the conventional method, the dereverberation step is performed independently of the subsequent sound source separation step, so that it is not possible to perform optimal processing overall when simultaneously performing dereverberation and sound source separation.

[0007] An object of the present invention is to provide an apparatus, method and program for enhancing an audio signal, which performs optimal processing overall. [Means for solving the problem]

[0008] The acoustic signal enhancement device according to an embodiment of the present invention receives, at each time t, an observed signal vector at time t and an inverse matrix of a time-space covariance matrix calculated at time t−1, and calculates a vector corresponding to the observed signal vector. , corresponding to sound source n or common to all sound sources A dereverberation unit that generates a dereverberation signal vector and a dereverberation unit that generates the dereverberation signal vector at each time t. Corresponding to sound source n or common to all sound sources The dereverberation signal vector is Using and a sound source separation unit that obtains an emphasis sound of sound source n and the power of sound source n, and a space-time parameter update unit that receives the power of sound source n and an observed signal vector at each time t and obtains an inverse matrix of a space-time covariance matrix corresponding to sound source n.

[0009] In an audio signal enhancement device according to an embodiment of the present invention, t is a time frame number, f is a frequency number, N is the number of sound sources, M is the number of microphones, n=1, . . . , N, and m=1, . . . , M. A dereverberation filter G n(f;t-1) and the observed signal x of microphone m m (f,t) and the observed signal vector X(f,t) m The dereverberation signal y corresponding to (f,t) n,m The dereverberation signal vector Y for the emphasized sound of sound source n, which is composed of (f,t), n The dereverberation unit generates (f,t) and the sound source separation filter Q generated one time before corresponding to the sound source n n (f;t-1) and the generated dereverberation signal vector Y n Using (f,t), the emphasis sound s of the sound source n n (f,t) and the power v of the sound source n n The source separation unit that obtains (t) and the power v of the sound source n n (t), the observed signal vector X(f,t), and the inverse matrix R of the time-space covariance matrix obtained one time before corresponding to the sound source n n -1 (f;t-1) and the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 The dereverberation unit includes a spatio-temporal parameter update unit that calculates a dereverberation filter G n (f;t-1) and the Kalman gain K corresponding to the sound source n n (f,t) and the power v of the sound source n n Using (t), the dereverberation filter G corresponding to the sound source n is n (f;t), and the sound source separation unit calculates the dereverberation signal vector Y n (f,t) and the power v of the sound source n n Using (t), the spatial covariance matrix Σ corresponding to the sound source n n (f,t) is calculated, and the spatial covariance matrix Σ corresponding to the obtained sound source n is calculated. n (f,t) and the sound source separation filter Q generated one time before corresponding to the sound source n n (f;t-1) and the sound source separation filter Q n Find (f;t).

[0010] In an audio signal enhancement device according to an embodiment of the present invention, t is a time frame number, f is a frequency number, N is the number of sound sources, M is the number of microphones, n=1, . . . , N, and m=1, . . . , M. A dereverberation filter G(f;t-1) obtained one time before and an observation signal x of microphone m are m (f,t) and the observed signal vector X(f,t) m The dereverberation signal y corresponding to (f,t) m The dereverberation unit generates a dereverberation signal vector Y(f,t) composed of (f,t), a sound source separation filter Q(f;t-1) generated one time before, and the generated dereverberation signal vector Y(f,t) are used to generate an emphasized sound s n The emphasis sound vector S(f,t) and the power v of the sound source n are n The source separation unit that obtains (t) and the power v of the sound source n n (t), the observed signal vector X(f,t), and the inverse matrix R of the time-space covariance matrix obtained one time before corresponding to the sound source n n -1 (f;t-1) and the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 The dereverberation unit includes a spatio-temporal parameter update unit that calculates a dereverberation filter G n (f;t-1) and the Kalman gain K corresponding to the sound source n n (f,t) and the power v of the sound source n n Using (t), the dereverberation filter G corresponding to the sound source n is n (f;t) is calculated, and the dereverberation filter G corresponding to each sound source n is calculated. n The dereverberation filter G(f;t) is calculated using the dereverberation signal vector Y(f,t) and the sound source separation filter Q(f;t-1) generated one time before. The sound source separation unit calculates the dereverberation signal vector Y(f,t) and the power v of the sound source n. n Using (t), the spatial covariance matrix Σ corresponding to the sound source n n (f,t) is calculated, and the spatial covariance matrix Σ corresponding to the obtained sound source n is calculated. n(f,t) and the sound source separation filter Q generated one time before corresponding to the sound source n n (f;t-1) and the sound source separation filter Q n Find (f;t). Effect of the Invention

[0011] Overall, optimal processing can be achieved. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a functional configuration of an acoustic signal enhancing device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a processing procedure of the audio signal enhancement method. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional configuration of the acoustic signal enhancing device according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a functional configuration of an acoustic signal enhancement device according to a superordinate concept of the first and second embodiments. [Diagram 5] FIG. 5 is a diagram illustrating an example of a functional configuration of a computer. [Figure 6] FIG. 6 is a diagram for explaining the background art. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described in detail. In the drawings, components having the same functions are designated by the same reference numerals, and duplicated explanations will be omitted.

[0014] [First embodiment] The audio signal enhancement device of the first embodiment uses a different dereverberation filter G for each sound source. n Dereverberation is performed using (f,t).

[0015] The acoustic signal enhancing device according to the first embodiment includes, for example, an initialization unit 1, a reverberation reduction unit 2, a sound source separation unit 3, and a spatio-temporal parameter update unit 4, as shown in FIG.

[0016] The acoustic signal enhancement method of the first embodiment is realized, for example, by each component of the acoustic signal enhancement device performing processes from step S1 to step S6 described below and shown in FIG.

[0017] In addition, the symbol " - " should be written directly above the character immediately following it, but due to limitations in text notation, it is written immediately before the character in question. In mathematical formulas, these symbols are written in their original position, that is, directly above the character. For example, " - X" is written in the formula as follows:

number

[0018] M is the number of microphones, and m (1≦m≦M) is the microphone number. M is a positive integer of 2 or more.

[0019] N is the number of sound sources, and n (1≦n≦N) is the number of the sound source. N is a positive integer of 2 or more.

[0020] The microphone number and the sound source number are indicated by subscripts at the bottom right.

[0021] t, τ (1≦t, τ≦T) are the time frame numbers. T is the total number of time frames and is a positive integer equal to or greater than 2.

[0022] f (1≦f≦F) is the frequency number. F is the frequency corresponding to the highest frequency bin.

[0023] (·) T is the anti-conjugate transpose of a matrix or vector, (·) H is the conjugate transpose of a matrix or vector. · is any matrix or vector.

[0024] The lowercase letters are scalar variables. For example, the observed signal x at microphone m at time t and frequency f is m (f,t) are scalar variables.

[0025] Capital letters represent vectors or matrices. For example, X(f,t)=[x1(f,t),x2(f,t),…,x N (f,t)] T ∈C M×1 is the observed signal vector at all microphones at time t and frequency f.

[0026] C M×N is the set of all M×N dimensional complex matrices. X∈C M×N is a notation that indicates that it is an element. That is, X is C M×N Indicates that this is an element of

[0027] - X(f,t) is a vector of past observed signal time series from time tD-L+1 to time tD, - X(f,t)=[X(f,tD) T ,x(f,t-D+1) T ,…,x N (f,tD-L+1) T ] T ∈C ML×1 D is the predicted delay, which is a positive integer equal to or greater than 1.

[0028] v n (t) is the power of source n at time t, a scalar.

[0029] s n (f,t) is the emphasis sound of sound source n at time t and frequency f, and is a scalar.

[0030] G n (f,t)∈C M(L-D)×M ,G(f,t)∈C M(L-D)×M is the estimate of the dereverberation filter for sound source n at frequency f at time t. L is the filter order, a positive integer equal to or greater than 2.

[0031] Q(f;t)=[Q1(f;t), Q2(f;t),…, Q N (f;t)] T ∈C M×N is the separation matrix for frequency f, and Q n (f;t) is the source separation filter for source n at frequency f.

[0032] R n -1 (f;t)∈C M(L-D)×M(L-D) is the inverse of the spatiotemporal covariance matrix for source n at frequency f and time t.

[0033] K n (f,t)∈C M(L-D)×1 is the Kalman gain for source n at frequency f and time t.

[0034] Each component of the acoustic signal enhancement device will be described below. Note that, in the following description, the time t and frequency f are assumed to be given, but in reality, the process described below is performed for each frequency f at each time t. That is, the acoustic signal enhancement device sequentially receives an acoustic signal that has been frequency-divided by short-time Fourier transform or the like, and performs the following process for each time t and each frequency f.

[0035] <Initialization part 1> For n=1,...,N, the initialization unit 1 initializes the dereverberation filter G n All elements of (f;0) are set to a predetermined value (e.g., 0), and the sound source separation filter Q(f;0) = [Q1(f;0),Q2(f;0),…,Q N (f;0)] and the inverse of the time-space covariance matrix R n -1 (f;0) and the spatial covariance matrix Σ of the dereverberated sound for each sound source n Let (f, 0) be a predetermined matrix (for example, a unit matrix).

[0036] The predetermined value may be a value other than 0. The predetermined matrix may be a matrix other than an identity matrix.

[0037] The initialized dereverberation filter G n (f;0) is output to the dereverberation unit 2 and stored therein.

[0038] The initialized source separation filter Q(f;0) and spatial covariance matrix Σ n (f, 0) is output to the sound source separation unit 3 and stored.

[0039] Inverse of time-space covariance matrix R n -1 (f;0) is output to the spatio-temporal parameter update unit 4 and stored therein.

[0040] The process of the initialization unit 1 is performed only once at time t = 0. The processes of the dereverberation unit 2, the sound source separation unit 3 and the spatio-temporal parameter update unit 4 described below are performed at each time t.

[0041] <Dereverberation section 2 (first processing)> The dereverberation unit 2 performs a first process and a second process, which will be described later. Here, the first process performed by the dereverberation unit 2 will be described.

[0042] The dereverberation unit 2 receives the observed signal x m An observation signal vector X(f,t) consisting of (f,t) is input.

[0043] For n=1,...,N, the dereverberation unit 2 uses the dereverberation filter G n (f;t-1) and the observed signal vector X(f,t), the observed signal x m The dereverberation signal y corresponding to (f,t) n,m The dereverberation signal vector Y for the emphasized sound of sound source n, which is composed of (f,t), n (f, t) is generated (step S2).

[0044] That is, the dereverberation unit 2 calculates the dereverberation signal vectors Y1(f,t), ..., Y N (f,t), where Y n(f,t)=[y n,1 (f,t),y n,2 (f,t),…,y n,M (f,t)], where m = 1,…,M, y n,m (f,t) is the observed signal x for the emphasis sound of sound source n m is the dereverberated signal corresponding to (f,t).

[0045] The generated dereverberation signal vector Y n (f, t) is output to the sound source separation unit 3.

[0046] The dereverberation filter G obtained one time before corresponding to the sound source n n (f;t-1) is stored in the dereverberation unit 2. The dereverberation unit 2 uses the dereverberation filter G n Processing is performed using (f;t-1).

[0047] The dereverberation unit 2 calculates the dereverberation signal vector Y n Find (f,t).

number

[0048] The sound source separation unit 3 receives the dereverberation signal vector Y n (f,t) is input.

[0049] For n=1,...,N, the sound source separation unit 3 uses a sound source separation filter Q generated one time before and corresponding to the sound source n. n (f;t-1) and the dereverberation signal vector Y n Using (f,t), the emphasis sound s of the sound source n n (f,t) and the power v of the sound source n n (t) is calculated (step S3).

[0050] That is, the sound source separation unit 3 separates the emphasized sounds s1(f,t), ..., s N (f,t) and v1(f),…,v corresponding to each sound source 1,…,N N Generate (t).

[0051] Emphasis sound s of generated sound source n n (f,t) is the output of the audio signal enhancement device. The power v of the generated sound source n n (t) is output to the spatio-temporal parameter update unit 4.

[0052] The source separation filter Q obtained one time before corresponding to the sound source n n (f;t-1) is stored in the sound source separation unit 3. The sound source separation unit 3 uses the sound source separation filter Q obtained one time before corresponding to the stored sound source n. n Processing is performed using (f;t-1).

[0053] The sound source separation unit 3 separates the emphasized sound s of the sound source n based on, for example, the following equation: n Find (f,t).

number

number

[0054] For n=1,...,N, the spatiotemporal parameter update unit 4 calculates the power v n (t), the observed signal vector X(f,t), and the inverse matrix R of the time-space covariance matrix obtained one time before corresponding to the sound source n n -1(f;t-1) and the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 (f;t) is calculated (step S4).

[0055] That is, the spatio-temporal parameter update unit 4 calculates the Kalman gains K1(f,t), ..., K N (f,t) and the inverse time-space covariance matrix R1 corresponding to each sound source 1,…,N -1 (f;t),…,R N -1 Find (f;t).

[0056] The Kalman gain K corresponding to the obtained sound source n n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 (f;t) is output to the dereverberation unit 2.

[0057] The inverse matrix R of the time-space covariance matrix obtained one time before corresponding to sound source n n -1 (f;t-1) is stored in the spatiotemporal parameter update unit 4. The spatiotemporal parameter update unit 4 updates the inverse matrix R n -1 Processing is performed using (f;t-1).

[0058] The spatio-temporal parameter update unit 4 calculates the Kalman gain K corresponding to the sound source n based on, for example, the following equation: n Find (f,t), where β is the forgetting factor, 0<β<1.

number

number

[0059] The dereverberation unit uses the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 (f;t) and the power v of the sound source n generated by the sound source separation unit 3 n (t) is entered.

[0060] For n=1,...,N, the dereverberation unit 2 uses the dereverberation filter G n (f;t-1) and the Kalman gain K corresponding to the sound source n n Using (f, t), the dereverberation filter G corresponding to the sound source n is n (f;t) is calculated (step S5).

[0061] That is, the dereverberation unit 2 includes dereverberation filters G1(f;t), ..., G N Find (f;t).

[0062] The dereverberation filter G corresponding to the obtained sound source n n (f;t) is stored in the dereverberation unit 2. The dereverberation filter G n (f;t) is used as a dereverberation filter for the previous time point corresponding to sound source n in the processing for the next time point t+1.

[0063] The dereverberation unit 2 calculates a dereverberation filter G corresponding to the sound source n based on, for example, the following equation: n Find (f;t).

number

[0064] For n=1,...,N, the sound source separation unit 3 separates the dereverberation signal vector Y n (f,t) and the power v of the sound source n n Using (t), the spatial covariance matrix Σ corresponding to the sound source n n (f,t) is calculated, and the spatial covariance matrix Σ corresponding to the obtained sound source n is calculated. n (f,t) and the sound source separation filter Q generated one time before corresponding to the sound source n n (f;t-1) and the sound source separation filter Q n (f;t) is calculated (step S6).

[0065] That is, the sound source separation unit 3 generates sound source separation filters Q1(f;t), ..., Q N Find (f;t).

[0066] The source separation filter Q corresponding to the obtained source n n (f;t) is stored in the sound source separation unit 3. The sound source separation filter Q n (f;t) is the sound source separation filter Q corresponding to sound source n at the next time t+1. n Used as (f;t).

[0067] The sound source separation unit 3 calculates the spatial covariance matrix Σ n Find (f,t). Here, L b is the block length, a positive integer.

number

number

number

number

[0068] In this way, the sound source separation results are fed back to the processing of the spatio-temporal parameter update unit 4, thereby making it possible to perform optimal processing overall. In addition, the Kalman gain K n (f,t) and the inverse of the time-space covariance matrix R n -1 By calculating (f;t) individually through online processing, it is no longer necessary to consider the relationship between the sound sources, so the size of the matrix required for optimization can be made smaller than in the background art, thereby reducing the overall computational cost.

[0069] In the first embodiment, in order to perform global optimization, all parameters are optimized using one optimization criterion. An example of the one optimization criterion is the criterion shown in the following formula (5).

number

[0070] In the above process, at each time, a dereverberation filter G that maximizes, for example, Eq. (5) is selected. n (f,t), sound source separation filter Q n (f,t), the power of each separated sound v n It can be said that (t) is calculated through online processing.

[0071] In addition, equation (5) states that (i) each separated sound has its power v n This criterion is derived based on the maximum likelihood method, taking into account, for example, the processing according to equations (1) and (2), under the assumption that (t) follows a time-varying complex Gaussian distribution, and (ii) that a forgetting factor β is used at each time instant that emphasizes the most recent observation signal (= forgets older observation signals from the past).

[0072] Since dereverberation and sound source separation are processed separately, modifications may be made such as optimizing using partially different criteria (for example, using different forgetting coefficients).

[0073] [Second embodiment] The audio signal enhancement device of the second embodiment differs from the audio signal enhancement device of the first embodiment in that it uses a dereverberation filter G(f,t−1) common to all sound sources to simultaneously dereverberate all sound sources and generate a dereverberation signal vector Y(f,t)∈C M×1 This is what is being sought.

[0074] The following description will focus on the differences from the acoustic signal enhancing device of the first embodiment, and will not repeat the same description of the same parts as in the first embodiment.

[0075] As shown in FIG. 3, the acoustic signal enhancement device of the second embodiment includes, for example, an initialization unit 1, a reverberation reduction unit 2, a sound source separation unit 3, and a spatio-temporal parameter update unit 4, similar to the acoustic signal enhancement device of the first embodiment.

[0076] <Initialization part 1> The initialization unit 1 initializes the dereverberation filter G(f;0) by setting all elements of the dereverberation filter G(f;0) to a predetermined value (for example, 0). Similarly to the first embodiment, the initialization unit 1 initializes the sound source separation filter Q(f;0) and the inverse matrix R n -1 (f;0) and the spatial covariance matrix Σ of the dereverberated sound for each sound source n Initialize (f,0).

[0077] <Dereverberation section 2 (first processing)> In the first process, the dereverberation unit 2 uses the dereverberation filter G(f;t-1) obtained one time ago and the observed signal x m (f,t) and the observed signal vector X(f,t) m The dereverberation signal y corresponding to (f,t) m A dereverberation signal vector Y(f,t) composed of (f,t) is generated (step S2).

[0078] Here, Y(f,t) = [y1(f,t), …, y M The dereverberation signal vector Y(f,t) can be considered as a dereverberation signal common to all sound sources.

[0079] The generated dereverberation signal vector Y(f,t) is output to the sound source separation unit 3.

[0080] The dereverberation unit 2 calculates a dereverberation signal vector Y(f,t) based on, for example, the following equation.

number

[0081] The sound source separation unit 3 calculates the emphasis sound vector S(f,t) based on, for example, the following formula: where S(f,t)=[s1(f,t),...,s N (f,t)]. Also, Q(f;t-1) = [Q1(f;t-1),…,Q N (f;t-1)].

number

[0082] The generated emphasis sound vector S(f,t) is composed of the emphasis sound s n (f,t) is the output of the audio signal enhancement device. The power v of the generated sound source n n (t) is output to the spatio-temporal parameter update unit 4.

[0083] <Spatial-temporal parameter update unit 4> The process of the spatio-temporal parameter update unit 4 is the same as that of the first embodiment. That is, for n=1,...,N, the spatio-temporal parameter update unit 4 updates the power v n (t), the observed signal vector X(f,t), and the inverse matrix R of the time-space covariance matrix obtained one time before corresponding to the sound source n n -1 (f;t-1) and the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 Calculate (f;t) (step S4) The Kalman gain K corresponding to the obtained sound source n n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 (f;t) is output to the dereverberation unit 2.

[0084] <Dereverberation section 2 (second processing)> In the second embodiment, the sound source separation filter Q(f;t) initialized by the initialization unit 1 or updated by the sound source separation unit 3 is also input to the dereverberation unit 2.

[0085] For n=1, . . . , N, the dereverberation unit 2 performs the second process by performing the same process as in the first embodiment to obtain a dereverberation filter G n (f;t-1) and the Kalman gain K corresponding to the sound source n n (f,t) and the power v of the sound source n n Using (t), the dereverberation filter G corresponding to the sound source n is n (f;t) is calculated (step S51).

[0086] Then, the dereverberation unit 2 calculates a dereverberation filter G corresponding to each sound source n. n A dereverberation filter G(f;t) is calculated using (f;t) and a sound source separation filter Q(f;t-1) generated one time before (step S52).

[0087] The dereverberation unit 2 calculates the dereverberation filter G(f;t) based on, for example, the following equation.

number

[0088] The dereverberation filter G(f,t) is a dereverberation filter common to all sound sources.

[0089] <Sound source separation unit 3 (second processing)> For n=1, ..., N, the sound source separation unit 3 performs a second process by extracting the dereverberation signal vector Y(f,t) and the power v n Using (t), the spatial covariance matrix Σ corresponding to the sound source n n (f, t) is calculated (step S61).

[0090] The sound source separation unit 3 of the second embodiment differs from the first embodiment in that, for example, the sound source separation unit 3 calculates the spatial covariance matrix Σ n Find (f,t).

number

[0091] The processes in steps S61 and S62 correspond to the process in step S6.

[0092] In the second embodiment, in order to perform global optimization, all parameters at each time are optimized based on one optimization criterion. An example of the optimization criterion is the criterion shown in formula (5) as in the first embodiment.

[0093] In the above process, the dereverberation filter G(f,t), the sound source separation filter Q(f;t), and the power v(t) of each separated sound that maximize, for example, equation (5) are found by online processing at each time.

[0094] As in the first embodiment, dereverberation and sound source separation are processed separately, and therefore modifications such as optimization using partially different criteria (eg, using different forgetting coefficients) may be added.

[0095] [Generic Concept of First and Second Embodiments] The acoustic signal enhancement device and method of the first and second embodiments have at least the following points in common.

[0096] In other words, the dereverberation unit 2, the sound source separation unit 3, and the spatio-temporal parameter update unit 4 of the audio signal enhancement device according to the first and second embodiments perform the following processes.

[0097] At each time t, the dereverberation unit 2 receives an observed signal vector at time t and the inverse matrix of the time-space covariance matrix calculated at time t-1, and generates a dereverberation signal vector corresponding to the observed signal vector.

[0098] The sound source separation unit 3 receives the dereverberation signal vector generated by the dereverberation unit 2 at each time t, and obtains an emphasized sound of the sound source n and the power of the sound source n.

[0099] The spatio-temporal parameter update unit 4 receives the power of the sound source n obtained by the sound source separation unit 3 and the observed signal vector at each time t, and obtains the inverse matrix of the spatio-temporal covariance matrix corresponding to the sound source n.

[0100] The acoustic signal enhancing device of the present invention may have at least the configuration common to the first and second embodiments.

[0101] In other words, the acoustic signal enhancement device only needs to include a dereverberation unit 2 that receives, at each time t, an observed signal vector at time t and an inverse matrix of the time-space covariance matrix determined at time t−1, and generates a dereverberation-reduction signal vector corresponding to the observed signal vector, a sound source separation unit 3 that receives, at each time t, the generated dereverberation-reduction signal vector, and determines an enhanced sound for sound source n and the power of the sound source n, and a spatio-temporal parameter update unit 4 that receives, at each time t, the power of sound source n and the observed signal vector, and determines the inverse matrix of the time-space covariance matrix corresponding to the sound source n.

[0102] By the spatiotemporal parameter updating unit 4 using the power of the sound source n obtained by the sound source separation unit 3, the result of the sound source separation is fed back to the processing of the spatiotemporal parameter updating unit 4, so that optimal processing can be performed overall.

[0103] [Variations] Although the embodiments of the present invention have been described above, the specific configurations are not limited to these embodiments, and it goes without saying that the present invention includes appropriate design changes and the like as long as they do not deviate from the spirit of the present invention.

[0104] The various processes described in the embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as necessary.

[0105] For example, data may be exchanged directly between the components of the audio signal enhancing device, or may be exchanged via a storage unit (not shown).

[0106] [Programs, recording media] The processing of each unit of each of the above-mentioned devices may be realized by a computer, in which case the processing contents of the functions that each device should have are described by a program. Then, by loading this program into the storage unit 1020 of the computer 1000 shown in Fig. 5 and operating the arithmetic processing unit 1010, the input unit 1030, the output unit 1040, etc., various processing functions of each of the above-mentioned devices are realized on the computer.

[0107] The program describing the processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a non-transitory recording medium, specifically, a magnetic recording device, an optical disk, etc.

[0108] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.

[0109] A computer that executes such a program, for example, first stores a program recorded on a portable recording medium or a program transferred from a server computer in the auxiliary recording unit 1050, which is a non-transient storage device of the computer. Then, when executing a process, the computer reads the program stored in the auxiliary recording unit 1050, which is a non-transient storage device of the computer, into the storage unit 1020, and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium into the storage unit 1020 and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the above-mentioned process by a so-called ASP (Application Service Provider) type service that realizes a processing function only by issuing an execution instruction and obtaining a result. Note that the program in this embodiment includes information used for processing by an electronic computer and equivalent to a program (data that is not a direct instruction to a computer but has a nature that specifies the processing of the computer, etc.).

[0110] In addition, in this embodiment, the present device is configured by executing a specific program on a computer, but at least a part of the processing contents may be realized by hardware.

[0111] Needless to say, other modifications can be made without departing from the spirit of the present invention.

[0112] [Experimental Results] The above embodiment performed dereverberation and sound source separation from the voices of two people speaking simultaneously recorded in a reverberant environment. The processing delay of online processing when the number of microphones was four was 8.89 milliseconds. In addition, the signal-to-distortion ratio (SDR) of the separated sounds was 6.81 dB. Considering that the SDR using the conventional method was 3.81 dB, it can be seen that the SDR was improved compared to the conventional method with low-delay (8.98 milliseconds) online processing.

Claims

1. a dereverberation unit that receives, at each time t, an observation signal vector at time t and an inverse matrix of a time-space covariance matrix calculated at time t−1, and generates a dereverberation signal vector corresponding to the observation signal vector, corresponding to sound source n, or common to all sound sources; a sound source separation unit that obtains an enhanced sound of the sound source n and a power of the sound source n by using the generated dereverberation signal vector corresponding to the sound source n or common to all sound sources at each time t; a space-time parameter update unit that receives the power of the sound source n and the observed signal vector at each time t, and calculates an inverse matrix of a space-time covariance matrix corresponding to the sound source n; 16. An acoustic signal enhancement device comprising:

2. where t is the time frame number, f is the frequency number, N is the number of sound sources, M is the number of microphones, n = 1, ..., N, m = 1, ..., M, The dereverberation filter G obtained one time before corresponding to the sound source n n (f;t-1) and the observed signal x of microphone m m (f,t) and the observed signal vector X(f,t) m The dereverberation signal y corresponding to (f,t) n,m The dereverberation signal vector Y for the emphasized sound of sound source n, which is composed of (f,t), n A dereverberation unit that generates (f,t); A sound source separation filter Q generated one time before corresponding to sound source n n (f;t-1) and the generated dereverberation signal vector Y n Using (f,t), the emphasis sound s of the sound source n n (f,t) and the power v of the sound source n n A sound source separation unit for obtaining (t); The power v of the sound source n n (t), the observed signal vector X(f,t), and the inverse matrix R of the time-space covariance matrix obtained one time before corresponding to the sound source n n -1 (f;t-1) and the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 A spatio-temporal parameter update unit for determining (f;t), The dereverberation unit uses a dereverberation filter G n (f;t-1) and the Kalman gain K corresponding to the sound source n n (f,t) and the power v of the sound source n n Using (t), the dereverberation filter G corresponding to the sound source n is n Find (f;t), The sound source separation unit extracts the dereverberation signal vector Y n (f,t) and the power v of the sound source n n Using (t), the spatial covariance matrix Σ corresponding to the sound source n n (f,t) is calculated, and the spatial covariance matrix Σ corresponding to the obtained sound source n is calculated. n (f, t) and the sound source separation filter Q generated one time before corresponding to the sound source n n (f;t-1) and the sound source separation filter Q n Search for (f;t), 13. An audio signal enhancement device comprising:

3. where t is the time frame number, f is the frequency number, N is the number of sound sources, M is the number of microphones, n = 1, ..., N, m = 1, ..., M, The dereverberation filter G(f;t-1) calculated one time ago and the observed signal x of microphone m m (f,t) and the observed signal vector X(f,t) m The dereverberation signal y corresponding to (f,t) m A dereverberation unit that generates a dereverberation signal vector Y(f,t) composed of (f,t), Using the sound source separation filter Q(f;t-1) generated one time before and the generated dereverberation signal vector Y(f,t), the emphasized sound s n The emphasis sound vector S(f,t) and the power v of the sound source n are n A sound source separation unit for obtaining (t); The power v of the sound source n n (t), the observed signal vector X(f,t), and the inverse matrix R of the time-space covariance matrix obtained one time before corresponding to the sound source n n -1 (f;t-1) and the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 A spatio-temporal parameter update unit for determining (f;t), The dereverberation unit uses a dereverberation filter G n (f;t-1) and the Kalman gain K corresponding to the sound source n n (f,t) and the power v of the sound source n n Using (t), the dereverberation filter G corresponding to the sound source n is n (f;t) is calculated, and the dereverberation filter G corresponding to each sound source n is calculated. n A dereverberation filter G(f;t) is calculated using the sound source separation filter Q(f;t-1) generated one time before. The sound source separation unit separates the dereverberation signal vector Y(f,t) and the power v n Using (t), the spatial covariance matrix Σ corresponding to the sound source n n (f,t) is calculated, and the spatial covariance matrix Σ corresponding to the obtained sound source n is calculated. n (f, t) and the sound source separation filter Q generated one time before corresponding to the sound source n n (f;t-1) and the sound source separation filter Q n Search for (f;t), 13. An audio signal enhancement device comprising:

4. a dereverberation step in which a dereverberation unit receives an observation signal vector at time t and an inverse matrix of a time-space covariance matrix calculated at time t−1, and generates a dereverberation signal vector corresponding to the observation signal vector, corresponding to sound source n, or common to all sound sources; a sound source separation step in which a sound source separation unit obtains an enhanced sound of the sound source n and a power of the sound source n by using the generated dereverberation signal vector corresponding to the sound source n or common to all sound sources at each time t; a space-time parameter updating step in which a space-time parameter updating unit receives the power of the sound source n and the observed signal vector at each time t, and calculates an inverse matrix of a space-time covariance matrix corresponding to the sound source n; 2. A method for enhancing an acoustic signal comprising:

5. where t is the time frame number, f is the frequency number, N is the number of sound sources, M is the number of microphones, n = 1, ..., N, m = 1, ..., M, The dereverberation unit uses the dereverberation filter G obtained one time before corresponding to the sound source n. n (f;t-1) and the observed signal x of microphone m m (f,t) and the observed signal vector X(f,t) m The dereverberation signal y corresponding to (f,t) n,m The dereverberation signal vector Y for the emphasized sound of sound source n, which is composed of (f,t), n a dereverberation step to generate (f,t); The sound source separation unit uses the sound source separation filter Q generated one time before corresponding to the sound source n. n (f;t-1) and the generated dereverberation signal vector Y n Using (f,t), the emphasis sound s of the sound source n n (f,t) and the power v of the sound source n n A source separation step to obtain (t); The spatio-temporal parameter update unit updates the power v n (t), the observed signal vector X(f,t), and the inverse matrix R of the time-space covariance matrix obtained one time before corresponding to the sound source n n -1 (f;t-1) and the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 A spatio-temporal parameter update step for determining (f;t), The dereverberation unit uses a dereverberation filter G n (f;t-1) and the Kalman gain K corresponding to the sound source n n (f,t) and the power v of the sound source n n Using (t), the dereverberation filter G corresponding to the sound source n is n Find (f;t), The sound source separation unit extracts the dereverberation signal vector Y n (f,t) and the power v of the sound source n n Using (t), the spatial covariance matrix Σ corresponding to the sound source n n (f,t) is calculated, and the spatial covariance matrix Σ corresponding to the obtained sound source n is calculated. n (f, t) and the sound source separation filter Q generated one time before corresponding to the sound source n n (f;t-1) and the sound source separation filter Q n Search for (f;t), 13. A method for enhancing an acoustic signal comprising the steps of:

6. where t is the time frame number, f is the frequency number, N is the number of sound sources, M is the number of microphones, n = 1, ..., N, m = 1, ..., M, The dereverberation unit uses the dereverberation filter G(f;t-1) calculated one time ago and the observed signal x m (f,t) and the observed signal vector X(f,t) m The dereverberation signal y corresponding to (f,t) m a dereverberation step for generating a dereverberation signal vector Y(f,t) consisting of (f,t); A sound source separation unit separates an emphasis sound s of a sound source n by using a sound source separation filter Q(f;t-1) generated one time before and the generated dereverberation signal vector Y(f,t). n The emphasis sound vector S(f,t) and the power v of the sound source n are n A source separation step to obtain (t); The spatio-temporal parameter update unit updates the power v n (t), the observed signal vector X(f,t), and the inverse matrix R of the time-space covariance matrix obtained one time before corresponding to the sound source n n -1 (f;t-1) and the Kalman gain K n (f,t) and the inverse matrix R of the time-space covariance matrix corresponding to sound source n n -1 A spatio-temporal parameter update step for determining (f;t), The dereverberation unit uses a dereverberation filter G n (f;t-1) and the Kalman gain K corresponding to the sound source n n (f,t) and the power v of the sound source n n Using (t), the dereverberation filter G corresponding to the sound source n is n (f;t) is calculated, and the dereverberation filter G corresponding to each sound source n is calculated. n A dereverberation filter G(f;t) is calculated using the sound source separation filter Q(f;t-1) generated one time before. The sound source separation unit separates the dereverberation signal vector Y(f,t) and the power v n Using (t), the spatial covariance matrix Σ corresponding to the sound source n n (f,t) is calculated, and the spatial covariance matrix Σ corresponding to the obtained sound source n is calculated. n (f, t) and the sound source separation filter Q generated one time before corresponding to the sound source n n (f;t-1) and the sound source separation filter Q n Search for (f;t), 13. A method for enhancing an acoustic signal comprising the steps of:

7. A program for causing a computer to function as each unit of the audio signal enhancing device according to any one of claims 1 to 3.

Images