Audio signal enhancement device, method and program

The acoustic signal enhancement device optimizes dereverberation and sound source separation by estimating individual spatio-temporal covariance matrices for each sound source and noise, achieving efficient and optimal acoustic signal processing.

JP7582439B2Active Publication Date: 2024-11-13NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023501919
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-02-25
Publication Date
2024-11-13
Estimated Expiration
2041-02-25

AI Technical Summary

Technical Problem

Existing dereverberation and sound source separation methods do not perform optimal processing as a whole, as they are performed independently without considering the subsequent sound source separation step.

Method used

An acoustic signal enhancement device that estimates individual spatio-temporal covariance matrices for each sound source and noise, using these matrices for dereverberation and sound source separation, with a control unit to repeatedly perform processing for optimal results.

Benefits of technology

Enables optimal processing by determining spatio-temporal covariance matrices for individual sound sources and noise, reducing overall calculation cost and improving acoustic signal enhancement efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582439000022
    Figure 0007582439000022
  • Figure 0007582439000023
    Figure 0007582439000023
  • Figure 0007582439000024
    Figure 0007582439000024
Patent Text Reader

Abstract

An acoustic signal enhancement device that, when j (j=1, ... , J+1, 1≤j≤J) represents sound sources that correspond to target sound, and J+1 represents a sound source that corresponds to noise, comprises a time / space covariance matrix estimation unit 2 that, for each sound source j, uses the power of the sound source j and an observation signal vector Xt,f from an observation signal xt,f (m) from a microphone m to estimate a time / space covariance matrix Rf (j),Pf (j), a reverberation suppression unit 3 that, for each sound source j, uses the estimated time / space covariance matrix Rf (j),Pf (j) to find a reverberation suppression filter Gf (j) for the sound source j and uses the found reverberation suppression filter Gf (j) and the observation signal vector Xt,f to generate a reverberation suppression signal vector, a sound source separation unit 4 that, for each sound source j (1≤j≤J) that corresponds to target sound, uses the generated reverberation suppression signal vector to find an enhanced sound yt,f (j) for the sound source j and the power of the sound source j, and a control unit 5 that performs control such that the processing of the time / space covariance matrix estimation unit, the processing of the reverberation suppression unit, and the processing of the sound source separation unit are repeated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an acoustic signal enhancement technology that separates individual sounds from an acoustic signal in which a plurality of sounds collected by a plurality of microphones, reverberations thereof, and other noises are mixed, without prior information on each component sound, and simultaneously suppresses reverberations.

Background Art

[0002] Conventionally, a reverberation suppression method that simultaneously suppresses reverberations related to all component sounds without prior information on each component sound is known (see, for example, Non-Patent Document 1).

[0003] Also, a method that simultaneously realizes noise suppression and sound source separation in a situation without reverberation is known (see, for example, Non-Patent Document 2).

[0004] Therefore, as shown in FIG. 6, by applying these two processes in order as a reverberation suppression step and a sound source separation noise suppression step, it was possible to simultaneously realize sound source separation, reverberation suppression, and noise suppression.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

[0006] However, in the dereverberation step of the background art, processing is performed independently of the type of processing performed in the subsequent sound source separation step, so the background art was unable to perform optimal processing overall when performing dereverberation and sound source separation simultaneously.

[0007] An object of the present invention is to provide an apparatus, method and program for enhancing an acoustic signal that performs optimal processing overall. [Means for solving the problem]

[0008] In an audio signal enhancement device according to one aspect of the present invention, t is a time frame number, f is a frequency number, M is the number of microphones, m=1,...,M, sound sources include target sounds and noise, J is the number of target sounds, M>J, j=1,...,J+1, where 1≦j≦J, j represents a sound source corresponding to the target sound, and J+1 represents a sound source corresponding to the noise. (1≦j≦J+1) , the power of the sound source j and the observed signal x of microphone m t,f (m) The observed signal vector X consists of t,f and the time-space covariance matrix R f (j) ,P f (j) and a time-space covariance matrix estimator for each sound source j (1≦j≦J+1)For the estimated time-space covariance matrix R f (j) ,P f (j) dereverberation filter G f (j) The dereverberation filter G f (j) and the observed signal vector X t,f and a dereverberation unit that generates a dereverberation signal vector using the generated dereverberation signal vector for each sound source j (1≦j≦J) corresponding to the target sound, and t,f (j) and a control unit that controls the processing of the time-space covariance matrix estimation unit, the processing of the dereverberation unit, and the processing of the sound source separation unit to be repeatedly performed. [Effects of the Invention]

[0009] By calculating the time-space covariance matrices for only the individual sound sources and noise and using them for dereverberation, it is possible to perform optimal processing overall. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the functional configuration of an acoustic signal enhancing device according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a processing procedure of the acoustic signal enhancement method. [Figure 3] FIG. 3 is a diagram illustrating an example of the functional configuration of the acoustic signal enhancing device according to the second embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the functional configuration of an acoustic signal enhancement device in a higher-level concept of the first and second embodiments. [Figure 5] FIG. 5 is a diagram illustrating an example of a functional configuration of a computer. [Figure 6] FIG. 6 is a diagram for explaining the background art. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings, in which like reference numerals are used to designate like components having the same functions, and redundant description will be omitted.

[0012] [First embodiment] As shown in FIG. 1, the acoustic signal enhancement device includes, for example, an initialization unit 1, a time-space covariance matrix estimation unit 2, a dereverberation unit 3, a sound source separation unit 4, and a control unit 5.

[0013] In the audio signal enhancing device of the first embodiment, a different dereverberation filter is determined and used for each sound source.

[0014] The acoustic signal enhancement method is realized, for example, by each component of the acoustic signal enhancement device performing the processes from step S1 to step S5 described below and shown in FIG.

[0015] The symbol used in this text is " - " should be written directly above the character immediately following it, but due to limitations in text notation, it is written immediately before the character in question. In mathematical formulas, these symbols are written in their proper position, that is, directly above the character. For example, " - X" is written in the formula as follows:

number

[0016] M is the number of microphones, and m (1≦m≦M) is the microphone number. M is a positive integer of 2 or more. In principle, the microphone number is expressed by an upper right subscript. For example, x t,f (m) It is expressed as follows.

[0017] J is the number of target sounds.

[0018] j is the sound source number. j in the range 1≦j≦J indicates the sound source that is the target sound, and J+1 indicates the sound source that is noise.

[0019] t, τ (1≦t, τ≦T) are the time frame numbers. T is the total number of time frames and is a positive integer equal to or greater than 2.

[0020] f (1≦f≦F) is the frequency number. The sound source is represented by the upper right subscript, and the time and frequency are represented by the lower right subscript. For example, x t,f (n) where F is the frequency corresponding to the highest frequency bin.

[0021] (·) T is the non-conjugate transpose of a matrix or vector, and (·) H is the conjugate transpose of a matrix or vector. · is any matrix or vector.

[0022] Lowercase letters are scalar variables. For example, the observed signal x at microphone m at time t and frequency f is t,f (m) is a scalar variable.

[0023] Capital letters represent vectors or matrices, e.g., X t,f =[x t,f (1) , x t,f (2) ,…,x t,f (M) ] T ∈C M×1 is the observed signal vector at all microphones at time t and frequency f.

[0024] C M×N is the set of all M×N dimensional complex matrices. X∈C M×N is a notation that indicates that it is an element. That is, X is C M×N Indicates that the element is

[0025] - X t-D,f =[x t-D,f T ,…,x t-L+1,f T ] T ∈CM(L-D)×1 is a past observed signal time series vector from time t-L+1 to time tD.

[0026] λ t (j) is the power of sound source j at time t, and is a scalar.

[0027] y t,f (j) is the emphasis sound of sound source j at time t and frequency f, and is a scalar.

[0028] G f (n) ∈C M(L-D)×M is a dereverberation filter for sound source j at frequency f. L is the filter order and is a positive integer greater than or equal to 2. D is a prediction delay and is a positive integer greater than or equal to 1.

[0029] Q f =[Q f (1) , Q f (2) ,…, Q f (M) ] T ∈C M×M is the separation matrix for frequency f. Q f (j) is the separation filter for sound source j.

[0030] R f (j) ∈C M(L-D)×M(L-D) , P f (j) ∈C M(L-D)×M is the spatiotemporal covariance matrix for each sound source at frequency f.

[0031] Each component of the acoustic signal enhancing device will be described below.

[0032] <Initialization section 1> For j=1,...,J, the initialization unit 1 calculates the power λ of each sound source j. t (j) and the dereverberation filter G f (j) and the separation matrix Qf =[Q f (1) , Q f (2) ,…, Q f (M) ] T ∈C M×M Initialize and.

[0033] The power λ of the initialized sound source j t (j) is output to the time-space covariance matrix estimation unit 2. The initialized dereverberation filter G f (j) is output to the dereverberation unit 3. The initialized separation matrix Q f is output to the sound source separation unit 4. If necessary, the power λ of the initialized sound source j is t (j) may be output to the sound source separation unit 4.

[0034] For example, the initialization unit 1 initializes the power λ of the sound source j. t (j) The observed signal x t,f (m) The power of the dereverberation filter G f (j) is a matrix with all elements being 0, and the separation matrix Q f These variables are initialized by setting x to a unit matrix. Of course, the initialization unit 1 may initialize these variables using other methods.

[0035] <Temporal-space covariance matrix estimator 2> The time-space covariance matrix estimation unit 2 receives the power λ of the sound source j initialized by the initialization unit 1 or updated by the sound source separation unit 4. t (j) and the observed signal x of microphone m t,f (m) The observed signal vector X consists of t,f is entered.

[0036] The time-space covariance matrix estimation unit 2 estimates the power λ of each sound source j. t (j) and the observed signal x of microphone m t,f(m) The observed signal vector X consists of t,f and the time-space covariance matrix R f (j) , P f (j) is estimated (step S2).

[0037] That is, the time-space covariance matrix estimation unit 2 estimates the time-space covariance matrix R f (1) , P f (1) ,…,R f (J) , P f (J) For each sound source 1,…,J corresponding to the target sound, the time-space covariance matrix R f (j) , P f (j) By estimating and using it for dereverberation, it is possible to realize a computationally efficient acoustic signal enhancement method while performing global optimization.

[0038] The time-space covariance matrix estimation unit 2 estimates the time-space covariance matrix R corresponding to the sound source J+1, which corresponds to noise. f (J+1) , P f (J+1) Even if there are multiple noises, the time-space covariance matrix estimation unit 2 estimates one time-space covariance matrix R f (J+1) , P f (J+1) This estimates the time-space covariance matrix R f (J+1) , P f (J+1) This can reduce the amount of calculation compared to estimating

[0039] Estimated time-space covariance matrix R f (j) , P f (j) is output to the dereverberation unit 3.

[0040] The time-space covariance matrix estimation unit 2 calculates the time-space covariance matrix R based on, for example, the following equation: f (j) , P f (j) Estimate.

number

[0041] In the first processing, the time-space covariance matrix estimation unit 2 estimates the power λ of the sound source j initialized by the initialization unit 1. t (j) In the second and subsequent processing, the time-space covariance matrix estimation unit 2 uses the power λ of the sound source j updated by the sound source separation unit 4. t (j) Processing is performed using

[0042] <Dereverberation section 3> The dereverberation unit 3 receives the spatiotemporal covariance matrix R estimated by the spatiotemporal covariance matrix estimation unit 2. f (j) , P f (j) and the observed signal x of microphone m t,f (m) The observed signal vector X consists of t,f is entered.

[0043] The dereverberation unit 3 calculates the estimated time-space covariance matrix R for each sound source j. f (j) , P f (j) dereverberation filter G f (j) The dereverberation filter G f (j) and the observed signal vector X t,f and the observed signal x t,f (m) The dereverberation signal vector Z corresponding to t,f(j) is generated (step S3).

[0044] That is, the dereverberation unit 3 uses dereverberation filters G corresponding to the respective sound sources 1, . . . , J corresponding to the target sounds. f (1) ,…,G f (J) and the dereverberation signal vector Z t,f (1) ,…,Z t,f (J) Generate.

[0045] The dereverberation unit 3 also uses a dereverberation filter G corresponding to the sound source J+1, which corresponds to noise. f (J+1) and the dereverberation signal vector Z t,f (J+1) Even if there are multiple noises, the dereverberation unit 3 generates a single dereverberation filter G f (J+1) and one noise separation matrix Q N,f The noise separation matrix Q is calculated as follows: N,f This will be discussed later.

[0046] The generated dereverberated signal vector Z t,f (j) is output to the sound source separation unit 4.

[0047] where Z t,f (j) =[z 1,t,f (j) ,…, z M,t,f (j) ], where m = 1,…,M, z m,t,f (j) is the observed signal x for the emphasis sound of sound source j t,f (m) is the dereverberated signal corresponding to

[0048] The dereverberation unit 3 calculates a dereverberation filter G based on, for example, the following equation: f (j) Generate.

number

number

[0049] The sound source separation unit 4 separates the generated dereverberation signal vector Z t,f (j) Using this, emphasized sound y of sound source j is t,f (j) and the power λ of sound source j t (j) is calculated (step S4).

[0050] That is, the dereverberation unit 3 generates the emphasized sounds y t,f (1) ,…,y t,f (J) and power λ t (1) ,…,λ t (1) Generate.

[0051] The emphasized sound y of the obtained sound source j t,f (j) is output from the audio signal enhancement device. Also, the power λ of the obtained sound source j is t (j) is output to the time-space covariance matrix estimation unit 2.

[0052] An example of the processing performed by the sound source separation unit 4 will be described below. Note that the sound source separation unit 4 may also be configured to separate the emphasized sound y of the sound source j using an existing method other than the method described below. t,f (j) and the power λ of sound source j t (j)You may ask for:

[0053] In this example, the sound source separation unit 4 receives the power λ of the sound source j initialized by the initialization unit 1. t (j) is further input.

[0054] The sound source separation unit 4 separates the dereverberation signal vector Z t,f (j) and the power λ of sound source j t (j) Using the spatial covariance matrix Σ corresponding to the sound source j, f (j) (2) For j = 1, ..., J, the spatial covariance matrix Σ f (j) Using the separation filter Q corresponding to the sound source j, f (j) and update the updated separation filter Q f (j) and the dereverberation signal vector Z t,f (j) Using the sound source j, emphasized sound y t,f (j) and updated emphasis y t,f (j) Using the power λ of sound source j t (j) (3) For j=1,...,J, the updated separation filter Q f (j) Using the noise separation matrix Q N,f By repeating the process of updating the emphasis sound y t,f (j) Ask for.

[0055] That is, the sound source separation unit 4 (1) calculates the dereverberation signal vector Z t,f (1) ,…,Z t,f (J+1) and the power λ of sound source 1,...,J+1 t (1) ,…,λ t (J+1) Using the spatial covariance matrix Σ corresponding to sound sources 1, ..., J+1, f (1) ,…,Σf (J+1) (2) the process of calculating the spatial covariance matrix Σ f (1) ,…,Σ f (J) , the separation filters Q corresponding to the sound sources 1,...,J are f (1) ,…,Q f (J) and update the updated separation filter Q f (1) ,…,Q f (J) and the dereverberation signal vector Z t,f (1) ,…,Z t,f (J) Using the sound source 1,...,J, the emphasized sound y t,f (1) ,…,y t,f (J) and updated emphasis y t,f (1) ,…,y t,f (J) Using the power λ of sound source 1,…,J t (1) ,…,λ t (J) (3) updating the updated separation filter Q f (1) ,…,Q f (J) Using the noise separation matrix Q N,f By repeating this process, we finally obtain the emphasized sound y t,f (1) ,…,y t,f (J) Ask for.

[0056] It should be noted that the processes (1) to (3) do not need to be repeated, that is, the processes (1) to (3) may be performed only once in one processing of step S4.

[0057] The final emphasis sound y of sound source j is t,f (j) is output from the audio signal enhancement device. The final updated power λ of the audio source j is t (j)is output to the time-space covariance matrix estimation unit 2. Furthermore, the updated separation matrix Q f is output to the dereverberation unit 3.

[0058] The sound source separation unit 4 calculates the spatial covariance matrix Σ corresponding to the sound source j based on, for example, the following equation: f (j) Ask for.

number

number

number

[0059] The sound source separation unit 4 separates the emphasized sound y of the sound source j based on, for example, the following equation: t,f (j) Update.

number

number

number

[0060] In this way, the noise separation matrix Q can be obtained in one step regardless of the number of noises. N,f By calculating

[0061] <Control unit 5> The control unit 5 controls the processing by the time-space covariance matrix estimation unit 2, the processing by the dereverberation unit 3, and the processing by the sound source separation unit 4 to be repeated (step S5).

[0062] For example, the control unit 5 repeats the process until a predetermined termination condition is met. An example of the predetermined termination condition is that the emphasized sound y of the sound source j ist,f (j) Another example of the predetermined termination condition is that the number of times the iteration process has been performed reaches a predetermined number.

[0063] In this way, the sound source separation results are fed back to the processing of the dereverberation unit 3 and all processing is repeated, thereby making it possible to perform optimal processing overall. f (j) , P f (j) By estimating the matrix size, it is no longer necessary to consider the relationship between each sound source, and the overall computational cost can be reduced.

[0064] In the first embodiment, all parameters are optimized using one optimization criterion to perform overall optimization. An example of the one optimization criterion is the criterion shown in the following formula (3).

number

[0065] Note that formula (3) is a criterion derived based on the maximum likelihood method, taking into account the processing by formulas (A) and (B) under the following two assumptions.

[0066] The first assumption is that the separated sounds of each target sound have a power λ f (j) follows a time-varying complex Gaussian distribution.

[0067] The second assumption is that the noise follows a complex Gaussian distribution whose power is time-invariant.

[0068] Generally, when comparing the dereverberation step (step S3) and the sound source separation step (step S4), the former requires a large computational cost per iteration, while the latter requires many iterations until convergence. In the first embodiment, by executing the sound source separation step multiple times in one iteration, it is possible to achieve control such that, overall, computational costs can be reduced (= fewer updates to the dereverberation step) while achieving faster convergence (= increasing the number of updates to the sound source separation noise suppression step).

[0069] In the above example, the power λ of the sound source j t (j) was calculated using equation (C). Since equation (C) takes the power average in the frequency domain, the time-space covariance matrix calculated based on it has low frequency resolution. This may result in a degradation of the estimation accuracy of the dereverberation filter.

[0070] To avoid this, in the calculation of the time-space covariance matrix used to estimate the dereverberation filter, the power λ of the sound source j, which differs for each frequency, is used. t,f (j) may also be used.

[0071] Specifically, the sound source separation unit 4 calculates the power λ of the sound source j used in calculating the time-space covariance matrix by the following equation: t,f (j) may be further determined.

number

[0072] Then, the time-space covariance matrix estimation unit 2 calculates the time-space covariance matrix R based on, for example, the following equation: f (j) , P f (j) Here too, for example, the noise power λ t (J+1) =1.

number

[0073] On the other hand, in the processing of the sound source separation unit 4, the power λ of the sound source j calculated based on the formula (C) is t (j) is used.

[0074] In addition, the power λ of the target sound obtained using other means such as a neural network t,f (j) may be used as prior information.

[0075] Specifically, the power of the target sound is assumed to take different values ​​at each time-frequency point, and λ t,f (j) Then, the prior distribution is modeled as an inverse gamma distribution, γ t,f (j) Let be the scale parameter. For example, γ t,f (j) is the power of the target sound obtained using only other means such as a neural network (i.e., prior information on the power of the target sound).

[0076] As a result, in the sound source separation and noise suppression step, the power of the target sound can be updated by the following equation: where α is a shape parameter of the inverse gamma distribution, and is set to α=1, for example.

number

[0077] In this case, the power λ of sound source j t (j) Instead, the power λ of the sound source j, which differs for each frequency, t,f (j) is output to the time-space covariance matrix estimation unit 2.

[0078] Then, the time-space covariance matrix estimation unit 2 calculates the time-space covariance matrix R based on, for example, the following equation: f (j) , P f (j) Here too, for example, the noise power λ t (J+1) =1.

number

number

[0079] The following description will focus on the differences from the acoustic signal enhancing device of the first embodiment, and will omit redundant description of the same parts as in the first embodiment.

[0080] As shown in FIG. 3 , the acoustic signal enhancement device of the second embodiment includes, for example, an initialization unit 1, a spatiotemporal covariance matrix estimation unit 2, a dereverberation unit 3, a sound source separation unit 4, and a control unit 5, similar to the acoustic signal enhancement device of the first embodiment.

[0081] <Initialization section 1> The processing of the initialization unit 1 is the same as in the first embodiment.

[0082] <Temporal-space covariance matrix estimator 2> The processing of the time-space covariance matrix estimation unit 2 is the same as in the first embodiment.

[0083] <Dereverberation section 3> As in the first embodiment, the dereverberation unit 3 receives the spatiotemporal covariance matrix R estimated by the spatiotemporal covariance matrix estimation unit 2. f (j) , P f (j) and the observed signal x of microphone m t,f (m) The observed signal vector X consists of t,f In the second embodiment, the dereverberation unit 3 receives the separation matrix Q initialized by the initialization unit 1. f and the separation matrix Q updated by the sound source separation unit 4 f is entered.

[0084] The dereverberation unit 3 calculates the estimated time-space covariance matrix R for each sound source j. f (j) ,P f (j) dereverberation filter G f (j) The dereverberation filter G f (j) A dereverberation filter G common to all sound sources f The dereverberation filter G f and the observed signal vector X t,f and the observed signal x t,f (m) The dereverberated signal z corresponding to t,f (m) The dereverberation signal vector Z consists of t,f is generated (step S3).

[0085] where Z t,f =[z t,f(1) ,…,z t,f (M) ]. The dereverberation signal vector Z t,f can be said to be a dereverberation suppression sound common to all sound sources.

[0086] The generated dereverberated signal vector Z t,f is output to the sound source separation unit 4.

[0087] The dereverberation unit 3 generates a dereverberation filter G f (j) Ask for.

[0088] The dereverberation unit 3 calculates a dereverberation filter G common to all sound sources based on, for example, the following equation: f Ask for.

number

number

[0089] The sound source separation unit 4 separates the dereverberation signal vector Z generated by the dereverberation unit 3 for each sound source j (1≦j≦J) corresponding to the target sound. t,f Using this, emphasized sound y of sound source j is t,f (j) and the power λ of sound source j t (j) is calculated (step S4).

[0090] For example, the sound source separation unit 4 (1) generates a dereverberation signal vector Z t,f and the power of sound source j, the spatial covariance matrix Σ corresponding to sound source j is calculated. f (j)(2) the process of calculating the spatial covariance matrix Σ f (j) Using the separation filter Q corresponding to the sound source j, f (j) and update the updated separation filter Q f (j) and the generated dereverberation signal vector Z t,f Using the sound source j, emphasized sound y t,f (j) and updated emphasis y t,f (j) (3) updating the power of the sound source j using the updated separation filter Q f (j) Using the noise separation matrix Q N,f By repeating the process of updating the emphasis sound y t,f (j) Ask for.

[0091] That is, the sound source separation unit 4 (1) separates the generated dereverberation signal vector Z t,f and the power λ of sound source 1,...,J+1 t (1) ,…,λ t (J+1) Using the spatial covariance matrix Σ corresponding to sound sources 1, ..., J+1, f (1) ,…,Σ f (J+1) (2) the process of calculating the spatial covariance matrix Σ f (1) ,…,Σ f (J) , the separation filters Q corresponding to the sound sources 1,...,J are f (1) ,…,Q f (J) and update the updated separation filter Q f (1) ,…,Q f (J) and the dereverberation signal vector Z t,f Using the sound source 1,...,J, the emphasized sound y t,f (1) ,…,y t,f (J) and updated emphasis y t,f (1),…,y t,f (J) Using the power λ of sound source 1,…,J t (1) ,…,λ t (J) (3) updating the updated separation filter Q f (1) ,…,Q f (J) Using the noise separation matrix Q N,f By repeating this process, we finally obtain the emphasized sound y t,f (1) ,…,y t,f (J) Ask for.

[0092] Unlike the first embodiment, the sound source separation unit 4 of the second embodiment calculates the spatial covariance matrix Σ based on, for example, the following equation: f (j) Ask for.

number

number

number

[0093] The other processes performed by the sound source separation unit 4 are the same as those in the first embodiment.

[0094] <Control unit 5> The processing of the control unit 5 is the same as in the first embodiment.

[0095] [Experimental Results] The voices of two people speaking simultaneously in a noisy and reverberant environment were recorded using eight microphones, and noise suppression, reverberation suppression, and sound source separation were performed using the acoustic signal enhancement device of the first embodiment.

[0096] The average word error rate for speech recognition without acoustic signal enhancement was 62.49%, while the average word error rate for speech recognition with conventional acoustic signal enhancement was 19.54%.

[0097] In contrast, the average word error rate of speech recognition when acoustic signal enhancement was performed using the acoustic signal enhancement device of the first embodiment was 25.65%, the average word error rate of speech recognition when acoustic signal enhancement was performed using the acoustic signal enhancement device of the first modified example of the first embodiment was 16.31%, and the average word error rate of speech recognition when acoustic signal enhancement was performed using the acoustic signal enhancement device of the first modified example of the first embodiment was 13.24%.

[0098] These results show that the above-described acoustic signal enhancement device can perform optimal processing overall, and can enhance acoustic signals more efficiently than conventional devices.

[0099] [Variations] The above describes the embodiments of the present invention, but the specific configuration is not limited to these embodiments, and it goes without saying that even if design changes are made as appropriate within the scope of the present invention, they are still included in the present invention.

[0100] The various processes described in the embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as necessary.

[0101] For example, data may be exchanged directly between the components of the acoustic signal enhancing device, or may be exchanged via a storage unit (not shown).

[0102] [Programs, recording media] The processing of each unit of each of the above-mentioned devices may be realized by a computer, in which case the processing content of the functions that each device should have is described by a program. Then, by loading this program into storage unit 1020 of computer 1000 shown in Fig. 5 and running it on arithmetic processing unit 1010, input unit 1030, output unit 1040, etc., various processing functions of each of the above-mentioned devices are realized on the computer.

[0103] The program describing the processing contents can be recorded on a computer-readable recording medium, such as a non-transitory recording medium, specifically a magnetic recording device, an optical disk, or the like.

[0104] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.

[0105] A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or transferred from a server computer in auxiliary recording unit 1050, which is its own non-transitory storage device. Then, when executing a process, the computer loads the program stored in auxiliary recording unit 1050, which is its own non-transitory storage device, into storage unit 1020 and executes processing in accordance with the loaded program. Alternatively, as another form of execution of this program, the computer may load the program directly from a portable recording medium into storage unit 1020 and execute processing in accordance with the program. Furthermore, each time a program is transferred from a server computer to this computer, the computer may execute processing in accordance with the received program. Alternatively, the above-described processing may be executed by a so-called ASP (Application Service Provider) type service, which realizes processing functions by issuing execution instructions and obtaining results without transferring the program from the server computer to this computer. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0106] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0107] It goes without saying that other modifications are possible without departing from the spirit of the present invention.

Claims

1. t is the time frame number, f is the frequency number, M is the number of microphones, m = 1, ..., M, there are target sounds and noises as sound sources, J is the number of target sounds, M > J, j = 1, ..., J + 1, 1 ≤ j ≤ J, j represents the sound source corresponding to the target sound, J + 1 represents the sound source corresponding to the noise, for each sound source j (1 ≤ j ≤ J + 1), the power λ of sound source j is t (j) and the observed signal x of microphone m t,f (m) The observed signal vector X consists of t,f Using the above, the time-space covariance matrix R f (j) , P f (j) A time-space covariance matrix estimator for estimating For each sound source j (1≦j≦J+1), the estimated time-space covariance matrix R f (j) , P f (j) Using the above, the dereverberation filter G f (j) The dereverberation filter G f (j) and the observed signal vector X t,f a dereverberation unit that generates a dereverberation signal vector using For each sound source j (1≦j≦J) corresponding to the target sound, the generated dereverberation signal vector is used to generate an emphasis sound y t,f (j) and a sound source separation unit for calculating the power of sound source j; a control unit that controls the process of the time-space covariance matrix estimation unit, the process of the reverberation reduction unit, and the process of the sound source separation unit to be performed repeatedly; 16. An acoustic signal enhancement device comprising:

2. 2. The audio signal enhancement device of claim 1, The dereverberation unit calculates the estimated time-space covariance matrix R f (j) , P f (j) Using the above, the dereverberation filter G f (j) The dereverberation filter G f (j) and the observed signal vector X t,f Using the above, we obtain the observed signal x t,f (m) The dereverberation signal vector Z t,f (j) Generate The sound source separation unit separates the generated dereverberation signal vector Z t,f (j) Using the above, the emphasis sound y of the sound source j is t,f (j) and find the power of sound source j, Acoustic signal enhancement device.

3. 3. The audio signal enhancing device according to claim 2, The sound source separation unit (1) extracts the generated dereverberation signal vector Z t,f (j) and the power of sound source j, the spatial covariance matrix Σ f (j) (2) The process of calculating the spatial covariance matrix Σ f (j) Using the above, the separation filter Q corresponding to the sound source j is f (j) and update the updated separation filter Q f (j) and the generated dereverberation signal vector Z t,f (j) Using the sound source j, emphasized sound y t,f (j) Updated the emphasis y t,f (j) (3) updating the power of the sound source j using the updated separation filter Q f (j) Using the noise separation matrix Q N,f By repeating the process of updating the emphasis sound y t,f (j) Ask for, Acoustic signal enhancement device.

4. 2. The audio signal enhancement device of claim 1, The dereverberation unit calculates the estimated time-space covariance matrix R f (j) , P f (j) Using the above, the dereverberation filter G f (j) The dereverberation filter G f (j) A dereverberation filter G common to all sound sources f The dereverberation filter G f and the observed signal vector X t,f Using the above, the observed signal x t,f (m) The dereverberated signal z t,f (m) The dereverberation signal vector Z t,f Generate The sound source separation unit separates the generated dereverberation signal vector Z t,f Using the above, the emphasis sound y of the sound source j is t,f (j) and find the power of sound source j, Acoustic signal enhancement device.

5. 5. The audio signal enhancing device according to claim 4, The sound source separation unit (1) extracts the generated dereverberation signal vector Z t,f and the power of sound source j, the spatial covariance matrix Σ f (j) (2) The process of calculating the spatial covariance matrix Σ f (j) Using the above, the separation filter Q corresponding to the sound source j is f (j) and update the updated separation filter Q f (j) and the generated dereverberation signal vector Z t,f Using the sound source j, emphasized sound y t,f (j)Updated the emphasis y t,f (j) (3) updating the power of the sound source j using the updated separation filter Q f (j) Using the noise separation matrix Q N,f By repeating the process of updating the emphasis sound y t,f (j) Ask for, Acoustic signal enhancement device.

6. The time-space covariance matrix estimator estimates the power of sound source j (1≦j≦J+1) and the observed signal x of microphone m, where t is the time frame number, f is the frequency number, M is the number of microphones, m=1,…,M, the sound sources include target sounds and noise, J is the number of target sounds, M>J, j=1,…,J+1, 1≦j≦J, j represents the sound source corresponding to the target sound, and J+1 represents the sound source corresponding to noise. t,f (m) The observed signal vector X consists of t,f Using the above, the time-space covariance matrix R f (j) , P f (j) A time-space covariance matrix estimation step for estimating The dereverberation unit estimates the time-space covariance matrix R f (j) , P f (j) Using the above, the dereverberation filter G f (j) The dereverberation filter G f (j) and the observed signal vector X t,f and a dereverberation step of generating a dereverberated signal vector using A sound source separation unit separates, for each sound source j (1≦j≦J) corresponding to the target sound, an emphasis sound y t,f (j) and a source separation step of calculating the power of the sound source j; a control step of controlling a control unit to repeatedly perform the process of the time-space covariance matrix estimation unit, the process of the dereverberation unit, and the process of the sound source separation unit; 2. A method for enhancing an acoustic signal comprising:

7. A program for causing a computer to function as each unit of the audio signal enhancement device according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Multi-Channel Speech Signal Enhancement for Robust Voice Trigger Detection and Automatic Speech Recognition

    US20180350379A1