BLASTING FORGE

Description

BACKGROUND 1. FIELD

[0001] The present invention relates to a beamforming device2. DESCRIPTION OF RELATED ART

[0002] A sound input signal input through a microphone may include not only a target speech required for speech recognition but also noise that interferes with speech recognition. Various researches are being conducted to improve the performance of the speech recognition by removing noise from the sound input signal and extracting only the desired target speech.[Related Art Document][Patent Document]

[0003] Korean Patent No. 10-1133308 (Registration Date: March 28, 2012)

[0004] US 2018 / 090158 A1 discloses a voice activity detection unit configured to receive at least two electric input signals in a number of frequency bands and a number of time instances, k and m being frequency band and time indices, respectively, (k, m) defining a specific time-frequency tile of said electric input signal.

[0005] CN 111 816 200 A discloses a multichannel speech enhancement method based on a time-frequency domain binary mask for receiving speech signals by an array. In the document it is described that a binary mask is calculated by utilizing a network model to output speech existence probability estimation, and classification of a signal time-frequency domain and corresponding beamforming parameter estimation are realized through the binary mask. CN 112 735 460 A discloses a beamforming method and system based on time-frequency masking value estimation. The method is described to comprise obtaining a multi-channel speech sequence, and extracting amplitude spectrum features and spatial domain features through Fourier transform, carrying out logarithm transformation on the amplitude spectrum characteristics to obtain a multi channel voice frequency spectrum characteristic sequence, and sending the multi-channel voice frequency spectrum characteristic sequence to a pre-trained and optimized neural network model to obtain a complex value time-frequency masking value, converting the complex value time-frequency masking value into a voice existence probability, and obtaining a time-frequency masking value by utilizing a probability model, calculating a voice signal covariance matrix according to the time-frequency masking value and the multi-channel voice feature sequence, and performing eigenvalue decomposition on the covariance matrix to obtain a beamforming filter coefficient, and in combination with the beamforming filter coefficient, performing filtering processing on the multi-channel voice sequence voice features by using a beamforming filter to obtain an enhanced voice signal.SUMMARY

[0006] The present invention provides a beamforming device capable of more accurately extracting a target speech signal from an input signal by estimating a speech existence probability corresponding to a probability that the target speech signal exists based on an input vector to provide a steering vector and a weight vector.

[0007] The invention is defined by the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0008] FIGS. 1 and 2 are diagrams for describing a beamforming device according to embodiments of the present invention. FIG. 3 is a diagram illustrating an example of a probability estimation unit included in the beamforming device of FIG. 2. FIG. 4 is a diagram illustrating an example of a steering vector unit included in the beamforming device of FIG. 2. FIG. 5 is a diagram illustrating a determination unit included in the beamforming device of FIG. 2. FIGS. 6 to 8 are diagrams for describing an input vector in a single channel applied to the beamforming device of FIG. 2. DETAILED DESCRIPTION

[0009] In the specification, in adding reference numerals to components throughout the drawings, it is to be noted that like reference numerals designate like components even though components are shown in different drawings.

[0010] On the other hand, the meaning of the terms described in the present specification should be understood as follows.

[0011] Singular expressions should be understood as including plural expressions, unless the context clearly defines otherwise, and the scope of rights should not be limited by these terms.

[0012] Also, it should be understood that terms such as "include" and "have" do not preclude the existence or addition possibility of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

[0013] Hereinafter, preferred embodiments of the present invention designed to solve the above problems will be described in detail with reference to the accompanying drawings.

[0014] FIGS. 1 and 2 are diagrams for describing a beamforming device according to embodiments of the present invention.

[0015] Referring to FIGS. 1 and 2, a beamforming device 10 according to an embodiment of the present invention includes a probability estimation unit 100, a steering vector unit 200, and a beamforming unit 300. The probability estimation unit 100 estimates a speech existence probability SPP corresponding to a probability that a target speech signal TSS exists based on an input vector X. For example, the target speech signal may be provided as a microphone input through a space (transfer function, steering vector) between a target speech and a microphone, and the microphone input may include noise. Here, the microphone input may be the input vector X according to the present invention.

[0016] In addition, the speech existence probability (SPP) may be defined as a posterior probability of the existence of the target speech signal TSS in the input vector X at time t and frequency f, and may be expressed as [Equation 1] below using a Bayes rule. p t , f = P H t , f s x t , f = 1 + 1 Λ t , f − 1

[0017] Here, p t,f may be the speech existence probability, P H t , f s x t , f may be a posterior probability for when the target speech signal exists in the input vector, and Λ t,f may be a generalized likelihood ratio. The generalized likelihood ratio may be expressed as [Equation 2] below. Λ t , f = 1 − P H t , f n P H t , f n p x t , f H t , f s p x t , f H t , f n

[0018] Here, P H t , f n may be a prior probability when there is no target speech signal and may be set to a constant between 0 and 1, p x t , f H t , f s may be a likelihood of when the target speech signal existing in the input vector, and p x t , f H t , f n may be the likelihood of when the target speech signal does not exist in the input vector.

[0019] According to an embodiment of the present invention, the speech existence probability SPP is determined according to a target speech signal spatial covariance matrix TGM for the target speech signal TSS included in the input vector X. Summarizing [Equation 1] above, it may be expressed as [Equation 3] below. p t , f = 1 + P H t , f n 1 − P H t , f n 1 + ξ t , f exp − μ t , f 1 + ξ t , f − 1 ξ t , f = tr R _ t , f n − 1 R _ t , f s μ t , f = x t , f H R _ t , f n − 1 R _ t , f s R _ t , f n − 1 x t , f

[0020] Here, R _ t , f n may be a noise spatial covariance matrix, and R _ t , f s may be the target speech signal spatial covariance matrix.

[0021] According to an embodiment of the present invention, the target speech signal spatial covariance matrix TGM for the target speech signal TSS included in the input vector X is calculated according to the noise spatial covariance matrix. For example, the target speech signal spatial covariance matrix TGM for the target speech signal (TSS) may be expressed as [Equation 4] below: R _ t , f s = R t , f x − R _ t , f n

[0022] Here, R _ t , f s may be the target speech signal spatial covariance matrix, R _ t , f n may be the noise spatial covariance matrix, and R t , f x may be the spatial covariance matrix for the input vector. The spatial covariance matrix for the input vector X may be expressed as [Equation 5] below. R t , f x = 1 ∑ l = 1 t γ t − l ∑ l = 1 t γ t − l x l , f x l , f H = 1 Γ t , f x γΓ t − 1 , f x R t − 1 , f x + x t , f x t , f H Γ t , f x = ∑ l = 1 t γ t − l = γΓ t − 1 , f x + 1

[0023] Here, x t,f may be the input vector, R t − 1 , f x may be the spatial covariance matrix for the input vector in the previous frame, Γ t , f x may be a weight for normalizing the spatial covariance matrix for the input vector, and γ may be a forgetting factor. Here, the forgetting factor may be a constant that may have a value between 0 and 1.

[0024] According to an embodiment of the present invention, the noise spatial covariance matrix for noise included in the input vector X is calculated according to the noise spatial covariance matrix estimate of the previous frame corresponding to the previous frame of the current frame. For example, the noise spatial covariance matrix may be expressed as [Equation 9] below. R _ t , f n = 1 Γ ^ t , f n γΓ t − 1 , f n R t − 1 , f n + x t , f x t , f H λ ^ t , f

[0025] Here, R t − 1 , f n may be the noise spatial covariance matrix estimate of the previous frame, Γ ^ t , f n may be the estimated weight for normalizing the noise spatial covariance matrix, Γ ^ t − 1 , f n may be the weight for normalizing the noise spatial covariance matrix in the previous frame, λ̂ t,f may be the estimated time-varying variance, x t,f may be the input vector, and γ may be the forgetting factor.

[0026] According to an embodiment, the noise spatial covariance inverse matrix for the noise included in the input vector X may be calculated according to the variance-weighted spatial covariance inverse matrix in the previous frame. For example, the noise spatial covariance inverse matrix may be expressed as [Equation 5] below. R _ t , f n − 1 = Γ ^ t , f n γ Ψ t − 1 , f − P t , f γ λ ^ t , f + Q t , f

[0027] Here, Ψ t-1,f may be the variance-weighted spatial covariance inverse matrix in the previous frame, λ̂ t,f may be the estimated time-varying variance, and γ may be the forgetting factor. Γ ^ t , f n is the estimated weight for normalization of the noise spatial covariance matrix and may be expressed as [Equation 6] below. Γ ^ t , f n = γ Γ t − 1 , f n + 1 / λ ^ t , f

[0028] Here, Γ t − 1 , f n may be a weight for normalizing the noise spatial covariance inverse matrix in the previous frame, λ̂ t,f may be the estimated time-varying variance, and γ may be the forgetting factor.

[0029] According to an embodiment, the estimated time-varying variance included in the noise spatial covariance inverse matrix may be calculated by weighted-averaging the time-varying variance in the previous frame. For example, the estimated time-varying variance may be expressed as [Equation 7] below. λ ^ t , f = max βλ t − 1 , f + 1 − β Y ^ t , f 2 , ϵ f

[0030] Here, λ̂ t,f may be the estimated time-varying variance, λ t-1,f may be the time-varying variance in the previous frame, β may be a constant between 0 and 1, and ε f may be a constant greater than 0. |Ŷ t,f | 2< may be the power of the estimated output signal, and may be expressed as [Equation 8] below. Y ^ t , f 2 = 1 2 N f + 1 ∑ r = f − N f f + N f w t − 1 , r H x t , r 2

[0031] Here, w t − 1 H may be the weight vector in the previous frame, (·) H< may be the Hermitian transpose, and N f may be the number of adjacent frequencies. The number of adjacent frequencies may be a constant greater than zero.

[0032] FIG. 3 is a diagram illustrating an example of the probability estimation unit included in the beamforming device of FIG. 2, and FIG. 4 is a diagram illustrating an example of the steering vector unit included in the beamforming device of FIG. 2.

[0033] Referring to FIGS. 1 to 4, according to an embodiment, the beamforming device 10 may further include the probability providing unit 110. The probability providing unit 110 may provide the speech existence probability SPP based on the target speech signal spatial covariance matrix TGM.

[0034] In addition, according to an embodiment, the beamforming device 10 may further include a mask unit 210. The mask unit 210 may provide a target speech mask MSK according to the speech existence probability SPP. For example, when it is unclear whether it is the target speech signal TSS, the speech existence probability SPP may have a value around 0.5. In this case, to extract the frame t and frequency f where the target speech signal TSS clearly exists, the target speech mask MSK as illustrated in [Equation 9] below may be used. M t , f = p t , f , if p t , f ≥ η k . ϵ p , otherwise .

[0035] Here, η k may be a threshold value (e.g., 0.8) with a constant between 0 and 1, and ε p may be a lower limit value (e.g., 0.1) with a constant between 0 and 1.

[0036] According to an embodiment of the present invention, the steering vector unit 200 provides an estimated steering vector CSV according to the speech existence probability SPP and the input vector X. In one embodiment, the estimated steering vector CSV may be determined according to the re-estimated time-varying variance calculated based on the target speech mask MSK. For example, the re-estimated time-varying variance may be expressed as [Equation 10] below. λ ^ t , f = max βλ t − 1 , f + 1 − β Y ˜ t , f 2 , ϵ f

[0037] Here, λ̃ t,f may be the re-estimated time-varying variance, λ t-1,f may be the time-varying variance in the previous frame, β may be a constant between 0 and 1, and ε f may be a constant greater than 0. |Ỹ t,f | 2< may be the power of the re-estimated output signal, and may be expressed as [Equation 11] below. Y ˜ t , f 2 = 1 2 N f + 1 ∑ r = f − N f f + N f M t , r w t − 1 , r H x t , r 2

[0038] Here, M t,r may be the target speech mask. According to the re-estimated time-varying variance, the noise spatial covariance matrix estimate in the current frame may be expressed according to [Equation 12] below. R t , f n = 1 Γ t , f n γΓ t − 1 , f n R t − 1 , f n + x t , f x t , f H λ ˜ t , f

[0039] Here, R t , f n may be the noise spatial covariance matrix estimate in the current frame, R t − 1 , f n may be the noise spatial covariance matrix estimate in the previous frame, Γ t − 1 , f n may be the weight for normalizing the noise spatial covariance matrix in the previous frame, λ̃ t,f may be the re-estimated time-varying variance, x t,f may be the input vector, γ may be the forgetting factor, and Γ t , f n may be the weight for normalizing the noise spatial covariance matrix in the current frame. The weight for normalizing the noise spatial covariance matrix in the current frame may be expressed according to [Equation 13] below. Γ t , f n = γΓ t − 1 , f n + 1 / λ ˜ t , f

[0040] Here, Γ t , f n may be the weight for normalizing the noise spatial covariance matrix in the current frame, Γ t − 1 , f n may be the weight for normalizing the noise spatial covariance matrix in the previous frame, and λ̃ t,f may be the re-estimated time-varying variance. In addition, the target speech signal spatial covariance matrix estimate TGME may be expressed according to [Equation 14] below. R t , f s = R t , f x − R t , f n

[0041] Here, R t , f s may be the target speech signal spatial covariance matrix estimate, R t , f x may be the spatial covariance matrix for the input vector, and R t , f n may be the noise spatial covariance matrix estimate in the current frame. The estimated steering vector CSV may be calculated based on an eigen vector corresponding to a maximum eigen value of the target speech signal spatial covariance matrix estimate TGME, and may be calculated as [Equation 15] according to a power method. h ˜ t , f = h t − 1 , f h ¯ t , f = R t , f s h ˜ t , f R t , f s h ˜ t , f h t , f = h ¯ t , f / h ¯ t , f 1

[0042] Here, h̃ t,f may be the estimated steering vector of the previous frame, h t,f may be an eigen vector corresponding to the maximum eigen value of the target speech signal spatial covariance matrix estimate, h ¯ t , f 1 may be a first component of h t,f , and h t,f may be the estimated steering vector.

[0043] According to an embodiment of the present invention, the beamforming unit 300 calculates the weight vector based on the speech existence probability SPP, the input vector X, and the estimated steering vector CSV to provide an output vector Y. In one embodiment, the weight vector may be determined according to the re-estimated time-varying variance calculated based on the target speech mask MSK. For example, the weight vector may be expressed as [Equation 16] and [Equation 17] below. w t , f = Ψ t , f h t , f h t , f H Ψ t , f h t , f Y t , f = w t , f H x t , f

[0044] Here, w t,f may be the weight vector, Y t,f may be the output vector, and Ψ t,f may be the variance-weighted spatial covariance inverse matrix.

[0045] In one embodiment, the variance-weighted spatial covariance inverse matrix may be determined according to the re-estimated time-varying variance calculated based on the target speech mask (MSK). The variance-weighted spatial covariance inverse matrix may be expressed as [Equation 17] below. Ψ t , f = 1 γ Ψ t − 1 , f − P t , f γ λ ˜ t , f + Q t , f

[0046] Here, λ̃ t,f may be the re-estimated time-varying variance.

[0047] According to an embodiment, the time-varying dispersion may be determined according to the power of the output signal calculated based on the target speech mask MSK. For example, the time-varying variance may be expressed as [Equation 18] below. λ t , f = βλ t − 1 , f + 1 − β Y ¯ t , f 2

[0048] Here, λ t-1,f may be the time-varying variance in the previous frame, and |Y t,f | 2< may be the power of the output signal. The power of the output signal may be expressed as [Equation 19]. Y ¯ t , f 2 = 1 2 N f + 1 ∑ r = k − N f k + N f M t , r Y t , r 2

[0049] Here, Y t,r may be the output vector and M t,r may be the target speech mask.

[0050] FIG. 5 is a diagram illustrating a determination unit included in the beamforming device of FIG. 2.

[0051] Referring to FIGS. 1 to 5, according to an embodiment, the beamforming device 10 may further include the determination unit 400. The determination unit 400 may determine whether the diagonal component of the target speech signal spatial covariance matrix estimate TGME is a negative number. According to an embodiment, when the diagonal component of the target speech signal spatial covariance matrix estimate TGME is the negative number, in the beamforming device 10 according to the present invention, the target speech mask MSK of the current frame may be the same as the target speech mask MSK of the previous frame, and the estimated steering vector CSV of the current frame may be the same as the estimated steering vector CSV of the previous frame.

[0052] FIGS. 6 to 8 are diagrams for describing an input vector in a single channel applied to the beamforming device of FIG. 2.

[0053] Referring to FIGS. 1 to 8, according to an embodiment, when the beamforming device 10 operates in a single channel, the input vector X is configured by changing the frame and frequency based on the current frame and reference frequency. For example, the current frame may be t and the reference frequency may be f. In this case, in the input vector X, corresponding values for the same frame may be arranged by moving a frequency up and down step by step based on X m,t,f , and values corresponding to previous frames may be arranged by changing only the frame at the same frequency on the left based on X m,t,f . Here, the single channel may mean that there is only one target sound source.

[0054] According to an embodiment, the input vector X may be composed of a portion of the input vector X. For example, in the input vector X, only the frame may be configured differently based on the same frequency f, or only the frequency may be configured differently at the same frame t. In addition, as illustrated in FIG. 8, the input vector X may not only be configured by extracting the frame or frequency every one step, but may also be configured in various ways.

[0055] According to the beamforming device 10 of the present invention, it is possible to more accurately extract the target speech signal TTS from the input signal by estimating the speech existence probability SPP corresponding to the probability that the target speech signal TSS exists based on the input vector X to provide the steering vector and the weight vector.

[0056] According to the present invention as described above, there are the following effects.

[0057] According to the beamforming device of the present invention, it is possible to more accurately extract the target speech signal from the input signal by estimating the speech existence probability corresponding to the probability that the target speech signal exists based on the input vector to provide the steering vector and the weight vector.

[0058] In addition, other features and advantages of the present invention may be newly understood through the embodiments of the present invention.

Claims

1. A beamforming device (10), comprising: a probability estimation unit (100) that estimates a speech existence probability corresponding to a probability that a target speech signal exists based on an input vector; a steering vector unit (200) that provides an estimated steering vector according to the speech existence probability and the input vector; and a beamforming unit (300) that calculates a weight vector based on the speech existence probability, the input vector, and the estimated steering vector to provide an output vector, wherein the speech existence probability is determined according to a target speech signal spatial covariance matrix for the target speech signal included in the input vector, wherein the target speech signal spatial covariance matrix for the target speech signal included in the input vector is calculated according to a noise spatial covariance matrix, and wherein the noise spatial covariance matrix for the noise included in the input vector is calculated according to a noise spatial covariance matrix estimate of a previous frame corresponding to the previous frame of a current frame.

2. The beamforming device (10) of claim 1, wherein a noise spatial covariance inverse matrix for the noise included in the input vector is calculated according to a variance-weighted spatial covariance inverse matrix in the previous frame.

3. The beamforming device (10) of claim 2, wherein an estimated time-varying variance included in the noise spatial covariance inverse matrix is calculated by weighted-averaging a time-varying variance in the previous frame.

4. The beamforming device (10) of any one of claims 1 to 3, further comprising: a probability providing unit that provides the speech existence probability based on the target speech signal spatial covariance matrix.

5. The beamforming device (10) of any one of claims 1 to 4, further comprising: a mask unit that provides a target speech mask according to the speech existence probability.

6. The beamforming device (10) of any one of claims 3 to 5, wherein the estimated steering vector is determined according to the re-estimated time-varying variance calculated based on the target speech mask.

7. The beamforming device (10) of any one of claims 3 to 6, wherein the weight vector is determined according to the re-estimated time-varying variance calculated based on the target speech mask.

8. The beamforming device (10) of any one of claims 3 to 7, wherein the time-varying variance is determined according to power of an output signal calculated based on the target speech mask.

9. The beamforming device (10) of any one of claims 3 to 8, wherein the variance-weighted spatial covariance inverse matrix is determined according to the re-estimated time-varying variance calculated based on the target speech mask.

10. The beamforming device (10) of any one of claims 1 to 9, further comprising: a determination unit (400) that determines whether a diagonal component of the target speech signal spatial covariance matrix is a negative number.

11. The beamforming device (10) of claim 10, wherein when the diagonal component of the target speech signal spatial covariance matrix is the negative number, the target speech mask of the current frame is the same as the target speech mask of the previous frame, and the estimated steering vector of the current frame is the same as the estimated steering vector of the previous frame.

12. The beamforming device (10) of any one of claims 1 to 11, wherein when the beamforming device (10) operates in a single channel, the input vector is configured by changing the frame and frequency based on the current frame and a reference frequency, or wherein the input vector is composed of a portion of the input vector.