Acoustic signal processing device, acoustic signal processing method, and program

Abstract

The method does not depend on a voice recognition result, and the recognition rate of voice recognition can be improved even in a noise environment. An acoustic signal processing device according to an embodiment includes a spatial filter control unit, a spatial filter storage unit, and an acoustic processing unit. The spatial filter control unit outputs a spatial filter for emphasizing a target speech component and suppressing a noise component with respect to N (N > = 2) audio signals recorded at different positions and synchronized in time. The spatial filter storage unit stores the spatial filter. The acoustic processing unit uses the spatial filter read out from the spatial filter storage unit to emphasize a target speech component of the acoustic signal and suppress a noise component of the acoustic signal. The spatial filter control unit includes a determination unit, a voice spatial correlation calculation unit, a noise spatial correlation calculation unit, a spatial correlation storage unit, and a spatial filter calculation unit.

Description

Technical Field

[0001] Embodiments of the present invention relate to an acoustic signal processing device, an acoustic signal processing method, and a program. Background Art

[0002] The following technology has been known in the past: the recognition rate of speech recognition is improved by performing sound processing on sound signals recorded by two or more microphones set at different positions, and the sound processing emphasizes the target speech that should be the object of speech recognition and suppresses the noise that should not be the object of speech recognition. For example, there is a technology that, when a keyword is detected by speech recognition, assumes that the signal in the time interval when the keyword is spoken is the target speech, and assumes that the signal outside the time interval when the keyword is spoken is the noise, and calculates the spatial filter of the sound processing for emphasizing the target speech and suppressing the noise.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 6711789 Summary of the invention

[0006] Technical problem to be solved by the invention

[0007] However, in the prior art, it is difficult to improve the recognition rate even in a noisy environment without relying on the speech recognition result.

[0008] Technical means for solving technical problems

[0009] The sound signal processing device of the embodiment includes a spatial filter control unit, a spatial filter storage unit, and a sound processing unit. The spatial filter control unit outputs a spatial filter for emphasizing a target speech component and suppressing a noise component for N sound signals recorded at different positions and synchronized in time, wherein N≥2. The spatial filter storage unit stores the spatial filter. The sound processing unit uses the spatial filter read from the spatial filter storage unit to emphasize the target speech component of the sound signal and suppress the noise component of the sound signal. The spatial filter control unit includes a determination unit, a speech spatial correlation calculation unit, a noise spatial correlation calculation unit, a spatial correlation storage unit, and a spatial filter calculation unit. The determination unit determines whether the sound signal is a target speech or noise. The speech spatial correlation calculation unit calculates a speech spatial correlation matrix using a speech interval determined as the target speech in the sound signal. The noise spatial correlation calculation unit calculates a noise spatial correlation matrix using a noise interval determined as the noise in the sound signal. The spatial correlation storage unit stores the speech spatial correlation matrix and the noise spatial correlation matrix. The spatial filter calculation unit calculates a spatial filter for emphasizing a target speech component and suppressing a noise component based on the speech spatial correlation matrix and the noise spatial correlation matrix read out from the spatial correlation storage unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a diagram showing an example of the functional configuration of the speech recognition device according to the first embodiment.

[0011] Figure 2 It is a diagram showing an example of the functional configuration of the spatial filter control unit according to the first embodiment.

[0012] Figure 3 This is a flowchart showing an example of the sound signal processing method according to the first embodiment.

[0013] Figure 4 This is a flowchart showing an example of a method for updating the spatial filter according to the first embodiment.

[0014] Figure 5 This is a diagram showing an example of the functional configuration of the speech recognition device according to the second embodiment.

[0015] Figure 6 It is a diagram showing an example of the functional configuration of a spatial filter control unit according to the second embodiment.

[0016] Figure 7 This is a flowchart showing an example of the sound signal processing method according to the second embodiment.

[0017] Figure 8This is a flowchart showing an example of a method for updating a spatial filter according to the second embodiment.

[0018] Fig. 9 This is a diagram showing an example of the hardware configuration of the speech recognition device according to the first and second embodiments. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of an acoustic signal processing device, an acoustic signal processing method, and a program will be described in detail with reference to the drawings.

[0020] (First embodiment)

[0021] The sound signal processing device of the first embodiment determines whether the sound signal input at each time is a target speech or noise, and calculates a spatial filter to emphasize the sound signal of the interval determined as the target speech and suppress the sound signal of the interval determined as the noise. Then, the sound signal processing device outputs the sound signal after the noise is suppressed using the spatial filter. The sound signal processing device of the first embodiment is used, for example, Figure 1 The speech recognition device shown.

[0022] [Example of Functional Configuration of Speech Recognition Device]

[0023] Figure 1 1 is a diagram showing an example of the functional configuration of the speech recognition device 100 according to the first embodiment. The speech recognition device 100 according to the first embodiment includes a microphone array 10, a short-time Fourier transform unit 11, an audio processing unit 12, a spatial filter control unit 13, a spatial filter storage unit 14, an inverse short-time Fourier transform unit 15, a speech recognition unit 16, a display control unit 17, and a display 18.

[0024] The sound processing unit 12, the spatial filter control unit 13, and the spatial filter storage unit 14 correspond to the sound signal processing device 1 of the first embodiment. The speech recognition device 100 of the first embodiment uses the output sound signal to perform speech recognition in the subsequent stage, thereby improving the recognition rate of speech recognition without using the speech recognition result.

[0025] The microphone array 10 uses N (N≥2) microphones arranged at different positions to obtain N time-synchronized sound signals x m (t) (m=1, 2, ..., N) Here, m represents the number of the microphone.

[0026] The short-time Fourier transform unit 11 applies a window function to the N acoustic signals xm(t) to generate a plurality of frames, performs a short-time Fourier transform on each frame to transform it into a time-frequency domain, and outputs a spectrum sequence X m (f, k) Here, f represents the number of the frequency bin, and k represents the number of the frame.

[0027] The sound processing unit 12 emphasizes the target speech component and suppresses the noise component contained in the N spectrum sequences, and outputs one spectrum sequence Y (f, k). In addition, the sound processing unit 12 may also output the sound signal itself. As a method for suppressing noise, for example, a GEV (generalized eigenvalue) beamformer, an MVDR (minimum variance distortionless) beamformer, and their derivative methods can be considered.

[0028] The spatial filter control unit 13 uses the N acoustic signals to update the value of the spatial filter storage unit 14. For example, when using the GEV beamformer, the spatial filter control unit 13 calculates the average values ​​of the spatial correlation matrices corresponding to the target speech and the noise, respectively, based on the frequency spectra of the frames corresponding to the speech interval and the noise interval, respectively, and calculates the spatial filter based on these average values.

[0029] The spatial filter storage unit 14 stores the value of a spatial filter for emphasizing a target speech sound and suppressing noise.

[0030] The inverse short-time Fourier transform unit 15 performs an inverse short-time Fourier transform on the spectrum sequence output from the sound processing unit 12 , and outputs one sound signal y(t) in which the target speech sound is emphasized and noise is suppressed.

[0031] The speech recognition unit 16 performs speech recognition on the acoustic signal y(t) to obtain a speech recognition result. In particular, in the first embodiment, a detection result of the utterance of a keyword is obtained.

[0032] The display control unit 17 performs control for displaying the speech recognition result on the display 18. The display 18 (an example of a display unit) displays the speech recognition result.

[0033] Next, use Figure 2 The function of the spatial filter control unit 13 which is a characteristic portion of this embodiment will be described.

[0034] [Example of Functional Configuration of Spatial Filter Control Unit]

[0035] Figure 2 1 is a diagram showing an example of the functional configuration of the spatial filter control unit 13 of the first embodiment. The spatial filter control unit 13 of the first embodiment is connected to the microphone array 10 and the spatial filter storage unit 14. The spatial filter control unit 13 includes a determination unit 131, a speech spatial correlation calculation unit 132, a noise spatial correlation calculation unit 133, a spatial correlation storage unit 134, and a spatial filter calculation unit 135.

[0036] The determination unit 131 determines whether the sound signal of each frame input from the microphone array 10 is a target speech to be recognized or noise to be suppressed. For example, the determination unit 131 calculates a value of a speech score indicating a speech fit degree for the sound signal, and determines that the sound signal is the target speech when the speech score is greater than a speech threshold, and determines that the sound signal is noise when the speech score is less than the speech threshold.

[0037] Specifically, the determination unit 131 uses a pre-learned DNN (deep neural network) to determine whether the sound signal is a target speech to be recognized or noise to be suppressed. For example, the determination unit 131 uses a DNN that uses a single frame of sound signals as input to determine whether it is a target speech or noise. When the speech score obtained by inputting the sound signal of each frame is greater than the speech threshold, it is determined to be the target speech to be recognized. By making a judgment based on the speech score output by a model such as a pre-learned DNN, a judgment using complex information can be made. In addition, the threshold used in the judgment can be installed as a constant, or an interface for setting the threshold from the outside can be provided. In addition, the determination unit 131 can also make a judgment based on the frequency spectrum of each frame output from the short-time Fourier transform unit 11.

[0038] In addition, continuous values ​​may be allowed as the judgment results, and weights may be assigned according to the values ​​of the judgment results in the speech space correlation calculation unit 132 and the noise space correlation calculation unit 133 for updating. Furthermore, the judgment of the target speech and the judgment of the noise may be performed independently by different judges. As an input signal for judgment, one of the N sound signals or one sound signal after noise suppression may be used. For the N sound signals, a signal from which the noise component has been removed by another method such as independent component analysis may also be used for judgment.

[0039] In addition, several other determination methods can also be cited. For example, the determination unit 131 can also determine the sound signal of each microphone, and use the statistics such as the average value, maximum value and minimum value of each determination result as the overall determination result. The determination unit 131 can also use the method [M.WaxandT.Kailath1985] or its derivative method to estimate the number of sound sources based on the number of dominant eigenvalues ​​(eigenvalues ​​with large values) contained in the spatial correlation matrix calculated from N sound signals to determine that the target speech exists when there is a dominant eigenvalue. That is, the determination unit 131 can also be equipped with a sound source number estimator that estimates the number of sound sources contained in the sound signal, and the speech score is represented by a function of the number of sound sources. By using the number of sound sources to determine the presence or absence of the target speech, the determination accuracy of the determination unit 131 can be improved when the noise is diffuse and the power of the target speech is sufficiently large compared to the noise.

[0040] In addition, for example, the determination unit 131 may calculate the power of N acoustic signals, and if the speech score represented by the function based on the statistic of the power of the acoustic signal is a value greater than the speech threshold, it is determined that the target speech exists. As the above-mentioned statistic, the average and dispersion of the power in each frequency bin in the time direction may be considered. In the case where the statistics are greatly different between the target speech and the noise, the determination unit 131 can be implemented with less calculation amount.

[0041] In addition, for example, the determination unit 131 may apply a sound source localization method such as the MUSIC (Multiple Signal Classification) method [ROSchmidt 1986] to the N sound signals, and determine that the target speaker is speaking only when the sound source comes from the direction of the target speaker, when it can be assumed that the direction of the target speaker relative to the microphone array 10 remains unchanged. In this case, the spatial filter control unit 13 may further include: a speaker direction setting unit that maintains the relative direction of the target speaker observed from the microphone; and a sound source localization unit that performs sound source localization on the sound signal and outputs the detected sound source direction. In addition, the determination unit 131 determines that at least one of the sound source directions is a target speech when the angle difference with the relative direction set by the speaker direction setting unit is less than an angle threshold, and determines that it is noise when it is greater than the angle threshold. By using the sound source direction information to determine the presence or absence of the target speech, it is possible to determine whether the speech is coming from the direction of the known target speaker, so that when there is a speaker other than the target speaker, the speech of the speaker can be suppressed using only the sound signal.

[0042] When the determination unit 131 determines that the sound signal is the target speech, the speech spatial correlation calculation unit 132 uses the sound signal of each frame to update the speech spatial correlation matrix stored in the spatial correlation storage unit 134. Specifically, when the sound signal is determined to be the target speech, the speech spatial correlation calculation unit 132 reads the speech spatial correlation matrix from the spatial correlation storage unit 134, calculates the speech spatial correlation matrix, and writes the calculated speech spatial correlation matrix into the spatial correlation storage unit 134 (speech spatial correlation matrix update processing).

[0043] As an example of an updating method, a method can be considered in which the acoustic signals of a certain time in the past and the determination results of the determination unit 131 are stored in advance, and the speech space correlation matrix is ​​calculated using only the acoustic signals of the interval determined to be speech. For example, the determination unit 131 may also include an acoustic signal storage unit, which stores the acoustic signals included from the current time to a predetermined period of time. In addition, the determination unit 131 determines whether the acoustic signals of each frame included from the current time to a predetermined period of time are target speech. The speech space correlation calculation unit 132 calculates the speech space correlation matrix using the acoustic signals included from the current time to a predetermined period of time, and stores the calculated speech space correlation matrix in the space correlation storage unit 134. An effective speech recognition result is obtained only when the acoustic signal includes the target speech. Therefore, in the example of this updating method, it is assumed that the target speech is included in the most recent acoustic signal (the acoustic signal included from the current time to a predetermined period of time). For example, if the predetermined period is set to T frames, when the determination unit 131 determines that the acoustic signal of the kth frame is the target speech, the speech space correlation matrix is ​​calculated according to the following formula (1):

[0044]

Mathematical formula 1

[0045]

[0046] Here, X(f, k) represents the longitudinal vector [X1(f, k), ..., X N (f, k)] T , H represents conjugate transposition, and s(k) represents a function that returns 1 when the determination result of the determination unit 131 for the kth frame is the target speech, and returns 0 when it is noise. Since s(k)=1 is assumed, it can be assumed that the denominator of equation (1) is not 0.

[0047] In addition, as another example, in order to reduce the amount of buffering and calculation of the sound signal, it is possible to consider using an exponential moving average to update the speech space correlation matrix one by one. For example, when s(k)=1, the speech space correlation matrix is ​​calculated according to the following formula (2): When s(k)=0, Here, α s is satisfied 0<α s A constant < 1.

[0048]

Mathematical formula 2

[0049] Φ s (f, k) = (1-α s )Φ s (f, k-1)+α s X(f, k)X(f, k)H …(2)

[0050] The speech space correlation calculation unit 132 may also set the judgment result s(k) of the judgment unit 131 to a continuous value indicating the degree to which the sound signal is the target speech instead of the two values ​​of 0 or 1. Thus, the larger the continuous value is, the greater the weight of the proportion of the value of updating the speech space correlation matrix is, and the speech space correlation matrix is ​​updated. For example, the range of the continuous value indicating the judgment result is 0 to 1, and the closer it is to 1, the greater the degree to which it is the target speech. For example, using the judgment result s(k) above, the speech space correlation matrix is ​​calculated according to the following formula (3):

[0051]

Mathematical formula 3

[0052] Φ s (f, k) = (1-α s s(k))Φ s (f, k-1)+α s s(k)X(f,k)X(f,k) H …(3)

[0053] By setting the output of the determination unit 131 to a continuous value instead of two values, the calculation of the speech space correlation matrix can be weighted according to the reliability of the determination. This makes it possible to calculate the speech space correlation more accurately and further improve the speech emphasis performance of the acoustic processing.

[0054] When the determination unit 131 determines that the sound signal is noise, the noise spatial correlation calculation unit 133 uses the sound signal of each frame to update the noise spatial correlation matrix stored in the spatial correlation storage unit 134. Specifically, when the sound signal is determined to be noise, the noise spatial correlation calculation unit 133 reads the noise spatial correlation matrix from the spatial correlation storage unit 134, calculates the noise spatial correlation matrix, and writes the calculated noise spatial correlation matrix to the spatial correlation storage unit 134 (noise spatial correlation matrix update processing). In addition, the sound signal processing device 1 of the first embodiment may perform both the noise spatial correlation matrix update processing and the above-mentioned speech spatial correlation matrix update processing, or may perform either one.

[0055] The method of updating the noise spatial correlation matrix is ​​the same as that of the speech spatial correlation calculation unit 132. For example, the noise spatial correlation matrix is ​​calculated according to the following formula (4) using the acoustic signal included from the current time to a predetermined period of time, and the calculated noise spatial correlation matrix is ​​stored in the spatial correlation storage unit 134.

[0056]

Mathematical formula 4

[0057]

[0058] In order to reduce the amount of buffering and calculation of the acoustic signal, the noise space correlation matrix may be updated successively using an exponential moving average. In this case, the determination result of the determination unit 131 is set as a continuous value, and the noise space correlation calculation unit 133 may also calculate the noise space correlation matrix by increasing the weight of the ratio of the value of the updated noise space correlation matrix as the continuous value decreases. For example, the noise space correlation matrix is ​​calculated by the exponential moving average according to the following formula (5):

[0059]

Mathematical formula 5

[0060]

[0061] Here, α n is satisfied 0<α n <1. The above judgment result s(k) can take two values, 0 or 1, or a continuous value from 0 to 1. As in the case of the speech space correlation calculation unit 132, by setting the output of the judgment unit 131 to a continuous value instead of two values, the calculation of the noise space correlation matrix can be weighted according to the reliability of the judgment. As a result, the calculation of the noise space correlation can be performed more accurately, and the noise suppression performance of the sound processing can be further improved.

[0062] In addition, in order to more reliably emphasize the most recent target speech, the speech space correlation calculation unit 132 may update the speech space correlation matrix according to formula (1) or formula (2) regardless of the determination result of the determination unit 131, and the noise space correlation calculation unit 133 may update the noise space correlation matrix using the past sound signal when the above determination result for the sound signal at a certain time in the past is noise. For example, the determination unit 131 may also include a sound signal storage unit that stores the sound signal from the current moment to a predetermined period of time ago. In addition, the determination unit 131 determines whether the sound signal before the predetermined period of time is noise. When the sound signal before the predetermined period of time is determined to be noise, the noise space correlation calculation unit 133 calculates the noise space correlation matrix using the sound signal before the predetermined period of time, and stores the calculated noise space correlation matrix in the spatial correlation storage unit 134. For example, when the above-mentioned specified period is set to D frames (D>0), the speech space correlation matrix is ​​calculated according to the above formula (2) regardless of the judgment result of the judgment unit 131. When the judgment unit 131 determines that the sound signal before D frames is noise (that is, s(kD)=0), the noise space correlation matrix is ​​calculated according to the following formula (6).

[0063]

Mathematical formula 6

[0064] Φ n (f, k) = (1-αn )Φ n (f, k-1)+α n X(f,kD)X(f,kD) H …(6)

[0065] Since effective speech recognition results can be obtained only when the sound signal contains the target speech, the target speech can be reliably included in the calculation of speech space correlation. On the other hand, for noise space correlation, the noise suppression effect can be improved when the target speech is included by using a time-delayed sound signal (the sound signal before a specified period) for calculation.

[0066] The spatial correlation storage unit 134 stores the values ​​of the spatial correlation matrices calculated by the speech spatial correlation calculation unit 132 and the noise spatial correlation calculation unit 133 .

[0067] The spatial filter calculation unit 135 calculates a spatial filter that emphasizes the target speech and suppresses noise using the spatial correlation matrices stored in the spatial correlation storage unit 134, and updates the spatial filter storage unit 14 with the calculated spatial filter value. The calculation of the spatial filter is performed based on various methods such as the GEV beamformer and the MVDR beamformer.

[0068] [Example of the method for processing acoustic signals]

[0069] Figure 3 : is a flowchart showing an example of the sound signal processing method of the first embodiment. First, the short-time Fourier transform unit 11 transforms the sound signal input from the microphone array 10 into a spectrum sequence (step S1). Next, the sound processing unit 12 uses the spatial filter stored in the spatial filter storage unit 14 to perform noise suppression on the spectrum sequence input from the short-time Fourier transform unit 11 (step S2). Next, the inverse short-time Fourier transform unit 15 transforms the spectrum sequence output by the sound processing unit 12 (the spectrum sequence after noise suppression) into an sound signal (step S3). Next, the speech recognition unit 16 performs speech recognition on the sound signal (the sound signal after noise suppression) output by the inverse short-time Fourier transform unit 15 (step S4).

[0070] Next, the display control unit 17 causes the display 18 to display the information of the keyword detected by the voice recognition of the voice recognition unit 16 (step S5). Next, the spatial filter control unit 13 updates the value of the spatial filter of the spatial filter storage unit 14 using the sound signal input from the microphone array 10 (step S6). The details of the processing of step S6 (the updating method of the spatial filter) will be referred to in detail. Figure 4 This will be described later.

[0071] Next, the sound processing unit 12 determines whether the input of the sound signal has reached the terminal (step S7). If the input of the sound signal has reached the terminal (step S7, "Yes"), the sound processing unit 12 ends the processing. If the input of the sound signal has not reached the terminal (step S7, "No"), the sound processing unit 12 returns to the processing of step S1 and repeats the same processing for subsequent inputs.

[0072] [Example of a method for updating a spatial filter]

[0073] Figure 4 1 is a flowchart showing an example of a method for updating a spatial filter according to the first embodiment. First, the determination unit 131 calculates a speech score indicating a speech fit of an acoustic signal input from the microphone array 10 (step S11). Next, the determination unit 131 compares the speech score with a predetermined speech threshold (or a speech threshold set externally) (step S12).

[0074] When the speech score is greater than the speech threshold (step S12, "yes"), the speech spatial correlation calculation unit 132 uses the sound signal input from the microphone array 10 to update the spatial correlation matrix of the target speech according to the above formula (1), (2) or (3) (step S13).

[0075] When the speech score is below the speech threshold (step S12, No), the noise spatial correlation calculation unit 133 uses the sound signal input from the microphone array 10 to update the noise spatial correlation matrix according to the above formula (4), (5) or (6) (step S14).

[0076] Next, the spatial filter calculation unit 135 calculates the value of the spatial filter using the spatial correlation matrix of the target speech and the spatial correlation matrix of the noise, and updates the value of the spatial filter storage unit 14 (step S15). The spatial correlation matrix of the target speech and the spatial correlation matrix of the noise are updated each time (successively updated) according to the sound signal input in each frame, so the spatial filter is calculated each time according to the time change of the position, direction and frequency characteristics of the target speech and the noise.

[0077] As described above, the spatial filter control unit 13 calculates a spatial filter for emphasizing the target speech component and suppressing the noise component based on the speech space correlation matrix calculated from the speech interval of the target speech to be recognized included in N (N≥2) sound signals recorded at different positions and synchronized in time, and the noise space correlation matrix calculated from the noise interval of the noise to be suppressed included in the sound signal. The spatial filter storage unit 14 stores the spatial filter. The sound processing unit 12 uses the spatial filter to emphasize the target speech component of the sound signal and suppress the noise component of the sound signal.

[0078] Therefore, according to the sound signal processing device 1 of the first embodiment, it is possible to improve the recognition rate even in a noisy environment without relying on the speech recognition result. Specifically, according to the sound signal processing device 1 of the first embodiment, only the microphone array 10 is used as an input device, and the spatial filter can be calculated without relying on the output of the speech recognition unit 16, so that the time change of the target speech and the noise can be tracked.

[0079] In the prior art, the calculation process of the spatial filter is triggered by keyword detection. Therefore, there is a problem that the spatial filter cannot be calculated when the keyword is not detected, such as when a speech unrelated to the keyword is made. In addition, since the sound processing is not performed before the initial keyword is detected, there is a problem that it cannot be used in an environment where the power of the noise is large and speech recognition is difficult without sound processing. Furthermore, since the spatial filter at the moment of detecting the first keyword is maintained until the second keyword is detected next, there is a problem that the effect of proper noise suppression cannot be obtained when the next speech is made from a position different from the speech of the first keyword.

[0080] On the other hand, according to the spatial filter control unit 13 of the first embodiment, it is possible to directly determine the target speech and noise based on the sound signal, and to obtain the effect of noise suppression without relying on the speech recognition unit 16. Thus, it is possible to obtain the following speech recognition device 100: for the input sound signal, it is possible to obtain a higher speech recognition rate without relying on the speech recognition result.

[0081] In addition, at least one of the determination unit 131, the speech space correlation calculation unit 132, and the noise space correlation calculation unit 133 may be input with the sound signal output from the sound processing unit 12. By using the result of performing speech emphasis and noise suppression on the sound signal once, the speech space correlation and the noise space correlation can be calculated more accurately, and the speech emphasis and noise suppression performance of the sound processing can be further improved.

[0082] In addition, the spatial filter control unit 13 may also include a sound source separation unit, which performs sound source separation on the input sound signal using a method such as independent component analysis, and outputs a separated sound signal separated into a target speech component and a noise component. In addition, the separated sound signal may be input to at least one of the determination unit 131, the speech space correlation calculation unit 132, the noise space correlation calculation unit 133, and the sound processing unit 12. By separating the sound signal into the target speech component and the noise component, the speech space correlation and the noise space correlation can be calculated more accurately, and the speech emphasis and noise suppression performance of the sound processing can be further improved.

[0083] In addition, the determination unit 131 may also calculate a target speech score indicating the degree of fit to the target speech and a noise score indicating the degree of fit to the noise for the acoustic signal. In this case, the determination unit 131 determines that the acoustic signal is the target speech when the target speech score is greater than the speech threshold, and determines that the acoustic signal is noise when the noise score is greater than the noise threshold. The determination unit 131 outputs the target speech score and the noise score respectively, so different algorithms can be used in the determination of the target speech and the noise, respectively. In addition, by not using data that is difficult to determine for the calculation of either the speech space correlation matrix or the noise space correlation matrix, it is possible to prevent adverse effects on the processing of the acoustic signal due to erroneous determination.

[0084] (Second embodiment)

[0085] Next, a second embodiment will be described. In the description of the second embodiment, the same description as the first embodiment will be omitted, and the differences from the first embodiment will be described.

[0086] The sound signal processing device of the second embodiment takes as input the image of a camera that has captured the target speaker, determines whether the target speaker is speaking at each moment, and calculates a spatial filter to emphasize the sound signal of the interval determined to be speaking and suppress the sound signal of the interval determined not to be speaking. Then, the sound signal processing device performs speech recognition on the sound signal that has been subjected to noise suppression using the spatial filter. Thus, the recognition rate of speech recognition can be improved without using the speech recognition result. In addition, the speech of speakers other than the target speaker that is difficult to suppress based on the speech score can also be suppressed.

[0087] [Example of Functional Configuration of Speech Recognition Device]

[0088] Figure 5 1 is a diagram showing an example of the functional configuration of a speech recognition device 100-2 according to the second embodiment. The speech recognition device 100-2 according to the second embodiment includes a microphone array 10, a camera 20, an audio processing unit 12, a spatial filter control unit 13-2, a spatial filter storage unit 14, a speech recognition unit 16, a display control unit 17, and a display 18. Among them, the audio processing unit 12, the spatial filter control unit 13-2, and the spatial filter storage unit 14 are equivalent to the audio signal processing device 1-2 according to the second embodiment.

[0089] The camera 20 is set to always capture the face of the target speaker and output the facial image of the target speaker at each moment. In this embodiment, the relative position of the camera 20 and the target speaker is considered to be fixed, and the camera 20 is fixed in the direction of the speaker to always obtain the facial image of the target speaker. In addition, instead of allowing the target speaker to move, the camera 20 can track the face of the target speaker to always obtain the facial image of the target speaker. For facial tracking, for example, a well-known technology such as KLT (Kanade-Lucas-Tomasi) Tracker [BDLucas and T.Kanade 1981] is used.

[0090] The spatial filter control unit 13 - 2 updates the value of the spatial filter storage unit 14 using the facial image of the target speaker and the N acoustic signals input from the microphone array 10 .

[0091] Next, use Figure 6 The function of the spatial filter control unit 13 - 2 which is a characteristic portion of this embodiment will be described.

[0092] [Example of Functional Configuration of Spatial Filter Control Unit]

[0093] Figure 6 1 is a diagram showing an example of the functional configuration of the spatial filter control unit 13-2 of the second embodiment. The spatial filter control unit 13-2 of the second embodiment is connected to the microphone array 10, the camera 20, and the spatial filter storage unit 14. The spatial filter control unit 13-2 includes a determination unit 131-2, a speech spatial correlation calculation unit 132, a noise spatial correlation calculation unit 133, a spatial correlation storage unit 134, and a spatial filter calculation unit 135.

[0094] The determination unit 131-2 determines whether the target speaker is speaking for the facial image of the target speaker in each frame input from the camera 20. For example, the determination unit 131-2 extracts the image of the lip area from the facial image of each frame, and determines that the target speaker is speaking when it is determined that the lip area is moving, and determines that the target speaker is not speaking when it is determined that the lip area is not moving. As in the first embodiment, the determination unit 131-2 outputs the determination result as two values ​​of 0 or 1 or a continuous value from 0 to 1, and the speech space correlation calculation unit 132 updates the spatial correlation storage unit 134 by the above formula (1), (2) or (3), and the noise space correlation calculation unit updates the spatial correlation storage unit 134 by the above formula (4), (5) or (6). The frame interval used by the spatial filter control unit 13 for sound signal processing and the frame interval used for image processing may be different. For example, the spatial filter control unit 13 may also include a determination result storage unit for storing the determination result of the determination unit 131-2. Then, the spatial filter control unit 13 updates the spatial correlation storage unit 134 using the determination result stored in the determination result storage unit.

[0095] As a method for extracting and detecting the movement of the lip region, the Viola-Jones method [P. Viola and M. Jones 2001] is widely known. Here, in order to prevent the false detection of the target speech, it is also possible to determine that the target speaker is speaking only when the sound signal is determined to be the target speech by the method described in the sound signal processing device 1 of the first embodiment. That is, the determination unit 131-2 may extract the image of the lip region from the facial image, and determine that the target speaker is speaking when it is determined that the lip region is moving and the sound signal is determined to be the target speech.

[0096] In addition, the parameters and thresholds for detecting the movement of the lip region may be implemented as constants, or an interface that enables external specification may be provided.

[0097] [Example of the method for processing acoustic signals]

[0098] Figure 7 2 is a flowchart showing an example of the sound signal processing method of the second embodiment. The processing of steps S21 to S25 is similar to that of steps S1 to S5 of the first embodiment (see Figure 3 ) is the same, so the description is omitted.

[0099] Next, the spatial filter control unit 13-2 updates the value of the spatial filter of the spatial filter storage unit 14 using the sound signal input from the microphone array 10 and the facial image of the target speaker input from the camera 20 (step S26). The details of the processing of step S26 (the updating method of the spatial filter) will be referred to in detail. Figure 8 This will be described later.

[0100] The process of step S27 is the same as the process of step S7 in the first embodiment, and therefore the description thereof is omitted.

[0101] [Example of a method for updating a spatial filter]

[0102] Figure 8 1 is a flowchart showing an example of a method for updating a spatial filter according to the second embodiment. First, the determination unit 131-2 calculates a speech score indicating whether the target speaker is speaking for the face image of the target speaker input from the camera 20 (step S31). Next, the determination unit 131 compares the speech score with a predetermined image threshold (or an image threshold set externally) (step S32).

[0103] When the speech score is greater than the image threshold value (step S32 , “Yes”), the speech spatial correlation calculation unit 132 updates the spatial correlation matrix of the target speech using the acoustic signal input from the microphone array 10 (step S33 ).

[0104] When the speech score is equal to or less than the image threshold (step S32 , No), the noise spatial correlation calculation unit 133 updates the noise spatial correlation matrix using the acoustic signal input from the microphone array 10 (step S34 ).

[0105] Next, the spatial filter calculation unit 135 calculates the value of the spatial filter using the spatial correlation matrix of the target speech and the spatial correlation matrix of the noise, and updates the value of the spatial filter storage unit 14 (step S35).

[0106] As described above, in the sound signal processing device 1-2 of the second embodiment, the determination unit 131-2 calculates the speech score using the facial image of the target speaker. Thus, by using the microphone array 10 and the camera 20 as input devices, not only background noise can be suppressed, but also speech other than the target speaker can be regarded as noise and suppressed. Specifically, by using image features such as the movement of the lip area of ​​the target speaker input from the camera 20 to determine whether it is the target speech or noise, when there is a speaker other than the target speaker, the speech of the speaker can be suppressed.

[0107] Finally, an example of the hardware configuration of the speech recognition apparatus 100 (100-2) according to the first and second embodiments will be described. The speech recognition apparatus 100 (100-2) according to the first and second embodiments can be realized by using, for example, any computer device as basic hardware.

[0108] [Hardware configuration example]

[0109] Fig. 9 1 is a diagram showing an example of the hardware configuration of the speech recognition device 100 (100-2) of the first and second embodiments. The speech recognition device 100 (100-2) of the first and second embodiments includes a processor 201, a main storage device 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main storage device 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0110] In addition, the speech recognition device 100 (100-2) may not include a part of the above-mentioned structure. For example, when the speech recognition device 100 (100-2) can use the input function and display function of an external device, the speech recognition device 100 (100-2) may not include the display device 204 and the input device 205.

[0111] The processor 201 executes a program read from the auxiliary storage device 203 to the main storage device 202. The main storage device 202 is a memory such as a ROM and a RAM. The auxiliary storage device 203 is a HDD (Hard Disk Drive) or a memory card.

[0112] The display device 204 is, for example, a liquid crystal display. The input device 205 is an interface for operating the speech recognition device 100 (100-2). In addition, the display device 204 and the input device 205 can also be implemented by a touch panel having a display function and an input function. The communication device 206 is an interface for communicating with other devices.

[0113] For example, the program executed by the speech recognition device 100 (100-2) is recorded in a computer-readable storage medium such as a memory card, hard disk, CD-RW, CD-ROM, CD-R, DVD-RAM, and DVD-R in an installable or executable form, and is provided as a computer program product.

[0114] Furthermore, for example, a configuration may be adopted in which the program executed by the speech recognition apparatus 100 ( 100 - 2 ) is stored in a computer connected to a network such as the Internet, and is provided by being downloaded via the network.

[0115] In addition, for example, the program executed by the speech recognition device 100 (100-2) may be provided via a network such as the Internet without being downloaded. Specifically, the speech recognition process may be performed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by its execution instructions and result acquisition, without transmitting the program from the server computer.

[0116] Furthermore, for example, the program of the speech recognition device 100 ( 100 - 2 ) may be installed in advance in a ROM or the like and provided.

[0117] The program executed by the speech recognition device 100 (100-2) is a module structure including the functions that can be realized by the program in the above-mentioned functional structure. Each of these functions is actual hardware, and the processor 201 reads the program from the storage medium and executes it, and the above-mentioned functional blocks are loaded into the main storage device 202. That is, the above-mentioned functional blocks are generated on the main storage device 202.

[0118] Furthermore, part or all of the above functions may be realized not by software but by hardware such as IC (Integrated Circuit).

[0119] Furthermore, each function may be implemented using a plurality of processors 201 . In this case, each processor 201 may implement one of each function, or may implement two or more of each function.

[0120] Several embodiments of the present invention have been described, and these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the invention. These embodiments and their variations are included in the scope and purpose of the invention, and are included in the invention described in the scope of the patent claim and the scope equivalent thereto.

Claims

1. An audio signal processing device, comprising: The spatial filter control unit outputs a spatial filter for emphasizing a target voice component and suppressing a noise component for N acoustic signals recorded at different positions and synchronized in time, wherein: N≥2； A spatial filter storage unit for storing the spatial filter; as well as The sound processing unit uses the spatial filter read from the spatial filter storage unit to emphasize the target voice component of the sound signal and suppress the noise component of the sound signal. The above-mentioned spatial filter control unit comprises: a determination unit for determining whether the sound signal is a target speech or noise; a speech space correlation calculation unit that calculates a speech space correlation matrix using the speech section determined to be the target speech in the acoustic signal; a noise spatial correlation calculation unit that calculates a noise spatial correlation matrix using a noise section determined to be the noise in the acoustic signal; A spatial correlation storage unit, storing the speech spatial correlation matrix and the noise spatial correlation matrix; as well as The spatial filter calculation unit calculates a spatial filter for emphasizing a target speech component and suppressing a noise component based on the speech spatial correlation matrix and the noise spatial correlation matrix read out from the spatial correlation storage unit.

2. The sound signal processing device according to claim 1, wherein: performing at least one of a speech space correlation matrix update process and a noise space correlation matrix update process, In the speech space correlation matrix update process, when the sound signal is determined by the determination unit to be the target speech, the speech space correlation calculation unit reads the speech space correlation matrix from the space correlation storage unit, updates the speech space correlation matrix, and writes the updated speech space correlation matrix into the space correlation storage unit. In the noise space correlation matrix update process, when the sound signal is determined by the determination unit to be the noise, the noise space correlation calculation unit reads the noise space correlation matrix from the spatial correlation storage unit, updates the noise space correlation matrix, and writes the updated noise space correlation matrix into the spatial correlation storage unit.

3. The sound signal processing device according to claim 1 or 2, wherein: The determination unit includes an acoustic signal storage unit that stores acoustic signals from a current time to a predetermined period of time ago. The determination unit determines whether the sound signal included in each time from the current time to a predetermined period before is the target voice. The above-mentioned speech space correlation calculation unit and the above-mentioned noise space correlation calculation unit use the sound signal contained from the above-mentioned current moment to the specified period before to update the above-mentioned speech space correlation matrix and the above-mentioned noise space correlation matrix, and store the updated above-mentioned speech space correlation matrix and the above-mentioned noise space correlation matrix in the above-mentioned space correlation storage unit.

4. The sound signal processing device according to claim 1 or 2, wherein: The determination unit includes an acoustic signal storage unit that stores acoustic signals from a current time to a predetermined period of time ago. The determination unit determines whether the sound signal before the predetermined period is the noise. The speech space correlation calculation unit updates the speech space correlation matrix using the acoustic signal at the current moment, and stores the updated speech space correlation matrix in the space correlation storage unit. When the acoustic signal before the predetermined period is determined to be noise, the noise spatial correlation calculation unit updates the noise spatial correlation matrix using the acoustic signal before the predetermined period, and stores the updated noise spatial correlation matrix in the spatial correlation storage unit.

5. The sound signal processing device according to claim 1 or 2, wherein: The determination unit calculates a value of a speech score indicating a degree of speech fitting for the acoustic signal, and determines that the acoustic signal is the target speech when the speech score is greater than a speech threshold, and determines that the acoustic signal is the noise when the speech score is less than the speech threshold.

6. The sound signal processing device according to claim 1 or 2, wherein: The determination unit calculates a value of a speech score indicating a degree of speech fit and a noise score indicating a degree of noise fit for the acoustic signal, and determines that the acoustic signal is the target speech when the speech score is greater than a speech threshold, and determines that the acoustic signal is the noise when the noise score is greater than a noise threshold.

7. The sound signal processing device according to claim 5, wherein: The determination unit includes a sound source number estimator for estimating the number of sound sources included in the acoustic signal. The above-mentioned speech score is represented by a function of the above-mentioned number of sound sources.

8. The sound signal processing device according to claim 5, wherein: The voice score is represented by a function based on statistics of the power of the acoustic signal.

9. The sound signal processing device according to claim 1 or 2, wherein: The judgment unit uses a facial image of the target speaker to calculate a speech score indicating whether the target speaker is speaking, and when the speech score is greater than an image threshold, the sound signal is judged to be the target speech; and when the speech score is less than the image threshold, the sound signal is judged to be the noise.

10. The sound signal processing device according to claim 1 or 2, wherein: The above-mentioned spatial filter control unit comprises: a speaker direction setting unit that maintains a relative direction of a target speaker as viewed from a microphone; and The sound source localization unit localizes the sound source of the sound signal and outputs the detected sound source direction. The determination unit determines at least one of the sound source directions as the target speech when the angle difference from the relative direction set by the speaker direction setting unit is less than an angle threshold, and determines it as the noise when the angle difference is greater than the angle threshold.

11. The sound signal processing device according to claim 1 or 2, wherein: The spatial filter control unit includes a sound source separation unit that performs sound source separation on the input sound signal and outputs a separated sound signal separated into the target speech component and the noise component. The separated sound signal is input to at least one of the determination unit, the speech space correlation calculation unit, the noise space correlation calculation unit, and the sound processing unit.

12. The sound signal processing device according to claim 1 or 2, wherein: The sound signal output from the sound processing unit is input to at least one of the determination unit, the speech space correlation calculation unit, and the noise space correlation calculation unit.

13. The sound signal processing device according to claim 1 or 2, wherein: The determination unit outputs a continuous value indicating the degree to which the acoustic signal is the target speech. The speech space correlation calculation unit calculates the speech space correlation matrix by increasing the weight of the ratio of updating the value of the speech space correlation matrix as the continuous value increases. The noise space correlation calculation unit calculates the noise space correlation matrix by increasing the weight of the ratio of updating the value of the noise space correlation matrix as the continuous value decreases.

14. The sound signal processing device according to claim 1 or 2, wherein: The determination unit calculates a target speech score indicating a degree of fit to the target speech and a noise score indicating a degree of fit to the noise for the acoustic signal. When the target speech score is greater than the speech threshold, it is determined that the sound signal is the target speech. When the noise score is greater than a noise threshold value, it is determined that the acoustic signal is the noise.

15. A method for processing an audio signal, comprising the following steps: The sound signal processing device outputs a spatial filter for emphasizing a target voice component and suppressing a noise component for N sound signals recorded at different positions and synchronized in time, wherein: N≥2； The above-mentioned sound signal processing device stores the above-mentioned spatial filter in a spatial filter storage unit; The sound signal processing device uses the spatial filter read from the spatial filter storage unit to emphasize the target speech component of the sound signal and suppress the noise component of the sound signal. The step of outputting the above spatial filter comprises the following steps: Determine whether the sound signal is a target voice or noise; calculating a speech space correlation matrix using the speech section determined to be the target speech in the acoustic signal; calculating a noise spatial correlation matrix using a noise section determined to be the noise in the acoustic signal; Storing the speech spatial correlation matrix and the noise spatial correlation matrix in a spatial correlation storage unit; as well as A spatial filter for emphasizing a target speech component and suppressing a noise component is calculated based on the speech spatial correlation matrix and the noise spatial correlation matrix read out from the spatial correlation storage unit.

16. A program, wherein: The above program causes the computer to function as a spatial filter control unit, a spatial filter storage unit, and an audio processing unit. The spatial filter control unit outputs a spatial filter that emphasizes a target voice component and suppresses a noise component for N acoustic signals recorded at different positions and synchronized in time, wherein N ≥ 2, The spatial filter storage unit stores the spatial filter. The sound processing unit emphasizes a target voice component of the sound signal and suppresses a noise component of the sound signal using the spatial filter read from the spatial filter storage unit. The spatial filter control unit has: a determination unit for determining whether the sound signal is a target speech or noise; a speech space correlation calculation unit that calculates a speech space correlation matrix using the speech section determined to be the target speech in the acoustic signal; a noise spatial correlation calculation unit that calculates a noise spatial correlation matrix using a noise section determined to be the noise in the acoustic signal; A spatial correlation storage unit for storing the speech spatial correlation matrix and the noise spatial correlation matrix; and The spatial filter calculation unit calculates a spatial filter for emphasizing a target speech component and suppressing a noise component based on the speech spatial correlation matrix and the noise spatial correlation matrix read out from the spatial correlation storage unit.

Images