Estimation device, estimation method, and program

Abstract

To enable robust estimation of an arrival direction of a sound source at lower calculation cost.SOLUTION: An estimation device according to an embodiment includes: a conversion part; a spatial correlation calculation part; a spatial correlation filter part; and a direction estimation part. The conversion part acquires a frequency spectrum by performing time frequency conversion on acoustic signals of a plurality of channels. The spatial correlation calculation part calculates a spatial correlation matrix from the frequency spectrum. The spatial correlation filter part calculates a spatial correlation filter from the spatial correlation matrix. The direction estimation part estimates general direction information from a partial element included in the spatial correlation filter.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] An embodiment of the present invention relates to an estimation device, an estimation method, and a program. [Background technology]

[0002] A method has been known in the art in which acoustic signals input from multiple microphones are frequency-converted, and beamforming is performed from the frequency-converted time-frequency converted signal using a spatial correlation filter based on target sound source direction information that indicates the direction relative to the target sound source contained in the sound signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7182168 [Patent Document 2] Patent No. 3677143 [Patent Document 3] Patent No. 7191793 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, it has been difficult to robustly estimate the direction of arrival of a sound source at a low computational cost. [Means for solving the problem]

[0005] The estimation device according to the embodiment includes a transform unit, a spatial correlation calculation unit, a spatial correlation filter unit, and a direction estimation unit. The transform unit acquires a frequency spectrum by performing a time-frequency transformation on sound signals of a plurality of channels. The spatial correlation calculation unit calculates a spatial correlation matrix from the frequency spectrum. The spatial correlation filter unit calculates a spatial correlation filter from the spatial correlation matrix. The direction estimation unit estimates general direction information from subelements included in the spatial correlation filter. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing an example of an arrangement of a plurality of microphones according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of the estimation device according to the first embodiment. [Figure 3A] FIG. 3A is a diagram for explaining a phase spectrum. [Figure 3B] FIG. 3B is a diagram for explaining the phase spectrum. [Figure 4] FIG. 4 is a diagram showing an example of the spatial correlation filter according to the first embodiment and an application example thereof. [Diagram 5] FIG. 5 is a diagram illustrating an example of a functional configuration of the direction estimation unit according to the first embodiment. [Figure 6A] FIG. 6A is a diagram showing an estimation example 1 of general directional information according to the first embodiment. [Figure 6B] FIG. 6B is a diagram showing an estimation example 2 of general direction information according to the first embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of the estimation method according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a functional configuration of a direction estimation unit according to the second embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of the estimation method according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a hardware configuration of the estimation device according to the first and second embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of an estimation device, an estimation method, and a program will be described in detail with reference to the accompanying drawings.

[0008] (First embodiment) When controlling the notification result of a voice recognition pattern, etc., it is not necessary to perform beamforming, and it may be sufficient to obtain general directional information such as left / right estimation. In addition, when the target sound source is not completely fixed and moves slightly, or when the distance between the microphones changes, strict directional estimation may not be suitable for the purpose. On the other hand, the direction determination may be required to be robust against ambient noise.

[0009] An estimation device according to a first embodiment that can reduce calculation costs and enhance noise resistance will be described below.

[0010] First, an example of the arrangement of a plurality of microphones in the first embodiment will be described.

[0011] [Example of multiple microphone placement] FIG. 1 is a diagram showing an example of the arrangement of multiple microphones L and R in the first embodiment. The example in FIG. 1 shows an in-vehicle use case. The estimation device of the first embodiment estimates the general arrival direction of a sound source (left or right in the example in FIG. 1) from information acquired by two microphones L and R. The general arrival direction of the sound source is utilized, for example, for device control by a keyword detected by voice recognition.

[0012] Specifically, for example, in-car entertainment operations such as music, television, and radio (for example, operations by user voice input such as "play music") can be controlled to respond to any voice. In the example of Figure 1, in-car entertainment operation device control responds to both voices coming from the right (i.e., the voice of the user sitting in the passenger seat) and voices coming from the left (i.e., the voice of the driver sitting in the driver's seat).

[0013] Also, for example, in driving assistance control using driving operation keywords (such as "Riamonitor"), it is possible to make the system respond only to the voice of the driver in the driver's seat. In the example of driving assistance control in Figure 1, the system responds to voices coming from the left (i.e., the voice of the driver sitting in the driver's seat) and does not respond to voices coming from the right (i.e., the voice of the user sitting in the passenger seat).

[0014] Depending on the arrangement of the microphones, the estimation device of the first embodiment can estimate not only left and right but also general directions of arrival such as up and down, front and back, etc. Also, the number of microphones (channels) is not limited to two, and may be three or more.

[0015] [Example of functional configuration] 2 is a diagram illustrating an example of a functional configuration of the estimation device 10 according to the first embodiment. The estimation device 10 according to the first embodiment includes a conversion unit 1, a delay unit 2, a spatial correlation calculation unit 3, a spatial correlation calculation unit 4, a spatial correlation filter unit 5, a direction estimation unit 6, and a control unit 7.

[0016] The converter 1 converts the acoustic signal input from the microphone L into a frequency spectrum X[0][size] by performing a time-frequency conversion. Here, [0] indicates the channel number indicating that the input is from the microphone L. [size] indicates the frequency bin number. The time-frequency conversion is calculated by processing such as a fast Fourier transform and a discrete Fourier transform.

[0017] Similarly, the converter 1 converts the acoustic signal input from the microphone R into a frequency spectrum X[1][size] by performing a time-frequency conversion, where [1] indicates the number of channels indicating that the input is from the microphone R.

[0018] Hereinafter, when there is no need to distinguish between the frequency spectra X[0][size] and X[1][size], they will be represented as frequency spectrum X[ch][size].

[0019] The components of the frequency spectrum X[ch][size], that is, the components of the time-frequency transformed acoustic signal, are expressed as a complex spectrum as shown in the following equation (1).

[0020]

number

[0021] Here, re denotes a real part and im denotes an imaginary part. This complex spectrum is used to calculate, for example, a power spectrum (amplitude component for each frequency) according to the following formula (2) and a phase spectrum (phase component for each frequency) according to the following formula (3).

[0022]

number

[0023]

number

[0024] 3A and 3B are diagrams for explaining the phase spectrum. The phase spectrum is an index showing how much an acoustic signal is ahead (or behind) a reference time. As shown in FIG. 3A, if there is a directional difference in the direction of arrival of a sound from a sound source, the arrival distance to each of microphones L and R changes by Δd. Therefore, a phase delay Δd occurs in the acoustic signal arriving at microphone R, as shown in FIG. 3B.

[0025] Returning to FIG. 2, when the delay unit 2 receives the frequency spectrum X[ch][size] from the conversion unit 1, it delays the frequency spectrum X[ch][size] by a predetermined time. The delay time may be set so that the duration of the command word utterance, which is the sound to be emphasized, is approximately equal to or shorter than the delay time. If the command word is a short word such as "switch on," it can be assumed that the duration of the command word utterance is 1 second or less. In this case, the delay time is set to, for example, 1 second. A delay time of 1 second is equal to 125 (=16000×1 / 128) delay frames. That is, the delay unit 2 buffers 125 frames of the frequency spectrum X[ch][size] and outputs X[ch][size] delayed by 125 frames.

[0026] The spatial correlation calculation unit 3 calculates a spatial correlation matrix between speech and noise based on the frequency spectrum X[ch][size]. The spatial correlation matrix is ​​information that indicates the spatial correlation between channels, and expresses the spatial energy distribution. Specifically, the spatial correlation calculation unit 3 first calculates a mixing matrix signal Conv[size][f] from the time-frequency converted acoustic signal (frequency spectrum X[ch][size]). The mixing matrix Conv[size][f] is calculated so that information from multiple channels is mixed, as shown in the following formula (4).

[0027]

number

[0028] Here, f indicates the element number. Note that the mixing matrix signal has elements that contain information on all of the multiple channels. In the above formula (4), the element number is f 3 and f 4 The mixing matrix includes information including the phase differences of a plurality of channels (channels 0 and 1 in the first embodiment).

[0029] Next, the spatial correlation calculation unit 3 calculates spatial correlation matrices Φs[size][f] and Φn[size][f] from the mixing matrix Conv[size][f] by the following formula (5), where s represents the signal and n represents the noise.

[0030]

number

[0031] Similarly, the spatial correlation calculation unit 4 calculates the spatial correlation matrix of the voice and noise based on the frequency spectrum X[ch][size] received from the delay unit 2. As in the spatial correlation calculation units 3 and 4 of the first embodiment, a plurality of spatial correlation matrix signals may be calculated. For example, the spatial correlation matrix at the current time calculated by the spatial correlation calculation unit 3 may be the spatial correlation matrix signal component, and the spatial correlation matrix calculated by the spatial correlation calculation unit 4 a certain time ago (a predetermined number of frames ago) may be the spatial correlation noise component (see Patent Document 3 for details).

[0032] In addition, when the spatial correlation matrix for a certain time before (a predetermined number of frames before) is not calculated, the delay unit 2 and the spatial correlation calculation unit 4 do not have to be provided in the estimation device 10.

[0033] The spatial correlation filter unit 5 calculates one or more spatial correlation filters (two spatial correlation filters in the first embodiment) based on the spatial correlation matrices Φs[size][f] and Φn[size][f]. Specifically, the spatial correlation filter unit 5 first calculates an eigenvalue vector signal from the spatial correlation matrix signals Φs[size][f] and Φn[size][f]. In calculating the eigenvalue vector signal, in order to reduce the processing load, an eigenvalue vector of approximately two dimensions may be used, such as the two-dimensional eigenvalue vector M[size] of the following formula (6).

[0034]

number

[0035] For each element of the eigenvalue vector, there is an element that contains all the information of multiple channels. In the example of the above formula (6), the element number is f 3 and f 4 Each eigenvalue M using the mixing matrix above contains phase difference information.

[0036] Next, the spatial correlation filter unit 5 calculates spatial correlation filter coefficients from the eigenvalue vector signal. For example, in the case of a two-dimensional eigenvalue, four element components are generated.

[0037] Fig. 4 is a diagram showing an example of the spatial correlation filter Vector[size] of the first embodiment and an application example thereof. In the example of Fig. 4, the spatial correlation filter coefficients are calculated so as to maximize the signal-to-noise ratio (SNR) of the signal component at the current time.

[0038] It should be noted that a general direction difference can be determined from one element of the spatial correlation filter Vector[size]. The method of determining a general direction difference will be described in detail below with reference to FIG.

[0039] In addition, multiple elements of the spatial correlation filter Vector[size] may be used. For example, one or more elements of the spatial correlation filter Vector[size] may be used to adjust parameters of the general direction estimation.

[0040] Moreover, by multiplying each element of the time-frequency converted acoustic signal by a spatial correlation filter Vector[size], it is possible to emphasize the signal at the current time.

[0041] Returning to Fig. 2, the direction estimation unit 6 estimates general direction information from one or more partial elements of the spatial correlation filter. The general direction information is not strict direction information, but general information such as left / right information. For example, the general direction information is a direction indicated by an angle of θ or -θ, left or right, front or back, and up or down.

[0042] The general directional information is used, for example, to control the notification of the result of the recognition pattern. The spatial correlation filter obtained from the spatial correlation matrix of the voice and noise is used to strengthen the resistance to ambient noise. In addition, since the estimation is limited to the general directional information, the calculation cost can be reduced. In addition, the estimation of the general directional information is not easily affected by the distance between the microphones or the movement of the target sound source.

[0043] The control unit 7 changes the control by the voice recognition pattern recognized from the acoustic signal based on the general direction information. For example, the control unit 7 estimates whether the voice is coming from the driver's seat or the passenger seat based on the general direction information, and changes the control by the voice recognition pattern based on the estimation result.

[0044] 5 is a diagram showing an example of a functional configuration of the direction estimation unit 6 according to the first embodiment. The direction estimation unit 6 according to the first embodiment includes an extraction unit 61, an estimation unit 62, and a smoothing processing unit 63.

[0045] The extraction unit 61 extracts one element of the spatial correlation filter coefficient as a major element. As described above, when it is a two-dimensional eigenvalue, four element components are generated. The extraction unit 61 narrows down the four element components to one element. Specifically, the extraction unit 61 extracts an element that includes information of all of a plurality of channels (for example, information indicating the phase difference of a plurality of channels) as a major element. When there are a plurality of elements that include information of all of a plurality of channels, the extraction unit 61 may extract a unique element based on the effect (the estimation accuracy of general directional information), or may extract or change one element later.

[0046] Of course, the extraction unit 61 may extract not only one type of element but also multiple elements at the same time and calculate the difference between them. Also, the extraction unit 1 may extract auxiliary elements other than the main elements, and the estimation unit 62 may weight the auxiliary elements to correct the local direction information.

[0047] The estimation unit 62 outputs the local direction estimated based on the main elements extracted by the extraction unit 61 or the main elements and auxiliary elements as general direction information. For example, the estimation unit 62 identifies a trend of a specific frequency at a specific time from the main elements, estimates local direction information indicated by the trend, and outputs general direction information based on the local direction information. Specifically, since the main elements of the spatial correlation filter coefficients are information for each frequency, the estimation unit 62 adds up and averages the main elements in a grouped frequency band. Then, the estimation unit 62 outputs the local direction information indicating the averaged main elements as general direction information.

[0048] 6A and 6B are diagrams showing estimation examples 1 and 2 of general direction information in the first embodiment. The values ​​of the principal components in Figs. 6A and 6B indicate local direction information obtained from a partial element (e.g., one element) included in a spatial correlation filter. The local direction information includes a trend of a specific frequency at a specific time. Figs. 6A and 6B show examples in which the general direction information estimated from the principal element of the spatial correlation filter is in the +30 degree direction and the -30 degree direction.

[0049] The estimation unit 62 estimates the local directional information based on whether the trend of a particular frequency at a particular time is greater than a predetermined value. In the example of Figures 6A and 6B, the predetermined value is 0.

[0050] In Fig. 6A, the main components indicating the local direction have more positive values ​​than negative values. Therefore, when the estimation unit 62 obtains the local direction information in Fig. 6A, it estimates that the sound is from a +30 degree direction (for example, a sound from the passenger seat).

[0051] On the other hand, in Fig. 6B, the main component indicating the local direction has more negative values ​​than positive values. Therefore, when the estimation unit 62 obtains the local direction information of Fig. 6B, it estimates that the voice is from a +30 degree direction (for example, a voice coming from the driver's seat).

[0052] The estimation unit 62 may add an arbitrary adjustment value to the local direction information. By adding the adjustment value, it becomes possible to adjust the judgment threshold. For example, in the example of Figs. 6A and 6B, the adjustment value can adjust whether a value near 0 for the main component indicates that the sound is in the +30 degree direction or the -30 degree direction. The adjustment value can further improve the estimation accuracy of the general direction information.

[0053] The frequency band to be collected may be the entire band from low to high, or may be limited to the range of the voice band that contains a large amount of voice components. In addition, the estimation unit 62 may give weighting to specific frequency components. The estimation unit 62 may also average the local direction information in the time direction. By averaging in this way, the local direction can be output as a more general direction.

[0054] [Example of estimation method] 7 is a flowchart showing an example of the estimation method of the first embodiment. First, the conversion unit 1 accepts input of a multi-channel voice (acoustic signal) (step S1) and performs a time-frequency conversion of the input acoustic signal (step S2). Next, the spatial correlation calculation unit 3 calculates a spatial correlation matrix between the voice and noise based on the frequency spectrum X[ch][size] obtained by the time-frequency conversion (step S3).

[0055] Next, the extraction unit 61 extracts a partial element (for example, one element) of the spatial correlation filter coefficient as a main element (step S4). Next, the estimation unit 62 calculates general directional information from the partial element included in the spatial correlation filter (step S5).

[0056] Next, the control unit 7 changes the control of the voice recognition pattern based on the general direction information (step S6).

[0057] As described above, in the estimation device 10 of the first embodiment, the conversion unit 1 acquires a frequency spectrum by performing a time-frequency conversion on the acoustic signals of multiple channels. The spatial correlation calculation unit 3 calculates a spatial correlation matrix from the frequency spectrum. The spatial correlation filter unit 5 calculates a spatial correlation filter from the spatial correlation matrix. Then, the direction estimation unit 6 estimates general direction information from subelements included in the spatial correlation filter.

[0058] As a result, the estimation device 10 of the first embodiment can robustly estimate the direction of arrival of the sound source with lower calculation cost.

[0059] Second embodiment Next, a second embodiment will be described. In the description of the second embodiment, the description of the same parts as in the first embodiment will be omitted, and only the parts different from the first embodiment will be described.

[0060] [Example of functional configuration] 8 is a diagram illustrating an example of a functional configuration of a direction estimation unit 6 according to the second embodiment. The direction estimation unit 6 according to the second embodiment includes an extraction unit 61, an estimation unit 62, and a smoothing processing unit 63. In the second embodiment, the smoothing processing unit 63 is further added to the configuration of the first embodiment.

[0061] The estimation unit 62 estimates a local direction indicated by a trend of a specific frequency at a specific time, based on the subelements extracted by the extraction unit 61.

[0062] The smoothing processing unit 63 adjusts the local direction information obtained by the estimation unit 62, and finally outputs the adjusted local direction information as general direction information. For example, the smoothing processing unit 63 performs smoothing correction in at least one of the time direction and the band direction, and outputs general direction information based on the smoothing-corrected local direction information. By smoothing correcting the local direction information in at least one of the time direction and the band direction (the time direction, the band direction, or both), it is possible to further reduce the strictness of the estimation result of the direction information. This makes it possible to output general direction information that is more resistant to disturbance factors.

[0063] [Example of estimation method] 9 is a flowchart showing an example of an estimation method according to the second embodiment. Steps S11 to S14 are the same as steps S1 to S4 in the first embodiment, and therefore a description thereof will be omitted.

[0064] The estimation unit 62 estimates a local direction including a trend of a specific frequency at a specific time based on the subelements extracted by the extraction unit 61 (step S15).

[0065] Next, the smoothing processing unit 63 smoothes the local direction obtained by the estimation unit 62 to calculate general direction information (step S16).

[0066] Step S17 is the same as step S6 in the first embodiment, and therefore a description thereof will be omitted.

[0067] Finally, an example of the hardware configuration of the estimation device 10 according to the first and second embodiments will be described.

[0068] [Example of hardware configuration] 10 is a diagram illustrating an example of a hardware configuration of the estimation device 10 of the first and second embodiments. The estimation device 10 of the first and second embodiments includes a processor 201, a main storage device 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main storage device 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0069] Note that some of the above configurations may not be included in the estimation device 10. For example, when the estimation device 10 can use an input function and a display function of an external device, the estimation device 10 does not need to be provided with the display device 204 and the input device 205.

[0070] The processor 201 executes a program read from the auxiliary storage device 203 to the main storage device 202. The main storage device 202 is a memory such as a ROM and a RAM. The auxiliary storage device 203 is a hard disk drive (HDD), a memory card, or the like.

[0071] The display device 204 is, for example, a liquid crystal display. The input device 205 is an interface for operating the estimation device 10. The display device 204 and the input device 205 may be realized by a touch panel or the like having a display function and an input function. The communication device 206 is an interface for communicating with other devices.

[0072] For example, the program executed by the estimation device 10 is provided as a computer program product in the form of an installable or executable file recorded on a computer-readable storage medium such as a memory card, a hard disk, a CD-RW, a CD-ROM, a CD-R, a DVD-RAM, or a DVD-R.

[0073] Also, for example, the program executed by the estimation device 10 may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0074] Also, for example, the program executed by the estimation device 10 may be provided via a network such as the Internet without being downloaded. Specifically, the estimation process may be performed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by issuing an execution instruction and obtaining the result, without transferring the program from a server computer.

[0075] Also, for example, the program for the estimation device 10 may be provided by being pre-installed in a ROM or the like.

[0076] The program executed by the estimation device 10 has a modular configuration including functions that can be realized by the program among the above-mentioned functional configurations. As for each function, as actual hardware, the processor 201 reads the program from a storage medium and executes it, and the above-mentioned functional blocks are loaded onto the main storage device 202. In other words, the above-mentioned functional blocks are generated on the main storage device 202.

[0077] Note that some or all of the above-described functions may be realized by hardware such as an integrated circuit (IC) instead of by software.

[0078] Furthermore, each function may be realized using a plurality of processors 201. In this case, each processor 201 may realize one of the functions, or may realize two or more of the functions.

[0079] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0080] L,R microphone 1. Conversion section 2 Delay section 3. Spatial correlation calculation section 4. Spatial correlation calculation section 5 Spatial correlation filter section 6 Direction estimation part 7 Control section 10 Estimation device 61 Extraction part 62 Estimation part 63 Smoothing processing section 201 Processor 202 Main storage 203 Auxiliary storage device 204 Display device 205 Input Device 206 Communication Equipment 210 Bus

Claims

1. A conversion unit that performs a time-frequency conversion on an audio signal of a plurality of channels to obtain a frequency spectrum; A spatial correlation calculation unit that calculates a spatial correlation matrix from the frequency spectrum; a spatial correlation filter unit that calculates a spatial correlation filter from the spatial correlation matrix; a direction estimation unit that estimates general direction information from partial elements included in the spatial correlation filter; An estimation device comprising:

2. The direction estimation unit is an extractor for extracting a key component including information indicating a phase difference between the plurality of channels from the spatial correlation filter; an estimation unit that identifies a trend of a specific frequency at a specific time from the main elements, estimates local directional information indicated by the trend, and outputs the general directional information based on the local directional information; The estimation device according to claim 1 , comprising:

3. The direction estimation unit is an extractor for extracting a key component including information indicating a phase difference between the plurality of channels from the spatial correlation filter; an estimation unit that identifies a trend of a specific frequency at a specific time from the main elements and estimates local directional information indicated by the trend; a smoothing processing unit that performs smoothing correction on the local direction information in at least one of a time direction and a band direction, and outputs the general direction information based on the smoothing-corrected local direction information; The estimation device according to claim 1 , comprising:

4. The estimation unit estimates the local directional information based on whether a trend of a specific frequency at the specific time is greater than a predetermined value. The estimation device according to claim 2 or 3.

5. The estimation unit adjusts the predetermined value by an adjustment value that improves estimation accuracy of the general direction information. The estimation device according to claim 4.

6. The general direction information indicates a direction indicated by an angle of +θ or −θ, left or right, front or back, and up or down. The estimation device according to any one of claims 1 to 3.

7. a control unit that changes control based on a voice recognition pattern recognized from the acoustic signal based on the general direction information; The estimation device according to claim 1 , further comprising:

8. The estimation device performs a time-frequency transform on a multi-channel acoustic signal to obtain a frequency spectrum; a step of the estimation device calculating a spatial correlation matrix from the frequency spectrum; the estimation device calculating a spatial correlation filter from the spatial correlation matrix; The estimation device estimates general directional information from subelements included in the spatial correlation filter; An estimation method including:

9. Computer, A conversion unit that performs a time-frequency conversion on an audio signal of a plurality of channels to obtain a frequency spectrum; A spatial correlation calculation unit that calculates a spatial correlation matrix from the frequency spectrum; a spatial correlation filter unit that calculates a spatial correlation filter from the spatial correlation matrix; a direction estimation unit that estimates general direction information from partial elements included in the spatial correlation filter; A program that functions as a

Images