Sound processing device, sound processing method and program

Abstract

To estimate a transfer function that is modified in a real sound environment.SOLUTION: In an acoustic processing device, a storge part stores a sound from sound source indicating a transmission characteristics as a first transfer function in each sound source direction, a sound source direction estimation part calculates a spatial spectrum in each sound source direction on the basis of a conversion coefficient in a frequency region of an acoustic signal in ech channel and the first transfer function, the sound direction of which the spatial spectrum becomes the maximum is estimated as an estimation sound source direction, a transfer function estimation part normalizes the conversion coefficient between channels, and estimates the transfer function to the estimation sound source direction as a second transfer function, and a transfer function updating part updates the first transfer function to the estimation sound source direction by using the second transfer function.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a sound processing device, a sound processing method, and a program. [Background technology]

[0002] Sound source localization and sound source separation are elemental technologies in acoustic signal processing. Sound source localization is a method for estimating the direction of a sound source from a multi-channel sound signal received using a microphone array. Sound source separation is a method for extracting components coming from individual sound sources from a multi-channel sound signal. When multiple sound sources are produced simultaneously, such as speech in a noisy environment, sound source localization and sound source separation are useful for focusing on a specific sound. Sound source localization and sound source separation are applied to various fields, including robot audition, smart speakers, teleconferencing systems, and minutes creation. In robot audition, they are sometimes used for communication with people or understanding auditory scenes.

[0003] In sound source localization and sound source separation, a transfer function that indicates the transfer characteristics from a sound source to a sound receiving point is used. Since the positional relationship between the sound source and the sound receiving point is fixed, the transfer function is defined as a static function. Since the transfer function is generally unknown in a real acoustic environment, it is common to obtain a series of transfer functions in advance. The transfer function is obtained, for example, by calculating using a mathematical model assuming a free sound field (Patent Document 1), or by measuring the transfer function of different sound source directions in a laboratory. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-144044 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the transfer function acquired in advance inevitably differs from the transfer function measured in the actual acoustic environment. This may result in a significant decrease in the performance of sound source localization and sound source separation. On the other hand, measuring the transfer function every time the acoustic environment used is changed results in a burden in terms of time and work. Even if the transfer function can be measured appropriately, it is likely to change depending on the arrangement of various objects in the acoustic environment. In addition, the transfer function may vary depending on the indoor environment, such as temperature, air pressure, and humidity.

[0006] The present embodiment has been made in consideration of the above points, and has an object to provide a sound processing device, a sound processing method, and a program capable of estimating a transfer function that varies in a real acoustic environment. [Means for solving the problem]

[0007] (1) The present invention has been made to solve the above problems, and one aspect of the present invention is a first transfer function indicating a transfer characteristic of a sound from a sound source. of A storage unit that stores information for each sound source direction; Supports multiple microphones a sound source direction estimating unit that calculates a spatial spectrum for each sound source direction based on a transform coefficient in the frequency domain of the acoustic signal for each channel and the first transfer function, and estimates the sound source direction in which the spatial spectrum is maximum as an estimated sound source direction; and The obtained value is With respect to the estimated sound source direction The A transfer function estimation unit that estimates the transfer function as two transfer functions; When the number of sound sources detected from the acoustic signal is one, a transfer function update unit that updates the first transfer function with respect to the estimated sound source direction by using the second transfer function. (2) Another aspect of the present invention is The first transfer function indicates the transfer characteristics of sound from a sound source. of A storage unit that stores information for each sound source direction; Supports multiple microphonesa sound source direction estimating unit that calculates a spatial spectrum for each sound source direction based on a transform coefficient in the frequency domain of the acoustic signal for each channel and the first transfer function, and estimates the sound source direction in which the spatial spectrum is maximum as an estimated sound source direction; and The obtained value is With respect to the estimated sound source direction The a transfer function estimating unit that estimates the first transfer function with respect to the estimated sound source direction as a second transfer function, and a transfer function updating unit that updates the first transfer function with respect to the estimated sound source direction by using the second transfer function. The transfer function estimation unit normalizes the amplitude of the transform coefficient for each channel by an inter-channel norm of the transform coefficient, and normalizes the phase of the transform coefficient for each channel by a phase of a sum of the transform coefficients for each channel. It is an audio processing device.

[0008] (3) Another aspect of the present invention is (1) Or (2) In the sound processing device, the transfer function update unit may update at least a portion of components of the first transfer function with the components of the second transfer function at predetermined time intervals.

[0011] (4) Another aspect of the present invention is the (3) In any one of the sound processing devices described above, the sound source direction estimating unit may calculate, as the spatial spectrum, a multi-signal classification spectrum based on the transformation coefficients and the first transfer function.

[0012] (5) Another aspect of the present invention is the (4) Any of the above sound processing devices, a sound source separation unit that determines a separation matrix for the estimated sound source direction based on a first transfer function for the estimated sound source direction, and outputs a vector calculated by applying the separation matrix to an input vector having the transformation coefficients as elements as an output vector having sound source components arriving for each sound source as elements. This is also fine.

[0013] (6) Another aspect of the present invention is to provide a method for controlling a computer to execute a program comprising: (5) The present invention may also be a program for causing the device to function as any one of the above sound processing devices.

[0014] (7) Another aspect of the present invention is a first transfer function that indicates a transfer characteristic of a sound from a sound source. of A method for a sound processing device having a storage unit that stores information for each sound source direction, Supports multiple microphonesa sound source direction estimating step of calculating a spatial spectrum for each sound source direction based on a transform coefficient in the frequency domain of the acoustic signal for each channel and the first transfer function, and estimating the sound source direction in which the spatial spectrum is maximum as an estimated sound source direction; and normalizing the transform coefficient between channels to obtain a sound source direction. The obtained value is With respect to the estimated sound source direction The A transfer function estimation step in which the transfer function is estimated as two transfer functions; When the number of sound sources detected from the acoustic signal is one, and a transfer function updating step of updating the first transfer function with respect to the estimated sound source direction using the second transfer function. (8) Another aspect of the present invention is a first transfer function that indicates a transfer characteristic of a sound from a sound source. of A method for a sound processing device having a storage unit that stores information for each sound source direction, Supports multiple microphones a sound source direction estimating step of calculating a spatial spectrum for each sound source direction based on a transform coefficient in the frequency domain of the acoustic signal for each channel and the first transfer function, and estimating the sound source direction in which the spatial spectrum is maximum as an estimated sound source direction; and normalizing the transform coefficient between channels to obtain a sound source direction. The obtained value is With respect to the estimated sound source direction The and a transfer function updating step of updating the first transfer function for the estimated sound source direction using the second transfer function. death, The transfer function estimation step includes normalizing the amplitude of the transform coefficient for each channel by an inter-channel norm of the transform coefficient; The phase of the transform coefficient for each channel is normalized by the phase of the sum of the transform coefficients across channels. An acoustic processing method. Effect of the Invention

[0015] As mentioned above, (2)、(6)- According to the configuration of (8), a transfer function for an estimated sound source direction estimated from an acquired acoustic signal for each channel is estimated as a second transfer function, and the first transfer function is updated using the estimated second transfer function. Therefore, it is possible to estimate a transfer function that varies in a real acoustic environment based on an acquired acoustic signal.

[0016] As mentioned above (3)According to the configuration, since some components of the first transfer function are updated at one time, the influence of fluctuations and erroneous estimation of the second transfer function is mitigated.

[0017] As mentioned above (1)、(7) According to this configuration, it is possible to more reliably estimate the second transfer function indicating the relative transfer characteristic between channels with respect to the estimated sound source direction.

[0018] As mentioned above (2)、(8) According to this configuration, the amplitude and phase of the transform coefficients between channels can be normalized to estimate the second transfer function.

[0019] As mentioned above (4) According to the configuration, the sound source direction can be accurately estimated using the multi-signal classification spectrum calculated using the first transfer function that reflects the actual acoustic environment.

[0020] As mentioned above (5) According to the configuration, it is possible to accurately extract the sound source components arriving from the estimated sound source direction by using the separation matrix calculated using the first transfer function that reflects the actual acoustic environment. [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram showing an example of the configuration of a sound processing system according to a first embodiment. [Diagram 2] 4 is a data flowchart showing an example of acoustic processing according to the first embodiment. [Diagram 3] FIG. 11 is a block diagram showing an example of the configuration of a sound processing system according to a second embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a sound collection unit. [Diagram 5] FIG. 13 is a diagram showing another example of the sound pickup section. [Figure 6] FIG. 13 is a diagram illustrating an example of an evaluation result of a transfer function. [Figure 7] FIG. 11 is a diagram showing an example of an evaluation result of sound source localization. [Figure 8] FIG. 11 is a diagram showing an example of an evaluation result of sound source separation. [Figure 9] FIG. 1 is a diagram illustrating an example of an implementation of sound source localization and sound source separation. [Figure 10] FIG. 13 is a diagram illustrating another example of performing sound source localization and sound source separation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] (First embodiment) A first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a sound processing system S1 according to this embodiment. The sound processing system S1 includes a sound processing device 10 and a sound collection unit 20.

[0023] The sound processing device 10 stores a transfer function indicating the transfer characteristics of the sound from the sound source for each sound source direction. The sound processing device 10 acquires sound signals of multiple channels, and calculates a spatial spectrum for each sound source direction based on the conversion coefficient in the frequency domain of the sound signal for each channel and the stored transfer function. The sound processing device 10 estimates the sound source direction in which the spatial spectrum is maximum as the estimated sound source direction (sound source localization). On the other hand, the sound processing device 10 normalizes the calculated conversion coefficient between channels to estimate it as a transfer function for the estimated sound source direction, and updates the transfer function stored in advance for the estimated sound source direction using the estimated transfer function. A transfer function set including the updated transfer function is used to estimate the sound source direction from a newly acquired sound signal. Thus, the estimation of the sound source direction and the update of the transfer function are sequentially repeated.

[0024] The sound processing device 10 has a function of extracting sound source components from individual sound sources from a multi-channel sound signal acquired using an estimated sound source direction (sound source separation). The sound processing device 10 may generate a sound signal having the extracted sound source components as a sound source signal. Depending on the method of sound source separation processing, the sound processing device 10 may use a transfer function related to the estimated sound source direction among the transfer functions included in the transfer function set. In this application, the transfer function stored in the sound processing device 10 may be referred to as a "first transfer function," and the transfer function estimated by the sound processing device 10 may be referred to as a "second transfer function," to distinguish between the two.

[0025] The sound processing device 10 may use the estimated sound source direction and one or both of the sound source components and the sound source signal for other processing in the device itself, or may output them to another device (not shown, hereinafter, sometimes referred to as an "output destination device") as an output destination. As another processing, the sound processing device 10 may estimate the presence of an object in the estimated sound source direction. The sound processing device 10 may perform a voice recognition process on the sound source components or sound source signal from a specific sound source direction (speaker), obtain a spoken text indicating the contents of the utterance, or estimate the speaker. The output destination device may be an information and communication device such as a PC (Personal Computer) or a multi-function mobile phone, or may be a measuring instrument, a monitoring device, or the like.

[0026] The sound collection unit 20 has a plurality of microphones 20-1 to 20-M and functions as a microphone array. The number M of microphones is an integer equal to or greater than 2. Each microphone is disposed at a different position and includes an actuator that collects sound waves arriving at the microphone. The actuator converts the arriving sound waves into an acoustic signal. The converted acoustic signal is output to the sound processing device 10 wirelessly or via a wired connection. Each microphone corresponds to a channel of the acoustic signal.

[0027] The arrangement of the microphones may be fixed or variable. The positions of the microphones may be different from each other. In the example shown in FIG. 4, eight microphones are arranged on a circumference parallel to a horizontal plane so that they are equally spaced from the center. In FIG. 4, each microphone is indicated by a black circle. The eight microphones are arranged on the side of a housing and formed as one microphone array. The housing has a shape that has rotational symmetry with respect to a rotation axis oriented in the vertical direction, that is, a so-called egg-shaped shape. The microphone array includes an output interface for collecting eight-channel acoustic signals recorded by the individual microphones and outputting them to the sound processing device 10 in parallel via a wired connection.

[0028] Next, an example of the functional configuration of the sound processing device 10 according to the present embodiment will be described. The sound processing device 10 includes an input / output unit 110, a control unit 120, and a storage unit 140. The input / output unit 110 is connected wirelessly or by wire to other devices so as to be able to input and output various data. The input / output unit 110 outputs M-channel acoustic signals from the sound collection unit 20 to the control unit 120 as input data. The input / output unit 110 can output, for example, the estimation information input from the control unit 120 as output data to an output destination device (not shown). The input / output unit 110 may be, for example, any one of an input / output interface, a communication interface, and the like, or a combination thereof.

[0029] The control unit 120 executes processes for implementing the functions of the sound processing device 10, processes for controlling the functions, and the like. The control unit 120 may be configured using dedicated members as a whole or for each individual function, or may be configured as a computer system including a processor such as a CPU (Central Processing Unit) and various storage media. The processor reads out a predetermined program stored in advance in the storage medium, and executes processes instructed by various commands written in the read program to implement the functions of the control unit 120.

[0030] The control unit 120 includes a frequency analysis unit 122, a transfer function estimation unit 124, a transfer function update unit 126, a sound source direction estimation unit 132, a sound source separation unit 134, and a sound source signal generation unit 136. Unless otherwise specified, the processes of the transfer function estimation unit 124, the transfer function update unit 126, the sound source direction estimation unit 132, and the sound source separation unit 134 are executed independently for each frequency.

[0031] The frequency analysis unit 122 receives M-channel acoustic signals from the sound collection unit 20 via the input / output unit 110. The acquired M-channel acoustic signals each represent a time series (waveform) of amplitude at each sample time in the time domain. The frequency analysis unit 122 performs frequency analysis for each frame of a predetermined period (e.g., 20 ms to 100 ms) for the time domain for each channel, and converts the frequency into a conversion coefficient for each frequency in the frequency domain. A set of conversion coefficients for each channel across frequencies indicates a frequency spectrum. For example, the frequency analysis unit 122 can use a method such as a discrete Fourier transform in the frequency analysis. The frequency analysis unit 122 outputs input information indicating the conversion coefficients obtained by the conversion to the transfer function estimation unit 124, the sound source direction estimation unit 132, and the sound source separation unit 134.

[0032] The transfer function estimation unit 124 receives input information from the frequency analysis unit 122. For each frequency, the transfer function estimation unit 124 estimates a transfer function from the sound source to the microphone corresponding to that channel based on the conversion coefficient for each channel indicated in the input information. As described later, the estimated transfer function is associated with the estimated sound source direction estimated in the sound source direction estimation unit 132 as a second transfer function. When estimating the second transfer function, the transfer function estimation unit 124 normalizes, for example, the amplitude and phase of the conversion coefficient for each channel between channels. In the example shown in equation (1), the input vector X is divided by its norm |X| to normalize the amplitude of the conversion coefficient. For example, the square root of the sum of squares can be applied as the norm. The input vector X is converted into a conversion coefficient X for each channel m at a certain frequency. m The normalized amplitude is a real value between 0 and 1. The conversion coefficient Xm The sum of the channels Σ m X m The absolute value of |Σ m X m The phase of the transform coefficient is normalized by multiplying it by the complex conjugate of the quotient obtained by dividing by |. By normalizing the phase, the average value of the phase between channels weighted by the amplitude of the transform coefficient of each channel becomes 0. In this embodiment, each transfer function may be a value relative to the other channels, and is not necessarily an absolute value. The transfer function estimation unit 124 outputs second transfer function information indicating the estimated second transfer function to the transfer function update unit 126.

[0033]

number

[0034] The transfer function update unit 126 receives second transfer function information from the transfer function estimation unit 124 and estimated sound source direction information from the sound source direction estimation unit 132. The estimated sound source direction information is information indicating the sound source direction estimated by the sound source direction estimation unit 132. The transfer function update unit 126 specifies the second transfer function for each channel indicated by the input second transfer function information as the second transfer function corresponding to the estimated sound source direction indicated in the estimated sound source direction information for each frequency. The transfer function update unit 126 uses the specified second transfer function to update the first transfer function corresponding to the estimated sound source direction among the transfer function sets stored in the storage unit 140. For example, the transfer function update unit 126 replaces the second transfer function of the frequency and channel to be updated with the first transfer function of the frequency and channel.

[0035] However, if the second transfer function is simply replaced by the first transfer function for each frame, the replaced first transfer function may fluctuate significantly. The first transfer function may be directly affected by, for example, the presence or absence of a sound from a sound source, a temporary change in the acoustic environment, an erroneous estimation of the sound source direction, and the like. Therefore, the transfer function update unit 126 may determine the first transfer function after updating so that some components of the first transfer function of the frequency and channel to be updated are replaced with some components of the second transfer function in one calculation. The transfer function update unit 126 may determine the first transfer function after updating so that some components of the first transfer function of the frequency and channel to be updated are replaced with some components of the second transfer function by using, for example, exponential smoothing. E The first transfer function H E In the example shown in equation (2), the weighting coefficient α by which the second transfer function H′ is multiplied is a predetermined real value that is greater than 0 and less than 1. E (θ') is multiplied by the weighting coefficient (1-α). Therefore, the first transfer function H E The time average value of the smoothed transfer function is obtained so that the newer second transfer function H' is emphasized as (θ'). The transfer function update unit 126 calculates the time average value of the smoothed transfer function H E Instead of (θ'), a new first transfer function H E (θ′) is stored in the storage unit 140 in association with the estimated sound source direction θ′.

[0036]

number

[0037] The sound source direction estimation unit 132 refers to the transfer function set stored in the storage unit 140, and calculates a spatial spectrum S for each frequency using the conversion coefficient of each channel indicated in the input information input from the frequency analysis unit 122. sp The spatial spectrum can be considered as an index showing the degree of possibility that a sound source exists in each direction based on the position of the sound pickup unit 20. The sound source direction estimation unit 132 calculates the transfer function set H E, sound source direction θ, and input vector X. As shown in equation (3), the sound source direction estimation unit 132 estimates the direction in which the spatial spectrum is maximum as the estimated sound source direction θ'. A specific example of a method for calculating the spatial spectrum will be described later. The sound source direction estimation unit 132 outputs estimated sound source direction information indicating the estimated sound source direction to the transfer function update unit 126 and the sound source separation unit 134.

[0038]

number

[0039] The sound source direction estimation unit 132 estimates the spatial spectrum S sp There may be cases where multiple directions in which (θ) is maximum and exceeds a predetermined spatial spectrum threshold are detected. In such a case, the sound source direction estimation unit 132 may output estimated sound source direction information indicating each of the multiple sound source directions as an estimated sound source direction to the sound source separation unit 134. This is because in such a case, it is estimated that multiple significant sound sources exist.

[0040] In addition, the sound source direction estimation unit 132 calculates the spatial spectrum S sp Only when one direction in which (θ) is maximized and is greater than a predetermined spatial spectrum threshold is detected, the transfer function update unit 126 may output estimated sound source direction information indicating the detected direction as an estimated sound source direction θ' to the transfer function update unit 126. As described above, the transfer function update unit 126 updates the first transfer function H E (θ′) can be updated using the second transfer function H′.

[0041] In other words, the sound source direction estimation unit 132 calculates the spatial spectrum S sp When two or more directions in which (θ) is maximum and is greater than a predetermined spatial spectrum threshold are detected, or when the spatial spectrum S spIf no direction is detected in which (θ) is maximum and exceeds the threshold value of a predetermined spatial spectrum, the estimated sound source direction information is not output to the transfer function update unit 126. In this case, the transfer function update unit 126 does not receive the estimated sound source direction information from the sound source direction estimation unit 132, and stops updating the first transfer function based on the second transfer function estimated by the transfer function estimation unit 124 from the input information from the frequency analysis unit 122. sp The direction in which (θ) is maximized and exceeds a predetermined spatial spectrum threshold is estimated as the sound source direction, but when two or more sound source directions are detected, the sounds arriving from the multiple sound sources are superimposed at the microphone, so the ratio of the conversion coefficients between the channels is not the ratio of the transfer function for the sound source direction relating to one specific sound source. When the sound source direction is not detected, no significant sound arrives at the microphone from the sound source in the first place. Therefore, by limiting the estimation and update of the transfer function when one sound source is detected, the deterioration of the estimation accuracy of the transfer function can be suppressed. Even when two or more sound sources are detected, the execution of sound source separation in the sound source separation unit 134 is permitted.

[0042] The sound source separation unit 134 receives input information from the frequency analysis unit 122 and estimated sound source direction information from the sound source direction estimation unit 132. The sound source separation unit 134 extracts sound source components arriving from the estimated sound source direction from the conversion coefficients for each channel indicated in the input information. The sound source separation unit 134 extracts sound source components arriving from the estimated sound source direction from the conversion coefficients for each channel indicated in the input information. E , and the separation matrix W(H E , θ′) to the input vector X. As shown in equation (4), the sound source separation unit 134 calculates a separation matrix W(H E, θ') to calculate, for each frequency, an output value Y (separated sound source) estimated as a sound source component coming from a sound source present in the estimated sound source direction θ'. The input vector X includes, as elements, conversion coefficients for each channel indicated in the input information. When multiple estimated sound source directions are detected, the sound source separation unit 134 can determine an output value for each sound source (estimated sound source direction). The sound source separation unit 134 outputs output information indicating the output value determined for each frequency for each sound source to the sound source signal generation unit 136.

[0043]

number

[0044] The sound source signal generating unit 136 converts the output value for each frequency indicated in the output information input from the sound source separation unit 134 for each sound source into a time series of amplitude for each sample time in the time domain. When converting the output value for each frequency in the frequency domain into a time series of amplitude, the sound source signal generating unit 136 can use an inverse process of the frequency analysis, for example, an inverse discrete Fourier transform. The sound source signal generating unit 136 can generate a sound source signal by connecting the time series of amplitude obtained for each frame for each sound source between frames. The sound source signal generating unit 136 may output the generated sound source signal to an output destination device via the input / output unit 110, or may store it in the storage unit 140.

[0045] The storage unit 140 includes a storage medium that temporarily or permanently stores various data. The storage unit 140 stores various data (including parameters, etc.) used by the control unit 120, and various data acquired by the control unit 120 or other functional units (including input data input from the outside, intermediate data during processing, and generated data generated as a processing result). A transfer function set is stored in the storage unit 140. The transfer function set includes a first transfer function for each microphone (channel) for each frequency for each sound source direction. As the initial value of the transfer function set, a transfer function measured in advance may be used, or a transfer function calculated in advance using a predetermined geometric model may be used. As the geometric model, a plane wave model assuming the propagation of a plane wave in a free sound field, a spherical wave model assuming the propagation of a spherical wave from a sound source present at a predetermined distance from the sound collection unit 20, or the like may be used. The initial transfer function set H , exemplified by Equation (4), T is the first transfer function H T (θ1)~H T (θ N ) as an element. H T (θ1) etc. indicate transfer functions calculated based on a geometric model related to the sound source direction θ1. N indicates the number of sound source directions. The interval between adjacent sound source directions directly affects the accuracy of the sound source direction estimated by sound source localization. The more the number of sound source directions, the more the accuracy of the sound source direction is expected to improve, but the amount of calculation related to the calculation of the spatial spectrum in sound source localization increases.

[0046]

number

[0047] The arrangement of the sound source directions associated with each of the first transfer functions constituting the transfer function set may be, for example, a one-dimensional arrangement distributed on a circumference parallel to a horizontal plane centered on the position of the sound collection unit 20. In that case, each sound source direction is expressed by an azimuth angle. The arrangement of the sound source directions may be a two-dimensional arrangement distributed on a sphere centered on the position of the sound collection unit 20. In that case, the sound source direction is expressed by an azimuth angle and an elevation angle. The transfer function set may also be configured to include a first transfer function for each sound source position. In that case, the arrangement of the sound source positions is a three-dimensional distribution distributed in a three-dimensional space. The sound source position is expressed by three-dimensional coordinates based on the position of the sound collection unit 20, and corresponds to a combination of the sound source direction and the distance from the reference position. However, in this embodiment, the case where the distribution of the sound source positions is a one-dimensional arrangement will be described as an example, but it is also applicable to a two-dimensional or three-dimensional arrangement.

[0048] When the transfer function set includes the first transfer function for each sound source position, the sound source direction estimation unit 132 can estimate the sound source position as information to be estimated. The sound source direction estimation unit 132 may calculate a spatial spectrum for each sound source position instead of the sound source direction, and identify the sound source position where the spatial spectrum is maximized (or maximum). The transfer function update unit 126 may take the identified sound source position as the estimated sound source position, and update the first transfer function related to the estimated sound source position using the second transfer function estimated by the transfer function estimation unit 124 using the above-mentioned method.

[0049] (Example of sound source localization) Next, the MUSIC (Multiple Signal Classification) method will be described as an example of a sound source localization method. In the MUSIC method, the spatial spectrum S sp (θ) is calculated. The sound source direction estimation unit 132 calculates an input correlation matrix R as shown in equation (6) from an input vector X including the calculated conversion coefficients as elements. XX Calculate.

[0050]

number

[0051] In equation (6), E[...] denotes the expectation of... * denotes the conjugate transpose of a matrix or vector... The sound source direction estimation unit 132 calculates the input correlation matrix R XX The eigenvalue δ of p and the eigenvector ξ p Calculate the input correlation matrix R XX , eigenvalue δ p , and the eigenvector ξ p has the relationship shown in equation (7).

[0052]

number

[0053] In formula (7), p is an integer between 1 and M. The order of the index p is determined by the eigenvalue δ p In descending order. The sound source direction estimation unit 132 estimates the transfer function vector H(θ) and the calculated eigenvector ξ for each sound source direction. p Based on this, the spatial spectrum S exemplified by Eq. (8) is sp (θ) is calculated. In equation (8), D m corresponds to the maximum number of detectable sound sources and is a predetermined natural number smaller than M. The transfer function vector H(θ) is a first transfer function H E It is an M-dimensional vector that contains (θ) as an element. That is, equation (8) expresses the square of the norm of the transfer function vector H(θ) as the D m +1st to DMth eigenvectors ξ p The spatial spectrum S is normalized by the sum of the dot products with each of sp (θ) is calculated.

[0054]

number

[0055] The sound source direction estimation unit 132 may use a method such as a beam forming (BF) method as another example of a sound source localization method involving the calculation of a spatial spectrum using a transfer function for each sound source direction, in addition to the MUSIC method. In the BF method, as shown in equation (9), the product of an input vector X and a pseudo inverse matrix of a transfer function vector H(θ) is a spatial spectrum S sp (θ) In equation (9), + denotes the pseudoinverse of a vector or matrix...

[0056]

number

[0057] (Example of sound source separation) Next, the GHDSS (Geometric-contrained High-order Decorrelation-based Source Separation) method will be described as an example of a sound source separation method. The GHDSS method includes a process of adaptively calculating the separation matrix W so that the cost function J(W) decreases. The cost function J(W) is expressed as the separation sharpness J as shown in equation (10): SS (W) and Geometric Constrain J GC It is the weighted sum of (W).

[0058]

number

[0059] In equation (10), β is the separation sharpness J SS , which indicates the degree of contribution of (W) to the cost function J(W). Separation sharpness J SS (W) is an index value exemplified by formula (11).

number

[0060] |…| 2 indicates the Frobenius norm. The Frobenius norm is the sum of squares of each element value of a matrix. diag(...) indicates the sum of the diagonal elements of the matrix.... In other words, the separation sharpness J SS (W) is an index value indicating the degree to which a sound source component Y of a certain sound source is mixed with a component of another sound source. Geometric constraint J GC (W) is an index value exemplified by formula (12).

[0061]

number

[0062] In formula (12), I represents the unit matrix. That is, the geometric constraint degree J GC (W) is an index value that indicates the degree of error between the output sound source signal and the original sound source signal emitted from the sound source.

[0063] The sound source separation unit 134 extracts transfer functions corresponding to the sound source directions of each sound source indicated in the estimated sound source direction information from the transfer function set stored in the storage unit 140, and generates a transfer function matrix D by integrating the sound sources and channels with the extracted transfer functions as elements. Here, each row and each column corresponds to a channel and a sound source (sound source direction), respectively. Based on the generated transfer function matrix D, the sound source separation unit 134 generates an initial separation matrix W init Calculate.

[0064]

number

[0065] In equation (13), ... -1 denotes the inverse of the matrix .... Therefore, D * If D is a diagonal matrix whose off-diagonal elements are all zero, then the initial separation matrix W init is the pseudo-inverse matrix of the transfer function matrix D. The sound source separation unit 134 uses a step size μ SS , μ GC Complex gradient J' by SS (W t ), J' GC (W t ) is the weighted sum of the separation matrix W t+1 The separation matrix W at the next time t+1 is obtained by subtracting t+1 Calculate.

[0066]

number

[0067] In equation (14), the separation matrix W t The component μ to be subtracted from SS J' SS (W t )+μ GC J' GC (W t ) corresponds to the update amount ΔW. Complex gradient J' SS (W t ) is the separation sharpness J SS is derived by differentiating with respect to the input vector X. GC (W t ) is the geometric constraint degree J GC It is derived by differentiating with respect to the input vector X.

[0068] The sound source separation unit 134 uses a separation matrix W t+1 When it is determined that the separation matrix W t+1 Let W(H E , θ′). For example, when the Frobenius norm of the update amount ΔW becomes equal to or smaller than a predetermined threshold, the sound source separation unit 134 updates the separation matrix W t+1 Alternatively, the sound source separation unit 134 determines that the separation matrix W t+1 When the ratio of the Frobenius norm to the separation matrix W t+1 It may be determined that convergence has occurred.

[0069] The sound source separation unit 134 can use, not only the GHDSS method, but also other sound source separation methods, such as the BF method, which involves the calculation of a separation matrix based on a transfer function related to the estimated sound source direction. The BF method uses a pseudo inverse matrix H(θ') of a transfer function vector H(θ') related to the estimated sound source direction θ' estimated by the sound source direction estimation unit 132. + This is a method that adopts (θ') as a separation matrix.

[0070] (Audio Processing) Next, the acoustic processing according to this embodiment will be described. Fig. 2 is a data flow chart showing an example of the acoustic processing according to this embodiment. The acoustic processing device 10 according to this embodiment is divided into a transfer function adaptive estimation block B10 and an acoustic processing block B12. Among the steps described below, steps S102, S106, S110, and S122 belong to the transfer function adaptive estimation block B10. Steps S122 and S124 belong to the acoustic processing block B12. Step S122 belongs to the transfer function adaptive estimation block B10 and the acoustic processing block B12, and may be executed asynchronously and independently in each block, or may be executed synchronously between the blocks.

[0071] (Step S102) The control unit 120 acquires in advance an initial value of a transfer function set, and stores the acquired transfer function set in the storage unit 140. The control unit 120 calculates a transfer function for each channel and frequency for each sound source direction, for example, by using a predetermined geometric model. (Step S104) The frequency analysis unit 122 converts each of the M-channel time-domain acoustic signals into a frequency-domain transform coefficient for each frame. The frequency analysis unit 122 provides input information X indicating the transform coefficient of each channel to the transfer function adaptive estimation block B10.

[0072] (Step S106) The transfer function estimation unit 124 estimates a second transfer function (estimated transfer function H') for each frequency based on the conversion coefficient for each channel indicated in the input information. In estimating the second transfer function, for example, the relationship shown in Equation (1) is used. (Step S110) The transfer function update unit 126 updates the first transfer function (updated transfer function H E (θ′) is updated. In updating the first transfer function, for example, the relationship shown in equation (2) is used. (Step S112) The transfer function update unit 126 stores the updated first transfer function in the transfer function set in the storage unit 140 in place of the original first transfer function before the update in association with the estimated sound source direction θ′.

[0073] (Step S122) The sound source direction estimating unit 132 refers to the transfer function set and calculates a spatial spectrum for each frequency using the conversion coefficient of each channel indicated in the input information. The sound source direction estimating unit 132 determines the sound source direction in which the spatial spectrum is maximum as the estimated sound source direction θ′. In determining the estimated sound source direction, for example, the relationship shown in Equation (3) is used. (Step S124) The sound source separation unit 134 refers to the transfer function set and calculates a separation matrix from the transfer function related to the estimated sound source direction θ'. The sound source separation unit 134 multiplies the input vector based on the input information by the separation matrix, and calculates an output value (separated sound source) estimated as a sound source component arriving from the estimated sound source direction θ' for each frequency.

[0074] Each time the processing of steps S104-S124 is repeated for each frame, an estimated sound source direction θ' and an output value Y indicating a sound source component are obtained. The estimated sound source direction θ' and the output value Y may be used for other processing by the control unit 120, or may be output to an output destination device and used in the output destination device. The estimated sound source direction θ' and the output value Y may be temporarily or permanently stored in the storage unit 140. The control unit 120 or the output destination device may use, for example, the estimated sound source direction θ′ as a target direction or a blind spot for directivity control of an M-channel acoustic signal. The control unit 120 or the output destination device may perform, for example, speech recognition processing on the output value Y or a sound source signal based on the output value Y to obtain spoken text, a type of sound source, and / or a speaker. The control unit 120 or the output destination device may perform dialogue processing using information on the spoken text and the speaker obtained as a result of speech recognition.

[0075] As described above, according to this embodiment, the following effects can be achieved. (1) Any kind of sound source can be used to estimate the transfer function, not just a predetermined known test signal (e.g., applause (impulse), time stretched pulse (TSP), etc.). (2) The transfer function can be updated directly without calibrating the positional relationship between the sound source and each microphone. (3) The transfer function can be adaptively learned online without prior processing such as calibration. (4) The adaptive learning of the transfer function can be performed in parallel with microphone array processing such as sound source localization and sound source separation.

[0076] Second Embodiment Next, a second embodiment of the present invention will be described. In the following description, differences from the above-mentioned embodiment will be mainly described, and unless otherwise specified, the same reference numerals as those in the above-mentioned embodiment will be used to refer to the description. The sound processing system S2 according to this embodiment is exemplified as a control system or sub-system of a robot (not shown) equipped with an operating mechanism 40.

[0077] FIG. 3 is a block diagram showing an example of the configuration of a sound processing system S2 according to this embodiment. The sound processing system S2 includes a sound processing device 10b and a sound collection unit 20. One or both of the sound processing device 10b and the sound collection unit 20 may be built into the housing of a robot. In the example shown in FIG. 5, a microphone array serving as the sound collection unit 20 is embedded in the head of a humanoid robot. Individual microphones are indicated by black circles. In this example, the number of microphones is 16. The 16 microphones are arranged on two concentric circles with different radii. Eight microphones each are arranged on each concentric circle at 45° intervals. A group of microphones arranged on one concentric circle has an azimuth angle shift of 22.5° from other microphones arranged on the other concentric circle. In the sound processing devices 10 and 10b, sound signals acquired from some of the microphones constituting the sound collection unit 20 may be used as M-channel (e.g., 15-channel) sound signals.

[0078] Returning to Fig. 3, the sound processing device 10b includes an input / output unit 110, a control unit 120b, and a storage unit 140. The control unit 120b may include a frequency analysis unit 122, a transfer function estimation unit 124, a transfer function update unit 126, a sound source direction estimation unit 132, a sound source separation unit 134, a sound source signal generation unit 136, and an operation control unit 138. In addition, the sound processing block B12 (Fig. 2) realized by the sound source direction estimation unit 132 and the sound source separation unit 134 may function as a robot audition function block that realizes robot audition.

[0079] The sound processing block B12 may execute a known speech recognition process on the sound source components related to each sound source to identify the type of the sound source (sound source identification). A human speaker may be identified as the type of the sound source. The sound processing block B12 may notify another device of estimated sound source direction information indicating an estimated sound source direction for the identified type of sound source, or may output a sound source signal converted from output information for the identified type of sound source to another device.

[0080] The sound source direction estimation unit 132 can estimate the sound source position as described above, and the operation control unit 138 receives estimated sound source direction information indicating the estimated sound source position from the sound source direction estimation unit 132 and receives output information indicating the sound source components from the sound source separation unit 134. The operation control unit 138 controls the operation of the operating mechanism 40 using one or both of the estimated sound source position and the sound source components. The operation control unit 138 may, for example, execute self-location estimation and environmental map creation based on the estimated sound source position and the sound source components (SLAM: Simultaneous Localization and Mapping, simultaneous localization map creation). The operation control unit 138 can estimate the presence of an object (including a person) that is the sound source at the estimated sound source position by executing sound source identification. The operation control unit 138 may determine the existence probability of the object that is the sound source using a predetermined density function model so that the closer to the estimated sound source position, the higher the existence probability. The operation control unit 138 can, for example, create an environmental map by superimposing the spatial distribution of the existence probability of each object between the objects. In the path planning, the movement control unit 138 may determine a travel path so as not to pass through an area where the existence probability of an object is higher than a predetermined existence probability. The travel path is represented by a target position for each time. The movement control unit 138 may determine an estimated direction of a predetermined type of sound source as a target direction relative to the front of the robot. The movement control unit 138 outputs a control signal indicating one or both of the target position and the target direction at that time to the movement mechanism 40.

[0081] The movement mechanism 40 is built into the robot's housing, and controls the movement of the robot based on control signals input from the movement control unit 138. The movement mechanism 40 is equipped with a motor (not shown) that serves as a power source, and an encoder (not shown) that detects the position and direction of the movement mechanism 40. The motor moves the robot so as to approach a target position or target direction specified by the control signal. The encoder sequentially outputs movement information indicating the position and direction detected at that time as the movement state to the movement control unit 138.

[0082] (Evaluation experiment) Next, an evaluation experiment performed to evaluate the effectiveness of the above embodiment will be described. The evaluation experiment was performed in a laboratory that had a rectangular parallelepiped space with length, width, and height of 4, 7, and 3 m. The reverberation time RT 60 is 0.3 [s]. Depending on the evaluation item, the microphone array illustrated in FIG. 4 (hereinafter referred to as "egg-shaped array") and the microphone array illustrated in FIG. 5 (hereinafter referred to as "robot-built-in array") were used as the sound collection unit 20. The egg-shaped array was 0.9 [m] high from the floor and was installed on a desk located in the approximate center of the laboratory, together with a notebook personal computer and other items. When using the robot-built-in array, all items other than the sound source were removed, and only the robot was installed in the center of the laboratory.

[0083] Prior to the evaluation experiment, the following data was prepared. The egg-shaped array used as the sound pickup unit 20 acquires acoustic signals with a sampling frequency of 16 kHz and a bit width of 24 bits per sample for each channel. Two types of transfer function sets TF T L , T.F. T M , White noise W recorded while moving around the egg-shaped array T , Speech recorded while moving around the egg-shaped array T , and mixed voice M T Mixed voice M T is used for sound source separation.

[0084] Transfer function set TF T L When obtaining the low position, the sound reproduced based on the TSP signal for each sound source direction was collected. Here, the sound source positions were set at 30° intervals on a circle parallel to the horizontal plane so that the distance from the center of the egg-shaped array was 0.78 m and the height from the floor was 0.78 m. This height corresponds to 15.8° downward from the center of the egg-shaped array. Transfer function set TF T M (Middle Position) is also a transfer function set TF TL The measurements were taken under the same conditions as in Example 1. However, the height of the sound source from the floor was set to 1.0 m. This height corresponds to 7.3° above the center of the egg-shaped array, and is equivalent to the height of the mouth of a person sitting on a chair.

[0085] White Noise W T When obtaining the data, the subject held a speaker that played white noise and rotated around the egg-shaped array clockwise once, then reversed the direction of movement and rotated counterclockwise once. The subject repeated this action six times. The distance from the center of the egg-shaped array to the speaker position (sound source position) and the height from the floor were set to 0.78m and 1.0m, respectively. The total recording time was 6.8 minutes.

[0086] Speech audio S T When acquiring the utterance, a male voice selected from the Corpus of Spontaneous Japanese (CSJ) was played from the loudspeaker. The distance of the loudspeaker from the egg-shaped array and the height from the floor were adjusted by using white noise W T The settings were the same as when obtaining the 1st row. However, the recording time of the male voice was 20 minutes, and the person was made to circle the egg-shaped array clockwise three times.

[0087] Mixed audio M T When obtaining the above data, two speakers were placed 0.78 m away from the egg-shaped array and 0.78 m above the floor, at azimuths of 0° and 60° from the front. Two male voices selected from the CSJ were used as the two sound sources, and were played simultaneously on different speakers. The recording time was 100 seconds. White noise was added to the two male voices. However, the SNR (Signal-to-Noise Ratio) with the voice played from 0° was set to 20 dB.

[0088] For the robot-mounted array, acoustic signals are acquired with a sampling frequency of 48 kHz and a bit width of 24 bits per sample for each channel. For the robot-mounted array, one transfer function set, TF TH and white noise W recorded while moving around the robot. H was prepared. Transfer function set TF T H When acquiring the high position, the sound played based on the TSP signal for each sound source direction was recorded. Here, the sound source position was set at 5° intervals on a circle parallel to the horizontal plane so that the distance from the center of the robot's built-in array was 1.5 m and the height from the floor was 1.5 m. This height corresponds to the height of the mouth of a person standing upright.

[0089] White Noise W H To obtain the data, a subject was asked to repeatedly circle the egg array clockwise twice while holding a speaker playing white noise. The total recording time was 15 minutes. Other transfer function sets TF T G We prepared a transfer function set TF T G is composed of transfer functions that are pre-calculated using a geometric model for each sound source direction.

[0090] Next, a method for evaluating the transfer function will be described. In this evaluation experiment, the proposed method in the above embodiment was used to evaluate the transfer function using white noise W T The transfer function estimated using and the preset transfer function set TF T L , T.F. T M , T.F. T G The transfer functions belonging to each of the above were evaluated using the mean squared error (MSE). In the evaluation of the transfer functions, two transfer function sets TF i , T.F. jIn equation (15), M and F indicate the number of microphones and the number of frequency bins, respectively, and m and f are the indices of the microphone (channel) and frequency, respectively. In the example shown in equation (15), the estimation errors for each channel and frequency are averaged across channels and frequencies. Here, the white noise W T The set of transfer functions estimated using TF i and set the transfer function TF T L , T.F. T M , T.F. T G Each of these is TF j was substituted.

[0091]

number

[0092] FIG. 6 is a diagram showing an example of an evaluation result of a transfer function. FIG. 6 shows an estimated transfer function set and a transfer function set TF T L , T.F. T M , T.F. T G The MSE for each sound source direction is shown for each of the transfer function sets TF T G The MSE for the other transfer function set TF T L , T.F. T M This indicates that the estimated transfer function is closer to the measured transfer function than the transfer function based on the geometric model. In other words, this proves that the proposed method estimates a transfer function that is adapted to the real acoustic environment. However, the transfer function set TF T L , T.F. T M No significant difference was observed in the MSE between the two groups. One reason for this is believed to be that the height of the sound source could not be accurately controlled because the sound source was moved manually.

[0093] Next, we explain the evaluation method for sound source localization. In this evaluation experiment, we used a set of transfer functions calculated by a geometric model, and a white noise W H and a transfer function set TF for the measured transfer functions. T H Using these, the localization error rate L E was calculated as the evaluation scale. E is the total number of frames N of valid acoustic signals (whose power exceeds a predetermined threshold (e.g., −5 dB, −10 dB, etc.)) used in the evaluation as exemplified in Equation (16). T For the number of frames in which localization errors occurred, N E As a measure of the localization error, the sound source direction was estimated by the sound source direction estimating unit 132 using a known DS (Delay-and-Sum) method in sound source localization.

[0094]

number

[0095] FIG. 7 shows an example of the evaluation result of sound source localization. FIG. 7 shows the results of the evaluation of the geometric model, the proposed method, and the transfer function set TF T H The average localization error rate is shown in the upper part for each of the above, and the estimated sound source direction is shown. The average localization error rate is calculated using the geometric model, the proposed method, and the transfer function set TF T H The transfer function set TF T H According to the transfer function set TF, the average localization error rate is almost zero. T H According to the proposed method, the estimated sound source direction faithfully tracks the real sound source direction. The sound source direction estimated by the proposed method has less variation than the geometric model. This proves that the accuracy of sound source localization can be improved by estimating the transfer function accurately using the proposed method. In addition, the proposed method and the transfer function set TF T HFor , the average localization error rate is lower when the threshold is set to -5 dB than when it is set to -10 dB. This indicates that significant signal components are included when sufficient signal strength is ensured, so the effects of ambient noise can be suppressed.

[0096] Next, a method for evaluating sound source separation will be described. In this evaluation experiment, the sound source separation unit 134 extracts mixed sounds M T For this purpose, we performed source separation using the GHDSS method, the DS method, the Linear Constrained Minimum Variance (LCMV) method, the NULL method (null beamformer), and the Minimum Variance Distortionless Response (MVDR) method. These methods are classified as follows according to the characteristics of the beamforming used to extract the source components from the sound source. The DS method and the NULL method are characterized by fully-fixed beamforming. The MVDR method is characterized by semi-fixed beamforming. The LCMV method and the GHDSS method are characterized by adaptive beamforming.

[0097] In this evaluation experiment, for each method, we use a set of transfer functions calculated by a geometric model, white noise W H and the transfer function set TF for the egg-shaped array. T M For each of the above, the Signal-to-Distortion Ratio (SDR) and the Signal-to-Interference Ratio (SIR) are used as metrics. The SDR and SIR can be calculated using Equations (17) and (18), respectively.

[0098]

number

[0099]

number

[0100] In equations (17) and (18), s target indicates the target sound source signal of the clean sound source, that is, the original sound source component, among the sound source signals s obtained by sound source separation. e residue corresponds to the residual signal obtained by subtracting the target sound source signal from the sound source signal s obtained by sound source separation, that is, the residual noise term. e interf is the residual signal e residue In this evaluation experiment, the difference between the SDR and SIR obtained from the sound source signals obtained by sound source separation and the raw audio signals obtained by recording was evaluated as the improvement of the SDR and SIR.

[0101] FIG. 8 shows an example of the evaluation result of sound source separation. FIG. 8 shows the results of the evaluation of the geometric model, the proposed method, and the transfer function set TF T M The improvement in SDR and SIR for each source separation method is shown. The improvement in SDR and SIR is shown for the transfer function set TF T M is the best, followed by the proposed method and the geometric model. It is shown that the transfer function estimated by the proposed method can extract higher quality source components than the transfer function calculated by the geometric model for any sound source separation method. In the case of the geometric model, the improvement is actually negative for SDR. In particular, there is a tendency for the sound components from the sound source placed at 0° to not be sufficiently separated from the sound from the sound source placed at 60° and from the white noise. This tendency occurs in common regardless of the sound source separation method.

[0102] Next, the geometric model, the proposed method, and the transfer function set TF T MFor each of the above, we will explain an example of the sound source direction for each sound source estimated by sound source localization and sound source separation. Figures 9 and 10 show the time change of the sound source direction for each of the two trial periods (laps). In the execution example shown in Figure 9, the white noise W T However, the sound processing device 10 was not made to update the transfer function simultaneously with the execution of sound source localization and sound source separation, and a transfer function set including the sound source direction estimated by the geometric model was set as the initial value of the transfer function set. In the first trial period, the time change of the estimated sound source direction by the proposed method was almost the same as the time change of the estimated sound source direction by the geometric model, and the transfer function set TF T M The estimated sound source direction is significantly different from that estimated by the In contrast, in the second trial period, the estimated sound source direction by the proposed method was higher than that by the geometric model using the transfer function set TF T M This also shows that more accurate sound source localization and separation can be achieved by using a transfer function estimated in a real acoustic environment.

[0103] In the execution example shown in FIG. 10, no calibration period is provided between the two trial periods, and the sound processing device 10 is made to update the transfer functions using the proposed method in parallel with the sound source localization and sound source separation. However, a transfer function set including a sound source direction estimated by a geometric model is set as the initial value of the transfer function set. In the first trial period, the proposed method detects a sound source direction similar to that of the geometric model, and detects a sound source direction that is no longer detected by the geometric model over time. However, the transfer function set TF T M In the second trial period, the sound source direction estimated by the proposed method was significantly different from that estimated by the transfer function set TF T M This indicates that accurate sound source localization and separation can be achieved by advancing the adaptive learning of the transfer function.

[0104] As described above, the sound processing device 10, 10b according to the present embodiment includes a storage unit 140 that stores a first transfer function indicating the transfer characteristic of the sound from the sound source for each sound source direction, and a sound source direction estimation unit 132 that calculates a spatial spectrum for each sound source direction based on the conversion coefficient in the frequency domain of the sound signal for each channel and the first transfer function, and estimates the sound source direction in which the spatial spectrum is maximum as the estimated sound source direction. The sound processing device 10, 10b includes a transfer function estimation unit 124 that normalizes the conversion coefficient between channels to estimate a transfer function for the estimated sound source direction as a second transfer function, and a transfer function update unit 126 that updates the first transfer function for the estimated sound source direction using the second transfer function. With this configuration, a transfer function for an estimated sound source direction estimated from an acquired acoustic signal for each channel is estimated as a second transfer function, and the first transfer function is updated using the estimated second transfer function. Therefore, it is possible to estimate a transfer function that varies in a real acoustic environment based on an acquired acoustic signal.

[0105] Furthermore, the transfer function update section 126 may update at least a portion of the components of the first transfer function with a portion of the components of the second transfer function at predetermined time intervals. With this configuration, some components of the first transfer function are updated at one time, thereby mitigating the effects of fluctuations and erroneous estimation of the second transfer function.

[0106] Furthermore, the transfer function update unit 126 may update the first transfer function when the number of sound sources detected from the acquired acoustic signal is one. This configuration makes it possible to more reliably estimate the second transfer function indicating the relative transfer characteristics between channels with respect to the estimated sound source direction.

[0107] Furthermore, the transfer function estimation unit 124 may normalize the amplitude of the transform coefficient for each channel by the inter-channel norm of the transform coefficient, and normalize the phase of the transform coefficient for each channel by the phase of the inter-channel sum of the transform coefficients. This configuration makes it possible to estimate the second transfer function by normalizing the amplitude and phase of the transform coefficients between channels.

[0108] Furthermore, the sound source direction estimating unit 132 may calculate a multi-signal classification spectrum as a spatial spectrum based on the transformation coefficients and the first transfer function. With this configuration, the sound source direction can be accurately estimated using the multi-signal classification spectrum calculated using the first transfer function that reflects the actual acoustic environment.

[0109] Furthermore, the sound processing device 10, 10b may be provided with a sound source separation unit 134 that determines a separation matrix for an estimated sound source direction based on a first transfer function for the estimated sound source direction, and converts a vector calculated by applying the separation matrix to an input vector having transformation coefficients as elements into an output vector having sound source components arriving for each sound source as elements. With this configuration, it is possible to accurately extract the sound source components arriving from the estimated sound source direction using the separation matrix calculated using the first transfer function that reflects the actual acoustic environment.

[0110] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0111] S1, S2...sound processing system, 10, 10b...sound processing device, 20...sound collection unit, 40...operation mechanism, 110...input / output unit, 120...control unit, 122...frequency analysis unit, 124...transfer function estimation unit, 126...transfer function update unit, 132...sound source direction estimation unit, 134...sound source separation unit, 136...sound source signal generation unit, 138...operation control unit, 140...storage unit

Claims

1. A storage unit that stores a first transfer function indicating a transfer characteristic of a sound from a sound source for each sound source direction; Calculating a spatial spectrum for each sound source direction based on a conversion coefficient in a frequency domain of an acoustic signal for each channel corresponding to each of a plurality of microphones and the first transfer function; a sound source direction estimating unit that estimates the sound source direction in which the spatial spectrum is maximized as an estimated sound source direction; a transfer function estimating unit that estimates a value obtained by normalizing the transform coefficients between channels as a second transfer function for the estimated sound source direction; a transfer function update unit that updates the first transfer function with respect to the estimated sound source direction by using the second transfer function when the number of sound sources detected from the acoustic signal is one; Equipped Sound processing equipment.

2. A storage unit that stores a first transfer function indicating a transfer characteristic of a sound from a sound source for each sound source direction; Calculating a spatial spectrum for each sound source direction based on a conversion coefficient in a frequency domain of an acoustic signal for each channel corresponding to each of a plurality of microphones and the first transfer function; a sound source direction estimating unit that estimates the sound source direction in which the spatial spectrum is maximized as an estimated sound source direction; a transfer function estimating unit that estimates a value obtained by normalizing the transform coefficients between channels as a second transfer function for the estimated sound source direction; a transfer function update unit that updates the first transfer function with respect to the estimated sound source direction by using the second transfer function, The transfer function estimation unit normalizing the amplitudes of the transform coefficients for each channel by an inter-channel norm of the transform coefficients; Normalizing the phase of the transform coefficient for each channel with the phase of the sum of the transform coefficients across channels Sound processing equipment.

3. The transfer function update unit is At least a portion of components of the first transfer function is updated with the components of the second transfer function at predetermined time intervals. The sound processing device according to claim 1 or 2.

4. The sound source direction estimation unit is A multi-signal classification spectrum is calculated based on the transform coefficients and the first transfer function as the spatial spectrum. The sound processing device according to claim 1 .

5. determining a separation matrix for the estimated sound source direction based on a first transfer function for the estimated sound source direction; A sound source separation unit that outputs a vector calculated by applying the separation matrix to an input vector having the transformation coefficients as elements as an output vector having sound source components arriving for each sound source as elements. The sound processing device according to any one of claims 1 to 4.

6. A program for causing a computer to function as the sound processing device according to any one of claims 1 to 5.

7. A method for a sound processing device including a storage unit that stores a first transfer function indicating a transfer characteristic of a sound from a sound source for each sound source direction, the method comprising: Calculating a spatial spectrum for each sound source direction based on a conversion coefficient in a frequency domain of an acoustic signal for each channel corresponding to each of a plurality of microphones and the first transfer function; a sound source direction estimating step of estimating the sound source direction in which the spatial spectrum is maximized as an estimated sound source direction; a transfer function estimating step of estimating a value obtained by normalizing the transform coefficients between channels as a second transfer function for the estimated sound source direction; a transfer function updating step of updating the first transfer function with respect to the estimated sound source direction by using the second transfer function when the number of sound sources detected from the acoustic signal is one; The acoustic processing method includes the steps of:

8. A method for a sound processing device including a storage unit that stores a first transfer function indicating a transfer characteristic of a sound from a sound source for each sound source direction, the method comprising: Calculating a spatial spectrum for each sound source direction based on a conversion coefficient in a frequency domain of an acoustic signal for each channel corresponding to each of a plurality of microphones and the first transfer function; a sound source direction estimating step of estimating the sound source direction in which the spatial spectrum is maximized as an estimated sound source direction; a transfer function estimating step of estimating a value obtained by normalizing the transform coefficients between channels as a second transfer function for the estimated sound source direction; a transfer function updating step of updating the first transfer function with respect to the estimated sound source direction by using the second transfer function, The transfer function estimation step includes: normalizing the amplitudes of the transform coefficients for each channel by an inter-channel norm of the transform coefficients; The phase of the transform coefficient for each channel is normalized by the phase of the sum of the transform coefficients across channels. Acoustic processing methods.

Images