Sound processing device, sound processing method and program
The sound processing device addresses high computational complexity in sound source separation by normalizing directional components and using machine learning to enhance target sound extraction quality, reducing the need for pre-defined spatial models and improving dynamic acoustic environment performance.
Patent Information
- Application Number
- JP2021035253
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-05
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-03-05
AI Technical Summary
Existing sound source separation methods require extensive preprocessing to account for varying spatial relationships between sound sources, leading to high computational complexity and inconsistent quality in extracting target sound components.
A sound processing device that includes a microphone array with a spatial normalization unit to normalize directional components into a standard direction, a mask function estimation unit using a machine learning model, and a mask processing unit to apply the mask function to acoustic signals, reducing the need for pre-defined spatial models and improving target sound source extraction.
This approach reduces spatial complexity in model learning while ensuring high-quality extraction of target sound components, even in dynamic acoustic environments, by normalizing directional components and using machine learning to estimate mask functions effectively.
Smart Images

Figure 0007577276000009 
Figure 0007577276000010 
Figure 0007577276000011
Abstract
Description
[Technical field]
[0001] The present invention relates to a sound processing device, a sound processing method, and a program. [Background technology]
[0002] Sound source separation is a technique for separating components based on individual sound sources from an acoustic signal that contains multiple components. Sound source separation is useful for acoustically analyzing the surrounding environment, and applications are being attempted in a wide range of fields and applications. Representative applications include autonomous driving, equipment operation, voice conferencing, and robot motion control. A method has been proposed for sound source separation that uses microphones positioned at different locations and utilizes the difference in sound transmission characteristics due to differences in the spatial relationship from the sound source to each microphone. Among these, selective sound separation is an important function in sound source separation.
[0003] Selective sound source separation is the separation of sound components coming from a sound source existing in a specific direction or position. Selective sound source separation is applied to, for example, acquiring a voice uttered by a specific speaker in a conversation robot. Non-Patent Document 1 proposes a method for separating a target source component from an acoustic input from two microphones in a reverberant environment (binaural sound source separation). Non-Patent Document 1 describes a method for estimating a mask that uses a neural network to extract a target sound from a spectral feature and a spatial feature obtained from the acoustic input. The estimated mask is used to act on the acoustic input to relatively emphasize a target sound from a specific direction and to reduce noise components from other directions. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] X. Zhang and D. Wang: “Deep Learning Based Binaural Speech Separation in Reverberant Environments”, IEEE / ACM TRANSACTIONS ON AUDIO, SPEECH, AND, LANGUAGE PROCESSING, VOL. 25, NO.5, MAY 2017 Summary of the Invention [Problem to be solved by the invention]
[0005] However, the number of sound sources in a real acoustic environment and the patterns of spatial relationships with their positions are generally diverse. On the other hand, if all patterns are assumed, it is necessary to set these patterns in advance and to train the model parameters of the neural network in advance so as to match each pattern. Therefore, the amount of processing and labor required for training the model parameters may be enormous. In addition, since the number and positions of sound sources may change dynamically, it is not always possible to obtain the components of the target sound source with sufficient quality using the preset patterns.
[0006] The present embodiment has been made in consideration of the above points, and has an object to provide a sound processing device, a sound processing method, and a program that can reduce the spatial complexity for sound source separation. [Means for solving the problem]
[0007] (1) The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is an audio processing device that includes a spatial normalization unit that normalizes a directional component toward a target direction of the microphone array, which is included in the spectrum of an audio signal acquired from each of a plurality of microphones that form a microphone array, to a directional component toward a predetermined standard direction to generate a normalized spectrum; a mask function estimation unit that determines a mask function for extracting a component of a target sound source arriving from the target direction based on the normalized spectrum using a machine learning model; and a mask processing unit that applies the mask function to the audio signal to estimate the component of the target sound source installed in the target direction.
[0008] (2) Another aspect of the present invention is the sound processing device of (1), wherein the spatial normalization unit may use a first steering vector indicating directivity toward the standard direction and a second steering vector indicating directivity toward the target direction in the normalization.
[0009] (3) In another aspect of the present invention, the sound processing device of (1) or (2) may further include a spatial filtering unit that applies a spatial filter indicating directivity toward the target direction to the normalized spectrum to generate a spatially corrected spectrum. The mask function estimation unit may input the spatially corrected spectrum to the machine learning model to determine the mask function.
[0010] (4) Another aspect of the present invention is an audio processing device according to any one of (1) to (3), which may include a model learning unit that determines a parameter set of the machine learning model so as to reduce a residual between an estimated value of a component of the target sound source obtained by applying the mask function to the audio signal indicating sounds arriving from a plurality of sound sources including the target sound source and a target value of the component of the target sound source.
[0011] (5) In another aspect of the present invention, in the sound processing device according to any one of (1) to (4), the model learning unit may determine a spatial filter for generating a spatially corrected spectrum from the normalized spectrum, and an estimated value of the component of the target sound source is obtained by applying the mask function to the spatially corrected spectrum.
[0012] (6) In another aspect of the present invention, the sound processing device according to any one of (1) to (5) may further include a sound source direction estimating unit that determines a sound source direction based on a plurality of the sound signals. The space normalizing unit may use the sound source direction as the target direction.
[0013] (7) Another aspect of the present invention may be a program for causing a computer to function as any one of the sound processing devices (1) to (6).
[0014] (8) Another aspect of the present invention is The sound processing device A first step of generating a normalized spectrum by normalizing a directional component in a target direction of the microphone array contained in a spectrum of an acoustic signal acquired from each of a plurality of microphones forming a microphone array to a directional component in a predetermined standard direction; a second step of determining a mask function for extracting a component of a target sound source arriving from the target direction based on the normalized spectrum using a machine learning model; and a third step of applying the mask function to the acoustic signal to estimate the component of the target sound source installed in the target direction. execution This is an acoustic processing method. Effect of the Invention
[0015] According to the above-mentioned configurations (1), (7), and (8), the normalized spectrum used to estimate the mask function is normalized to include directional components in a standard direction, so there is no need to prepare a machine learning model assuming all sound source directions. Therefore, it is possible to reduce the spatial complexity of the acoustic environment in model learning while ensuring the quality of the target sound source components obtained by sound source separation.
[0016] According to the above-mentioned configuration (2), by using the first and second steering vectors that can also be used for other microphone array processing such as sound source direction estimation, spatial normalization can be realized with simple processing and configuration.
[0017] According to the above-mentioned configuration (3), the components of the target sound source installed in the target direction contained in the acquired acoustic signal are reliably captured, so that the quality of the components of the estimated target sound source can be ensured.
[0018] According to the above-mentioned configuration (4), it is possible to learn a machine learning model for determining a mask function that acts on an acoustic signal to estimate a component of a target sound source.
[0019] According to the above-mentioned configuration (5), it is possible to simultaneously determine a parameter set of the machine learning model and a spatial filter for generating a spatially corrected spectrum to be input to the machine learning model.
[0020] According to the above-mentioned configuration (6), even if the target direction of the target sound source is unknown, the components of the target sound source can be estimated. [Brief description of the drawings]
[0021] [Figure 1] 1 is a block diagram showing an example of the configuration of a sound processing system according to an embodiment of the present invention. [Diagram 2] FIG. 11 is an explanatory diagram for explaining spatial normalization. [Diagram 3] 3 is a surface diagram showing an example of a sound collecting section according to the embodiment; FIG. [Figure 4] FIG. 2 is a side view illustrating an example of a sound collecting section according to the embodiment. [Diagram 5] 5 is a flowchart illustrating an example of acoustic processing according to the present embodiment. [Figure 6] 11 is a flowchart illustrating an example of model learning according to the present embodiment. [Figure 7] FIG. 2 is a plan view showing the positional relationship between a microphone array and a sound source. [Figure 8]FIG. 2 is a side view showing the positional relationship between a microphone array and a sound source. [Figure 9] 11 is a table showing the quality of extracted target sound source components. [Figure 10] FIG. 4 illustrates an example of the amplitude response of a spatial filter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0022] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a sound processing system S1 according to this embodiment. The sound processing system S1 includes a sound processing device 10 and a sound collection unit 20.
[0023] The sound processing device 10 determines the spectrum of the sound signal of multiple channels acquired from the sound collection unit 20. The sound processing device 10 determines a normalized spectrum by normalizing the directional component toward the target direction of the sound collection unit 20 included in the spectrum determined for each channel to a directional component toward a predetermined standard direction. The sound processing device 10 determines a mask function for each channel for extracting the component arriving from the target direction based on the normalized spectrum determined using a machine learning model. The sound processing device 10 estimates the component of the target sound source installed in the target direction by applying the mask function determined for each channel to the sound signal. The sound processing device 10 outputs the sound signal indicating the component of the estimated target sound source to the output destination device 30. The output destination device 30 is another device to which the sound signal is to be output.
[0024] The sound collection unit 20 has a plurality of microphones and is formed as a microphone array. Each microphone is located at a different position and collects sound waves arriving at the respective microphones. In the example shown in FIG. 1, each microphone is distinguished by a sub-number such as 20-1 and 20-2. Each microphone has an actuator that converts the collected sound waves into an acoustic signal, and outputs the converted acoustic signal to the sound processing device 10. In this embodiment, the unit of the acoustic signal collected by each microphone is called a channel. In the example shown in FIG. 3 and FIG. 4, the sound collection unit 20 has two microphones fixed to a spheroidal housing. The microphones 20-1 and 20-2 are installed on the outer edge of a cross section A-A' that crosses the central axis C of the housing. The intersection of the central axis C and the cross section A-A' is set as a representative point O. In this example, the angle between the representative point O and the direction of the microphone 20-1 and the direction of the microphone 20-2 is 135°.
[0025] In the present application, a case where the number of microphones is two will be mainly described as shown in Figures 1 and 3. One microphone 20-1 and the other microphone 20-2 may be referred to as microphones 20-1 and 20-2, respectively. The number of microphones may be three or more. The positions of the individual microphones are not limited to the example shown in the figure. The positional relationship between the multiple microphones may be fixed or variable.
[0026] Next, an example of the functional configuration of the sound processing device 10 according to the present embodiment will be described. The sound processing device 10 includes an input / output unit 110 and a control unit 120. The input / output unit 110 is connected wirelessly or by wire to other devices so that various data can be input and output. The input / output unit 110 outputs input data input from other devices to the control unit 120. The input / output unit 110 outputs output data input from the control unit 120 to other devices. The input / output unit 110 may be, for example, either an input / output interface or a communication interface, or a combination thereof. The input / output unit 110 may include both or one of an analog-to-digital (A / D) converter and a digital-to-analog (A / D) converter. The A / D converter converts an analog acoustic signal input from the sound collection unit 20 into a digital acoustic signal, and outputs the converted acoustic signal to the control unit 120. The D / A converter converts a digital acoustic signal input from the control unit 120 into an analog acoustic signal, and outputs the converted acoustic signal to the output destination device 30.
[0027] The control unit 120 executes processes for implementing the functions of the sound processing device 10, processes for controlling the functions, etc. The control unit 120 may be configured using a dedicated member, or may be configured including a processor such as a CPU (Central Processing Unit) and various storage media. The processor reads out a predetermined program stored in advance in the storage medium, and executes processes instructed by various commands written in the read program to implement the processes of the control unit 120.
[0028] The control unit 120 includes a frequency analysis unit 122 , a spatial normalization unit 124 , a spatial filtering unit 126 , a mask function estimation unit 128 , a mask processing unit 130 , and a sound source signal processing unit 132 .
[0029] The frequency analysis unit 122 performs frequency analysis on the acoustic signals input from the individual microphones for each frame at a predetermined time interval (for example, 10 to 50 msec) to determine a spectrum. In the frequency analysis, the frequency analysis unit 122 performs, for example, a Discrete Fourier Transform (DFT). The spectrum of the acoustic signal of channel k in frame t is expressed as a complex number x at frequency w. k,w,t A vector x containing w,t This vector is called the observed spectrum vector. w,t is [x k1,w,t ,x k2,w,t ] T where T denotes the transpose of a vector or matrix. The observed spectrum vector x w,t An element of, say, x k1,w,t The frequency analysis unit 122 outputs the spectrum of each channel to the spatial normalization unit 124 for each frame. The frequency analysis unit 122 also outputs the observed spectrum of a given channel (e.g., x k1,w,t ) to the mask processing unit 130 for each frame.
[0030] The spatial normalization unit 124 normalizes (spatial normalizes) the observation spectrum input from the frequency analysis unit 122 so that the directional component of the sound collection unit 20 toward the target direction included in the spectrum is converted into a directional component toward a predetermined standard direction, thereby generating a normalized spectrum. The target direction corresponds to the direction of the sound source from the reference position, which is the position of the sound collection unit 20. The standard direction corresponds to a predetermined reference direction (for example, the front direction) from the reference position. The directional component of the sound collection unit 20 can be controlled using a steering vector. The steering vector is a vector including a complex number indicating the gain and phase for each channel as element values. The steering vector is determined for each directional direction, and has a directivity such that the gain in the directional direction is higher than the gain in other directions as the directionality for that directional direction. The element values for each channel of the steering vector for the target direction are used as weighting coefficients to calculate the array output as a microphone array. The gain of the array output in the target direction is larger than the gain in other directions. The steering vector includes element values obtained by normalizing the transfer function from the sound source to the microphone corresponding to each channel. The transfer function may be an actual measurement value in the usage environment, or a calculated value calculated by a simulation assuming a physical model. The physical model may be a mathematical model that gives the acoustic transfer characteristics from the sound source to the sound receiving point where the microphone is installed.
[0031] In the spatial normalization, the spatial normalization unit 124 calculates a normalized spectrum vector x' using, for example, equation (1). w,t can be determined.
[0032]
number
[0033] In formula (1), a w (r'), a w (r c,t ) are the steering vector for the standard direction r' and the target direction r c,t The symbol circle combined with a cross sign indicates element-wise multiplication of the vectors before and after it. The symbol circle combined with a / sign indicates element-wise division of the vector immediately before it by the vector immediately after it. Steering vector a w (r c,t ) is, for example, [a k1,w (r c,t ),a k2,w (r c,t )] T It is expressed as: k1,w (r c,t ), a k2,w (r c,t ) indicate the transfer functions from the sound source installed in the target direction to the microphones 20-1 and 20-2, respectively. However, the steering vector a w (r c,t ), a w (r') is the norm ||a w (r c,t The spatial normalization unit 124 normalizes the determined normalized spectrum x' w,t to the spatial filtering unit 126.
[0034] The spatial filtering unit 126 filters the normalized spectrum x′ input from the spatial normalization unit 124. w,t With respect to the target direction r c,t A spatial filter that indicates the directionality to the corrected spectrum z w,t As a spatial filter, the target direction r c,t Alternatively, a vector or a matrix may be used, the elements of which are filter coefficients that provide directivity with respect to the target direction r. As such a filter, for example, a delay-and-sum beamformer (DS beamformer) may be used. c,t Steering vector a for w (r c,tThe spatial filtering unit 126 may use a spatial filter based on the normalized spectrum x′ as shown in equation (2). w,t Using the DS beamformer, we obtain the spatially corrected spectrum z w,t can be determined.
[0035]
number
[0036] In formula (2), a w (r c,t ) is the target direction r c,t H denotes the conjugate of a vector or a matrix. The spatial filtering unit 126 filters the determined correction spectrum z w,t to the mask function estimation unit 128.
[0037] The mask function estimation unit 128 receives the normalized spectrum x' w,t Corrected spectrum z determined based on w,t The mask function estimation unit 128 uses a predetermined machine learning model to derive a corrected spectrum z w,t The mask function m for frequency w and frame t is w,t The mask function m w,t is expressed as a real number or a complex number whose absolute value is normalized to a range of 0 to 1. As the machine learning model, for example, various neural networks (NN) can be used. The neural network may be any type, such as a convolutional neural network, a recurrent neural network, or a forward propagation neural network. In addition, the machine learning model is not limited to a neural network, and may be any method, such as a decision tree, a random forest, or association rule learning. The mask function estimation unit 128 estimates the calculated mask function m w,t to the mask processing unit 130.
[0038] The mask processing unit 130 calculates the spectrum of the acoustic signal input from the frequency analysis unit 122, i.e., the observed spectrum x k1,w,t , the mask function m input from the mask function estimation unit 128 w,t The spectrum (sometimes referred to as the "target spectrum" in this application) of the component of the target sound source (sometimes referred to as the "target component" in this application) installed in the target direction is obtained by applying the above formula. w,t The mask processing unit 130 estimates the observed spectrum x k1,w,t mask function m w,t Multiplying by y' to get the target spectrum w,t The mask processing unit 130 calculates the calculated target spectrum y' w,t to the sound source signal processing unit 132.
[0039]
number
[0040] The sound source signal processor 132 processes the target spectrum y′ input from the mask processor 130. w,t The sound source signal processing unit 132 performs an inverse discrete Fourier transform (IDFT) on the target sound source component in the time domain to generate a sound source signal of the target sound source component in the time domain. The sound source signal processing unit 132 outputs the generated sound source signal to the output destination device 30 via the input / output unit 110. The sound source signal processing unit 132 may store the generated sound source signal in a storage unit (not shown) of the output destination device 30. The output destination device 30 may be an audio device such as a speaker, or may be an information device such as a personal computer or a multi-function mobile phone.
[0041] (observation model) Next, an observation model that is the premise of this embodiment will be described. The observation model is a model that formulates the observation spectrum of sound waves arriving at the sound collection unit 20 from sound sources installed in an acoustic space. M (M is an integer of 2 or more) sound sources are located at different positions r m,tWhen the microphones are installed in the sound pickup section 20, the observed spectrum x of the acoustic signal received by each microphone of the sound pickup section 20 is w,t is formulated using equation (4).
[0042]
number
[0043] In equation (4), m indicates an index indicating an individual sound source. m denotes the spectrum of the acoustic signal output by the sound source m. w (r m,t ) denotes the transfer function vector. The transfer function vector h w (r m,t ) is the sound source position r m,t The vector [h k1,w (r m,t ),h k2,w (r m,t )] T n w,t denotes the noise vector. The noise vector n w,t is a vector [n k1,w,t ,n k2,w,t ] T Equation (4) expresses the spectrum s of the acoustic signal output by each sound source m. m and the transfer function h w (r m,t ) and the noise spectrum n w,t The sum of these is the observed spectrum x w,t In this application, the sound source signal generated by the sound source and its spectrum may be referred to as the "sound source signal" and the "sound source spectrum", respectively.
[0044] According to this model, the target direction r c,t Target spectrum y based on target sound source c placed at w,t is a transfer function h from a target sound source c to a given microphone (e.g., microphone 20-1) as shown in equation (5). k1,w(r c,t ) and the source spectrum s of the target sound source c c,w,t The sound processing device according to this embodiment is expressed as the product of the observed spectrum x w,t The components of the target sound source c contained in w,t The present invention is provided with a configuration for estimating the above.
[0045]
number
[0046] (spatial normalization) Next, spatial normalization will be described. Spatial normalization corresponds to converting the directional component of the sound pickup unit 20 toward the target direction contained in the observation spectrum into a directional component toward a predetermined standard direction. 2 shows a case where one of two sound sources is a target sound source Tg and the other is another sound source Sr, and the directional component of the target sound source Tg in the target direction θ is converted into a directional component in the standard direction 0°. However, the representative point of the sound collection unit 20 is the origin O, and the sound source direction of each sound source is shown by an azimuth angle that forms the standard direction 0° from the origin. The azimuth angle is determined counterclockwise with the standard direction as a reference.
[0047] In this case, the spectrum of the incoming components from the sound source installed at the target direction θ and the standard direction 0° is calculated by the transfer function h k,w (θ), h k,w In this embodiment, the steering vector a is used as a directional component in the spatial normalization. k,w (θ) steering vector a k,w (0°) ratio a k,w (0°) / a k,w (θ). The steering vector is proportional to the transfer function from the sound source to the microphone, so the transfer function h k,w (θ) and steering vector a k,w (θ) cancels out, and the steering vector a k,w (0°), i.e., the transfer function h k,w The component proportional to (0°) is retained.
[0048] As described above, a transfer function measured in advance or a transfer function synthesized using a physical model is used as the steering vector. On the other hand, in a real sound field, the transfer function varies depending on the environment, so the transfer function h k,w (θ) and steering vector a k,w (θ) are not completely cancelled. However, the steering vectors reflect the differences in magnitude and phase due to the difference in the microphone positions, and the dependence on the sound source position remains. By spatial normalization, the transfer function h k,w (θ) and steering vector a k,w (θ) is partially canceled, so the transfer function h k,w The sound source direction dependency of (θ) is mitigated.
[0049] (Model learning) Next, a description will be given of learning of a parameter set of a machine learning model used by the mask function estimation unit 128. As described above, the mask function estimation unit 128 uses a machine learning model to estimate the corrected spectrum z w,t is used as the input value, and the mask function m w,t is calculated as an output value. For this reason, a parameter set of a machine learning model is set in advance in the mask function estimation unit 128. The sound processing device 10 may include a model learning unit (not shown) for determining the parameter set using training data.
[0050] The model learning unit determines a parameter set of the machine learning model so as to reduce a residual between an estimated value of a component of a target sound source obtained by applying a mask function to an acoustic signal indicating a sound in which components coming from each of a plurality of sound sources including the target sound source are mixed, and a target value of the component of the target sound source. As the target value, an acoustic signal indicating a sound coming from the target sound source and not including components from other sound sources is used.
[0051] Therefore, the model learning unit constructs training data including multiple data sets (typically 100-1000 or more) each of which is a pair of a known input value and an output value corresponding to that input value. The model learning unit calculates an estimate of an output value from the input value included in each data set using a machine learning model. In model learning, the model learning unit repeats a process of updating a parameter set so that a loss function indicating the magnitude of the difference (estimation error) between the estimate calculated for each data set and the output value included in that data set becomes smaller. A parameter set Θ is determined for each set of training data. One set of training data is a set of observed spectrum vectors x w,t and a set of sound source directions r c,t Each data set is obtained using one frame of the sound source signal. The frames of the sound source signal used for each data set may be continuous in time or may be intermittent.
[0052] The observed spectrum vector x is used as input for the machine learning model. w,t The corrected spectrum z taken as input value from w,t is given by the above method. The observed spectrum vector x w,t is obtained by performing frequency analysis on the acoustic signals emitted from a plurality of sound sources at different positions and picked up by the individual microphones constituting the sound pickup section 20. The target spectrum y that is the output value for the machine learning model w,t is obtained by frequency analysis of an acoustic signal collected by at least one microphone of the sound collection unit 20 when sound is emitted from a target sound source that is one of multiple sound sources and no sound is emitted from the other sound sources. However, for the target sound source, a sound based on a sound source signal that is common to the sound source signal used when acquiring the input value is reproduced.
[0053] The acoustic signals used to obtain the input and output values do not necessarily have to be collected by a microphone, but may be synthesized by simulation. For example, in a simulation, a convolution operation is performed on the sound source signal using an impulse response indicating the transfer characteristics from the position of each sound source to each microphone, and an acoustic signal indicating the component coming from the sound source can be generated. Therefore, an acoustic signal indicating the sound from multiple sound sources is obtained by adding up the components of each sound source. As an acoustic signal indicating the sound from a target sound source, an acoustic signal indicating the component of the target sound source may be adopted.
[0054] The model learning unit determines whether the parameter set has converged based on whether the update amount, which is the difference before and after the parameter set update, is equal to or less than a predetermined update amount threshold. The model learning unit continues the process of updating the parameter set until it is determined that the parameter set has converged. The model learning unit uses, for example, the L1 norm shown in Equation (6) as the loss function G(Θ).
[0055]
number
[0056] Equation (6) is the output value of the known target spectrum y w,t The target spectrum y' is estimated from the logarithm of the amplitude of w,t The loss function G(Θ) is given by the sum of the logarithmic amplitude difference between the frequencies and sets (frames). w,t , y' w,t By taking the logarithmic values of the respective parameters, it is possible to reduce the difference in the amplitude range, which may be significantly different for each frequency. This is convenient for processing frequencies collectively. The model learning unit may omit the convergence determination of the parameter set and repeat the parameter set update process a predetermined number of times.
[0057] In the above example, the mask function estimation unit 128 and the model learning unit use the corrected spectrum zw,t In the example, we used the normalized spectrum x' w,t In this case, the mask function estimation unit 128 uses the normalized spectrum x' w,t For the target spectrum y' w,t can be defined as the output value. In that case, the spatial filtering unit 126 may be omitted.
[0058] The spatial filtering unit 126 uses a spatial filter matrix W w H and the bias vector b w Using the corrected spectrum z w,t may be determined.
[0059]
number
[0060] Spatial filter matrix W w is a J (J is a predetermined integer equal to or greater than 1) filter coefficient vector w j,w In each column, j is an integer between 1 and J. That is, the spatial filter matrix W w is [w 1,w ,…,w J,w ]. Each filter coefficient vector w j,w corresponds to each beamformer and indicates the directivity in a given direction. j,w The norm of ||w j,w || is normalized to 1. Therefore, equation (7) is the normalized spectrum x' w,t For the spatial filter matrix W w H The product obtained by multiplying w Add to correct spectrum z w,t The mask function estimation unit 128 calculates the corrected spectrum z w,t Or its absolute value |z w,t| is used as the input value, and the mask function m w,t can be calculated.
[0061] The model learning part learns the parameter set of the machine learning model, as well as the spatial filter matrix W w and the bias vector b w The target spectrum y w,t As described above, the normalized spectrum x' w,t The spatial filter matrix W w and the bias vector b w Using the corrected spectrum z w,t The calculated corrected spectrum z w,t Based on this, the estimated value y' of the target spectrum is w,t is further calculated using the parameter set of the machine learning model.
[0062] In the above embodiment, it is assumed that the target direction is determined in advance, but this is not limiting. The sound processing device 10 may include a sound source direction estimation unit (not shown) for estimating the sound source direction using the sound signal of each channel. The sound source direction estimation unit outputs target direction information indicating the determined sound source direction as the target direction to the space normalization unit 124 and the spatial filtering unit 126. The space normalization unit 124 and the spatial filtering unit 126 can specify the target direction using the target direction information input from the sound source direction estimation unit.
[0063] The sound source direction estimator can estimate the sound source direction using, for example, the MUSIC (Multiple Signal Classification) method. The MUSIC method is a method of calculating, as a spatial spectrum, the ratio of the absolute value of a transfer function vector to a residual vector obtained by subtracting a significant eigenvector component from a transfer function vector, and determining, as the sound source direction, a direction in which the power of the spatial spectrum for each direction is higher than a predetermined threshold and is maximum. The transfer function vector is a vector having transfer functions from the sound source to each microphone as elements. The sound source direction estimating unit may estimate the sound source direction using other methods, for example, the Weighted Delay and Sum Beam Forming (WDS-BF) method. q The squared value of the delay sum of the above is calculated as the power of the spatial spectrum, and the sound source direction where the power of the spatial spectrum is higher than a predetermined threshold and is maximum is searched for.
[0064] The sound source direction estimator can determine the sound source direction of each of a plurality of sound sources simultaneously by using the above-mentioned method. In the process, the number of significant sound sources is detected. Therefore, the spatial filtering unit 126 is provided with a spatial filter matrix W w and the bias vector b w In the model learning, the model learning unit sets the number of filters J to be equal to or greater than the number of sound sources, and sets the spatial filter matrix W w and the bias vector b w The spatial filtering unit 126 may determine the number of sound sources based on the sound source direction of each sound source indicated in the sound source direction information input from the sound source direction estimation unit, and may determine a spatial filter matrix W corresponding to a filter number J equal to or greater than the determined number of sound sources. w and the bias vector b w Since the directivity of the spatial filter as a whole covers the sound source directions of all sound sources, a stable corrected spectrum can be obtained even if the number of sound sources increases.
[0065] As described above, the mask processing unit 130 determines each of the detected sound sources as a target sound source and calculates a mask function m w,t Using the target spectrum y' w,t The sound source signal processing unit 132 calculates the target spectrum y' w,tThe sound source signal processing unit 132 may output sound source direction information indicating the sound source direction estimated by the sound source direction estimation unit to a display unit provided in the device itself or the output destination device 30, and may select one of the multiple sound sources according to an operation signal input from an operation input unit. The display unit is, for example, a display. The operation input unit is, for example, a pointing device such as a touch sensor, a mouse, or a button. The sound source signal processing unit 132 may output a sound source signal of a target sound source component with the selected sound source as the target sound source, and stop outputting other sound source signals.
[0066] Also, in the above example, the mask function m w,t In the above, it is assumed that x is a scalar value having one element, but it may be a vector having multiple elements. In that case, the mask processing unit 130 processes the observed spectrum x k,w,t , the mask function m k,w,t The sum of the products obtained by multiplying w,t However, the mask function estimation unit 128 uses a similar method in model learning to calculate the target spectrum y' w,t The machine learning model generated by calculating is set.
[0067] (Audio Processing) Next, an example of the acoustic processing according to this embodiment will be described below with reference to a flowchart shown in FIG. (Step S102) The frequency analysis unit 122 performs frequency analysis for each frame on the acoustic signals of each channel input from each microphone, and determines an observation spectrum. (Step S104) The space normalization unit 124 performs space normalization so that the directivity direction of the sound collection unit 20 toward the target direction included in the observation spectrum is converted into a directivity direction toward a predetermined standard direction, and determines a normalized spectrum.
[0068] (Step S106) The spatial filtering unit 126 applies a spatial filter in the target direction to the normalized spectrum to determine a corrected spectrum. (Step S108) The mask function estimation unit 128 uses a machine learning model to determine a mask function with the corrected spectrum as an input value. (Step S110) Mask processing unit 130 applies a mask function to the observed spectrum of a predetermined channel to determine a target spectrum. (Step S112) The sound source signal processing unit 132 generates a sound source signal of a target sound source component in the time domain based on the target spectrum, and then ends the process shown in FIG.
[0069] (Model learning) Next, an example of model learning according to this embodiment will be described below. Fig. 6 is a flowchart showing an example of model learning according to this embodiment. (Step S202) The model learning unit forms training data including a plurality of data sets including corrected spectra based on normalized spectra for each frame from a plurality of sound sources as input values and target spectra from a target sound source as output values. (Step S204) The model learning unit sets initial values of a parameter set. If model learning has been performed in the past, the model learning unit may set the parameter set obtained by the past model learning as the initial values.
[0070] (Step S206) The model learning unit determines the update amount of the parameter set to make the loss function smaller by using a predetermined parameter estimation method. As the parameter estimation method, for example, any of the back propagation method, the steepest descent method, the stochastic gradient descent method, and the like can be used. (Step S208) The model learning unit adds a determined update amount to the original parameter set to calculate an updated parameter set (parameter update).
[0071] (Step S210) The model learning unit determines whether the parameter set has converged based on whether the update amount is equal to or less than a predetermined update amount threshold. If it is determined that the parameter set has converged (step S210 YES), the process shown in FIG. 6 ends. The model learning unit sets the obtained parameter set in the mask function estimation unit 128. If it is determined that the parameter set has not converged (step S210 NO), the process returns to step S206.
[0072] In the above description, spatial normalization, spatial filtering, mask processing, sound source signal processing, etc. are mainly performed using a frequency domain spectrum and involve calculations in the frequency domain, but this is not limiting. A time domain signal may be used instead of a frequency domain spectrum. In that case, convolution and deconvolution operations in the time domain may be performed instead of multiplication and division in the frequency domain. For example, the mask processing unit 130 may perform the following operations on the observed spectrum x k1,w,t mask function m w,t Multiplying by y' to get the target spectrum w,t Instead of calculating , a transform coefficient of a mask function in the time domain may be convoluted with the sound signal from the sound collection unit 20 to generate a sound signal indicating the target component. In that case, the inverse Fourier transform and the frequency analysis unit 122 in the sound source signal processing unit 132 may be omitted.
[0073] (experiment) Next, an experiment conducted to evaluate the effectiveness of the sound processing device 10 will be described. In the experiment, two types of sound sources were used. One was a sound source signal indicating human voice, and the other was a sound source signal indicating non-voice. As human voice, speech sounds included in the Corpus of Spontaneous Japanese (CSJ) were used. Sound source signals for the test set were selected from the official evaluation set defined by the CSJ. The test set includes sound source signals indicating 100 minutes of voice by 10 men and 10 women as test signals. The duration of the test signal in each trial ranges from 3 seconds to 10 seconds. As non-voice, sound source signals selected from the Real World Computing Partnership Sound Scene Database in Real Acoustical Environments were used as the test set. The RWCP Real World Computing Partnership Sound Scene Database in Real Acoustical Environments is a corpus that includes about 60 types of non-voice signals. For example, the sounds include glass breaking sounds, bell sounds, etc. As training data, sounds from 223 hours of academic presentations were used. The academic presentation includes source signals representing 799 male voices and 168 female voices.
[0074] In this experiment, a two-channel acoustic signal (sometimes referred to as a binaural signal in the following explanation) was synthesized as the observed signal by convolving a two-channel impulse response with the sound source signal. The observed signals were used to generate the training data and test set, respectively. The two-channel impulse response was measured in advance in an anechoic chamber with a sampling frequency of 16 kHz for each sound source direction. For the measurements, a two-channel microphone array as shown in Figures 3 and 4 was used. The impulse response represents the transmission characteristics of sound waves from the sound source to each microphone in the time domain.
[0075] 7 is a plan view showing the positional relationship between the microphone array (sound collection unit 20) and the sound source. A representative point of the microphone array is used as the origin O, and the sound source direction can be set in 1° increments on a circumference of a circle with a radius of 1.0 m centered on the origin O. However, in this experiment, two sound sources Sr-1 and Sr-2 with different heights were set for each sound source direction. 8 is a side view showing the positional relationship between the microphone array (sound collection unit 20) and sound sources Sr-1 and Sr-2. The height of the cross section where the two microphones are arranged is 0.6 m from the floor, while the heights of the sound sources Sr-1 and Sr-2 are 1.35 m and 1.10 m, respectively.
[0076] The sound sources Sr-1 and Sr-2 were used to generate different test sets 1 and 2, respectively. However, the sound source Sr-1 was used to generate the training data, and the sound source Sr-2 was not used. Therefore, the test set 1 is a matched test set in which the same sound source Sr-1 as the training data is used. The test set 2 is an unmatched test set in which the sound source Sr-2 different from the training data is used.
[0077] As training data, we used audio signals obtained by mixing the speech signals of three speakers. Most of the signals were from the same speaker. c,t The target directions of the other two speakers were chosen as (θ c,t +20+u)° and (θ c,t +340-u), where u is an integer value randomly selected from the integer values 0 to 140 inclusive.
[0078] Four types of data sets were used as test sets. The four types of data sets include signals that are a mixture of acoustic signals that show components from multiple sound sources as test signals in each trial. None of these signals are included in the training data. The four types of data sets are called the 2-voice (sp2) set, the 3-voice (sp3) set, the 2-voice + non-voice (sp2+n1) set, and the 4-voice (sp4) set, respectively. The 2-voice set includes test signals that are a mixture of the voices of two people. The sound source direction patterns in each trial included in the 2-voice set include three patterns: [0°, 30°], [0°, 45°], and [0°, 60°]. The 3-voice set includes test signals that are a mixture of the voices of three people. The sound source direction patterns in each trial included in the 3-voice set include three patterns: [0°, 30°, 60°], [0°, 45°, 90°], and [0°, 60°, 120°]. The 2-voice + non-voice (sp2+n1) set includes a test signal that mixes two voices with one non-voice. The same pattern as in the 2-voice set is used as the sound source direction pattern for the two voices. The sound source signal is used as is as the acoustic signal indicating non-voice. The 4-voice set includes a test signal that mixes four voices. The sound source direction pattern for the four voices includes one type of pattern [0°, 45°, 270°, 315°]. In all cases, the standard direction in spatial normalization was set to 0°. When using a DS beamformer, its directivity was always oriented to 0°. The test set includes an error of ±2° in the target direction.
[0079] For comparison with the present embodiment, the following two types of methods that do not involve spatial normalization were also evaluated as a baseline. The two types of methods are called Process A and Process B. Process A omits spatial normalization and uses a spatially corrected spectrum z based on a DS beamformer generated in spatial filtering. w,t In process B, spatial normalization is omitted and the spatially corrected spectrum z based on the spatial filter (optimized beam, OptBeam) obtained by learning is input to the mask function. w,t In either case, the target direction θ c,tis made variable, and the target sound source components are separated independently for each target sound source. For this embodiment, four types of processing were evaluated: processing A with spatial normalization, processing B with spatial normalization (J=2), processing B with spatial normalization (J=3), and processing B with spatial normalization (J=4).
[0080] In this experiment, we used a neural network as the machine learning model, and the settings were common among model training, source separation, and the test set for source separation. The neural network includes a feature-extraction network and a fully connected network. The feature extraction network includes mel-filter bank feature extraction, and the parameters were trained using back-propagation.
[0081] In this experiment, the frame shift was 10 ms. The feature extraction network includes the following functions in order: discrete Fourier transform (512-point window function), absolute value calculation, linear projection (filter bank, 64 dimensions), absolute value calculation, power calculation, frame concatenation, and linear projection (bottleneck, 256 dimensions). Spatial filtering was applied to each feature extraction stream. The duration of the observed signal contained in each data set constituting the training data was 640 ms. The fully connected network has 7 layers with a sigmoid function as the activation function. The output layer has 256-dimensional output nodes and is activated by a mask function m w,t with a sigmoid function to output
[0082] In this experiment, the Signal-to-Distortion Ratio (SDR) and Cepstrum Distortion (CD) were used as indicators of effectiveness. The SDR is an index value that indicates the degree of distortion of the target sound source component from a known reference signal. The higher the SDR value, the better the quality. The SDR can be determined using equation (8).
[0083]
number
[0084] Equation (8) expresses the target sound source component y' w,t The amplitude of the reference signal y w,t The product of the amplitude of and the parameter α and the error e w,t The parameter α is expressed as the sum of the frequency w and the error e w,t That is, the parameter α is set so as to minimize the target sound source component y' w,t The SDR is the contribution of the reference signal to the error amplitude |e w,t The amplitude α of the reference signal component for |y w,t The ratio of | corresponds to the logarithm of the sum of the power over frequency w and frame t.
[0085] On the other hand, CD is calculated using the cepstral coefficients obtained by performing a discrete cosine transform on the logarithmic magnitude spectrum. The smaller the CD value, the better the quality. In this experiment, the dimension of the cepstral coefficients was set from 1 to 24, and the distance value was calculated based on the average L1 norm (absolute error value). For SDR and CD, we considered the averaged values over the isolated target source components for each test set. When the input data contained multiple sources, the target direction was used to extract the target source components for each source from the other sources.
[0086] Next, the experimental results will be described. Fig. 9 is a table showing the quality of the extracted target sound source components. Fig. 9 shows the SDR and CD for each method and test set. The upper and lower rows of each column show the SDR and CD, respectively. However, "No processing" indicates the SDR and CD for the observed signal obtained without any processing. The underlines indicate the best performance for each test set. Comparing the baseline and this embodiment, the embodiment generally achieves better performance.
[0087] First, the SDR and CD obtained by Processing A, which is the baseline, show improvements over the SDR and CD obtained without processing for both test sets 1 and 2. However, as the number of sound sources increases, the performance significantly deteriorates, and performance is poorest when non-speech is mixed in. This indicates that Processing A has difficulty separating non-speech. The CDR and CD for treatment B showed no improvement at all compared to the CDR and CD for no treatment. One of the reasons for this is presumably failure to learn the spatial filter.
[0088] The SDR and CD obtained by the spatial normalization + processing A according to the present embodiment show good performance for both test sets 1 and 2. For test set 1, all items are the best. For test set 2, the CD for three sound sources, the SDR and CD for two sound sources + non-voice, and the SDR and CD for three sound sources are the best. According to the spatial normalization + processing A, an improvement of about 1-3 dB in CD is observed compared to the processing A related to the baseline. For the spatial normalization + processing B, the SDR and CD tend to be better as the number of filters J increases. For the spatial normalization + processing B (J = 4), the SDR and CD for two sounds and the SDR for three sounds are the best. This indicates that the performance is expected to improve as the number of filters J increases. It is presumed that the performance deteriorates when the number of filters J for the spatial normalization + processing B is small due to overlearning of the training data and the fact that no constraints were used in learning. Overfitting can accentuate the directivity toward a specific sound source direction, which can prevent the capture of the components of the target sound source that has that direction as the target direction. For example, the performance can be improved by using sparseness in Independent Component Analysis (ICA) as a constraint.
[0089] The directivities of the learned spatial filters have complementary beam patterns. The complementary beam patterns have a combination of a flat gain pattern and a null pattern in which the gain is lower in a certain direction than in other directions. Figure 10 shows the amplitude responses of the first and fourth channels of the four spatial filters obtained by learning in the first and second rows, respectively. The vertical and horizontal axes indicate the frequency and the azimuth angle of the sound source direction, respectively. The shading indicates the gain. The darker the area, the higher the gain, and the lighter the area, the lower the gain. In Fig. 10, two null directions (blind spots) are recognized in the fourth filter, whereas no null direction is recognized in the corresponding direction in the first filter. This shows that by using multiple filters, it is possible to capture all the components of the target sound source without missing any, even if the target sound source has a null direction of some filters based on the complementary beam pattern using a neural network.
[0090] As described above, the sound processing device 10 according to this embodiment includes a spatial normalization unit 124 that acquires sound signals from each of a plurality of microphones forming a microphone array, and normalizes a directional component toward a target direction of the microphone array included in the spectrum of the acquired sound signal to a directional component toward a predetermined standard direction to generate a normalized spectrum. The sound processing device 10 includes a mask function estimation unit 128 that determines a mask function for extracting a component of a target sound source arriving from the target direction based on the normalized spectrum using a machine learning model. The sound processing device 10 includes a mask processing unit 130 that applies a mask function to the acquired sound signal to estimate a component of a target sound source installed in the target direction. According to this configuration, the normalized spectrum used to estimate the mask function is normalized to include directional components in a standard direction, so there is no need to prepare a machine learning model that assumes all sound source directions. Therefore, it is possible to reduce the spatial complexity of the acoustic environment in model learning while ensuring the quality of the target sound source components obtained by sound source separation.
[0091] In normalization, space normalization section 124 may use a first steering vector indicating a directivity in a standard direction and a second steering vector indicating a directivity in a target direction. With this configuration, spatial normalization can be achieved with simple processing and configuration by using the first and second steering vectors that can also be used for other microphone array processing such as sound source direction estimation.
[0092] The sound processing device 10 may include a spatial filtering unit that applies a spatial filter indicating directivity toward a target direction to the normalized spectrum to generate a spatially corrected spectrum. The mask function estimation unit 128 may input the spatially corrected spectrum to a machine learning model to determine a mask function. With this configuration, the components of the target sound source installed in the target direction that are included in the acquired acoustic signal are reliably captured, so that the quality of the components of the estimated target sound source can be ensured.
[0093] The sound processing device 10 may include a model learning unit that determines a parameter set of a machine learning model so as to reduce a residual between an estimated value of a component of a target sound source obtained by applying a mask function to an acoustic signal indicating sounds arriving from multiple sound sources including the target sound source and a target value of the component of the target sound source. This configuration makes it possible to learn a machine learning model for determining a masking function that acts on an acoustic signal to estimate the components of a target sound source.
[0094] The model training unit may define a spatial filter for generating a spatially corrected spectrum from the normalized spectrum. The estimate of the component of the target sound source is obtained by applying a mask function to the spatially corrected spectrum. With this configuration, it is possible to simultaneously determine a parameter set for the machine learning model and a spatial filter for generating a spatially corrected spectrum to be input to the machine learning model.
[0095] The sound processing device 10 may include a sound source direction estimating unit that determines a sound source direction based on the plurality of sound signals. The space normalizing unit may determine the sound source direction determined by the sound source direction estimating unit as a target direction. With this configuration, even if the target direction of the target sound source is unknown, the components of the target sound source can be estimated.
[0096] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.
[0097] As described above, the mask processing unit 130 determines each of the detected sound sources as a target sound source and calculates a mask function m w,t Using the target spectrum y' w,t The sound source signal processing unit 132 calculates the target spectrum y' w,t The sound source signal processing unit 132 may output sound source direction information indicating the sound source direction estimated by the sound source direction estimation unit to a display unit provided in the device itself or the output destination device 30, and may select one of the multiple sound sources according to an operation signal input from an operation input unit. The display unit is, for example, a display. The operation input unit is, for example, a pointing device such as a touch sensor, a mouse, or a button. The sound source signal processing unit 132 may output a sound source signal of a target sound source component with the selected sound source as the target sound source, and stop outputting other sound source signals.
[0098] Furthermore, the sound processing device 10 may be configured as an acoustic unit integrated with the sound collection section 20. The positions of the individual microphones constituting the sound collection section 20 may be variable. The individual microphones may be installed on a moving body. The moving body may be a dolly, an aircraft, or the like. When the positions of the individual microphones are variable, the sound processing device 10 may be connected to a position detector for detecting the positions of the individual microphones. The control section 120 may determine a steering vector based on the positions of the individual microphones.
[0099] Note that a part of the sound processing device 10 in the above-described embodiment, for example, a part or all of the frequency analysis unit 122, the spatial normalization unit 124, the spatial filtering unit 126, the mask function estimation unit 128, the mask processing unit 130, and the sound source signal processing unit 132, may be realized by a computer. In this case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into and executed by a computer system including a processor. In addition, a part or the whole of the sound processing device 10 in the above-mentioned embodiment and modified examples may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the sound processing device 10 may be individually processed, or may be integrated into a processor in part or in whole. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used. [Explanation of symbols]
[0100] S1...sound processing system, 10...sound processing device, 110...input / output section, 120...control section, 122...frequency analysis section, 124...spatial normalization section, 126...spatial filtering section, 128...mask function estimation section, 130...mask processing section, 132...sound source signal processing section
Claims
1. a spatial normalization unit that normalizes a directional component toward a target direction of the microphone array, which is included in a spectrum of an acoustic signal acquired from each of a plurality of microphones forming a microphone array, to a directional component toward a predetermined standard direction to generate a normalized spectrum; a mask function estimation unit that determines a mask function for extracting a component of a target sound source arriving from the target direction based on the normalized spectrum using a machine learning model; a mask processing unit that applies the mask function to the acoustic signal to estimate a component of a target sound source installed in the target direction. Sound processing equipment.
2. The spatial normalization unit uses a first steering vector indicating a directivity toward the standard direction and a second steering vector indicating a directivity toward the target direction in the normalization. The sound processing device according to claim 1 .
3. a spatial filtering unit that applies a spatial filter that indicates a directivity toward the target direction to the normalized spectrum to generate a spatially corrected spectrum; The mask function estimation unit The spatially corrected spectrum is input into the machine learning model to determine the mask function. The sound processing device according to claim 1 or 2.
4. A model learning unit that determines a parameter set of the machine learning model so that a residual between an estimated value of a component of the target sound source obtained by applying the mask function to the acoustic signal indicating a sound arriving from a plurality of sound sources including the target sound source and a target value of the component of the target sound source is small. The sound processing device according to claim 1 .
5. The model learning unit includes: determining a spatial filter for generating a spatially corrected spectrum from the normalized spectrum; and obtaining an estimate of the component of the target sound source by applying the mask function to the spatially corrected spectrum. The sound processing device according to claim 4 .
6. a sound source direction estimating unit that determines a sound source direction based on a plurality of the acoustic signals; The spatial normalization unit uses the sound source direction as the target direction. The sound processing device according to any one of claims 1 to 5.
7. To your computer A program for causing the sound processing device according to any one of claims 1 to 6 to function as such a device.
8. An audio processing device comprising: a first step of normalizing a directional component in a target direction of the microphone array included in a spectrum of an acoustic signal acquired from each of a plurality of microphones forming a microphone array to a directional component in a predetermined standard direction to generate a normalized spectrum; A second step of determining a mask function for extracting a component of the target sound source arriving from the target direction based on the normalized spectrum using a machine learning model; A third step of estimating a component of a target sound source installed in the target direction by applying the mask function to the acoustic signal. Acoustic processing methods.
Citation Information
Patent Citations
Sound source direction estimation device, sound processing system, sound source direction estimation method, sound source direction estimation program
JP2014056181A
Noise elimination device and noise elimination method
WO2019049276A1
