Sound collection device, sound collection program, and sound collection method

Abstract

To provide robust sound collection processing with respect to an environment change related to a sound source within a target area.SOLUTION: The present invention related to a sound collection device. The sound collection device includes target sound extraction means which uses a learning model so as to acquire a target sound emphasis signal where a component of target sound included in a first input signal is emphasized from a differential signal to be a difference between the first input signal from a first microphone constituting a microphone array and a second input signal from a second microphone constituting the microphone array.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a sound collection device, a sound collection program, and a sound collection method, and more particularly to an area sound collection process for collecting sound coming from a sound source present in a target area (hereinafter referred to as "target area sound"). [Background technology]

[0002] Conventionally, an area sound collection technology using a multi-channel microphone is MUBASE (Multiple Beam-forming Area Sound Enhancement) described in Non-Patent Document 1. MUBASE is a method for enhancing the area in the front direction by utilizing the ability to extract disturbing sounds from the surroundings by using the difference between the observation signals of two-channel microphones.

[0003] FIG. 7 is a diagram showing an example of acquiring sound components emphasizing the sound components in the front direction (the direction in which the target area sound exists) by MUBASE using observation signals of a microphone array MA having two microphones Ml and Mr.

[0004] Here, when the observed signals of microphones Ml and Mr are expressed as equations (1) and (2) below, respectively (f is the frequency bin index), the difference of the observed signals can be expressed as equation (3) below. The difference of the observed signals then becomes a filter (hereinafter referred to as a "difference filter") that orients the null in the front direction, and interference sounds from outside the area are extracted. However, it is known that the power of the estimated interference sounds obtained by the difference filter becomes weaker at lower frequencies compared to the actual interference sounds. Using the estimated interference sounds obtained by the difference filter, the sound source y f The sound from can be extracted using the subtraction expressed by the following equation (4).

number

[0005] In equation (4), the subtraction coefficient αf is a hyperparameter. In equation (4), the coefficient α f The width of the highlighted area changes depending on the value of α f The larger the value, the narrower the beam. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Kazuhiro Katagiri, Tokuo Yamaguchi, Takashi Yazu, and Yoong Keok Lee, “Multiple beam-forming area sound enhancement (MUBASE) and stereophonic area sound reproduction (SASR) system”, SIGGRAPH Asia 2015 Emerging Technologies, 2015. Summary of the Invention [Problem to be solved by the invention]

[0007] In the conventional area sound collection process using MUBASE, the optimal coefficient α f The values ​​are different.

[0008] For example, in the conventional area recording processing using MUBASE, the coefficient α f If we assume that we want to adjust the coefficient α f If is too large, over-subtraction occurs, and the signal obtained by the sound pickup processing (the signal that emphasizes the target area sound) becomes distorted. f If is too small, the suppression of interference sounds (non-target area sounds) will be insufficient. Therefore, the coefficient α f It is difficult to achieve optimal adjustment of the

[0009] In view of the above problems, there is a demand for a sound collection device, a sound collection program, and a sound collection method that are more robust against environmental changes (eg, movement of the sound source) related to the sound source within the target area. [Means for solving the problem]

[0010] The sound collecting device of the first invention includes a target sound extraction processing means for acquiring a target sound emphasis signal in which a component of a target sound included in a first input signal is emphasized from a first input signal from a first microphone constituting a microphone array and a difference signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array, using a learning model. a learning means for acquiring the learning model by performing learning processing on data including the first input signal, the difference signal, and the target sound signal as teacher data; The present invention is characterized by having the following.

[0011] A second sound collection program of the present invention includes a target sound extraction processing means for acquiring a target sound emphasis signal in which a component of a target sound included in a first input signal is emphasized from a first input signal from a first microphone constituting a microphone array and a difference signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array, using a learning model; a learning means for acquiring the learning model by performing learning processing on data including the first input signal, the difference signal, and the target sound signal as teacher data; The present invention is characterized in that it functions as a

[0012] The third aspect of the present invention is Revenue In the sound collection method performed by a sound collection device, the sound collection device includes a target sound extraction processing means and learning tools The target sound extraction processing means uses a learning model to acquire a target sound enhancement signal in which a component of the target sound included in a first input signal is enhanced from a first input signal from a first microphone constituting a microphone array and a difference signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array. and obtaining the learning model by performing learning processing using data including the first input signal, the difference signal, and the target sound signal as training data. It is characterized by: Effect of the Invention

[0013] According to the present invention, it is possible to provide a sound collection process that is more robust against environmental changes related to sound sources within a target area. [Brief description of the drawings]

[0014] [Figure 1] 4 is a block diagram showing a functional configuration of a first target area sound extraction section according to the embodiment. FIG. [Diagram 2] 10 is a block diagram showing a functional configuration of a second target area sound extraction section according to the embodiment. FIG. [Diagram 3] 1 is a block diagram showing a functional configuration of a sound collection device according to an embodiment. [Figure 4] 1 is a block diagram showing an example of a hardware configuration of a sound collection device according to an embodiment. [Diagram 5] FIG. 1 is a diagram showing an experimental environment for a sound collection device according to an embodiment. [Figure 6] 11A to 11C are diagrams showing experimental results of the sound collection device according to the embodiment. [Figure 7] FIG. 1 is a diagram showing sound collection processing using a conventional two-channel microphone array. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] (A) Main embodiment Hereinafter, an embodiment of a sound collecting device, a program, and a method according to the present invention will be described in detail with reference to the drawings.

[0016] (A-1) Configuration of the embodiment FIG. 3 is a block diagram showing the functional configuration of the sound collection device 100 of this embodiment.

[0017] The sound collection device 100 performs a target area sound collection process for collecting target area sound from a sound source in the target area, using a microphone array MA including two microphones Mr and Ml.

[0018] The microphone array MA is disposed at an arbitrary location in the space in which the target area exists. In this embodiment, for the sake of simplicity, it is assumed that there is only one target area (target sound source disposed in the target area) that is the target of sound pickup by the microphone array MA.

[0019] Next, the internal configuration of the sound collection device 100 will be described.

[0020] The sound collection device 100 includes a signal input unit 101, a target area sound extraction unit 102, and a signal output unit 103. The detailed processing of each functional block constituting the sound collection device 100 will be described later.

[0021] The signal input unit 101 has the function of converting the acoustic signals (analog signals) observed by each microphone into digital signals, and then converting them into signals in a format that can be processed by the target area sound extraction unit 102 (in this embodiment, frequency domain signals). The signal input unit 101 converts the acoustic signals (analog signals) observed by each microphone from analog signals to digital signals, and further converts them from the time domain to the frequency domain (for example, by fast Fourier transform, etc.), and supplies them to the target area sound extraction unit 102.

[0022] Here, the observation signals (acoustic signals converted into the frequency domain) of the microphones Ml and Mr supplied from the signal input unit 101 to the target area sound extraction unit 102 are respectively denoted by X r , X l This is represented as follows.

[0023] The target area sound extraction unit 102 has a function of estimating and extracting a target area sound component from the signal supplied from the signal input unit 101 .

[0024] The signal output unit 103 converts the signal output from the target area sound extraction unit 102 from the frequency domain to the time domain and outputs the signal in a predetermined format. Note that the format and method of the signal output by the signal output unit 103 are not limited.

[0025] Next, an example of the hardware configuration of the sound collection device 100 will be described.

[0026] The sound collection device 100 may be configured entirely of hardware (for example, a dedicated chip, etc.), or may be configured partially or entirely as software (programs). The sound collection device 100 may be configured, for example, by installing a program (including the sound collection program of the embodiment) in a computer having a processor and memory.

[0027] FIG. 4 is a block diagram showing an example of a hardware configuration of the sound collection device 100. As shown in FIG.

[0028] FIG. 4 shows an example of a hardware configuration when the sound collection device 100 is configured using software (computer).

[0029] 4 includes, as a hardware component, a computer 400 in which a program (including the sound collection program of the embodiment) is installed. The computer 400 may be a computer dedicated to the sound collection program, or may be configured to be shared with programs of other functions.

[0030] The computer 400 shown in FIG. 4 includes a processor 401, a primary storage unit 402, and a secondary storage unit 403. The primary storage unit 402 is a storage unit that functions as a working memory (work memory) for the processor 401, and can be, for example, a high-speed memory such as a dynamic random access memory (DRAM). The secondary storage unit 403 is a storage unit that records various data such as an operating system (OS) and program data (including data of the sound collection program according to the embodiment), and can be, for example, a non-volatile memory such as a FLASH (registered trademark) memory, HDD, or SSD. In the computer 400 of this embodiment, when the processor 401 starts up, the OS and programs (including the sound collection program according to the embodiment) recorded in the secondary storage unit 403 are read, deployed on the primary storage unit 402, and executed.

[0031] Note that the specific configuration of the computer 400 is not limited to the configuration in Fig. 4, and various configurations can be applied. For example, if the primary storage unit 402 is a non-volatile memory (e.g., FLASH (registered trademark) memory, etc.), the secondary storage unit 403 may be excluded.

[0032] Next, an overview of the target area sound extraction process performed by the target area sound extraction section 102 will be described.

[0033] The target area sound extraction process described here is designed to extract the target area sound from the observation signals of two microphones, similar to the conventional MUBASE. In the conventional MUBASE process, the above formula (4) is applied, but the optimal coefficient α f The settings were different, and it was sometimes difficult to set them manually. f If the value of α is too large, over-subtraction occurs, and the audio in the target area may become distorted. f If the value is small, interference sounds outside the area may not be suppressed very well.

[0034] In the target area sound extraction unit 102 of this embodiment, the calculation corresponding to the subtraction expressed by the above equation (4) is described as being applied to a configuration in which the target area sound is collected by learning using a deep neural network (DNN (Deep Neural Network) (hereinafter referred to as "deep area sound collection" or "DMUBASE"). In the target area sound extraction unit 10 of this embodiment, deep area sound collection (DMUBASE) can achieve highly accurate area sound collection regardless of the arrival angle of the target sound or interfering sound.

[0035] In area sound collection processing, it is desirable for the processing to be robust against the movement of sound sources within the target area. However, in deep area sound collection (DMUBASE), filters are learned in a data-driven manner, so constraints are required to ensure robustness.

[0036] Therefore, it is desirable for deep area sound collection (DMUBASE) to meet the requirements of "being able to suppress interfering sounds outside the area" and "being robust against the movement of the target sound source within the area."

[0037] In light of the above, in this embodiment, as the model architecture of the target area sound extraction unit 102, for example, either the first target area sound extraction unit 102A shown in FIG. 1 or the second target area sound extraction unit 102B shown in FIG. 2 is applied.

[0038] First, the first target area sound extraction unit 102A shown in FIG. 1 will be described.

[0039] The first target area sound extraction unit 102A includes an estimation processing unit 200, a mask processing unit 210, a phase processing unit 220, and a difference extractor 230.

[0040] The estimation processing unit 200 estimates the observed signal X l , X r Based on X rThe components of the interference sound (non-target area sound) contained in X r The signal (hereafter referred to as the "mask signal") is output with a coefficient (filter coefficient) set to suppress the non-target area sound components included in the mask signal. r A filter coefficient (a value between 0 and 1) is set to suppress the interference sound (non-target area sound) components contained in the signal.

[0041] Specifically, the estimation processing unit 200 uses a DNN to estimate the observed signal |X r | and X l , X r The difference (output of the difference filter) is |d|=|X r -X l Here, the mask signal is estimated based on |. Note that, in order to extract the target area sound components from the observation signal Xr of the microphone Mr, the observation signal |X r and the output of the difference filter |X r -X l In the following description, the mask signal is estimated based on the observed signal |X l Based on |, the filter output |X l -X r A process of estimating a mask signal based on | may be performed.

[0042] The mask processing unit 210 calculates |X r The components of the interfering sound (non-target area sound) contained in | are masked (attenuated; suppressed; filtered) and a signal that emphasizes the target area sound is output.

[0043] The phase processor 220 applies X to the signal supplied from the mask processor 210. r The signal output from the phase processing unit 220 is expressed as "y^". Here, y^ is the output signal of the first target area sound extraction unit 102A. In other words, y^ can be said to be the result of extracting (emphasizing; estimating) the target area sound.

[0044] The difference extractor 230 obtains the difference between y^ (the result of estimating the destination area sound) output from the phase processing unit 220 and the clean destination area sound (hereinafter referred to as "y") which becomes the teacher label (correct label) in machine learning, and feeds back this difference to the estimation processing unit 200 as a loss (loss in machine learning). In other words, the difference extractor 230 is an element which functions only when the estimation processing unit 200 is made to perform learning processing. Therefore, if new learning processing is not already performed in the estimation processing unit 200, the difference extractor 230 may be excluded from the first destination area sound extraction unit 102A.

[0045] As described above, in the first target area sound extraction unit 102A, the estimation processing unit 200 receives the observed signal |X r A neural network (estimation processing unit 200) inputs |d|, which is dominated by non-target area sound (interference sound) obtained as the output of a differential filter, and estimates a mask signal for extracting a sound source (target area sound) within the target area corresponding to a sector-shaped area in front of the microphone array MA (see Figure 7).

[0046] The first target area sound extraction unit 102A of this embodiment is compatible with both an operation mode in which the estimation processing unit 200 executes a learning process (hereinafter referred to as a "learning processing mode") and an operation mode in which the target area sound extraction process (extraction of a mask signal and y^) is performed based on the supplied observation signals Xl and Xr (hereinafter referred to as a "signal processing mode"). Note that the first target area sound extraction unit 102A may be configured not to support the learning processing mode (for example, a configuration in which a learning model has already been acquired or a learning model is acquired from outside, etc.).

[0047] When operating in the learning processing mode, the first target area sound extraction unit 102A uses an observed signal (X l , X r ) and a clean target area sound y as the teacher label (hereafter referred to as the teacher dataset). l, X r ) to |X r and |d| and supply them to the estimation processing unit 200, and also feeds back the loss (difference) extracted by the difference extractor 230 to the estimation processing unit 200. This allows the estimation processing unit 200 to acquire a learning model that has been learned (deep learning) based on the teacher dataset.

[0048] Next, an example of the internal configuration of the estimation processing unit 200 will be described with reference to FIG.

[0049] Here, the example of Figure 1 will be described as the internal configuration of the estimation processing unit 200, but as the estimation processing unit 200, various machine learning (deep learning) frameworks can be applied as long as learning processing and signal processing based on the above-mentioned teacher data set are possible.

[0050] In this embodiment, the neural network of the estimation processing unit 200 is described as having the five-layer configuration shown in FIG. 1, but various configurations can be applied as long as learning processing and signal processing based on the above teacher data set are possible.

[0051] In the estimation processing unit 200 shown in Fig. 1, "FC layers 211, 212", "FC layers 221, 222", "FC layer 231", "FC layer 241", and "FC layer 251" are arranged in this order from the input layer. In the estimation processing unit 200 shown in Fig. 1, |x r 1, the activation function of only the FC layer 251 is a sigmoid, and the activation functions of the other FC layers are ReLU (Rectified Linear Unit).

[0052] In the neural network of the estimation processing unit 200 shown in FIG. rAfter applying nonlinear transformation to each of | and |d| by FC layers 211, 212, 221, and 222, the two inputs are combined and input to the third layer, FC layer 231. Furthermore, in the neural network of the estimation processing unit 200 shown in Fig. 1, the next two FC layers 241 and 251 perform transformation (inverse transformation) and output a mask signal (time-frequency mask). As described above, the activation function of the FC layer 251 is sigmoid, so the estimation processing unit 200 can output a coefficient (filter coefficient) represented by a numerical value between 0 and 1 for each frequency.

[0053] In the estimation processing unit 200, by configuring a neural network as shown in FIG. 1, the interference sound dominant |d| and the observed signal (observed sound) |X r By learning from data the process of extracting a mask signal (filter coefficient) that emphasizes the target area sound whose sound source is in the front direction of the microphone array MA from |, the subtraction process equivalent to equation (4) can be learned by the DNN. In other words, in the estimation processing unit 200, by configuring a neural network as shown in Fig. 1, it is possible to perform area sound collection processing that is robust against the movement of the sound source in the target area in the front direction of the microphone array MA. In particular, in the estimation processing unit 200, the FC layer 231, which is an intermediate layer to which two inputs are connected, is configured to be responsible for the above subtraction process.

[0054] Next, a description will be given of the second target area sound extraction unit 102B shown in Fig. 2. In Fig. 2, the same or corresponding parts as those in Fig. 1 above are given the same or corresponding reference numerals.

[0055] The following describes the second target area sound extraction unit 102B and the differences between the first target area sound extraction unit 102A.

[0056] As shown in FIG. 2, the second target area sound extraction section 102B includes an estimation processing section 300, a phase processing section 220, and a difference extractor 230.

[0057] The estimation processor 200 estimates the observed signal |X r While the estimation processing unit 300 previously performed processing to estimate a masking signal based on | and |d|, the estimation processing unit 300 differs from the estimation processing unit 200 in that it outputs a power spectrum (a spectrum of a signal in which the components of the target area sound are emphasized; a signal in the frequency domain) obtained as a result of estimating the target area sound y.

[0058] The phase processor 220 adds X to the power spectrum supplied from the estimation processor 300. r The signal is output as a signal y^ after processing to add (synchronize) the phase (phase information) of the signal y^.

[0059] As described above, in the second target area sound extraction unit 102B, the estimation processing unit 300 receives the observed signal |X r A neural network (estimation processing unit 300) inputs |d|, which is dominated by non-target area sounds (interference sounds) obtained as the output of a differential filter, and estimates the power spectrum of the sound source (target area sound) in the target area corresponding to the sector-shaped region in front of the microphone array MA (see Figure 7).

[0060] Then, the second target area sound extraction unit 102B may be adapted to support a learning processing mode and a signal processing mode, similarly to the first target area sound extraction unit 102A.

[0061] Next, an example of the internal configuration of the estimation processing unit 300 will be described with reference to FIG.

[0062] Here, the example of FIG. 2 will be described as the internal configuration of the estimation processing unit 300, but various machine learning (deep learning) configurations can be applied to the estimation processing unit 300 as long as it is capable of performing learning processing and signal processing based on the above teacher data set.

[0063] 2, the neural network of the estimation processing unit 300 is different from that of the estimation processing unit 200 in that the FC layer 251 at the last stage (output layer) of the neural network of the estimation processing unit 300 is replaced with an FC layer 351. The FC layer 351 of the estimation processing unit 300 is different from that of the estimation processing unit 200 in that the activation function is ReLu instead of sigmoid. This allows the FC layer 351 of the estimation processing unit 300 to output a power spectrum.

[0064] In the estimation processing unit 300, the neural network shown in FIG. 2 is used to estimate |d|, which is the dominant interfering sound, and |X r By configuring a mechanism to output target area sound with the sound source coming from the front direction of the microphone array MA from |, it is possible to learn the subtraction process equivalent to equation (4) from the data.

[0065] (A-2) Operation of the embodiment Next, the operation of the sound collection device 100 in this embodiment having the above-mentioned configuration (the sound collection method according to the embodiment) will be described.

[0066] First, a process performed when the target area sound extraction unit 102 of the sound collection device 100 operates in the learning process mode will be described.

[0067] When a teacher data set is supplied to the target area sound extraction unit 102 operating in the learning processing mode, the target area sound extraction unit 102 extracts the observed signal (X l , X r ) to |X r | and |d| are obtained and input into the neural network to perform a learning process for deep area sound collection (learning the process of extracting target area sounds using the neural network).

[0068] When the first target area sound extraction unit 102A is applied to the sound collection device 100, the first target area sound extraction unit 102A extracts |X r| and |d| are input to the estimation processing unit 200. At this time, in the first target area sound extraction unit 102A, the difference extractor 230 extracts the loss between the signal y^ output from the phase processing unit 220 and the teacher label y, and feeds it back to the estimation processing unit 200. In the first target area sound extraction unit 102A, a learning process for deep area sound collection (learning of the process of extracting the target area sound by a neural network) is performed by the above-mentioned feedback.

[0069] On the other hand, when the second target area sound extraction unit 102B is applied to the sound collection device 100, the second target area sound extraction unit 102B extracts |X r | and |d| are input to the estimation processing unit 300. At this time, in the second target area sound extraction unit 102B, the difference extractor 230 extracts the loss of the power spectrum output from the estimation processing unit 300 and feeds it back to the estimation processing unit 300. In the second target area sound extraction unit 102B, a learning process for deep area sound collection (learning of the process of extracting target area sound by a neural network) is performed by the above-mentioned feedback.

[0070] Next, the operation of the target area sound extraction unit 102 of the sound collection device 100 when it operates in the signal processing mode will be described.

[0071] Here, an observation signal (X l , X r ) is supplied to the target area sound extraction unit 102. Then, the target area sound extraction unit 102 inputs |X r and |d|, and as a result, y^ is obtained and supplied to the signal output unit 103. The signal output unit 103 converts y^ from the frequency domain to the time domain and outputs it.

[0072] Next, the inventors will explain the experimental results and evaluation results of an experiment (hereinafter referred to as "this experiment") in which they actually constructed a sound collection device 100, performed a process to collect target area sound, and evaluated its quality.

[0073] FIG. 5 shows the environment of this experiment.

[0074] FIG. 5 shows an example in which microphones Mr, Ml, a target sound source, and an interfering sound source are all on the same plane. In addition, in FIG. 5, the position of the midpoint of a line L connecting the positions (center positions) of microphones Mr, Ml (center point of microphone array MA) is illustrated as P1. Furthermore, in FIG. 5, the direction of microphone Mr as viewed from P1 is 0°, and the direction of microphone Ml as viewed from P1 is 180°, and the target sound source and the interfering sound source arrive from any angle of 0° to 180° as viewed from P1. Hereinafter, the direction in which the apparent sound source and the interfering sound source exist from P1 will also be referred to as the "arrival angle" or "arrival direction". In addition, as shown in FIG. 5, the positions of the target sound source and the interfering sound source (sound source in a non-target area) are on a semicircular line at a distance of 1 m from P1.

[0075] In this experiment, in both the learning processing mode (during training) and the signal processing mode (during signal processing), the TIMIT corpus (see Reference 1 below) was used as the dry source (signal) of the target sound source, and the TMIT corpus or the DEMAND (Diverse Environments Multi-channel Acoustic Noise Database) corpus (see Reference 2 below) was used as the dry source (signal) of the interfering sound.

[0076] Reference 1: JS Garofolo, LF Lamel, WM Fisher, JGFiscus, DS Pallett, NL Dahlgren, V. Zue, “TIMIT acoustic phonetic continuous speech corpus,” Linguistic Data Consotrium, 1992. Reference 2: J. Thiemann, N. Ito, and E. Vincent, “The diverseenvironments multi-channel acoustic noise database(DEMAND): A database of multichannel environmental noise recordings”, The Journal of the Acoustical Society of America, vol. 133, p. 3591,05, 2013.

[0077] In this experiment, observation signals (acoustic signals) captured by microphones Ml and Mr in a sound field (model environment) such as that shown in Figure 5 were obtained by computer simulation, and the results of inputting the obtained observation signals to the sound collection device 100 were evaluated.

[0078] Specifically, in this experiment, we used PyRoomAcoustics (see Reference 3 below) to perform a simulation with a sound field (model environment) as shown in Figure 5 to obtain an impulse response, and convolved the obtained impulse response with the above dry sources (the dry sources of the target sound source and the interfering sound source) to obtain the observed signal X of microphones Ml and Mr. l , X r obtained.

[0079] Reference 3: Scheibler, E. Bezzam, I. Dokmani´c, “Pyroomacoustics: A Python package for audio room simulations and array processing algorithms”, Proc. IEEE ICASSP, 2018

[0080] In addition, in the simulation of this experiment, the observed signal X l , X r The SNR at is adjusted to be approximately 0.0 dB. In the following, the 3D space of the sound field used in this experiment is represented by a three-dimensional coordinate system of (x, y, z).

[0081] In the simulation of this experiment, the size of the room that constitutes the sound field of the model environment was set to (x,y,z) [m] = (5,3,3), and the coordinates of the 2ch microphones Ml and Mr were set to (x,y,z) [m] = (2.49,1.5,1) and (x,y,z) [m] = (2.51,1.5,1), respectively. This results in a distance between microphones Ml and Mr of 2 [cm]. In the simulation of this experiment, the sound absorption coefficient of the room was set to 0.2, and the number of reflections in the room was set to 3.

[0082] In this experiment, simulations were performed for each of the following cases: a first target area sound extraction unit 102A that estimates the target area sound by mask estimation is applied to the target area sound extraction unit 102 of the sound collection device 100 (hereinafter referred to as the ``first experimental model of the present invention''); a second target area sound extraction unit 102B that estimates the target area sound by power spectrum estimation is applied (hereinafter referred to as the ``second experimental model of the present invention''); and target area sound estimation using the conventional MUBASE is applied (hereinafter referred to as the ``experimental model of the conventional configuration'').

[0083] Next, the position of each sound source in the simulation of this experiment will be described.

[0084] In this experiment, the position of the target sound source was fixed at 90° during learning, and the target sound source was moved within the area (within a range of 80° to 90° at a distance of 1 m from P1) during testing to verify whether the sound collection device 100 satisfies the above two requirements. In addition, the interference sound source was randomly placed in 1 to 3 of a total of 8 positions, 0°, 15°, 30°, 45°, 135°, 150°, 165°, and 180°, both during learning and testing. In this experiment, such changes in the positions of the target sound source and the interference sound source were performed in data processing units (e.g., word units) on the corpus.

[0085] Next, detailed parameter settings in the simulation of this experiment will be described.

[0086] In a simulation applying the "conventional MUBASE model," when estimating non-target area sound (interference sound) using a differential filter in the target area sound extraction unit 102, the value of α was set to 200 / (f+0.01) based on the tendency for the power to be weaker at lower frequencies.

[0087] In addition, in the training of the "first experimental model of the present invention" and the "second experimental model of the present invention", the batch size was set to 32, the number of epochs was set to 200, and the mean square error was used as the loss function. In the training of the "first experimental model of the present invention" and the "second experimental model of the present invention", Adam (see Reference 4 below) was used as the optimization algorithm, and the learning rate was set to 0.001.

[0088] Reference 4: D. Kingma, and J. Ba, “Adam: A method for stochastic optimization”, International Conference on Learning Representations (ICLR), 2015.

[0089] In this experiment, an environment was constructed for the first experimental model of the present invention (mask estimation), the second experimental model of the present invention (power spectrum estimation), and an experimental model of a conventional configuration (MUBASE), and the above-mentioned learning process and signal processing (test process) were performed for each of them. In the signal processing (test process) of this experiment, two indexes, SNR (Signal-to-Noise Ratio) and STOI (Short-Time Objective Intelligibility), were measured for each of the three experimental models. In addition, in the test process of this experiment, test processes were performed for each experimental model in a pattern in which the position of the target sound source was fixed at 90° (hereinafter referred to as the "target sound source fixed pattern") and a pattern in which the target sound source was moved randomly between 80° and 90° (hereinafter referred to as the "target sound source moving pattern"). Figure 6 is a diagram showing the results of this experiment.

[0090] (A-3) Effects of the embodiment According to this embodiment, the following effects can be obtained.

[0091] In a conventional configuration using MUBASE, area sound collection processing was performed by spectral subtraction involving a predetermined coefficient, but in the sound collection device 100 of this embodiment, deep area sound collection (DMUBASE) is performed using a neural network trained with teacher data. In particular, in the sound collection device 100 of this embodiment, by taking the output d of a differential filter in a two-channel microphone array MA, interference sounds (non-target area sounds) coming from directions other than the front direction can be obtained, and by inputting the output d of the differential filter (information in which interference sounds are dominant) into the neural network together with the observation signal, an output in which the target area sound in the front direction is emphasized can be obtained.

[0092] As described above, in the area sound collection process, it is desirable to be robust against the movement of the sound source in the target area, but in the deep area sound collection (DMUBASE), since the filter is learned in a data-driven manner, a constraint to guarantee robustness is necessary. In the sound collection device 100 of this embodiment, robustness against environmental changes is improved by using physical information such as the output d of the differential filter as auxiliary information, rather than a simple data-driven (for example, only the observed signal). As shown in FIG. 6, both of the experimental models (the first and second experimental models of the present invention) that reproduce the configuration of this embodiment exceeded the accuracy of the experimental model (MUBASE) of the conventional configuration in both the target sound source fixed pattern and the target sound source moving pattern. In other words, it was confirmed that the experimental model of the present invention is more robust against the movement of the sound source of the target area sound than the conventional one.

[0093] (B) Other embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments such as those exemplified below.

[0094] (B-1) In the above embodiment, the sound collection device 100 has been described as supporting both the learning processing mode and the signal processing mode (test mode). However, if a learning model is stored in advance, the sound collection device 100 may be configured to support only the signal processing mode and to exclude the means (learning means) required for the learning processing mode. [Explanation of symbols]

[0095] 100...sound collection device, 101...signal input section, 102...target area sound extraction section, 103...signal output section, 102A...first target area sound extraction section, 200...estimation processing section, 212, 221, 222, 231, 241, 251, 211...FC layer, 210...mask processing section, 220...phase processing section, 230...difference extractor, 102B...second target area sound extraction section, ...estimation processing section 300, 212, 221, 222, 231, 241, 351, 211...FC layer, 230...difference extractor.

Claims

1. a target sound extraction processing means for acquiring a target sound emphasis signal in which a component of a target sound included in a first input signal is emphasized from a first input signal from a first microphone constituting a microphone array and a difference signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array, using a learning model; a learning means for obtaining the learning model by performing learning processing on data including the first input signal, the difference signal, and the target sound signal as teacher data; A sound collecting device comprising:

2. the learning model outputs a mask coefficient that suppresses components of non-target sounds other than the target sound included in the first input signal from the first input signal and the differential signal; The target sound extraction processing means obtains the target sound emphasis signal by suppressing the non-target sound components from the first input signal using the mask coefficient.

2. The sound pickup device according to claim 1 .

3. The sound collection device according to claim 1, characterized in that the learning model outputs a target sound emphasis signal that suppresses non-target sound components contained in the first input signal and emphasizes the target sound components from the first input signal and the differential signal.

4. Computer, a target sound extraction processing means for acquiring a target sound emphasis signal in which a component of a target sound included in a first input signal is emphasized from a first input signal from a first microphone constituting a microphone array and a difference signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array, using a learning model; a learning means for obtaining the learning model by performing learning processing on data including the first input signal, the difference signal, and the target sound signal as teacher data; A sound recording program characterized by functioning as follows.

5. A sound collection method performed by a sound collection device, The sound collection device includes a target sound extraction processing means and a learning means, the target sound extraction processing means acquires, using a learning model, a target sound enhancement signal in which a component of the target sound included in a first input signal is enhanced from a first input signal from a first microphone constituting a microphone array and a differential signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array; The learning means obtains the learning model by performing a learning process using data including the first input signal, the difference signal, and the target sound signal as teacher data. A sound collection method comprising:

Images