Sound collection device, sound collection program, and sound collection method

The RMC-MV beamformer with DNN mask estimation addresses coefficient adjustment issues in existing beamformers, enhancing target sound extraction and noise suppression by allowing controlled distortion, improving robustness and performance.

JP7589943B2Active Publication Date: 2024-11-26OKI ELECTRIC INDUSTRY CO LTD +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2021102528
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-06-21
Publication Date
2024-11-26
Estimated Expiration
2041-06-21

AI Technical Summary

Technical Problem

Existing beamforming technologies like SSBF and MVDR beamformers face challenges in optimally adjusting coefficients for sound source positions, leading to insufficient noise suppression and reduced robustness against environmental changes, particularly sound source movements.

Method used

A sound collection device using a Relaxed Multiple-Constraint Minimum Variance (RMC-MV) beamformer with a distortion tolerance, combined with a deep neural network (DNN) for mask estimation, to enhance target sound extraction by allowing controlled distortion, improving noise suppression performance and robustness.

Benefits of technology

The RMC-MV beamformer enhances target sound extraction by increasing robustness against environmental changes and sound source movements, improving noise suppression performance without a trade-off, as demonstrated in experimental results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589943000003
    Figure 0007589943000003
  • Figure 0007589943000004
    Figure 0007589943000004
  • Figure 0007589943000005
    Figure 0007589943000005
Patent Text Reader

Abstract

To provide a sound collection device, a sound collection program, and a sound collection method that are more robust against environmental changes regarding a sound source.SOLUTION: The present invention relates to a sound collection device. Then, the sound collection device according to the present invention includes non-target area sound dominant signal acquiring means that acquires a non-target area sound dominant signal in which a non-target area sound other than a target area sound having a target area as a sound source is dominant, from input signals from a plurality of microphones, mask estimating means that estimates a mask for acquiring a target area sound emphasized signal in which the target area sound is emphasized from the input signal, on the basis of the input signal of any microphone and the non-target area sound dominant signal, and extraction means that performs an extraction process of extracting a target area sound enhancement signal from an input signal using the mask. The extraction means performs extraction processing using an MVDR beamformer in which the constraint condition is replaced with a distortion allowance that allows distortion of the target area sound.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a sound collection device, a sound collection program, and a sound collection method, and more particularly to an area sound collection process for collecting sound (hereinafter referred to as "target area sound" or "target sound") coming from a sound source (hereinafter referred to as "target sound source") present in a target area. [Background technology]

[0002] Conventionally, a beamforming technology using a multi-channel microphone is SSBF (Spectrum-Subtraction based Beamformer) described in Non-Patent Document 1. SSBF is a method that emphasizes a target sound source in the front direction by utilizing the ability to extract interference sounds from the surroundings by using the difference between observed signals of two-channel microphones.

[0003] FIG. 9 is a diagram showing an example of acquiring sound by emphasizing frontal sound components using SSBF, using observed signals from a microphone array MA including two microphones Ml and Mr.

[0004] Here, when the observed signals of microphones Ml and Mr are expressed as equations (1) and (2), respectively, (f is the frequency bin index), the difference between the observed signals can be expressed as equation (3) below. Note that in the text of this specification (text other than drawings and formulas), for convenience of notation, the observed signals of microphones Ml and Mr (acoustic signals in the frequency domain; signals expressed by equations (1) and (2)) are expressed as x (l) f , x (r) f This is represented as follows.

[0005] The difference between the observed signals is then filtered to direct the null in the front direction (hereafter referred to as the "difference filter"), and interference sounds from directions other than the front direction are extracted. However, it is known that the estimated interference sounds obtained by the difference filter have weaker power at lower frequencies than the actual interference sounds. Using the estimated interference sounds obtained by the difference filter, the target sound source yf The sound from can be extracted using spectral subtraction, as shown in equation (4) below.

[0006]

number

[0007] Non-Patent Document 2 describes a technology (hereinafter referred to as "localization MVDR beamformer") that estimates the position of a target sound source using a deep neural network (DNN) and extracts the target sound source using an MVDR (Minimum Variance Distortionless Response) beamformer. The MVDR beamformer imposes a distortion-free constraint on the direction of the target sound source (target direction) and minimizes the average power of noise after processing under the constraint condition. Since the constraint is imposed on the target direction, a steering vector of the target sound source is usually required. On the other hand, localization MVDR does not require a steering vector of the target sound source by estimating a mask that separates the target sound source and noise by a DNN while taking into account a spatial model by the MVDR beamformer. Since the target sound source is separated by taking into account the spatial model, this DNN estimates the position of the target sound source.

[0008] Non-Patent Documents 3 and 4 describe an MVDR beamformer that imposes distortion-free constraints in multiple directions (hereinafter referred to as a "multiple constraint MVDR beamformer," "multipoint constraint MVDR beamformer," or "MC-MVDR beamformer"). This method makes it possible to give width to the beam by imposing distortion-free conditions in multiple directions, thereby enabling robust enhancement against slight movements of the sound source. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Kazuhiro Katagiri, Tokuo Yamaguchi, Takashi Yazu, and Yoong Keok Lee, “Multiple beam-forming area sound enhancement (MUBASE) and stereophonic area sound reproduction (SASR) system”, SIGGRAPH Asia 2015 Emerging Technologies, 2015. [Non-Patent Document 2] Tsubasa Ochiai, Shinji Watanabe, Takaaki Hori and John R. Hershey, “Multichannel End-to-end Speech Recognition”, arXiv:1703.04783, 2017. [Non-Patent Document 3] Futoshi Asano, "Acoustic Technology Series 16: Array Signal Processing of Sound - Localization, Tracking and Separation of Sound Sources", pp86-90, edited by the Acoustical Society of Japan, Corona Publishing, published on February 25, 2011 [Non-Patent Document 4] RG Lorenz and SP Boyd, “Robust minimum variance beamforming,” IEEE Trans. on Signal Process., vol. 53, pp.1684-1696, May. 2005. Summary of the Invention [Problem to be solved by the invention]

[0010] In the beamformer using SSBF described in Non-Patent Document 1, the optimal coefficient α f The values ​​are different.

[0011] For example, in the conventional area sound collection processing using SSBF, the coefficient α f If we assume that we want to adjust the coefficient α f If is too large, over-subtraction occurs, and the signal obtained by the sound pickup processing (the signal that emphasizes the target area sound) becomes distorted. f If is too small, the suppression of interference sounds (non-target area sounds) will be insufficient. Therefore, the coefficient α f It is difficult to achieve optimal adjustment of the

[0012] Furthermore, the localization MVDR described in Non-Patent Document 2 estimates the position of the target sound source implicitly, and therefore cannot extract the target sound source if the estimation of the position fails.

[0013] Furthermore, it is known that the multiple constraint MVDR beamformer described in Non-Patent Documents 3 and 4 has a significantly reduced degree of freedom due to multiple distortion-free constraint conditions, and therefore the noise suppression performance is also reduced. Therefore, the above-mentioned trade-off remains between the robustness of the target sound source movement and the noise suppression performance (target sound enhancement performance).

[0014] In view of the above problems, there is a demand for a sound collection device that is more robust against environmental changes (eg, movement of the sound source) related to the target sound source. [Means for solving the problem]

[0015] The first sound collection device of the present invention comprises (1) a non-destination area sound dominant signal acquisition means for acquiring a non-destination area sound dominant signal in which non-destination area sounds other than the destination area sound whose sound source is the destination area are dominant from input signals from a plurality of microphones; (2) a mask estimation means for estimating a mask for acquiring a destination area sound emphasis signal that emphasizes the destination area sound from the input signal based on the input signal of any of the microphones and the non-destination area sound dominant signal; and (3) an extraction means for performing an extraction process for extracting the destination area sound emphasis signal from the input signal using the mask, and (4) the extraction means is characterized in that the extraction process is performed using an MVDR beamformer whose constraint condition is a distortion tolerance that allows distortion of the destination area sound.

[0016] The second sound collection program of the present invention causes a computer to function as: (1) a non-destination area sound dominant signal acquisition means for acquiring a non-destination area sound dominant signal in which non-destination area sounds other than the destination area sound whose sound source is the destination area are dominant from input signals from a plurality of microphones; (2) a mask estimation means for estimating a mask for acquiring a destination area sound emphasis signal that emphasizes the destination area sound from the input signal based on the input signal of any of the microphones and the non-destination area sound dominant signal; and (3) an extraction means for performing an extraction process for extracting the destination area sound emphasis signal from the input signal using the mask; and (4) the extraction means is characterized in that it performs the extraction process using an MVDR beamformer whose constraint condition is a distortion tolerance that allows distortion of the destination area sound.

[0017] The third invention is a sound collection method performed by a sound collection device, which comprises (1) a non-target area sound dominant signal acquisition means, a mask estimation means, and an extraction means, (2) the non-target area sound dominant signal acquisition means acquires a non-target area sound dominant signal in which non-target area sounds other than the target area sound whose sound source is the target area are dominant from input signals from a plurality of microphones, (3) the mask estimation means estimates a mask for acquiring a target area sound emphasis signal that emphasizes the target area sound from the input signal based on the input signal of any of the microphones and the non-target area sound dominant signal, (4) the extraction means performs an extraction process for extracting the target area sound emphasis signal from the input signal using the mask, and (5) the extraction means performs the extraction process using an MVDR beamformer whose constraint condition is a distortion tolerance that allows distortion of the target area sound. Effect of the Invention

[0018] According to the present invention, it is possible to provide a sound collection device, a sound collection program, and a sound collection method that are more robust against environmental changes related to a target sound source. [Brief description of the drawings]

[0019] [Figure 1] FIG. 2 is a block diagram showing a functional configuration of a target area sound extraction unit according to the embodiment. [Diagram 2] 1 is a block diagram showing a functional configuration of a sound collection device according to an embodiment. [Diagram 3] 1 is a block diagram showing an example of a hardware configuration of a sound collection device according to an embodiment. [Figure 4] 1 is a diagram showing an example of a space model related to area sound collection processing performed by a sound collection device according to an embodiment; [Diagram 5] FIG. 2 is a block diagram showing a first architecture of a DNN constituting a target area sound extraction unit according to the embodiment. [Figure 6] FIG. 11 is a block diagram showing a second architecture of a DNN constituting a target area sound extraction unit according to the embodiment. [Figure 7] FIG. 2 is a diagram showing an experimental model of a sound collecting device according to an embodiment. [Figure 8] 11A to 11C are diagrams showing experimental results of the sound collection device according to the embodiment. [Figure 9] FIG. 1 is a diagram showing sound collection processing using a conventional two-channel microphone array. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] (A) Main embodiment Hereinafter, an embodiment of a sound collecting device, a program, and a method according to the present invention will be described in detail with reference to the drawings.

[0021] (A-1) Configuration of the First Embodiment FIG. 2 is a block diagram showing the functional configuration of the sound collection device 100 of this embodiment.

[0022] The sound collection device 100 performs a target area sound collection process for collecting target area sound from a sound source in the target area, using a microphone array MA including two microphones Mr and Ml.

[0023] The microphone array MA is disposed at an arbitrary location in the space in which the target area exists. In this embodiment, for the sake of simplicity, it is assumed that there is only one target area (target sound source disposed in the target area) that is the target of sound pickup by the microphone array MA.

[0024] Next, the internal configuration of the sound collection device 100 will be described with reference to FIG.

[0025] The sound collection device 100 includes a signal input unit 101, a target area sound extraction unit 102, and a signal output unit 103. The detailed processing of each functional block constituting the sound collection device 100 will be described later.

[0026] The signal input unit 101 has the function of converting the acoustic signals (analog signals) observed by each microphone into digital signals, and then converting them into signals in a format that can be processed by the target area sound extraction unit 102 (in this embodiment, signals in the frequency domain). The signal input unit 101 converts the acoustic signals (analog signals) observed by each microphone from analog signals to digital signals, and further converts them from the time domain to the frequency domain (for example, by fast Fourier transform, etc.), and supplies them to the target area sound extraction unit 102.

[0027] The target area sound extraction unit 102 extracts the observed signal (x (r) f , x (l) f ), and acquires an emphasis signal that emphasizes the target area sound (target sound). Note that in this specification, for convenience of notation, the emphasis signal output from BF203 is referred to as "^y f " shall be expressed as follows.

[0028] The signal output unit 103 outputs the emphasis signal ^y output from the target area sound extraction unit 102. f The signal output unit 103 converts the signal from the frequency domain to the time domain and outputs the converted signal in a predetermined format. Note that the format of the signal output by the signal output unit 103 is not limited.

[0029] Next, an example of the hardware configuration of the sound collection device 100 will be described.

[0030] The sound collection device 100 may be configured entirely of hardware (for example, a dedicated chip, etc.), or may be configured partially or entirely as software (programs). The sound collection device 100 may be configured, for example, by installing a program (including the sound collection program of the embodiment) in a computer having a processor and memory.

[0031] FIG. 3 is a block diagram showing an example of a hardware configuration of the sound collection device 100. As shown in FIG.

[0032] FIG. 3 shows an example of a hardware configuration when the sound collection device 100 is configured using software (computer).

[0033] 3 includes, as a hardware component, a computer 400 in which a program (including the sound collection program of the embodiment) is installed. The computer 400 may be a computer dedicated to the sound collection program, or may be configured to be shared with programs of other functions.

[0034] The computer 400 shown in FIG. 3 includes a processor 401, a primary storage unit 402, and a secondary storage unit 403. The primary storage unit 402 is a storage unit that functions as a working memory (work memory) for the processor 401, and can be, for example, a high-speed memory such as a dynamic random access memory (DRAM). The secondary storage unit 403 is a storage unit that records various data such as an operating system (OS) and program data (including data of the sound collection program according to the embodiment), and can be, for example, a non-volatile memory such as a FLASH (registered trademark) memory, HDD, or SSD. In the computer 400 of this embodiment, when the processor 401 starts up, the OS and programs (including the sound collection program according to the embodiment) recorded in the secondary storage unit 403 are read, deployed on the primary storage unit 402, and executed.

[0035] Note that the specific configuration of the computer 400 is not limited to the configuration in Fig. 3, and various configurations can be applied. For example, if the primary storage unit 402 is a non-volatile memory (e.g., a flash memory, etc.), the secondary storage unit 403 may be excluded from the configuration.

[0036] The sound collection device, sound collection program, and sound collection method according to the embodiment include at least a configuration equivalent to the target area sound extraction unit 102. Therefore, a device or program that performs only the processing equivalent to the target area sound extraction unit 102 may be configured as the sound collection device or sound collection program of the present invention.

[0037] Next, a model (space model) of the positional relationship between the microphone array MA (microphones Mr, Ml) and the target area (target sound source) in this embodiment will be described with reference to FIG.

[0038] In Fig. 4, the spatial model is shown as viewed from above when the microphones Mr, Ml, the target sound source (target area sound source), and the interfering sound source (non-target area sound source) are all present on the same plane. In Fig. 4, the position of the midpoint of the line L connecting the positions (center positions) of the microphones Mr, Ml (center point of the microphone array MA) is shown as P1. Furthermore, in Fig. 4, the direction of microphone Mr from P1 is set to 0°, and the direction of microphone Ml from P1 is set to 180°.

[0039] In the sound collection device 100 of this embodiment, sound collection processing is performed on the assumption that the target sound source and the interfering sound source are present in any direction (angle) of 0° to 180° as viewed from P1. Hereinafter, the direction in which the apparent sound source and the interfering sound source (sound source of non-target area sound) are present from P1 will also be referred to as the "arrival angle" or "arrival direction." In addition, below, the direction at 90° as viewed from P1 will be referred to as the front direction.

[0040] Next, an overview of the target area sound extraction process performed by the target area sound extraction section 102 will be described.

[0041] Here, first, a conventional multiple constraint MVDR beamformer will be described. As described above, as a multiple constraint MVDR beamformer, there are technologies described in, for example, Non-Patent Documents 3 and 4. The conventional multiple constraint MVDR beamformer is a method that enables robust emphasis against slight movement of a target sound source by giving a beam width by multiple constraint conditions (constraint conditions that make the output for signals arriving from multiple directions distortion-free), but the strong constraint conditions that impose distortion-free constraints on multiple directions greatly reduce the degree of freedom, and the target sound emphasis performance (noise suppression performance) is low, which is an issue.

[0042] On the other hand, in conventional MVDR beamformers (e.g., multiple constraint MVDR beamformers and localization MVDR beamformers), there is a trade-off between the strength of the constraint conditions and the target sound enhancement performance (noise suppression performance), and it is known that the target sound enhancement performance (noise suppression performance) can be significantly improved by allowing distortion of the target sound source.

[0043] Therefore, in the target area sound extraction unit 102 of this embodiment, a method (hereinafter also referred to as an "RMC-MV (Relaxed Multiple-Constraint MV) beamformer" or "distortion-tolerant multiple constraint minimum variance beamformer") is adopted in which the constraint conditions in a conventional MVDR beamformer (e.g., a multiple constraint MVDR beamformer) are replaced with parameters that allow distortion of the target area sound (target sound) in order to improve the target sound emphasis performance (noise suppression performance).

[0044] For example, the RMC-MV beamformer can be realized by replacing the constraint conditions in a conventional multiple constraint MVDR beamformer with a "distortion tolerance" (hereinafter referred to as "E") that allows distortion of the target area sound (target sound).

[0045] For example, the filter coefficients applied to the RMC-MV beamformer (hereafter referred to as w RMCMV The optimal MVDR beamformer (expressed as "Eq. (7)") can be obtained by solving the optimization problem (constrained optimization problem) of the following equations (5) and (6). In equations (5) and (6), the constraint condition part of the optimization problem in the conventional multiple constraint MVDR beamformer is replaced with the distortion tolerance E. Then, by solving the optimization problem of equations (5) and (6), a solution (optimal solution) such as equation (7) can be obtained.

[0046]

number

[0047] As described above, the target area sound extraction unit 102 extracts the emphasis signal ^y from the observation signal by using the RMC-MV beamformer described above. f is extracted. Also, in the target area sound extraction unit 102 of this embodiment, the above-mentioned RMC-MV beamformer is optimized using DNN.

[0048] FIG. 1 is a block diagram showing an example of the internal configuration of the target area sound extraction unit 102. As shown in FIG.

[0049] When the above-mentioned RMC-MV beamformer is realized using a DNN, the internal configuration of the target area sound extraction unit 102 is, for example, as shown in FIG.

[0050] As shown in FIG. 1, the target area sound extraction unit 102 includes a difference filter 201, a deep neural network (DNN) 202, a beam former (BF) 203, and a difference extractor 204.

[0051] Difference filter 201 is x (r) f , x (l) f Difference d f The difference d f can be calculated by the above formula (3). Assuming that the target sound source (target area) is located approximately in front of the microphone array MA (position P1), the output of the differential filter 201, d f In this case, non-target area sounds (interference sounds from non-target areas) are the dominant signal.

[0052] The DNN 202 uses the DNN to generate a signal d f and the observed signal x (l) f (x (r) fThe DNN 202 receives the signal d, and performs a process of estimating a time-frequency mask (hereinafter referred to as "mask") that separates the target area sound and the non-target area sound (interference sound; noise) while taking into account the spatial model shown in FIG. f and the observed signal x (l) f The above is used as input to perform processing to estimate a time-frequency mask (hereinafter referred to as "mask") used in RMC-MV beamformer processing (hereinafter referred to as "mask estimation processing").

[0053] The BF203 uses the spatial correlation matrix (SCM) of noise (non-target area sound; interference sound) obtained from the mask and a steering vector given in advance to calculate the filter coefficients w RMCMV The BF203 obtains the filter coefficient w RMCMV Based on the observed signal (x (r) f or x (l) f ) to emphasize the target area sound. f can be extracted.

[0054] The difference extractor 204 extracts the enhancement signal ^y output from the BF 203. f (The result of estimating the target area sound) and the clean target area sound (hereafter referred to as "y f ") and feeds back this difference (or a parameter based on the difference) to the DNN 202 as a loss (loss in machine learning). In other words, the difference extractor 204 is an element that functions only when the DNN 202 is made to perform learning processing. Therefore, if new learning processing is not already being performed by the DNN 202, the difference extractor 204 may be excluded from the sound collection device 100.

[0055] As for the specific processing of the difference extractor 204, for example, the processing based on the following Reference 1 can be applied, so a detailed explanation will be omitted.

[0056] Reference 1: J. Heymann, K. Drude, and R. Haeb-Umbach, “Neural network-based spectral mask estimation for acoustic beamforming,” in Proc. IEEE ICASSP, 2016, pp. 196-200.

[0057] In the sound collection device 100 of this embodiment, there is an operation mode in which the target area sound extraction unit 102 executes a learning process (hereinafter referred to as a "learning process mode"), and a mode in which the target area sound extraction unit 102 executes a learning process. (l) f (x (r) f Based on this, the enhancement signal ^y f The sound collection device 100 is assumed to be compatible with both an operation mode in which the sound collection device 100 extracts and outputs the signal from the input signal (hereinafter referred to as a "signal processing mode"). Note that the sound collection device 100 may be configured not to support the learning processing mode (for example, a learning model is already acquired or a learning model is acquired from the outside).

[0058] When the sound collection device 100 operates in the learning processing mode, the target area sound extraction unit 102 receives an observed signal (x (l) f (x (r) f ) and the clean signal y as the teacher label f A data set including the above (hereinafter referred to as a "teacher data set") is supplied to the target area sound extraction unit 102 to execute the learning process. At this time, the difference extractor 230 extracts the processing result based on the teacher data (the emphasis signal ^y f ) and the teacher label (clean signal y f ) is obtained and fed back to the DNN 202. This allows the DNN 202 to obtain a learning model that has been trained (deep learning) based on the teacher dataset.

[0059] When the DNN 202 operates in the learning mode, it learns (global optimization) based on the loss supplied from the difference extractor 204, and updates each parameter (parameters of each layer of the DNN) so that the mask estimation process takes into account the spatial model (the model shown in Figure 4).

[0060] Next, an example of the internal configuration of the DNN 202 will be described.

[0061] 5 and 6 are diagrams showing an example of the internal configuration of the DNN202. FIG. 5 is a block diagram showing an example of the architecture of the DNN202 using multiple fully connected layers (hereinafter, also referred to as the "first architecture"). FIG. 6 is a block diagram showing an example of the architecture of the DNN202 to which a U-net type architecture is applied (hereinafter, referred to as the "second architecture"). In FIG. 5 and FIG. 6, the DNN is illustrated in units of blocks consisting of one or more layers (functional layers constituting the DNN). In FIG. 5 and FIG. 6, the layers constituting each block are marked with symbols indicating the function of each layer. In Figures 5 and 6, each layer is labeled with one of the following symbols: "FC" indicating a fully connected layer, "Conv" indicating a 2D convolution layer, "UP-Conv" indicating an upsampling (2D deconvolution) layer, "BN" indicating a batch normalization layer, "Pool" indicating a max pooling layer, "ReLu" indicating a ReLU (Rectified Linear Unit) activation function layer, and "Sigmoid" indicating a sigmoid activation function layer. In the following, each layer will be referred to by its symbol. In Figures 5 and 6, the description in parentheses following Conv indicates the size (matrix size) of the filter (window) used for convolution. For example, "Conv(3×3)" indicates that convolution is performed using a 3-row × 3-column filter (matrix). 5 and 6, the description in parentheses following Pool indicates the size (matrix size) of the filter (window) used in the pooling process. Furthermore, in Fig. 5 and 6, the description in parentheses following UP-Conv indicates the size (matrix size) of the filter (window) used in the upsampling (deconvolution) process.

[0062] Next, the configuration of the DNN 202 according to the first architecture will be described with reference to FIG.

[0063] In the DNN202 of the first architecture, "blocks B111, B112", "blocks B121, B122", "block B131", "block B141", and "block B151" are arranged in order from the input side.

[0064] Blocks B111, B112, B121, B122, B131, and B141 are each configured with layers of FC, BN, and ReLu connected in series from the input side, while block B151 is configured with layers of FC, BN, and Sigmoid connected in series from the input side.

[0065] In the first architecture DNN202, input blocks B111 and B112 are each assigned x (r) f , d f In the first architecture, DNN202, x (r) f , d f After transformation is performed on each of the above by blocks B111, B112, B121, and B122, they are concatenated (CONCAT) and input to block B131. Furthermore, in the DNN202 shown in FIG. 5, transformation is performed by blocks B141 and B151 and the result is output as a mask.

[0066] Next, the configuration of the DNN 202 according to the second architecture will be described with reference to FIG. 6.

[0067] As shown in Fig. 6, the second architecture DNN202 includes a block (hereinafter referred to as "downsampling block") DB1 (DB1 to DBL) that performs L-stage (L is an integer equal to or greater than 1) downsampling processing, a block (hereinafter referred to as "upsampling block") UB (UB1 to UBL) that performs L-stage upsampling processing, blocks B201 and B202 that are arranged between the downsampling block DBL and the upsampling block UBL, and a block B203 that is located after the upsampling block UBL. In the second architecture DNN202, when viewed from the input side, the downsampling blocks DB1, DB2, ... DB(L-1), DBL, blocks B201, B202, upsampling blocks UBL, UB(L-1), ... , UB2, UB1, and block B203 are connected in series in this order.

[0068] Any number can be applied as the number of upsampling / downsampling stages L, but in this embodiment, the following description will be given assuming that L=4.

[0069] Each downsampling block DB (DB1~DBL) has x (r) f A sub-block SB11 processes the input line, and d f A sub-block SB12 that processes the input line is provided.

[0070] The sub-blocks SB11 and SB12 have the same configuration. As shown in Fig. 6, each of the sub-blocks SB11 and SB12 has a configuration in which, from the input side, the layers Conv(3x3), Conv(3x3), BN, ReLu, and Pool(2x2) are connected in series.

[0071] Block B201 contains x (r) f A sub-block SB21 processes the input line, and d fThe sub-block SB21 and the sub-block SB22 are arranged to process the input line. The sub-blocks SB21 and SB22 are each configured with the layers Conv(3×3), Conv(3×3), BN, and ReLu connected in series from the input side.

[0072] The outputs from each of the sub-blocks SB21 and SB22 of the block B201 are concatenated (CONCAT) and supplied to the block B202.

[0073] Block B202 has a configuration in which FC, BN, and Sigmoid layers are connected in series, in that order from the input side.

[0074] The upsampling blocks (UB1 to UBL) each have a configuration in which, from the input side, the Conv(2×2), UP-Conv(2×2), Conv(3×3), Conv(3×3), BN, and ReLu layers are connected in series.

[0075] As shown in Figure 6, in the upsampling block UBi in the i-th stage (i is an integer from 1 to L), the output of UP-Conv(2x2), the output of ReLu of sub-block SB11 in the downsampling block DBi, and the output of ReLu of sub-block SB12 in the downsampling block DBi are concatenated (CONCAT) and supplied to the first layer CONV(3x3).

[0076] Block B203 has a configuration in which, from the input side, Conv(1×1) and Sigmoid layers are connected in series.

[0077] (A-2) Operation of the First Embodiment Next, the operation of the sound collection device 100 in this embodiment having the above-mentioned configuration (the sound collection method according to the embodiment) will be described.

[0078] First, a process performed when the target area sound extraction unit 102 of the sound collection device 100 operates in the learning process mode will be described.

[0079] When a training data set is supplied to the target area sound extraction unit 102 operating in the learning processing mode, first, the training data of the training data set (observation signal x (l) f , x (r) f ) to get the difference d f is obtained and fed to DNN202.

[0080] In DNN202, the difference d f and the observed signal x (l) f Based on this, a process of estimating the mask is performed.

[0081] In BF203, the mask and the observed signal x (l) f Based on the enhancement signal ^y f is obtained and provided to the difference extractor 204.

[0082] In the difference extractor 204, the enhancement signal ^y f and the clean signal y as the teacher label f The difference between these is obtained as loss and fed back to DNN202.

[0083] In the DNN 202, a learning process (an update process of a learning model for estimating a mask with less loss using a neural network) is performed based on the loss.

[0084] Next, the operation of the target area sound extraction unit 102 of the sound collection device 100 when it operates in the signal processing mode will be described.

[0085] Here, an observation signal (x (l) f , x (r) f ) shall be deemed to have been supplied.

[0086] When a teacher data set is supplied to the target area sound extraction unit 102 operating in a signal processing mode, first, a difference filter 201 extracts an observed signal (x (l) f , x (r) f ) to get the difference d f is obtained and fed to DNN202.

[0087] In DNN202, the difference d f and the observed signal x (l) f Based on this, a process of estimating a mask is performed based on the retained learning model.

[0088] In BF203, the mask and the observed signal x (l) f Based on the enhancement signal ^y f is acquired and supplied to the signal output unit 103.

[0089] The signal output unit 103 outputs the emphasis signal ^y f is converted from the frequency domain to the time domain and output.

[0090] Next, the inventor constructed a sound collection device 100 in a simulation environment and collected the target area sound (emphasis signal ^y f The following describes the experimental results and evaluation results of an experiment (hereinafter referred to as "this experiment") to evaluate the quality (degree of robustness against movement of the sound source position) of the process to obtain the characteristic of the sound source.

[0091] In this experiment, the performance (target area sound enhancement performance; noise suppression performance; sound source separation accuracy) of target area sound collection processing using the sound collection device 100 (target area sound extraction unit 102) of this embodiment (i.e., sound collection processing using an "RMC-MV beamformer") will be compared with that of sound collection processing using a conventional MC-MVDR beamformer (multiple constrained MVDR beamformer).

[0092] FIG. 7 shows the environment of this experiment.

[0093] FIG. 7 shows an example in which a target sound source and an interfering sound source are arranged on a model similar to that in FIG.

[0094] As shown in FIG. 7, in this experiment, the positions of the target sound source and the interfering sound source (sound source in a non-target area) are on a semicircular line at a distance of 1 m from P1.

[0095] In this experiment, the TIMIT corpus (see Reference 2 below) was used as the dry source (signal) of the target sound source in both the learning processing mode (training) and the signal processing mode (signal processing). In addition, in this experiment, the TMIT corpus was used as the dry source signal of the interference sound (non-target area sound) in both the learning processing mode (training) and the signal processing mode (signal processing), and the DEMAND (Diverse Environments Multi-channel Acoustic Noise Database) corpus (see Reference 3 below) was used as the signal equivalent to the non-stationary noise. In this experiment, when the TMIT corpus was used as the dry source signal of the interference sound, an utterance different from the target sound source was selected. In addition, in this experiment, when the DEMAND corpus was applied as the non-stationary noise, one of NRIVER, NPARK, DLIVING, OOFFICE, and OMEETING was applied from the DEMAND corpus.

[0096] Reference 2: JS Garofolo, LF Lamel, WM Fisher, JGFiscus, DS Pallett, NL Dahlgren, V. Zue, “TIMIT acoustic phonetic continuous speech corpus,” Linguistic Data Consotrium, 1992. Reference 3: J. Thiemann, N. Ito, and E. Vincent, “The diverseenvironments multi-channel acoustic noise database(DEMAND): A database of multichannel environmental noise recordings”, The Journal of the Acoustical Society of America, vol. 133, p. 3591,05, 2013.

[0097] In this experiment, observation signals (acoustic signals) captured by microphones Ml and Mr in a sound field (model environment) such as that shown in Figure 7 were obtained by computer simulation, and the obtained observation signals were then input to the sound collection device 100 and the results were evaluated.

[0098] Specifically, in this experiment, we used PyRoomAcoustics (see Reference 4 below) to perform a simulation with a sound field (spatial model) as shown in Figure 7 to obtain an impulse response, and convolved the obtained impulse response with the above dry sources (the dry sources of the target sound source and the interfering sound source) to obtain the observed signals x (l) f , x (r) f obtained.

[0099] Reference 4: Scheibler, E. Bezzam, I. Dokmani´c, “Pyroomacoustics: A Python package for audio room simulations and array processing algorithms”, Proc. IEEE ICASSP, 2018

[0100] In addition, in the simulation of this experiment, the observed signal x (l) f , x (r) fThe SNR at is adjusted to be approximately 0.0 dB. In the following, the 3D space of the sound field used in this experiment is represented by a three-dimensional coordinate system of (x, y, z).

[0101] In the simulation of this experiment, the size of the room that constitutes the sound field (spatial model) was set to (x,y,z) [m] = (5,3,3), and the coordinates of the 2ch microphones Ml and Mr were set to (x,y,z) [m] = (2.485,1.5,1) and (x,y,z) [m] = (2.515,1.5,1), respectively. This results in a distance of 3 [cm] between microphones Ml and Mr. In the simulation of this experiment, the sound absorption coefficient of the room was set to 0.2, and the number of reflections in the room was set to 3.

[0102] Next, detailed parameter settings in the simulation of this experiment will be described.

[0103] In this experiment, the conventional multiple constraint MVDR beamformer was constrained so that the output for signals arriving from two directions, 80° and 100°, would be distortion-free. In addition, the RMC-MV beamformer imposes constraints on the output for signals arriving from the same two directions, but rather than trying to make the output completely distortion-free, the constraints were relaxed to allow distortion. In this experiment, the directions (angles) for these constraints were determined based on prior experiments using oracle masks to determine the most suitable directions (angles).

[0104] In this experiment, we performed simulations (learning) using multiple values ​​in the RMC-MV beamformer to experimentally determine the optimal value of the hyperparameter λ. Specifically, in this experiment, λ = 1.0 × 10 6 In this experiment, a fixed value was applied to the value of λ in both the learning processing mode and the signal processing mode.

[0105] In addition, in this experiment, the batch size was set to 16 and the number of epochs to 100 for training the DNN model, and the mean squared error between the enhanced signal and the ground truth signal was used as the loss function. In this experiment, Adam (see Reference 5 below) was used as the optimization algorithm for the DNN model, and the learning rate was set to 0.001.

[0106] Reference 5: D. Kingma, and J. Ba, “Adam: A method for stochastic optimization”, International Conference on Learning Representations (ICLR), 2015.

[0107] FIG. 8 shows the results of this experiment.

[0108] Figure 8 shows a table comparing the unprocessed microphone observation signal (hereinafter also referred to as "Observation") in this experiment, the multiple constrained MVDR beamformer output sound (MC-MVDR), and the output sound of the sound collection device 100 (RMC-MV beamformer output sound).

[0109] In FIG. 8, the signal-to-distortion ratio (SDR) and short-time objective intelligibility (STOI), which are evaluation measures of sound quality, are used as measures for evaluating the speech enhancement performance of the beamformer.

[0110] In Figure 8, the SDR and STOI values ​​themselves are shown for Observation (unprocessed observed signal). Also, in Figure 8, for the voice of two types of beamformers (MC-MVDR, RMC-MV), the improvement value SDRi (SDR improvement) for the Observation SDR and the improvement value STOIi (STOI improvement) for the Observation STOI are shown.

[0111] In this experiment, the target sound source was positioned in one of three directions, 80°, 90°, and 100°, in the learning processing mode of two types of beamformers (MC-MVDR, RMC-MV), and in the signal processing mode, the target sound source was positioned in one of 1° increments within the range of 80° to 90°. In addition, in this experiment, the interference sound sources were randomly placed in one to three locations out of a total of eight locations, 0°, 15°, 30°, 45°, 135°, 150°, 165°, and 180°, in both the learning processing mode and the signal processing mode. In this experiment, such changes in the positions of the target sound source and interference sound source (non-target area sound source) were performed in data processing units (e.g., word units) on the corpus.

[0112] In addition, in the signal processing mode of this experiment, the target sound source was placed at the same position as in the learning processing mode (either 80°, 90°, or 100°) (hereinafter referred to as the "first condition" or "80, 90, 100 matched"), and the target sound source was placed at a position slightly shifted from the learning processing mode in any direction in 1° increments within the range of 80° to 90° (hereinafter referred to as the "second condition" or "80to100 mismatched"). Figure 8 shows the evaluation results (SDRi / STOIi) for each of the first target sound source position (80, 90, 100 matched) and the second target sound source position (80to100 mismatched).

[0113] Here, the RMC-MV beamformer (performance of the sound collection device 100 of this embodiment (speech enhancement performance of target area sound)) will be considered based on the results of this experiment shown in Fig. 8. Specifically, based on the results of the MC-MVDR beamformer and the RMC-MV beamformer, the effect of relaxing the distortion-free constraint in a conventional MC-MVDR beamformer (increasing the freedom of the beamformer by allowing distortion) (i.e., the effect of the sound collection device 100 of this embodiment compared to the conventional MC-MVDR beamformer) will be considered.

[0114] As shown in Figure 8, in this experiment, the performance of the RMC-MV beamformer was significantly higher than that of the conventional MC-MVDR beamformer, demonstrating the effectiveness of relaxing the distortion-free constraint in the MVDR beamformer on the target sound enhancement performance.

[0115] In addition, the results of this experiment show that for the RMC-MV beamformer, there is almost no change in speech enhancement performance between the first condition (80,90,100 matched) and the second condition (80 to 100 mismatched) (approximately 3% for SDRi and 5% for STOIi). Therefore, this experiment clarifies that the RMC-MV beamformer is robust against the movement of the target sound source.

[0116] That is, it can be seen that the RMC-MV beamformer improves the target sound emphasis performance while maintaining the same degree of robustness against the movement of the target sound source compared to the conventional MC-MVDR beamformer. As described above, in the conventional MVDR beamformer, there is a trade-off between the robustness against the movement of the target sound source and the target sound emphasis performance (noise suppression performance), but it can be seen that the RMC-MV beamformer improves the target sound emphasis performance without causing such a trade-off compared to the conventional MC-MVDR beamformer.

[0117] (A-3) Effects of the embodiment According to this embodiment, the following effects are obtained.

[0118] As described above, in the sound collection device 100 according to this embodiment, the target sound emphasis performance can be improved without causing a trade-off between the robustness of the target sound source's movement and the target sound emphasis performance (noise suppression performance) compared to the conventional MVDR beamformer. In other words, in the sound collection device 100 according to this embodiment, the target sound emphasis performance can be improved without causing a trade-off between the robustness of the target sound source's movement and the target sound emphasis performance (noise suppression performance) by relaxing the distortion-free constraint in the conventional MC-MVDR beamformer (increasing the degree of freedom of the beamformer by allowing distortion).

[0119] (B) Other embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments such as those exemplified below.

[0120] (B-1) In the above embodiment, the sound collection device 100 has been described as supporting both the learning processing mode and the signal processing mode (test mode). However, if a learning model is stored in advance, the sound collection device 100 may be configured to support only the signal processing mode and to exclude the means (learning means) required for the learning processing mode. [Explanation of symbols]

[0121] 100...sound collection device, 101...signal input section, 102...target area sound extraction section, 103...signal output section, 201...difference filter, 202...DNN, 203...BF, 204...difference extractor, MA...microphone array, Ml, Mr...microphone.

Claims

1. a non-target area sound dominant signal acquiring means for acquiring a non-target area sound dominant signal in which non-target area sounds other than the target area sound having the target area as a sound source are dominant from input signals from a plurality of microphones; a mask estimation means for estimating a mask based on the input signal of any one of the microphones and the non-target area sound dominant signal, to obtain a target area sound enhanced signal in which the target area sound is enhanced from the input signal; Extraction means for extracting the target area sound emphasis signal from the input signal using the mask, The extraction means performs the extraction process using an MVDR beamformer whose constraint condition is a distortion tolerance that allows distortion of the target area sound. A sound collecting device characterized by the above.

2. 2. The sound pickup device according to claim 1, wherein the mask estimation means is configured using a neural network.

3. Computer, a non-target area sound dominant signal acquiring means for acquiring a non-target area sound dominant signal in which non-target area sounds other than the target area sound having the target area as a sound source are dominant from input signals from a plurality of microphones; a mask estimation means for estimating a mask based on the input signal of any one of the microphones and the non-target area sound dominant signal, to obtain a target area sound enhanced signal in which the target area sound is enhanced from the input signal; The mask is used to extract the target area sound emphasis signal from the input signal. The extraction means performs the extraction process using an MVDR beamformer whose constraint condition is a distortion tolerance that allows distortion of the target area sound. A sound recording program characterized by:

4. In the sound collection method performed by the sound collection device, The non-target area sound dominant signal acquisition means includes a mask estimation means and an extraction means. The non-target area sound dominant signal acquisition means acquires a non-target area sound dominant signal in which non-target area sounds other than the target area sound having a sound source in the target area are dominant from input signals from a plurality of microphones, The mask estimation means estimates a mask for acquiring a target area sound emphasis signal in which the target area sound is emphasised from the input signal based on the input signal of any one of the microphones and the non-target area sound dominant signal, The extraction means performs an extraction process to extract the target area sound emphasis signal from the input signal using the mask, The extraction means performs the extraction process using an MVDR beamformer whose constraint condition is a distortion tolerance that allows distortion of the target area sound. A sound collection method comprising:

Citation Information

Patent Citations

  • Sound pickup device, program and method

    JP2016127457A