Method and device for detecting or estimating a target sound source among multiple sound sources

Abstract

The present application discloses a method, an apparatus and a computer program for detecting or estimating a target sound source among a plurality of sound sources, wherein the target sound source is located in at least one target spatial region among a plurality of spatial regions, and the method comprises: a processor receives a plurality of signals, each of the plurality of signals is associated with one of a plurality of microphone signals, and the microphone signal comprises a sound event generated by the plurality of sound sources; the processor extracts a plurality of features through a neural network, wherein the plurality of features are extracted from the plurality of signals, and each of the plurality of features is obtained by training the neural network for different spatial regions among the plurality of spatial regions; the processor generates another plurality of features corresponding to the at least one target spatial region based on the extracted plurality of features; and the processor detects or estimates at least one sound source among the target sound sources in the target spatial region based on the another plurality of features.

Description

[0001] This application claims the priority of the European patent application filed with the European Patent Office on October 21, 2021, with application number EP21204004.2, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to a neural network-based audio processing system, which is used to estimate or detect signals generated by sound sources located in a defined, continuous spatial area by using spatially correlated feature extraction technology, a method for estimating or detecting signals generated by sound sources located in a defined, continuous spatial area by using spatially correlated feature extraction technology, and a program executed by a processor and including instructions for implementing the method. Background Art

[0003] The problem of audio extraction is usually solved using techniques such as single-microphone steady-state noise suppression or multi-microphone setups to remove the unwanted signal from the target signal. The extracted desired signal is used for speech enhancement, speech recognition, audio transcription, and many other audio-based applications.

[0004] Signals that exhibit high sparsity in the short-time Fourier transform (STFT) domain, such as speech or music, can be effectively separated with the help of a time-frequency (TF) mask that identifies the STFT points (bins) where the target signal dominates. To obtain such a mask, deep learning methods are usually used. TF masks are a known tool in audio processing for speech enhancement and localization, which allows supervised learning. The TF mask includes information about the activity of the target source at each time and frequency in the short-time Fourier transform (STFT) domain. Typically, each element of the TF mask is a number between 0 and 1. The TF mask can also be a complex number. The TF mask can be used to restore the corresponding clean source signal of the target source, for example, by element-by-element multiplication of the mask and the reference microphone signal, followed by an inverse transform. As explained, the TF mask identifies the short-time Fourier transform (STFT) points (bins) dominated by the signal of interest and can be obtained by exploiting spectral temporal correlation and spatial information. This can be exploited using deep learning methods. Besides estimating the TF mask, other known approaches are estimating the STFT bins of the signal of interest, estimating the time domain signal of interest or estimating the multi-channel beamformer weights applied to multi-channel microphone recordings.

[0005] In order to apply deep neural networks (DNNs) to this task, the selection of discriminative features that can identify the target component is a key aspect. For single microphone capture, it is possible to separate, for example, speech and background noise by exploiting typical structures in the spectro-temporal properties of speech. However, this approach fails when the target and the distractor are of the same type (e.g., separating two different speakers). When multiple microphone captures are available, spatial information can be additionally exploited to distinguish sources with similar spectro-temporal properties.

[0006] Regardless of which information is used, it is crucial to clearly define the target classification so that the DNN can learn to correctly suppress the unwanted components while preserving the target. For example, for the separation of speech from background noise, the distinction can be made based on spectro-temporal (speech / non-speech) and / or spatial (local / diffuse) signal properties. These discriminative properties can be incorporated during the training of the neural network.

[0007] For the separation of multiple local sound sources, the sources can be distinguished based on their locations, which can be described, for example, in terms of the corresponding directions of arrival (DOA). However, since any source (e.g., one of multiple talkers speaking at the same time) can be considered as the target, user input is required to resolve this ambiguity in the target viewing direction. A common approach is to train the network to extract signal components from various directions. The desired signal can then be obtained by selecting a combination of one or more of the multiple output streams (e.g., all streams in the target tracking beam). In this approach, the optimal results of all streams are generated without prior knowledge of the correlation of each stream, and then only a small part of these streams is used to reconstruct the target signal, i.e., the DNN output contains a lot of redundancy. Therefore, this approach does not effectively use the available computing resources. Another approach is to train the neural network for fixed, a priori known spatial angles. However, this approach cannot select angles or spaces at runtime and is therefore not flexible. A single angle is also usually considered.

[0008] "Mask-based Source Separation Using Neural Networks with Full-band and Sub-band Spatial Features" by A. Bohlender, A. Spriet, W. Tirry, and N. Madhu, EUSIPCO 2021, discloses the use of a deep neural network architecture (composed of convolutional (CNN) and recurrent layers) to estimate TF masks, such as Figure 1 shown. Figure 1The input to the CNN shown consists of the phase spectrograms of the N microphone signals. Since the masks for all directions of arrival (DOAs) are estimated simultaneously, the output size of each TF point (bin) is equal to I for a discrete grid with a total of I DOAs. Therefore, each of the I TF masks returned corresponds to a specific direction. To extract a single target, only the mask corresponding to the target source DOA is required, and all other estimated masks can be discarded. Therefore, the computational power of this method is not efficiently utilized because TF masks are calculated that will not be used. In addition, the performance may be less than ideal because all angles need to arrive at the same time for minimum performance.

[0009] “Online Multichannel Speech Enhancement with Convolutional Recurrent Neural Networks Based on Time-Frequency Masking” by S. Chakrabarty and EAP Habets, IEEE Special Topics in Signal Processing, Vol. 13, No. 4, pp. 787-799, August 2019, DOI: 10.1109 / JSTSP, discloses a method similar to Figure 1 A similar CNN architecture for mask estimation is shown in Fig. 2. However, only a single TF mask is returned because this method only considers separating a single local speech source from scattered and uncorrelated background noise. In this case, it is sufficient to distinguish only local and non-local signal components, and the specific location of the target source does not play a role. However, this method is not suitable for separating multiple local sources.

[0010] "Eigenvector-based speech mask estimation for multi-channel speech enhancement" by L. Pfeifenberger, M. Zeller and F. Pernkopf, IEEE / ACM Audio, Acoustic, and Speech Signal Processing, Vol. 27, No. 12, pp. 2162-2172, December 2019, DOI: 10.1109 / TASLP.2019.2941592, disclose using a neural network architecture for TF mask estimation, where the output of the neural network includes a total of three masks corresponding to the target source, the interference source, and the background noise. By training the network on well-defined scenarios, the ambiguity about which source is considered the target can be resolved. Figure 2 An example of a possible training scenario is shown, where the location of one source is confined to a specific area D1, while the location of another source is confined to a different area D2. However, this approach is not universal, as the target DOA range must be specified in advance and cannot be controlled by the user after the neural network is trained, as it makes assumptions about the location and movement of the sources to explicitly define which source is the target and which is the interferer.

[0011] "Distant Speech Separation Using Time-Frequency Masks Predicted from Spatial Features" by P. Pertile and J. Nikunen, Speech Communications 68 (2015), pp. 97–106, disclose a method in which the problem of separating multiple sources based on their DOAs is specifically considered. A simple feed-forward neural network is employed. The input features are based on the phase spectrogram of the microphone signals as well as the DOAs. In order to obtain a DOA representation suitable for use with the microphone signal phases, the “theoretical” inter-channel phase differences resulting from ideal plane wave propagation along the specified DOAs are first calculated. Then, as input to the neural network, scores (per TF bin) are determined for the observed and non-observed directions. The method can be extended to allow a target range of DOAs to be specified by averaging or aggregating the input scores for several directions. Alternatively, the aggregation can be based on the independently obtained outputs of multiple DOAs. However, the latter is not efficient due to its computational complexity, while the former approach is also suboptimal, as the neural network must reach a compromise to provide suitable performance for all possible spatial angles of interest.

[0012] “Multi-channel Overlapping Speech Recognition with Position-guided Speech Extraction Network” by Z. Chen, X. Xiao, T. Yoshioka, H. Erdogan, J. Li and Y. Gong, 2018 IEEE Symposium on Spoken Language Technologies (SLT), pp. 558-565, DOI: 10.1109 / SLT.2018.8639593, and “End-to-end Multi-channel Target Speech Separation with Direction Information” by R. Gu and Y. Zou, arXiv preprint arXiv:2001.00391, 2020, both disclose incorporating the definition of the target into the input features, which lacks generality and reduces efficiency. Three different types of input features are used, namely spatial features (inter-channel phase differences), spectral features (beamformed microphone signals) and so-called angular features (cosine distance between microphone signal spectrum and steering vector), two of which are (partially) based on the position of the target speaker. The viewing direction is added to the spectral and angular features to define the target. The so-called angular feature is a score that quantifies how well the microphone signal corresponds to the specified viewing direction. Furthermore, the beamformer is steered in the target direction in order to obtain a spectral feature specific to this direction. Therefore, these methods are also computationally complex. Moreover, since they have to work for user-defined angles, the performance for the user-defined angles needs to be compromised to achieve good performance at all considered angles. Summary of the invention

[0013] The present application provides a method and device for detecting or estimating a target sound source among multiple sound sources.

[0014] In a first aspect, the present application provides a method for detecting or estimating a target sound source among multiple sound sources, wherein the target sound source is located in at least one target spatial region among multiple spatial regions, the method comprising: a processor receives multiple signals, wherein each of the multiple signals is associated with one of a plurality of microphone signals, wherein the microphone signal comprises sound events generated by the multiple sound sources; the processor extracts multiple features through a neural network, wherein the multiple features are extracted from the multiple signals, and wherein each of the multiple features is obtained by training the neural network for different spatial regions among the multiple spatial regions; the processor generates another multiple features based on the extracted multiple features, wherein the another multiple features correspond to the at least one target spatial region; the processor detects or estimates at least one of the target sound sources in the target spatial region based on the another multiple features corresponding to the at least one target spatial region.

[0015] A plurality of additional features may be generated based on features corresponding to the target spatial region and features corresponding to other non-target spatial regions. The target sound sources may all be located in a single target spatial region or distributed in a plurality of spatial target regions. The neural network may have been trained for each of the plurality of spatial regions. The neural network may include a spatial region correlation layer for each of the plurality of spatial regions, and the spatial region correlation layer of the neural network may be trained for each spatial region. At least one target spatial region may be set by a user at runtime through a user interface or in any other suitable manner. Alternatively, several target spatial regions may be set at runtime. The plurality of spatial regions for training the spatial correlation layer may cover a wider space than the target spatial region. In fact, the plurality of spatial regions may cover all possible spatial regions that the user may set. As an example, the plurality of spatial regions may correspond to a grid covering a DOA region of 0 to 360 degrees, such that, for example, the plurality of spatial regions include a first spatial region covering a region including an angle from 0 to 10 degrees, a second spatial region covering a region including an angle from 10 to 20 degrees, a third spatial region covering a region including an angle from 20 to 30 degrees, and so on.

[0016] The plurality of spatial regions may include a plurality of directions of arrival. Alternatively, the spatial region may include a three-dimensional volume. For example, the three-dimensional volume may be defined as a plurality of points, where each point is defined by a set of three-dimensional coordinates in a Cartesian coordinate system, an elliptical coordinate system, or a spherical coordinate system.

[0017] In an embodiment of the present application, the multiple input signals of the neural network may include multiple phases of the time-frequency representation of the microphone signal, and / or multiple normalized amplitudes of the time-frequency representation of the microphone time domain signal, wherein the normalization is based on the norm of at least one of the multiple time-frequency representations of the microphone time domain signal and / or the multiple complex time-frequency representations of the microphone signal.

[0018] In another embodiment of the present application, the plurality of features are extracted from the plurality of signals by at least one of a plurality of convolutional neural network layers.

[0019] In one embodiment of the present application, the method may include: the processor estimating a time-frequency TF mask based on the additional multiple features, and detecting or estimating the at least one sound source in the target sound source in the at least one target spatial region based on the TF mask. The TF mask may have a complex or real value. The TF mask may be applied to a single channel or multiple channels.

[0020] In another embodiment of the present application, the method may include: the processor estimating a time domain representation of the at least one sound source based on the further plurality of features, and detecting or estimating the at least one sound source based on the time domain representation.

[0021] In another embodiment, the method may include the processor estimating a time-frequency representation of the at least one sound source based on the further plurality of features, and detecting or estimating the at least one sound source based on the time-frequency representation.

[0022] In another embodiment, the method may include: the processor estimating beamforming weights (e.g., multi-channel complex masks or time domain filter coefficients) based on the additional plurality of features, and detecting or estimating the at least one sound source based on the beamforming weights. The beamforming weights may be applied to a multi-channel signal.

[0023] Another plurality of features may be generated based on a weighted combination of the extracted plurality of features. That is, each of the other plurality of features may be obtained based on any suitable linear combination of the extracted plurality of features.

[0024] In another embodiment, the additional plurality of features are generated by weighting the plurality of extracted features and taking the maximum value of the weighted plurality of extracted features.

[0025] In another embodiment, the weighted combination may be generated based on the average value of the values ​​of the plurality of features extracted corresponding to the at least one target area. That is, the additional plurality of features may be obtained by calculating the average value of all values ​​of the plurality of features extracted corresponding to the at least one target area.

[0026] Another aspect of the present application relates to a device for detecting or estimating a target sound source among multiple sound sources, wherein the target sound source is located in at least one target spatial area among multiple spatial areas, the device comprising a memory and a processor communicatively connected to the memory, and configured to execute instructions to perform the above method.

[0027] Another aspect of the present application relates to a computer program for executing the above method.

[0028] By providing spatial, e.g., DOA, related feature extraction and user-controllable spatial (e.g., DOA) selection that can be integrated into a DNN or neural network, there is no need to change the input and output of the DNN to accommodate spatial (e.g., DOA) dependencies.

[0029] Target sources are defined by providing a target area range (at runtime) that can be defined by a set of target DOAs or spatial regions, providing versatility and scalability while still allowing separation of multiple local sources.

[0030] Furthermore, the locations of target sources do not need to be predefined, while still avoiding the need to train a separate DNN for each possible scenario, thus providing flexibility while saving resources.

[0031] In the present application, only one mask can be estimated for all target components, rather than estimating a mask for each of the I DOAs or spatially correlated features, thereby efficiently using computational and memory resources. In addition, since the final neural network layer only needs to focus on a single spatial region, improved performance is expected. Note that the present application is not limited to mask estimation. In addition to predicting the mask, the network can estimate the time domain or time-frequency domain representation of the target signal. Alternatively, the network can estimate the beamformer coefficients.

[0032] This application allows the DNN to learn on its own how to process each part of the input data according to the specified spatial region or DOA. In this way, spatial or DOA information is inherently combined with other types of information without the need for hand-crafted features.

[0033] The claimed application performs dimensionality reduction within a neural network (e.g., via an element-wise maximum operation). As a result, the DNN can better control which information to discard.

[0034] The claimed application also allows the incorporation of information for a range of spatial regions or DOAs, allowing for better use of the available information. This will improve source quality and allow for better noise and interference rejection.

[0035] Through the claimed spatial region-dependent feature extraction, the neural network can learn by itself how to combine any type of input features with spatial region information.

[0036] Obviously, other methods are available and known to those skilled in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present application will now be discussed in more detail with reference to the accompanying drawings, in which:

[0038] Figure 1 Schematically illustrates a neural network based audio processing system according to the prior art;

[0039] Figure 2 Schematically illustrates a training scenario of a neural network-based audio processing system according to the prior art;

[0040] Figure 3 Another audio processing system based on a neural network according to the prior art is schematically shown;

[0041] Figure 4A Schematically illustrates an audio processing system based on a neural network according to a first embodiment of the present application;

[0042] Figure 4B An example of a spatial region and a target spatial region according to an embodiment of the present application is shown;

[0043] Figure 5 Schematically illustrates an audio processing system based on a neural network according to a second embodiment of the present application;

[0044] Fig. 6A and Figure 6B The simulation results of signal estimation performed using the prior art method and the method according to the embodiment of the present application are shown;

[0045] Figure 7 A flowchart of a method for processing audio in a neural network-based audio processing system according to an embodiment of the present application is shown.

[0046] The drawings are for illustration purposes only and are not intended to limit the scope or protection provided by the claims. DETAILED DESCRIPTION

[0047] The present application can be used in any multi-microphone system to estimate and / or detect signals generated by sound sources located within a defined, continuous spatial target area. In the case of enhancement, this means suppressing all sources outside the specified target area, as well as suppressing non-local background noise. For example, one application can be to extract a single target source, whose position can be obtained using a source localization algorithm. The size of the target area can be adjusted to control the uncertainty of the position. Another application of the present application is to allow spatial audio capture, where the size of the target area is adapted / controlled by an application (e.g., acoustic scaling). Or, for example, in monitoring applications, the present application can be used to detect a specific sound source located in a spatial region of interest. Therefore, the present application can be applied to audio enhancement, source separation, and source activity detection of a specific sound coming from a certain direction. Other applications of the present application are also possible.

[0048] In this application, the target spatial extent is specified at runtime. This can be added as a set of additional input parameters to the neural network, thereby explicitly defining the region from which sources should be extracted.

[0049] Of particular interest here is that the spatial region of interest can be determined at runtime. To achieve this, it is proposed to perform spatial range-dependent feature extraction, where the first layer of the DNN uses weights according to the specified target spatial range. Without loss of generality, we will consider below the spatial range represented by the direction of arrival (DOA). However, this does not limit the scope of the present application to the target range specification based only on DOA. For example, abandoning the use of the spatial region represented by the direction of arrival, a three-dimensional (3D) volume in space is represented by a plurality of points, each of which can be defined according to Cartesian coordinates, spherical coordinates, or elliptical coordinates. Therefore, although the description may refer to a spatial region including a DOA, all described embodiments may alternatively be implemented by using any other type of suitable spatial region, such as a 3D volume.

[0050] Thus, a target DOA range is specified, and all sound sources located within the target DOA range are considered as target sources, while all sound sources located outside the target DOA range are considered as interfering sources. For the resulting source separation problem, DOA-dependent feature extraction and user-controllable DOA selection that can be integrated into the DNN are used. In this way, both the input and the output of the DNN do not need to be changed to accommodate the DOA dependency. In this way, the input does not need to include features containing target DOA range information (e.g., theoretical inter-channel phase differences) that are combined (e.g., by concatenation along one dimension or by calculating a score) with other features (i.e., on which information the source separation is performed, such as input features derived from microphone signals). Moreover, regarding the output, since a single target is defined based on the specified DOA range, the computational power of the DNN can be used more efficiently than when estimating multiple targets simultaneously, where all but a small subset representing the desired direction are subsequently discarded. This saves resources and improves performance.

[0051] Figure 4A The DNN-based signal processing system 10 according to an embodiment of the present application is schematically shown. The DNN-based signal processing system 10 includes an input 12, an output 14, a first DNN layer 16, a second DNN layer 18, and a further processing DNN layer 20.

[0052] The first DNN layer 16 includes a plurality of DOA-dependent DNN layers 22, wherein each DOA-dependent DNN layer 22 is associated with a different DOA in I discrete DOAs (i.e., I is the number of different DOAs). The trainable parameters of the plurality of DOA-dependent DNN layers 22 are specific to each of the I discrete DOAs. Since the definition target depends on the selected DOA range, the DNN is expected to learn the appropriate parameters for each DOA by itself during training. Figure 4B An example of a discrete DOA region and a target DOA range is shown. Figure 4B In the example, there are I = 12 DOA areas φ1, φ2, ..., φ 12 , covering a total of 360 degrees, of which each of the 12 DOA regions covers 30 degrees. Figure 4B In the example of , the target DOA range includes a range from -30 degrees to 30 degrees. As explained, Figure 4B are examples, and any other suitable number of DOA regions arranged in any other suitable manner may be used. For example, the grid may be finer so that each DOA region covers, for example, 5 or 10 degrees. The spatial region dependent DNN layer will be trained using sources from any location in the spatial region.

[0053] The second DNN layer 18 includes a target DOA range guided selection module for aggregating DOA-related features corresponding to a specified target DOA range to obtain a feature representation that no longer includes information from all DOAs. This can be achieved, for example, by discarding features for DOAs outside the target DOA range in I DOAs (i.e., multiplying them by 0) and then taking the maximum value (per feature) among the DOAs within the target DOA range in I DOAs.

[0054] Thus, the plurality of DOA-related DNN layers 22 may receive a plurality of signals, each associated with a microphone signal including a sound event generated by a sound source. The DOA-related DNN layer 22 may extract a plurality of features from the plurality of signals. The second DNN layer 18 may receive the extracted plurality of features from the DOA-related DNN layer 22 and generate a further plurality of features based on the extracted plurality of features, wherein the further plurality of signals correspond to at least one target spatial region. For example, a weight may be assigned to each of the extracted features, and the further plurality of features may be generated based on a weighted combination of the extracted plurality of features. Each weight may be a value between 0 and 1. In another embodiment, each weight may be 0 or 1. In one embodiment, the further plurality of features are generated by weighting the extracted plurality of features and taking the maximum value of the weighted extracted features. In another embodiment, the weighted combination may be generated by calculating the average of the values ​​of the extracted plurality of features. The further processing DNN layer 20 no longer needs to perform DOA-specific processing because the first DNN layer 16 and the second DNN layer 18 have already distinguished between DOAs. In another embodiment, the further processing DNN layer 20 may not exist. The DNN-based signal processing system 10 can estimate a TF mask calculated based on another plurality of features, and use the estimated TF mask to detect or estimate a sound source within a target DOA range. Alternatively, the DNN-based signal processing system 10 can estimate a time domain representation of a sound source within a specified target DOA range, and the time domain representation can be used to estimate or detect at least one target sound source. In another embodiment, a time-frequency representation can be estimated based on the other plurality of features, and the time-frequency representation can be used to estimate or detect a target sound source. Alternatively, beamforming weights can be estimated so that a sound source is detected or estimated based on the beamforming weights.

[0055] An example implementation according to an embodiment of the present application will now be explained. ref (μ,λ) and S(μ,λ) represent the reference microphone signal and the target signal in the short-time Fourier transform (STFT) domain, respectively, where μ is the frequency index and λ is the time frame index. In addition, Φ t is the set of target DOAs such that:

[0056]

[0057] Φ S is the set of (actual) source DOAs such that:

[0058]

[0059] Among them, φ1, φ2, ..., φ I are different discrete DOAs, where I is the number of discrete DOAs, Φ t and Φ s is the set φ1, φ2, ..., φ I The set of inner discrete DOAs. Φ t ∩Φ s Corresponds to including both belonging to the set Φ t and the set Φ s is a set of discrete DOAs of the DOA, and represents the set of target source DOAs, while Φs\(Φt∩Φs) is a set of interference source DOAs, and corresponds to a set of discrete DOAs, including those contained in the set φ1, φ2, ..., φ I The set of DOAs that are in but not in the target source Φ t ∩Φ s DOA in.

[0060] Figure 1 The target output of the CNN includes the TF mask for all i=1,...,I For all i, where The ideal TF mask is Use the target source direction (or estimate its ) The resulting estimate of the target signal is as follows:

[0061]

[0062] Therefore, although the CNN returns a total of I TF masks Only with the target source DOAφ i ∈Φ t ∩Φ s One of the corresponding ones is actually needed to obtain an estimate of the target signal.

[0063] In order to more effectively utilize the computational power of CNN, equation (3) can be rewritten as

[0064]

[0065] and

[0066]

[0067] Therefore, when Φ is pre-specified t When , the target output is reduced to a single TF mask

[0068] For a specific non-limiting embodiment of the present application, DOA-related feature extraction is integrated into Figure 1 In the TF mask estimator, as follows Figure 5 shown.

[0069] In addition to adding Φ to the input of DNN t , the parameters of the first layer are based on the DOA index i=1,...,I, so that the output features of this layer are DOA-dependent. The parameters (i.e., weights) of the DOA-dependent layers are obtained by training these layers using the sources from DOA index i. Subsequently, for all target DOAφ i ∈Φ t The obtained aggregation of features is used for further processing. The aggregation of these features is obtained by weighted combination of the extracted features. For example, the weighted combination can be generated by weighting the extracted features and taking the maximum value of all weighted extracted features. In addition, the weighted combination can be obtained by calculating the average value of all weighted extracted features. Other suitable ways of aggregating features can also be used.

[0070] In order to integrate the DOA-related feature extraction described in Figure 4 into Figure 1 The CNN of TF mask estimation is added by Φ t To specify the target DOA range, the modified DNN architecture is as follows Figure 1 shown.

[0071] The DOA correlation is introduced in the first layer (in this case, the first of the N-1 convolutional layers, where N is the number of microphones). Since this part of the DNN processes the subbands corresponding to each of the M' discrete frequencies independently, up to the Nyquist frequency, this is done for each μ=0,...,M'-1. After DOA selection, the following layers (including the other N-2 convolutional layers) can be similar to Figure 1 More generally, DOA correlation can be incorporated into the first L convolutional layers, so that only the remaining NL-1 convolutional layers are DOA-independent. Furthermore, since only one TF mask is required (instead of one TF mask for each of I DOAs), the output size (per frequency and per frame) is reduced from I to 1.

[0072] Figure 5The DNN-based signal processing system 40 according to an embodiment of the present application is schematically shown. The DNN-based signal processing system 40 includes an input 42, an output 44, a DNN layer 46 with DOA-related parameters, a target DOA range guidance selection layer 48, a further convolution layer 50, and a further processing DNN layer 52.

[0073] The DNN layer 46 with DOA-related parameters includes a collection of convolutional layers 54, each of which includes 64 filters of length 2 that operate only in the channel dimension (i.e., for the first layer, on the elements of the microphone signal phase vector ∠Y(μ,λ)). Each convolutional layer 54 may include any suitable number of filters, and each filter may have any suitable length. The input for each frequency is processed separately, but the filters are frequency-independent (i.e., training may be performed jointly for all frequencies). Figure 5 This is indicated by horizontal connections between some blocks. By introducing the DOA correlation parameters, the total number of filters in each DOA correlation layer increases to (64·I).

[0074] The target DOA range guidance selection layer 48 receives the outputs of the DNN layer with the DOA-related parameters 46 and discards from the outputs those outputs whose DOA is not within the specified target DOA range (i.e., those φ i The set Φ that does not belong to the target DOA t ). To this end, the portion of the output signal corresponding to the DOA makes φ i ∈Φ t (i.e., DOAφ i The set Φ belonging to the target DOA t ) can be multiplied by 1, and the portion of the output signal corresponding to the DOA makes (That is, this φ i The set Φ that does not belong to the target DOA t ) may be multiplied by 0, thereby providing a weighted DOA-related feature. In this way, a weighted combination of DOA-related features may then be generated. However, this may be performed in any other suitable manner. In practice, for the DOA-related features corresponding to , the computation of the DNN layer with DOA-related parameters 46 can be skipped completely, since these are not needed anyway. Since all further processing in the DNN-based signal processing system 40 is independent of DOA, the DOA dimension is no longer needed afterwards, so its size should be reduced to 1. For example, the maximum value of the features corresponding to all (selected) DOAs is used for further processing.

[0075] Another convolutional layer 50 can be similar to Figure 1 The corresponding part of the first layer is shown. Figure 1 In the (N-1) convolutional layers of φ, the DOA related parameters are only used in the first of these (N-1) convolutional layers, so that the following (N-2) convolutional layers and the rest of the DNN, i.e., the further convolutional layer 50 and the further processing DNN layer 52, are independent of Φ t Therefore, the architecture of these parts may be different from Figure 1 (e.g., including FC, flatten, stack, and LSTM layers). However, in Figure 1 In , since the TF masks for all DOAs are estimated simultaneously, all DOA information is still included in the processed data. Figure 4A In this case, the computational power of DNN is utilized more efficiently.

[0076] DOA correlation is enabled by training data consisting of pairs of target DOA ranges Φ t and target output For a ceteris paribus scenario (i.e., identical signal mix at the microphones), the target output can vary depending on which source DOAs fall within the target DOA range.

[0077] This application is not limited to Figure 5 In addition to (or instead of) the phase of the time-frequency representation of the microphone signal, the complex-valued time-frequency representation and / or the normalized amplitude of the time-frequency representation may be used as the phase input of FIGS. Figure 5 The normalization of the amplitude time-frequency representation can be based on the norm (e.g., 1-norm or 2-norm) of the time-frequency representation of all or a subset of the microphone signals, as shown in equation (6):

[0078]

[0079] This application is not limited to Figure 5 The architecture shown, into which the present application has been integrated as explained. Instead, according to the general block diagram depicted in FIG. 4 , it can be well integrated into any other DNN topology, such as an encoder / decoder based neural network (i.e., a DOA specific encoder layer). In addition, there may be no further processing DNN layer 52. In other embodiments, in addition to estimating the target output TF mask, other estimates may be made, such as a complex TF mask, a time domain or time-frequency representation of the target signal, or beamformer coefficients.

[0080] Some variations of alternative embodiments are listed below and are not specific to any particular DNN architecture but can be used for any suitable DNN.

[0081] When integrating the DNN layer with the DOA-related parameters 46 in the first layer of the DNN-based signal processing system 40, the increase in the number of trainable parameters is minimal. This is because for each frequency and frame, the input consists of a single vector of length N, whereas the input and output of the subsequent convolutional layers consist of vectors with the same number of filters ( Figure 5 For different architectures, it may be preferable to integrate the DNN layer with the DOA-related parameters 46 into different layers of the DNN-based signal processing system.

[0082] The goal of this application is to remove the DOA dependency of the data early in the chain, so that all further processing can be done without regard for the target DOA. By doing this directly based on the input, the remaining layers can use a DOA-independent representation, based on which computational power can be used more efficiently (and may even allow the use of transfer learning or other efficient learning methods). In contrast, in e.g. Figure 1 In the prior art shown, the DOA information of all I DOAs is still included in the data until the masks of all I DOAs are finally estimated. Since only a small subset of these masks are actually needed, there is considerable redundancy in the processed data, so the computing resources are not effectively utilized.

[0083] Nevertheless, DOA-related processing can also be integrated into DNN in different forms. For example, the present application can be integrated at a deeper level of the network (e.g., Figure 5 The fully-connected (FC) layer in , where a higher level of abstraction has been achieved.

[0084] Figure 5 The target DOA range guide selection layer 48 performs DOA selection by generating a weighted combination of the extracted features, for example, by taking the weighted DOA-related features I(μ, λ, φ i ) is the maximum value in all viewing directions, which can be expressed as

[0085]

[0086] Among them, I(μ,λ,φ i ) is the matrix input received by the target DOA range guidance selection layer 48 and is also the output of the DNN layer with DOA-related parameters 46, max(·) is an operation function whose output is the maximum value of its input and is applied element-wise, and the weight w(φ i ) is assigned to each extracted feature and is given by

[0087]

[0088] Regardless of tWhat is the number of elements in for all DOAφ i ∈Φ t The intuition behind taking the maximum value is Φ t Source activity in any direction contained in should be captured. However, other solutions are possible and some non-limiting alternatives are explained below.

[0089] The target output shown in equation (5) above is the sum of all individual TF masks. Therefore, to generate a weighted combination of the extracted features, one may consider summing or averaging the weighted extracted features over all DOAs rather than taking the maximum value. Equation (7) can be summarized as

[0090]

[0091] Here, op(·) is some operation, such as summation or averaging.

[0092] Real-valued weights can be used instead of binary values ​​w(φ i ). For example, when Φ t When covering a coherent DOA range, the weight can be set to the maximum value w(φ i )=1 and decreases monotonically towards the end of the target DOA range to achieve a smooth transition. Alternatively, soft information about the source activity at each DOA, such as information obtained from a separate system (e.g., a neural network), can be used directly to set the weights.

[0093] Finally, for this special case, Φ t is constrained so that |Φ t |=1(where, Representing a collection ), given a particular target source DOA and the information extracted from the microphone signals, an appropriate TF mask will be returned. Therefore, in this form, a direct comparison can be made. Because the TF mask is calculated separately for each DOA, the computational power of the neural network can be used more efficiently when using this application than when estimating masks for all directions simultaneously.

[0094] Fig. 6A shows the simulation results, where similar to Figure 1 The DNN is used to estimate TF masks in all directions simultaneously. Figure 6B The simulation results are shown, where, according to an embodiment of the present application, DNN is used to estimate the TF mask of the target DOA range, which covers an azimuth width of 15°. In the presence of additional noise (SNR = 5dB), Fig. 6A- The two simulated test scenarios in Figure B consist of two people speaking simultaneously (with a basic arrival angle difference of 60°). One of the speech signals serves the target, while the interfering talker and noise should be suppressed. Fig. 6A - In Figure B, the horizontal axis represents time in seconds, and the vertical axis represents frequency in kilohertz. Bars 60 and 62 represent the TF mask color legend, where the TF mask can have values ​​between 0 and 1, and each value is associated with a color shown in bars 60 and 62. Obviously, Figure 6B The DOA-related feature extraction used in the simulations shown allows for a more accurate capture of the harmonic structure of speech. Specifically, the following key improvements are observed, some of which are also illustrated in Fig. 6A -B indicates:

[0095] Improved low frequency separation : At low frequencies, the differences in spatial information between signal components (TF mask estimation is based only on this) are small, which makes their separation more difficult. Through DOA-related feature extraction, Figure 6B The low frequency band in the image still shows obvious structure, as indicated by 64, while Fig. 6A The TF mask in has been blurred, as indicated by 66.

[0096] Preserve a larger proportion of the target speech : Also at higher frequencies, it is clear that the DNN with DOA-related feature extraction can make better use of the limited information. In this case, the signal Fig. 6A is completely suppressed to higher frequencies (TF mask is anywhere close to 0), as indicated by 68, while Figure 6B The mask in the method used in only suppresses part of the signal, as indicated by 67 .

[0097] Capture fine structures more accurately : The background noise is broadband, while the target speech is relatively sparse. Therefore, to ensure that the noise is suppressed also in the TF region with significant speech activity, it is critical that there is also considerable suppression between the individual harmonics. Fig. 6A In the mask, the harmonics appear blurred, as indicated by 69. The resulting insufficient suppression between harmonics may cause perceptible distortion of the target speech. In contrast, with DOA-related feature extraction, more distinct harmonics can be seen in the mask, resulting in significantly reduced perceptible distortion of the target speech (see Figure 6B 63).

[0098] Figure 7 A flow chart of a method for detecting or estimating a target sound source among a plurality of sound sources is shown, wherein the target sound source is located in at least one target spatial region among a plurality of spatial regions.

[0099] At step 70 , the method includes receiving, by a processor, a plurality of signals, wherein each of the plurality of signals is associated with one of a plurality of microphone signals, wherein the microphone signal includes sound events generated by a plurality of sound sources.

[0100] In step 72, the method includes: the processor extracting a plurality of features through a neural network, wherein the plurality of features are extracted from the plurality of signals, and wherein each of the plurality of features is obtained by training the neural network for a different spatial region among the plurality of spatial regions.

[0101] In step 74 , the method includes: the processor generating another plurality of features based on the extracted plurality of features, wherein the another plurality of features corresponds to at least one target spatial region.

[0102] Finally, in step 76 , the method includes the processor detecting or estimating at least one of the target sound sources in the target spatial region based on a further plurality of features corresponding to the at least one target spatial region.

[0103] Although the present application has been described with reference to exemplary embodiments, it will be appreciated by those skilled in the art that various changes may be made and that equivalents may be substituted for its elements without departing from the scope of the present application. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present application without departing from the basic scope of the present application. Therefore, it is intended that the present application is not limited to the specific embodiments disclosed, but that the present application will include all embodiments falling within the scope of the appended claims.

[0104] In particular, the specific features of various aspects of the present application can be combined. One aspect of the present application can be further advantageously enhanced by adding the features described in relation to another aspect of the present application. It should be understood that the present application is only limited by the attached claims and their technical equivalents. In this document and its claims, the verb "comprise" and its conjugation are used in a non-restrictive sense, indicating that the items after the word are included, but items not specifically mentioned are not excluded. In addition, the reference to an element by the indefinite article "a" or "an" does not exclude the possibility of the presence of multiple elements, unless the context clearly requires the presence of one and only one element. Therefore, the indefinite article "a" or "an" usually means "at least one".

Claims

1. A method for detecting or estimating a target sound source among a plurality of sound sources, wherein the target sound source is located in at least one target spatial region among a plurality of spatial regions, the method comprising: a processor receiving a plurality of signals, wherein each of the plurality of signals is associated with one of a plurality of microphone signals, wherein the microphone signals include sound events generated by the plurality of sound sources; The processor extracts a plurality of features by a neural network, wherein the plurality of features are extracted from the plurality of signals, and wherein each of the plurality of features is obtained by training the neural network for a different spatial region among the plurality of spatial regions; The processor generates an additional plurality of features based on the neural network and the extracted plurality of features, wherein the additional plurality of features corresponds to the at least one target spatial region; The processor detects or estimates at least one of the target sound sources in the target spatial region based on the additional plurality of features corresponding to the at least one target spatial region.

2. The method according to claim 1, wherein: Each of the plurality of signals is associated with a plurality of phases of a time-frequency representation of a microphone time-domain signal.

3. The method according to claim 1 or 2, wherein: Each of the plurality of signals comprises at least one of a plurality of normalized amplitudes of a time-frequency representation of a microphone time-domain signal, wherein the normalization is based on a norm of at least one of the plurality of time-frequency representations of the microphone time-domain signal.

4. The method according to claim 1, wherein: Each of the plurality of signals is associated with a plurality of complex time-frequency representations of the microphone time-domain signals.

5. The method according to claim 1 or 2, wherein: The plurality of features are extracted from the plurality of signals by at least one of a plurality of convolutional neural network layers.

6. The method according to claim 1 or 2, wherein: Each of the plurality of spatial regions includes a plurality of directions of arrival.

7. The method according to claim 1 or 2, further comprising: The processor estimates a time-frequency TF mask based on the further plurality of features, and detects or estimates the at least one of the target sound sources in the at least one target spatial region based on the TF mask.

8. The method according to claim 1 or 2, further comprising: The processor estimates a time domain representation of the at least one sound source based on the further plurality of features, and detects or estimates the at least one sound source based on the time domain representation.

9. The method according to claim 1 or 2, further comprising: The processor estimates a time-frequency representation of the at least one sound source based on the further plurality of features, and detects or estimates the at least one sound source based on the time-frequency representation.

10. The method according to claim 1 or 2, further comprising: The processor estimates beamforming weights based on the further plurality of features, and detects or estimates the at least one sound source based on the beamforming weights.

11. The method according to claim 1 or 2, wherein: The additional plurality of features are generated based on a weighted combination of the extracted plurality of features.

12. The method according to claim 11, wherein: Generating the weighted combination of the extracted multiple features includes: A weight value is generated for each of the extracted plurality of features, wherein the weight value of a feature of the extracted plurality of features indicates whether the feature corresponds to the target spatial region.

13. The method according to claim 1 or 2, wherein: The additional multiple features are generated by weighting the extracted multiple features and taking the maximum value of the weighted multiple extracted features.

14. A device for detecting or estimating a target sound source among a plurality of sound sources, wherein: The target sound source is located in at least one target spatial area among a plurality of spatial areas, and the apparatus comprises a memory and a processor communicatively connected to the memory and configured to execute instructions to perform the method according to any one of claims 1 to 13.

15. A computer program product, wherein: A computer program is included for executing the method according to any one of claims 1 to 13.

Images