A method and system for localizing multiple moving sound sources based on spatial and spectral temporal information modeling

Abstract

The present invention relates to a method and system for localizing multiple mobile sound sources based on spatial and spectral time series information modeling. The present invention encodes the direct path phase difference and the sound source activity information into the learning target in a weighted summation manner, thereby guiding the network to learn the direct path phase difference characteristics of multiple mobile sound sources with competitiveness and time variation, avoiding the allocation ambiguity and output dimension uncertainty problems faced by the multi-target regression framework; the positions of multiple mobile sound sources are determined by iteratively detecting and locating the dominant sound source, reducing the mutual influence between multiple sound sources; the spectral feature extraction network is introduced to establish the association between the sound source position and the sound source identity, and reliable multi-mobile sound source tracking is achieved by modeling the spatial and spectral time series information.

Description

Technical Field

[0001] The invention belongs to the field of robot auditory audio signal processing, and in particular relates to a method and system for localizing multiple mobile sound sources based on spatial and spectral time series information modeling. Background Art

[0002] Robot hearing is an intelligent, friendly and natural way for robots to interact with the external environment. Since sound signals are diffractive and omnidirectional, hearing is not constrained by the linear field of vision and can still work normally when there is a field of vision obstruction. Sound source localization of the robot hearing system refers to the robot using the microphone array installed in the robot hearing system to determine the spatial relative position of the sound source. Sound source localization technology has a wide range of applications in the fields of intelligent robot hearing, human-computer interaction, etc., and as the front end of audio signal processing tasks such as speech enhancement, sound source separation, and automatic speech recognition, its performance is directly related to whether the robot's hearing function can operate well. In recent years, sound source localization technology has attracted the attention of many audio and acoustic signal processing academic organizations.

[0003] Sound source localization technology aims to determine the spatial position of the external sound source relative to the microphone array. Traditional sound source localization methods mainly include arrival time difference technology, controllable beamforming, high-resolution spectrum estimation, etc. With the development of deep learning technology, sound source localization methods based on supervised learning have attracted more and more attention from researchers. Compared with traditional unsupervised methods, sound source localization methods based on deep learning are data-driven, so they can adapt well to the acoustic conditions in the training data. According to the role of deep neural network (DNN) in sound source localization, sound source localization methods can be roughly divided into four categories, namely, methods based on feature-position mapping, methods based on signal-position mapping, methods based on feature-feature mapping, and methods based on signal-feature mapping. Methods based on feature-position mapping and methods based on signal-position mapping can learn nonlinear mapping functions from features or signals to sound source positions. Methods based on feature-feature mapping and methods based on signal-feature mapping provide a simple and effective way to recover effective positioning features from distorted features or signals. Sound source localization methods based on deep learning generally deal with the situation of multiple sound sources by changing the loss function or changing the network output. In the classification framework, a multi-classification cross entropy loss function is often used to train a single sound source localization network. By replacing the loss function with a multi-label binary cross entropy loss function, it can be directly applied to multiple sound source localization. The output dimension of the classification network is determined by the division accuracy of the positioning space, which increases with the improvement of the division accuracy of the positioning space. Compared with steering angle localization, the discrete angle categories of the steering angle and pitch angle joint estimation task will increase exponentially, which makes the output dimension of the classification network also become very large. In the regression framework, a single sound source localization network generally outputs a position or feature. In order to deal with multiple sound sources, one way is to change the single output of the localization network to an output that is the same as the number of sound sources (or the maximum number of sound sources that may exist). There is an allocation ambiguity problem between multiple network outputs and multiple training targets, which increases the difficulty of network training. In addition, when the number of sound sources is unknown and not fixed, it is difficult to use a unified network to regress a variable number of positions or features. Mobile sound source localization methods based on deep learning can be divided into two categories, namely short-term time series information processing methods and long-term time series information processing methods. Generally speaking, the movement of a mobile sound source is continuous, and the spatial characteristics of the sound source change slowly over time. The localization work based on short-term temporal information processing regards the spatial information of short-term signals as unchanged and uses static sound source localization methods for processing, but the long-term temporal background information will be ignored. The method based on long-term temporal information processing mainly uses temporal networks to model the temporal and spatial information of mobile sound sources.

[0004] In recent years, although great progress has been made in the research of sound source localization technology for robot auditory systems, most research works either propose multiple static sound source localization solutions for the complexity of interactive objects, or propose single mobile sound source localization solutions for the complexity of motion. The actual working scenarios of robot auditory systems often have multiple interactive sound sources and sound source motion. Multiple mobile sound source scenarios require the localization algorithm to estimate and track the dynamic positions of competing sound sources in a timely manner, which poses new challenges to the sound source localization task. Summary of the invention

[0005] Aiming at the problems of dynamic changes in spatial positions of multiple moving sound sources and failure in tracking sound source trajectories in complex scenes, the present invention proposes a method and system for localizing multiple moving sound sources based on spatial and spectral temporal information modeling.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A method for localizing multiple moving sound sources based on spatial and spectral temporal information modeling comprises the following steps:

[0008] 1) Using a causal convolutional recurrent neural network (CRNN) to predict the competitive and time-varying spatial features of multiple sound sources;

[0009] 2) Use iterative sound source detection and localization methods to separate the spatial features of multiple sound sources and determine the number and direction of arrival (DOA) of active sound sources;

[0010] 3) Based on the DOA of the sound source, a beamformer is used to separate the time-frequency representation of the microphone signal of each sound source;

[0011] 4) Based on the time-frequency representation of the microphone signal of each sound source, a spectrum feature extraction network is used to extract identity recognition features for each sound source, and the identity recognition features at the current moment are matched with the identity recognition features in the history to determine the identity of the active sound source at the current moment.

[0012] Furthermore, step 1) involves a causal convolutional recurrent neural network, the network input is the amplitude spectrum and phase spectrum of the dual-channel microphone signal, the network learning target encodes the direct path phase difference and activity information of multiple sound sources in the form of weighted summation, and the mean square error between the network output and the training target is taken as the loss of network training. The multi-sound source spatial features predicted by the network can be directly used to construct the multi-sound source spatial spectrum.

[0013] Furthermore, step 2) involves an iterative sound source detection and localization method. Each iteration of the iterative method mainly includes three steps: a) constructing a spatial spectrum based on the multi-sound source spatial characteristics; b) detecting a new sound source by maximizing the spatial spectrum, and determining the DOA, phase difference sequence and activity level of the new sound source; c) removing the contribution of the new sound source from the multi-sound source spatial characteristics.

[0014] Furthermore, step 4) involves a spectral feature extraction network, the network structure of which is based on a twin network, the input of which is the time-frequency representation of the single-channel microphone signal corresponding to each sound source, and the output of which is a spectral feature with identity recognition, namely, an identity recognition feature.

[0015] A multi-mobile sound source localization system based on spatial and spectral time series information modeling using the above method comprises:

[0016] A multi-sound source spatial feature extraction module, which is used to predict competitive and time-varying multi-sound source spatial features using a causal convolutional recurrent neural network;

[0017] Iterative sound source detection and localization module, used to separate the spatial features of multiple sound sources by using iterative sound source detection and localization method, and determine the number and arrival direction of active sound sources;

[0018] The sound source signal separation module is used to separate the time-frequency representation of the microphone signal of each sound source using a beamformer according to the arrival direction of the sound source;

[0019] A sound source spectrum feature extraction module is used to extract identity recognition features for each sound source using a spectrum feature extraction network based on the time-frequency representation of the microphone signal of each sound source;

[0020] The sound source identity determination module is used to match the identity recognition features at the current moment with the identity recognition features in the history to determine the identity of the active sound source at the current moment.

[0021] The present invention encodes the direct path phase difference and sound source activity information into the learning objectives in a weighted summation manner, thereby guiding the network to learn the direct path phase difference characteristics of multiple mobile sound sources that are competitive and time-varying, avoiding the allocation ambiguity and output dimension uncertainty problems faced by the multi-target regression framework; the positions of multiple mobile sound sources are determined by iteratively detecting and locating the dominant sound source, thereby reducing the mutual influence between the multiple sound sources; a spectral feature extraction network is introduced to establish the association between the sound source position and the sound source identity, and reliable multi-mobile sound source tracking is achieved by modeling the spatial and spectral temporal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 .Overall flow chart of the method of the present invention. DETAILED DESCRIPTION

[0023] In order to describe the method of the present invention more clearly and completely, the present invention will be further described below in conjunction with specific embodiments of the present invention and the accompanying drawings.

[0024] refer to Figure 1 , is a flow chart of the method of the present invention, which mainly includes the following steps:

[0025] Step 1: Multi-sound source spatial feature extraction

[0026] First, the short-time Fourier transform is used to transform the multi-channel microphone signal from the time domain to the time-frequency domain, and then the multi-source spatial features are estimated based on the phase spectrum and amplitude spectrum of each microphone pair. The network structure for multi-source spatial feature extraction is causal CRNN. ​​The network input is the logarithmic amplitude spectrum and phase spectrum of the dual-channel microphone signal. The input features are sent to 10 causal convolution modules, each of which consists of a causal convolution layer, a batch normalization layer, and a ReLU activation function. A maximum pooling layer is added after every two convolution modules to compress the frequency and time dimensions. The output of the convolutional neural network layer is expanded into one dimension in the frequency and filter dimensions and sent to a layer of unidirectional gated recurrent unit. Using a fully connected layer and K max The Tanh activation function outputs the direct path phase difference of the microphone pair. Here, K max Refers to the maximum number of possible sound sources. The direct path phase difference vectors of multiple sound sources and the sound source activity information are weighted summed to form the training target, that is,

[0027]

[0028] Among them, R mm' (n') represents the training target of the microphone pair mm' corresponding to the n'th output frame, where m and m' both represent microphone indices; the sum weight β k (n') represents the active probability of the kth sound source in the n'th output frame, and its value is between 0 and 1; K represents the actual number of sound sources. mm' (θ k ) is the direct path phase difference vector of the full frequency band, expressed as:

[0029]

[0030] Among them, θ k represents the arrival direction of the kth sound source, ω f represents the angular frequency of the fth frequency band, τ mm' represents the arrival time difference between the signals received by the m-th microphone and the m'-th microphone, and F represents the number of frequency bands.

[0031] Correspondingly, the value of the summed vector element is within [0, K]. The mean square error between the network output and the training target is taken as the loss of network training.

[0032] Step 2: Iterative sound source detection and localization

[0033] The iterative process of this method mainly includes three steps. The first step is the spatial spectrum estimation process. After obtaining the predicted values ​​of the spatial features of multiple sound sources, the spatial spectrum is constructed using the features of all microphone pairs, namely:

[0034]

[0035] Where P(θ,n') represents the spatial spectrum constructed using the features of all microphone pairs, θ represents the candidate arrival direction used to construct the spatial spectrum, represents the predicted value of the spatial features of multiple sound sources, and M represents the number of microphones.

[0036] The second step is to detect new sound sources by maximizing the spatial spectrum and determine the DOA, phase difference sequence and activity level of the new sound source, which are expressed as:

[0037]

[0038]

[0039]

[0040] The third step is to remove the contribution of the new sound source from the multi-sound source spatial features. For all microphone pairs, we have:

[0041]

[0042] When the activity of the newly detected sound source is small enough, When β TH Indicates the predefined threshold, the dominant sound source Inactive, iteration index k decreases by 1 and the iteration stops; otherwise, Assign to The DOA estimate of the sound source in the n'th frame is

[0043] Step 3: Sound source signal separation

[0044] Based on the DOA estimation value of each sound source, the minimum-variance distortionless response (MVDR) beamformer is used to extract the signal of each sound source. Where n and f represent the time frame and frequency index respectively.

[0045] Step 4: Extracting sound source spectrum features

[0046] The twin network is used as the spectral feature extraction network, and the input is the time-frequency representation of the single-channel microphone signal corresponding to each sound source. The output is a spectrum feature v with identity recognition k The network is trained using the contrastive loss function.

[0047] Step 5: Sound source identification

[0048] The sound source identity identification features output by the spectrum feature extraction network are matched with the identity identification features in the history. When the best matching sound source has a high enough match, it is used as the identity of the current active sound source; when the best matching sound source has a low match, the current active sound source is considered to be a new sound source. In this way, the sound source identity is associated with the location.

[0049] Based on the same inventive concept, another embodiment of the present invention provides a multi-mobile sound source localization system based on spatial and spectral temporal information modeling using the above method, which includes:

[0050] A multi-sound source spatial feature extraction module, which is used to predict competitive and time-varying multi-sound source spatial features using a causal convolutional recurrent neural network;

[0051] Iterative sound source detection and localization module, used to separate the spatial features of multiple sound sources by using iterative sound source detection and localization method, and determine the number and arrival direction of active sound sources;

[0052] The sound source signal separation module is used to separate the time-frequency representation of the microphone signal of each sound source using a beamformer according to the arrival direction of the sound source;

[0053] A sound source spectrum feature extraction module is used to extract identity recognition features for each sound source using a spectrum feature extraction network based on the time-frequency representation of the microphone signal of each sound source;

[0054] The sound source identity determination module is used to match the identity recognition features at the current moment with the identity recognition features in the history to determine the identity of the active sound source at the current moment.

[0055] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0056] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0057] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for localizing multiple mobile sound sources based on spatial and spectral temporal information modeling, the steps of which include: A causal convolutional recurrent neural network is used to predict the competitive and time-varying spatial features of multiple sound sources. Iterative sound source detection and localization method is used to separate the spatial features of multiple sound sources and determine the number and arrival direction of active sound sources; Based on the arrival direction of the sound source, a beamformer is used to separate the time-frequency representation of the microphone signal of each sound source; According to the time-frequency representation of the microphone signal of each sound source, a spectrum feature extraction network is used to extract identity recognition features for each sound source, and the identity recognition features at the current moment are matched with the identity recognition features in the history to determine the identity of the active sound source at the current moment; The causal convolutional recurrent neural network encodes the direct path phase difference and the sound source activity information into the learning target in a weighted summation manner, guides the network to learn the competitive and time-varying multi-mobile sound source spatial features, and takes the mean square error between the network output and the training target as the loss of network training; the multi-sound source spatial features predicted by the network can be directly used to construct the multi-sound source spatial spectrum; The iterative process of the iterative sound source detection and localization method includes the following three steps: Construct multi-sound source spatial spectrum according to multi-sound source spatial characteristics; Detect new sound sources by maximizing the spatial spectrum, and determine the arrival direction, phase difference sequence, and activity level of the new sound sources; The contribution of the new sound source is removed from the multi-sound source spatial feature.

2. The method according to claim 1, characterized in that The multi-sound source spatial spectrum is calculated according to the following formula: Where P(θ,n') represents the spatial spectrum constructed using the features of all microphone pairs, θ represents the candidate arrival direction used to construct the spatial spectrum, n' represents the n'th output frame; M represents the number of microphones; F represents the number of frequency bands; m, m' represents the mth microphone and the m'th microphone; Represents the predicted value of multi-sound source spatial features; r mm' (θ k ) is the direct path phase difference vector of the full frequency band.

3. The method according to claim 2, characterized in that The arrival direction, phase difference sequence and activity level of the new sound source are determined according to the following formula: in, They respectively represent the arrival direction, phase difference sequence and activity level of the new sound source.

4. The method according to claim 3, characterized in that The contribution of the new sound source is removed from the multi-sound source spatial feature according to the following formula:

5. The method according to claim 1, characterized in that The frequency spectrum feature extraction network is used to determine the identity of the sound source in a specific arrival direction at the current moment, so as to cope with the situation where a new sound source is added and the sound source trajectory tracking fails.

6. A multi-mobile sound source localization system based on spatial and spectral time series information modeling using the method described in any one of claims 1 to 5, characterized in that: include: A multi-sound source spatial feature extraction module, which is used to predict competitive and time-varying multi-sound source spatial features using a causal convolutional recurrent neural network; Iterative sound source detection and localization module, used to separate the spatial features of multiple sound sources by using iterative sound source detection and localization method, and determine the number and arrival direction of active sound sources; The sound source signal separation module is used to separate the microphone signal time-frequency representation of each sound source using a beamformer according to the arrival direction of the sound source; A sound source spectrum feature extraction module is used to extract identity recognition features for each sound source using a spectrum feature extraction network based on the time-frequency representation of the microphone signal of each sound source; The sound source identity determination module is used to match the identity recognition features at the current moment with the identity recognition features in the history to determine the identity of the active sound source at the current moment.

7. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 5 is implemented.

Images