Low signal-to-noise ratio sound source positioning system and method based on combination of GCC and neural network

Abstract

The invention discloses a low signal-to-noise ratio sound source positioning system and method based on combination of GCC and neural network, and the system comprises a microphone array which is used for collecting a time domain signal of a sound source; the generalized cross-correlation calculation unit is used for calculating a generalized cross-correlation matrix between each pair of microphones in the microphone array; the feature extraction unit is used for inputting the generalized cross-correlation matrix into a ResNet-34 neural network for feature extraction; and the classification unit is used for classifying the extracted features so as to determine the position of the sound source. According to the method, the time domain signal of the microphone array is converted into the generalized cross-correlation matrix, sound source positioning classification is carried out in combination with the neural network, and the method is particularly suitable for a low-signal-to-noise-ratio environment. According to the method, the traditional generalized cross-correlation technology and the feature extraction capability of deep learning are combined, the accuracy of sound source positioning can be remarkably improved in a low-signal-to-noise-ratio environment, and the method is particularly suitable for sound source positioning requirements in a complex environment.

Description

Technical Field

[0001] The present invention relates to the fields of sound source localization and neural network, and in particular to a low signal-to-noise ratio sound source localization system and method based on the combination of GCC and neural network. Background Art

[0002] Time Delay Estimation (TDE) is a key component in many acoustic localization applications, including sound source tracking, robotics, and self-calibration. In these applications, accurately estimating the propagation time of the signal from the sound source to the microphone is the basis for determining the location of the sound source. In a typical TDE setting, the signal is received by a set of synchronized microphones with known positions, while the source signal itself and its emission time are usually unknown. Therefore, it is impossible to directly obtain the absolute transmission time of the signal from the sound source to the microphone. Instead, the sound source is localized by measuring the Time Difference of Arrival (TDOA). By cross-correlating the signals received by the microphone array, the TDOA values ​​can be obtained, and these values ​​are further used to calculate the direction of arrival (DOA) of the signal or calculate the location of the sound source through multilateration.

[0003] The generalized cross-correlation (GCC) method has always been one of the most widely used methods in TDOA estimation. In recent years, some studies have begun to try to combine GCC with neural networks to improve the performance of traditional GCC methods. Some studies use autoencoder networks to denoise the frequency-sliding GCC output to improve the accuracy of TDOA estimation. Other studies use neural networks to predict speech masks for frequency-selective filtering of signals to reduce the interference of environmental noise on TDOA estimation. In addition, some studies have proposed calculating multiple GCCs and processing the outputs through convolutional neural networks (CNNs) to directly predict the TDOA between two signals. Although these methods reduce the average error to a certain extent, there is still the problem of inaccurate sample prediction in scenarios where high-precision positioning is required.

[0004] The traditional GCC method has long occupied an important position in acoustic localization due to its simple implementation and good robustness, but its adaptability to complex environments is limited, especially in conditions of severe noise and reverberation. Summary of the invention

[0005] The purpose of the present invention is to provide a low signal-to-noise ratio sound source localization system and method based on the combination of GCC and neural network, which can effectively solve the problems of sensitivity to noise and reverberation and high computational complexity in traditional TDOA estimation, thereby achieving high-precision, low-complexity sound source localization. The present invention is particularly suitable for low signal-to-noise ratio environments and can greatly improve the accuracy and robustness of sound source localization.

[0006] In one aspect of the present invention, the present invention proposes a low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network. According to an embodiment of the present invention, the system includes:

[0007] A microphone array is used to collect the time domain signal of the sound source;

[0008] A generalized cross-correlation calculation unit, used for calculating a generalized cross-correlation matrix between each pair of microphones in the microphone array;

[0009] A feature extraction unit, used for inputting the generalized cross-correlation matrix into a ResNet-34 neural network for feature extraction;

[0010] The classification unit is used to classify the extracted features to determine the location of the sound source.

[0011] In addition, the low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network according to the above embodiment of the present invention may also have the following additional technical features:

[0012] In some embodiments of the present invention, the number of array elements of the microphone array is 16, and the generalized cross-correlation calculation unit calculates the signals of the 16 microphone arrays to obtain a 120×43 feature matrix containing delay information, where 120 is the number of microphone pairs and 43 is the delay feature dimension.

[0013] In some embodiments of the present invention, the generalized cross-correlation calculation unit obtains a generalized cross-correlation matrix by performing Fourier transform on the time domain signal of the microphone pair and performing phase transform on the spectrum thereof, thereby extracting the time delay feature of the sound source.

[0014] In some embodiments of the present invention, the feature extraction unit uses a ResNet-34 neural network to perform layer-by-layer convolution operations on the GCC feature map to extract delay features; when the feature extraction unit performs a convolution operation on each GCC matrix, a 1×1 convolution kernel is used.

[0015] In some embodiments of the present invention, during the sound source localization process, the sound source plane is a 3×3 square grid, the size of the plane is 2×2 meters, the resolution is 1 meter, the number of microphone array elements is 16, and the sound source position is determined by classifying the GCC matrices of different microphone pairs.

[0016] In some embodiments of the present invention, the classification unit classifies the extracted features through a Softmax layer to determine the specific position of the sound source in a 3×3 square grid; the classification unit processes the features extracted by ResNet-34 through a fully connected layer, and uses a Softmax activation function to output the probability distribution of the sound source position.

[0017] In another aspect of the present invention, the present invention proposes a low signal-to-noise ratio sound source localization method based on the combination of GCC and neural network. According to an embodiment of the present invention, the method comprises the following steps:

[0018] (1) Calculate the generalized cross-correlation matrix between each pair of microphones in the microphone array to obtain the time delay information of the sound source;

[0019] (2) Each generalized cross-correlation matrix is ​​regarded as a feature map, and a ResNet-34 neural network is used to perform convolution operations on it to extract time delay features and retain the characteristic information of the sound source at different spatial positions;

[0020] (3) Classify the feature information processed by the neural network and determine the specific location of the sound source.

[0021] In addition, the low signal-to-noise ratio sound source localization method based on the combination of GCC and neural network according to the above embodiment of the present invention may also have the following additional technical features:

[0022] In some embodiments of the present invention, the microphone array is an array of 16 microphones, capable of locating the sound source position in a 3×3 grid.

[0023] In some embodiments of the present invention, in step (2), the ResNet-34 neural network includes multiple convolutional layers and residual modules, which are used to gradually extract the delay information in the GCC feature map and retain the spatial distribution characteristics.

[0024] In some embodiments of the present invention, in step (2), the input of the ResNet-34 neural network is a 120×43 feature matrix, which undergoes multi-layer convolution and pooling operations and finally outputs a one-dimensional feature vector for classification.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1) The present invention adopts the ResNet-34 neural network structure, which includes multiple convolutional layers and residual modules, which can effectively extract the time delay information in the input feature map and gradually learn more complex and abstract features. This deep network structure makes the sound source localization process not only highly accurate, but also more robust when dealing with complex acoustic environments (low signal-to-noise ratio environments), can accurately locate the sound source, and can adapt to complex environments.

[0027] 2) The present invention combines generalized cross-correlation calculation, ResNet-34 feature extraction module and Softmax classifier module, and the present invention can accurately realize sound source localization in a complex sound field environment. First, the cross-correlation matrix between microphone pairs is calculated by GCC, then spatial and temporal features are extracted from it using ResNet-34, and finally the features are classified by Softmax classifier to output the specific location of the sound source.

[0028] 3) The technical solution of the present invention significantly improves the accuracy and efficiency of sound source localization by combining classical signal processing technology and deep learning models, and has broad application prospects and practical value.

[0029] 4) The present invention performs convolution operations by combining the GCC matrix with the ResNet-34 neural network, which reduces the computational complexity while retaining the time delay characteristics of the sound source and improving the computational efficiency.

[0030] 5) The present invention is based on the GCC matrix of the multi-microphone array and adopts neural network multi-channel parallel processing, which greatly shortens the calculation time and improves the real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of simulating microphone array time domain signal positioning in Example 1 of the present invention;

[0032] Figure 2 This is a flow chart of a low signal-to-noise ratio sound source localization method based on the combination of GCC and neural network in Example 2 of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example 1

[0035] The low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network includes the following main modules: microphone array, generalized cross-correlation calculation unit, feature extraction unit and classification unit.

[0036] The microphone array is used to collect the time domain signal of the sound source, wherein the number of array elements of the microphone array is 16.

[0037] The generalized cross-correlation calculation unit is used to calculate the generalized cross-correlation (GCC) matrix between each pair of microphones in the microphone array to obtain the delay information of the sound source. The generalized cross-correlation calculation unit calculates the signals of the 16-microphone array to obtain a 120×43 feature matrix containing delay information, where 120 is the number of microphone pairs and 43 is the delay feature dimension. The generalized cross-correlation calculation unit obtains the generalized cross-correlation matrix by Fourier transforming the time domain signal of the microphone pair and performing phase transformation on its spectrum, thereby extracting the delay characteristics of the sound source.

[0038] The feature extraction unit is used to input the GCC matrix into the ResNet-34 neural network for feature extraction. The feature extraction unit uses the ResNet-34 neural network to perform layer-by-layer convolution operations on the GCC feature map to extract delay features. When the feature extraction unit performs convolution operations on each GCC matrix, a 1×1 convolution kernel is used.

[0039] The classification unit is used to classify the features processed by the neural network to determine the specific location of the sound source. In the process of sound source localization, the sound source plane is a 3×3 square grid, the size of the plane is 2×2 meters, the resolution is 1 meter, the number of microphone array elements is 16, and the sound source position is determined by classifying the GCC matrix of different microphone pairs. The classification unit classifies the extracted features through the Softmax layer to determine the specific location of the sound source in the 3×3 square grid; the classification unit processes the features extracted by ResNet-34 through the fully connected layer and uses the Softmax activation function to output the probability distribution of the sound source position.

[0040] Specifically as shown in Figure 1, Figure 1 The system structure for sound source localization is shown, including a 16-microphone annular microphone array and an N×N scanning plane. The microphone array is arranged in a ring and contains 16 microphones for collecting the time domain signal of the sound source. The position of each microphone can be expressed in polar coordinates, with a distance r from the center of the array. mn In this system, the scan plane is used to estimate the location of the sound source, dividing it into N×N grids for more accurate positioning. The center point of each grid is represented by the coordinates (x n ,y n,z) represents a possible sound source position in the scanning plane. By analyzing the acoustic signals received from different microphones, the system can determine the specific grid position of the sound source in the scanning plane.

[0041] Example 2

[0042] The low signal-to-noise ratio sound source localization method based on the combination of GCC and neural network specifically includes the following steps:

[0043] First, through Figure 2 In the processing flow shown in the figure, the microphone array is used to collect the time domain signals of the sound source, which contain the delay and direction information of the sound source. Subsequently, the system uses the GCC calculation unit to perform generalized cross-correlation calculations on the signals between each pair of microphones in the microphone array to extract the delay characteristics of the sound source and generate a 120×43 GCC feature matrix, where 120 represents the number of microphone pairs and 43 represents the dimension of the delay characteristics. The generated GCC matrix contains the delay information between different microphone pairs as the input feature for sound source localization. Finally, the Softmax classifier classifies the feature vector and outputs the specific position of the sound source in the grid, thereby realizing sound source localization.

[0044] 1. Generalized Cross-Correlation Calculation (GCC): Calculate the generalized cross-correlation matrix between each pair of microphones in the microphone array to obtain the time delay information of the sound source.

[0045] Assuming that the sound source position of the microphone array has spatial distribution characteristics, for the microphone pair (i, j), the sound signal received by each microphone can be expressed as x i (t) and x j (t), the generalized cross-correlation function can be calculated by the following formula:

[0046]

[0047] Among them, X i (f) and X j * (f) are x i (t) and x j The Fourier transform of (t), x j * (f) represents x i The complex conjugate of (f); G ij (f) is a weighting function used to enhance the signal to noise ratio. Common weighting functions include phase shift (PHAT) and cross-power spectrum phase weighting (Cross-Power Spectrum Phase, CPSP); represents the imaginary unit, represents the frequency, and τ represents the time delay.

[0048] 2. Residual network (ResNet-34) feature extraction: Each generalized cross-correlation matrix is ​​regarded as a feature map, and a convolution operation is performed on it using the ResNet-34 neural network to extract the time delay features and retain the characteristic information of the sound source at different spatial positions.

[0049] The input of the ResNet-34 network is a 120×43 GCC matrix, where 120 represents all microphone pairs (a total of 16 microphones, 120 microphone pairs are obtained by selective pairing), and 43 represents the number of sample points of time delay. By calculating the generalized cross-correlation matrix for each microphone pair, the input features for sound source localization are obtained.

[0050] ResNet-34 is a convolutional neural network with a residual structure, which is used to effectively extract features and avoid the gradient vanishing problem. The main process is as follows:

[0051] 2.1. Convolutional layer: The input GCC matrix first passes through a 7×7 convolutional layer with 64 filters and a convolution operation with a stride of 2 for preliminary feature extraction;

[0052] 2.2 Residual Block: The input passes through multiple residual blocks. Each residual block consists of two convolutional layers and a skip connection to extract deep features from the input signal. The entire network contains 34 convolutional layers, and the representation ability of the network is enhanced by continuously stacking convolutional layers.

[0053] 2.3. Pooling layer: At the end of the network, global average pooling (GAP) is used to reduce the dimension of the convolutional features into a one-dimensional feature vector.

[0054] The final feature vector is represented as F∈R d , where d is the feature dimension. This feature vector contains the spatial and temporal information of the sound source location and can provide efficient representation for subsequent classifier modules.

[0055] 3. Softmax classifier classifies features: classifies the feature information processed by the neural network to determine the specific location of the sound source.

[0056] In order to determine the specific location of the sound source, this embodiment uses a Softmax classifier to classify the features. The input of the classifier module is the feature vector extracted by ResNet-34. The Softmax classifier is a neural network layer used for multi-classification tasks. It realizes the classification of the sound source location by calculating the probability distribution of the input features in each category. The output of the classifier is expressed as:

[0057]

[0058] Where P(y=k|F) represents the probability that the input feature F belongs to category k; W k and b k are the weight and bias parameters of category k respectively; K is the total number of all categories, corresponding to the number of candidate locations of the sound source; y is the category label, indicating the target category of the current classification; F is the input feature vector, indicating the features extracted from the data; W j and b j are the weight and bias parameters of category j respectively; represents the base of the natural logarithm, used to calculate the exponent; represents the transpose operation.

[0059] Finally, the Softmax classifier outputs a probability distribution of size K, which indicates the probability of the sound source being located at different grid positions. The location of the sound source is finally determined by selecting the category corresponding to the maximum probability.

[0060] In the above system embodiment, each step and each unit included are only divided according to functional logic and are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the naming of each unit is only for the convenience of distinction and is not used to limit the scope of protection of the present invention.

[0061] The above contents are merely examples and explanations of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they shall all fall within the protection scope of the present invention.

Claims

1. A low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network, characterized in that: include: A microphone array is used to collect the time domain signal of the sound source; A generalized cross-correlation calculation unit, used for calculating a generalized cross-correlation matrix between each pair of microphones in the microphone array; A feature extraction unit, used for inputting the generalized cross-correlation matrix into the ResNet-34 neural network for feature extraction; The classification unit is used to classify the extracted features to determine the location of the sound source.

2. The low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network according to claim 1, characterized in that: The number of array elements of the microphone array is 16. The generalized cross-correlation calculation unit calculates the signals of the 16 microphone arrays to obtain a 120×43 feature matrix containing delay information, wherein 120 is the number of microphone pairs and 43 is the delay feature dimension.

3. The low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network according to claim 1, characterized in that: The generalized cross-correlation calculation unit obtains a generalized cross-correlation matrix by performing Fourier transform on the time domain signal of the microphone pair and performing phase transform on the spectrum thereof, thereby extracting the time delay feature of the sound source.

4. The low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network according to claim 1, characterized in that: The feature extraction unit uses a ResNet-34 neural network to perform layer-by-layer convolution operations on the GCC feature map to extract delay features; When the feature extraction unit performs a convolution operation on each GCC matrix, a 1×1 convolution kernel is used.

5. The low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network according to claim 1, characterized in that: In the process of sound source localization, the sound source plane is a 3×3 square grid, the size of the plane is 2×2 meters, the resolution is 1 meter, the number of microphone array elements is 16, and the sound source position is determined by classifying the GCC matrices of different microphone pairs.

6. The low signal-to-noise ratio sound source localization system based on the combination of GCC and neural network according to claim 5, characterized in that: The classification unit classifies the extracted features through a Softmax layer to determine the specific location of the sound source in a 3×3 square grid; The classification unit processes the features extracted by ResNet-34 through a fully connected layer and uses a Softmax activation function to output the probability distribution of the sound source position.

7. A low signal-to-noise ratio sound source localization method based on the combination of GCC and neural network, characterized in that: The following steps are involved: (1) Calculate the generalized cross-correlation matrix between each pair of microphones in the microphone array to obtain the time delay information of the sound source; (2) Each generalized cross-correlation matrix is ​​regarded as a feature map, and a ResNet-34 neural network is used to perform convolution operations on it to extract time delay features and retain the characteristic information of the sound source at different spatial positions; (3) Classify the feature information processed by the neural network and determine the specific location of the sound source.

8. The low signal-to-noise ratio sound source localization method based on the combination of GCC and neural network according to claim 7, characterized in that: The microphone array is an array of 16 microphones, which can locate the sound source position in a 3×3 grid.

9. The low signal-to-noise ratio sound source localization method based on the combination of GCC and neural network according to claim 7, characterized in that: In step (2), the ResNet-34 neural network includes multiple convolutional layers and residual modules, which are used to gradually extract the delay information in the GCC feature map and retain the spatial distribution characteristics.

10. The low signal-to-noise ratio sound source localization method based on the combination of GCC and neural network according to claim 7, characterized in that: In step (2), the input of the ResNet-34 neural network is a 120×43 feature matrix, which undergoes multi-layer convolution and pooling operations and finally outputs a one-dimensional feature vector for classification.

Images