A method, system, device and medium for direction of arrival estimation

Abstract

This invention relates to the field of deep learning technology and discloses a direction-of-arrival (DOA) estimation method, system, device, and medium. The method includes: acquiring multi-source signals in a target environment; converting the multi-source signals into a GCC-PHAT matrix, and then converting the GCC-PHAT matrix into a GCC-PHAT matrix image, the GCC-PHAT matrix image containing time delay information of the signals acquired by each microphone; converting the multi-source signals into a covariance matrix, and then converting the covariance matrix into a covariance matrix image, the covariance matrix image containing spatial distribution information of the signals acquired by each microphone; fusing the GCC-PHAT matrix image and the covariance matrix image and inputting them into a Vision Mamba network to extract the time delay features and spatial distribution features of the multi-source signals, and estimating the DOA of the multi-source signals based on the time delay features and spatial distribution features, thereby improving the accuracy of the DOA estimation.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a direction-of-arrival estimation method, system, device, and medium. Background Technology

[0002] Direction of Arrival (DOA) estimation is a crucial research area in array signal processing, with wide applications in radar, sonar, communications, and speech processing. Its fundamental goal is to determine the azimuth angle of a sound source signal arriving at each element of the array. Accurately estimating the channel's angle of arrival from noisy background noise or strong room reverberation is one of the main tasks in DOA algorithm estimation.

[0003] With the advent of the 5G era and the rapid updates and iterations of smart products, such as smart speakers and smart home devices, the requirements for DOA estimation accuracy are becoming increasingly stringent. In real-world environments, noise types are diverse, and signal-to-noise ratios (SNRs) ranges are wide and difficult to define: there exist quiet environments with high SNR and weak reverberation, as well as complex environments with low SNR and strong reverberation. Furthermore, within a given spatial range, the propagation of multiple sound sources can cause the received signal to include the superposition of reflected reverberant waves, making the spatial correlation structure complex, causing severe spectral overlap, and resulting in mutual masking between different sound sources. This significantly increases the difficulty of the DOA estimation task.

[0004] Existing DOA estimation algorithms have shown relatively ideal performance in relatively quiet environments with few sound sources (such as inside a train car, in a quiet bedroom, or in an open field). However, they are difficult to meet the requirements for estimation accuracy and robustness in complex environments with low signal-to-noise ratio, strong reverberation, and multiple sound sources, such as workshops, restaurants, and KTVs. Therefore, traditional DOA estimation algorithms perform poorly when directly applied in complex environments. Summary of the Invention

[0005] The purpose of this invention is to provide a direction of arrival (DOA) estimation method, system, device, and medium that can solve the problem of poor DOA estimation performance in complex environments with low signal-to-noise ratio, strong reverberation, and multiple sound sources.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a direction-of-arrival estimation method, comprising the following steps:

[0007] Multi-source signals from the target environment are acquired using a circular microphone array consisting of multiple microphones;

[0008] The multi-source signals are converted into a generalized cross-correlation-phase transform (GCC-PHAT) matrix, and then the GCC-PHAT matrix is ​​converted into a GCC-PHAT matrix image; wherein, the GCC-PHAT matrix image contains the time delay information of the signals acquired by each microphone;

[0009] The multi-source signals are converted into a covariance matrix, and then the covariance matrix is ​​converted into a covariance matrix image; wherein, the covariance matrix image contains the spatial distribution information of the signals acquired by each microphone;

[0010] The GCC-PHAT matrix image and the covariance matrix image are fused and then input into the Vision Mamba network. The time delay features and spatial distribution features of the multi-source signals are extracted from the fused GCC-PHAT matrix image and the covariance matrix image.

[0011] Based on the time delay characteristics and spatial distribution characteristics, the direction of arrival (DOA) of multi-source signals is estimated.

[0012] Optionally, the conversion of multi-source signals into a generalized cross-correlation-phase transform (GCC-PHAT) matrix includes:

[0013] Take any two microphones in the circular microphone array as a microphone pair. For each microphone pair, perform Fourier transform on the sound source signals obtained by the two microphones respectively, and then calculate the cross power spectrum.

[0014] Amplitude information in the cross-power spectrum is removed by normalization;

[0015] The inverse Fourier transform of the normalized cross-power spectrum yields the generalized cross-correlation function of the microphone pair in the time domain.

[0016] The GCC-PHAT matrix is ​​constructed based on the generalized cross-correlation function of all microphone pairs.

[0017] Optionally, the conversion of the GCC-PHAT matrix into a GCC-PHAT matrix image includes:

[0018] Obtain the generalized cross-correlation function of any microphone in the GCC-PHAT matrix that forms a microphone pair with all other microphones, and use it as the target generalized cross-correlation function;

[0019] Multiple target generalized cross-correlation functions are stacked into a matrix, and the matrix is ​​converted into an image, which is called the GCC-PHAT matrix image. Each row of the GCC-PHAT matrix image represents the time delay range of a microphone pair, and each column represents the correlation of all microphone pairs under a preset time delay.

[0020] Optionally, the step of converting multi-source signals into a covariance matrix and then converting the covariance matrix into a covariance matrix image includes:

[0021] The multi-source signal is divided into multiple sub-source signals. For each sub-source signal, the inner product of the transpose matrix of the sub-source signal and the sub-source signal itself is calculated to obtain the covariance matrix.

[0022] The average covariance matrix is ​​obtained by normalizing the covariance matrix;

[0023] The average covariance matrices of multiple sound source signals are concatenated column-wise into a single matrix, and then adjusted to an RGB image to obtain the covariance matrix image.

[0024] Optionally, the step of fusing the GCC-PHAT matrix image and the covariance matrix image and inputting them into the Vision Mamba network, and extracting the time delay features and spatial distribution features of the multi-source signals from the fused GCC-PHAT matrix image and the covariance matrix image, includes:

[0025] The fused GCC-PHAT matrix image and covariance matrix image are used as the fused image. The fused image is divided into multiple two-dimensional image blocks, and each two-dimensional image block is flattened into a one-dimensional vector.

[0026] Each one-dimensional vector is mapped to an embedding vector of a preset length using a preset projection matrix, and each embedding vector is positionally encoded.

[0027] By capturing the global dependencies between all embedded vectors through a linear attention mechanism, the temporal delay features and spatial distribution features in the image are modeled and fused.

[0028] Embodiments of the present invention also provide a direction-of-arrival estimation system, comprising:

[0029] The sound source signal acquisition module is used to acquire multi-sound source signals in the target environment through a circular microphone array composed of multiple microphones;

[0030] The first signal conversion module is used to convert multi-source signals into a generalized cross-correlation-phase transform (GCC-PHAT) matrix, and then convert the GCC-PHAT matrix into a GCC-PHAT matrix image; wherein, the GCC-PHAT matrix image contains the time delay information of the signals acquired by each microphone.

[0031] The second signal conversion module is used to convert multi-source signals into a covariance matrix, and then convert the covariance matrix into a covariance matrix image; wherein, the covariance matrix image contains spatial distribution information of the signals acquired by each microphone;

[0032] The signal feature extraction module is used to fuse the GCC-PHAT matrix image and the covariance matrix image and input them into the Vision Mamba network to extract the time delay features and spatial distribution features of the multi-source signals from the fused GCC-PHAT matrix image and the covariance matrix image.

[0033] The direction-of-arrival (DOA) estimation module is used to estimate the DOA of multi-source signals based on time delay and spatial distribution characteristics.

[0034] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described direction-of-arrival estimation method.

[0035] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described direction-of-arrival estimation method.

[0036] The direction-of-arrival estimation method provided by this invention has at least the following beneficial effects:

[0037] This invention converts multi-source signals from a target environment acquired by a circular microphone array composed of multiple microphones into GCC-PHAT matrices and covariance matrices, and then converts the two matrices into corresponding images. The GCC-PHAT matrix image and the covariance matrix image respectively contain important time delay information and spatial distribution information in the direction-of-arrival (DOA) estimation problem. The GCC-PHAT matrix image and the covariance matrix image are fused and input into the Vision Mamba network, which facilitates the neural network to simultaneously learn the time delay characteristics and spatial distribution characteristics of multi-source signals. This effectively captures the complex feature information in the image, better adapts to this multimodal data (i.e., multi-source signals), and improves the accuracy of DOA estimation in complex environments and under multi-source conditions. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0039] Figure 1 A flowchart illustrating a direction-of-arrival estimation method provided by the present invention. Figure 1 ;

[0040] Figure 2 An environmental schematic diagram provided for this invention;

[0041] Figure 3 A flowchart illustrating a direction-of-arrival estimation method provided by the present invention. Figure 2 ;

[0042] Figure 4 A schematic diagram of a GCC-PHAT image provided by the present invention;

[0043] Figure 5 This is a schematic diagram of a covariance matrix image provided by the present invention;

[0044] Figure 6 A schematic diagram of a fused feature image provided by the present invention;

[0045] Figure 7 This invention provides a schematic diagram of the average accuracy results of DOA estimation for different signal-to-noise ratios in an experiment. Figure 1 ;

[0046] Figure 8 This invention provides a schematic diagram of the average F1 score results for DOA estimation at different signal-to-noise ratios in an experiment.

[0047] Figure 9 This invention provides a schematic diagram of the average accuracy results of DOA estimation at different signal-to-noise ratios in an experiment. Figure 2 ;

[0048] Figure 10 This invention provides a schematic diagram of the average recall results for DOA estimation at different signal-to-noise ratios in an experiment.

[0049] Figure 11 This invention provides a schematic diagram of the average accuracy results for DOA estimation of 2-6 sound sources in a multi-source experiment. Figure 1 ;

[0050] Figure 12 A schematic diagram of the average F1 score of DOA estimation for 2-6 sound sources in a multi-source experiment provided by the present invention;

[0051] Figure 13 This invention provides a schematic diagram of the average accuracy results for DOA estimation of 2-6 sound sources in a multi-source experiment. Figure 2 ;

[0052] Figure 14 This is a schematic diagram of the average recall results for DOA estimation of 2-6 sound sources in a multi-source experiment provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0054] Currently, traditional DOA estimation methods are mainly based on model-driven frameworks, achieving DOA estimation by establishing a mathematical model of the signal arrival array. Representative algorithms include Capon's minimum variance beamformer, time difference of arrival (TDOA) based positioning methods, and subspace algorithms such as Multiple Signal Classification (MUSIC), Estimation of Signal Parameters via Rotational Invariance Techniques (ESPRIT), and some improved MUSIC algorithms. These traditional algorithms can achieve high-accuracy estimation under ideal conditions and approach the theoretical performance limit.

[0055] However, traditional algorithms are highly sensitive to signal-to-noise ratio (SNR) and the number of sound sources, performing well only when the SNR is high and the number of sound sources is small. Under unfavorable conditions such as low SNR, multiple sound sources, source coherence, or model mismatch, estimation performance deteriorates significantly, and computational complexity increases significantly. Furthermore, subspace-based algorithms involve complex matrix decomposition, resulting in high computational complexity, and accuracy is further reduced when non-ideal factors such as array element coupling, amplitude and phase errors, or array element position errors exist. While many improved algorithms have enhanced the accuracy of direction-of-arrival (DOA) estimation, they still cannot solve the problem of multi-source localization in reverberant environments.

[0056] In recent years, deep learning technology has been widely used in DOA estimation, employing a combination of data-driven and model-driven approaches. Common architectures include multilayer perceptrons, convolutional neural networks, traditional algorithms, and joint deep learning methods. These methods demonstrate good performance in scenarios with low signal-to-noise ratios, a small number of sound sources, and the processing of coherent sources, with some implementations also exhibiting low computational complexity.

[0057] However, deep learning methods require a large amount of training data, the training process is time-consuming, and the generalization ability of the model is limited by the diversity of the training data. When the signal-to-noise ratio is too low, the noise power will be much greater than the signal power, the signal characteristics will be masked by noise, making it difficult to learn the key signal characteristics, and too many sound sources will lead to unstable performance in complex or unknown environments. In addition, overfitting is prone to occur in high-dimensional state spaces, limiting its effectiveness in some practical applications. Therefore, improving the accuracy, robustness, and stability of direction-of-arrival estimation has become a current research focus.

[0058] This invention aims to address the problems of poor robustness and insufficient stability in DOA estimation under complex noise environments and with a large number of sound sources. Inspired by fused feature images, a deep learning-based DOA estimation method for a circular microphone array is proposed. Because of the data imbalance between angle categories and angle combinations, the DOA estimation problem is first treated as a multi-label classification task by converting the received signal into a GCC-PHAT matrix image and a covariance matrix image. Then, the weighted fused feature image of the two types of images is used as input and fed into a Vision-Mamba network. The bidirectional state-space model and attention mechanism of the ViM network are used to capture the rich temporal and spatial features of the fused image. By focusing on the effective information regions in the image, the nonlinear mapping relationship between the data input and the label is learned, resulting in a multi-hot encoded label classification of angle combinations, thus achieving high-precision DOA estimation under low signal-to-noise ratio and multiple sound sources.

[0059] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0060] One embodiment of the present invention relates to a direction-of-arrival (DOA) estimation method. The specific flow of the DOA estimation method in this embodiment can be as follows: Figure 1 As shown, it includes:

[0061] Step 101: Acquire multi-source signals in the target environment using a circular microphone array consisting of multiple microphones.

[0062] Step 102: Convert the multi-source signals into a generalized cross-correlation-phase transform (GCC-PHAT) matrix, and then convert the GCC-PHAT matrix into a GCC-PHAT matrix image; wherein, the GCC-PHAT matrix image contains the time delay information of the signals acquired by each microphone.

[0063] Step 103: Convert the multi-source signals into a covariance matrix, and then convert the covariance matrix into a covariance matrix image; wherein, the covariance matrix image contains the spatial distribution information of the signals acquired by each microphone.

[0064] Step 104: After fusing the GCC-PHAT matrix image and the covariance matrix image, input them into the Vision Mamba network to extract the time delay features and spatial distribution features of the multi-source signals from the fused GCC-PHAT matrix image and the covariance matrix image.

[0065] Step 105: Based on the time delay characteristics and spatial distribution characteristics, estimate the direction of arrival (DOA) of the multi-source signal.

[0066] The following is a detailed description of the implementation details of the direction-of-arrival estimation method in this embodiment. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0067] Since the number of angle categories in the direction of arrival estimation is much greater than the number of angle combinations, causing data imbalance, the direction of arrival estimation is treated as a multi-class labeling task. The direction of arrival estimation for multiple signal sources is achieved by encoding the output into multi-hot encoded labels.

[0068] Specifically, traditional DOA estimation methods (such as MUSIC and ESPRIT) typically output one or more angle values. In multi-source scenarios, directly predicting multiple angles may face problems such as data imbalance (some angle combinations occur much less frequently than others) and combination explosion. Treating the problem as a multi-label classification means assigning a label to each possible angle (or angle range), and during prediction, outputting which labels are true (i.e., which angles contain sound sources).

[0069] Will Figure 2 The environment shown is the target environment of this embodiment (a closed room). The room size is set to 6 meters x 6 meters x 4 meters (length x width x height). A circular microphone array is located in the center of the room, at a height of 2 meters. It is an omnidirectional microphone with the same sensitivity to sound sources in all directions, and the reverberation time RT is set to 0.4 seconds. This invention does not involve elevation angles; therefore, the height of the sound source and the microphone array are at the same height. The target sound source will randomly appear at positions between 0 degrees and 359 degrees, that is, any position on the circumference of a circle with a radius of 2 meters from the microphone array. The number of target sound sources in this study ranges from 2 to 6. After determining the simulation environment, a clean speech dataset and a microphone array signal generator are used to generate the corresponding clean microphone array speech through convolution. Noise is generated by adding the clean microphone array speech to three typical noise types: pink noise, white noise, and multi-person overlapping speech, to obtain a mixed signal, along with the corresponding DOA angle label.

[0070] by Figure 3 The following flowchart illustrates the subsequent processing. First, the received target sound source signal (i.e., multi-source signal) is expressed as follows:

[0071] ;

[0072] In the formula, This refers to the room's impact response, including characteristics such as reverberation and attenuation. Indicates noise signal, Represents speech signals. Indicates the first The time delay of the signal received by each microphone element.

[0073] Then, the obtained signal is converted into a GCC-PHAT matrix and a covariance matrix, and then converted into a GCC-PHAT image and a covariance matrix image.

[0074] The signal is converted into a GCC-PHAT matrix: any two microphones in the circular microphone array are taken as a microphone pair. For each microphone pair, the sound source signals obtained by the two microphones are subjected to Fourier transforms and the cross power spectrum is calculated. The amplitude information in the cross power spectrum is removed by normalization. The inverse Fourier transform is performed on the normalized cross power spectrum to obtain the generalized cross-correlation function of the microphone pair in the time domain. Based on the generalized cross-correlation function of all microphone pairs, the GCC-PHAT matrix is ​​established.

[0075] In practical implementation, generalized cross-correlation has become an important algorithm for time delay estimation due to its high computational efficiency, simple model structure, good portability, ease of transfer learning, and compatibility with other signal processing tools. The microphone array consists of 8 elements. Convolving the room pulses at corresponding angles with clean speech signals and adding noise yields the signals received by the 8 elements. The signals are first normalized to prevent signal clipping distortion and ensure the accuracy of subsequent feature extraction. The time delay is calculated using the GCC-PHAT algorithm on the normalized signal matrix. Because the computation between microphone pairs is large, a simplified explanation is as follows: Assume the signals of a pair of microphone elements are represented as follows: and First, perform a Fourier transform on the two signals to obtain... and Then calculate the cross power spectrum. To obtain phase information related to time delay, the phase information is obtained by normalizing and removing the amplitude information of the cross-power spectrum. Finally, an inverse Fourier transform is performed on the phase-transformed spectrum to obtain the microphone's generalized cross-correlation function in the time domain:

[0076] ;

[0077] In the formula, This represents the cross-power spectrum after PHAT weighting. This represents the inverse Fourier transform. This represents the generalized cross-correlation function.

[0078] Convert the matrix into the corresponding GCC-PHAT image (e.g.) Figure 4 (See the image shown): Obtain the generalized cross-correlation function of any microphone in the GCC-PHAT matrix and the other microphones to form a microphone pair, as the target generalized cross-correlation function; stack multiple target generalized cross-correlation functions into a matrix, and convert the matrix into an image, as the GCC-PHAT matrix image; where each row of the GCC-PHAT matrix image represents the time delay range of a microphone pair, and each column represents the correlation of all microphone pairs at a specific time delay.

[0079] In the specific implementation, for an array of 8 microphones, we select the signal between the first microphone and each of the other microphones (i.e., microphones 2 to 8, a total of 7 pairs), and use the GCC-PHAT algorithm to calculate the generalized cross-correlation function of each microphone pair. (i=2,3,...,8), thus extracting the time delay features associated with the first microphone. Each GCC-PHAT function is a time-domain vector of length N (the number of time delay sampling points), representing the microphone pair at different time delays. The correlation is calculated by stacking the seven GCC-PHAT functions into a 7×N matrix, where each row corresponds to the time delay feature of a microphone pair. To feed these time delay features into a convolutional neural network (CNN), this matrix is ​​transformed into a 7×N image, where the intensity of each pixel corresponds to the value of the GCC-PHAT function. Each row of the image represents the time delay range of a microphone pair, and each column represents the correlation of all microphone pairs at a preset time delay. Images generated in this way are suitable for input into a CNN to learn sound source direction.

[0080] The signal is converted into a covariance matrix, and then into the corresponding covariance matrix image (e.g., Figure 5 (See the image shown): The multi-source signal is divided into multiple sub-source signals. For each sub-source signal, the inner product of the transpose matrix of the sub-source signal and the sub-source signal itself is calculated to obtain the covariance matrix. The average covariance matrix is ​​obtained by normalizing the covariance matrix. The average covariance matrices of the multiple sub-source signals are concatenated column-wise into a matrix, and then adjusted to an RGB image to obtain the covariance matrix image.

[0081] In practical implementation, the covariance matrix represents the spatial correlation of the signal over time, which helps in analyzing the dynamic characteristics of the sound source and reflects the correlation between channels in the microphone array. This is crucial for determining the direction of the sound source. First, the number of segments T into which the signal will be divided is determined. Then, the received signal matrix X is transposed... Calculate the inner product with itself. The covariance matrix R is obtained, and then the average covariance matrix can be obtained by normalization. Because segmentation is required, the covariance matrix of each segment can be similarly represented as:

[0082] ;

[0083] In the formula, L represents the number of snapshots. To convert it into a covariance matrix image, T covariance matrices are used. The columns are concatenated into a matrix, and finally adjusted to an RGB image. It is suitable for input into a convolutional neural network (CNN) to learn the direction of the sound source.

[0084] After obtaining the GCC-PHAT image and the covariance matrix image, the GCC-PHAT image and the covariance matrix image are fused at the pixel level to generate a fused feature image, such as... Figure 6 As shown. The fused feature image contains the temporal delay characteristics and spatial feature information required for DOA estimation, containing rich information. This method is no longer limited to calculating only one type of feature information, but can more comprehensively capture information related to DOA estimation, which is more advantageous for DOA estimation. The fusion process can be represented as:

[0085] ;

[0086] In the formula, gcc_img refers to the GCC-PHAT image, and cov_img refers to the covariance matrix image.

[0087] Finally, the obtained feature image samples are divided into training, validation, and test sets in an 8:1:1 ratio and fed into the Vision Mamba network for training.

[0088] Figure 3The lower half of the framework is the specific implementation of the Vision Mamba network. To handle visual pattern processing tasks, Vision Mamba first flattens the fused feature image into two-dimensional image patches. Next, these patches are input into a linear projection layer to project a vector of length D, embedding tokens representing the class and location within the vector. This token sequence is then fed into the Vision Mamba Encoder. Finally, the output class labels are normalized and input into the head of a multilayer perceptron (MLP) to obtain the final prediction. Because the number of training samples is relatively small, overfitting may occur during network training. Therefore, the number of epochs is set to 100 during training, and an early stopping mechanism is introduced. Training stops if the F1 score does not show significant improvement within 20 epochs, reducing unnecessary resource consumption. Choosing an appropriate batch size affects the model's training speed and resource consumption; too small a batch size hinders convergence, while too large a batch size consumes a large amount of memory. Here, the batch size is set to 64. Because of the small sample dataset, all training does not use transfer learning and starts from scratch. The optimizer selected is the Adam optimizer with decoupled weightdecay regularization (AdamW) to accelerate convergence.

[0089] Step 1: Flatten the obtained fused feature image into individual two-dimensional image patches. The network input is the fused feature image. Its size is Where H is the height of the input image, W is the width of the input image, and 3 represents the number of channels in the image. Image I is divided into a series of two-dimensional image patches, each patch having a size of... The image is segmented into... Patches. Then each The patch is flattened into a one-dimensional vector with a length of Arrange all the flattened patch vectors in order to form a sequence:

[0090] ;

[0091] In the formula, It represents the number of patches, i.e., the sequence length.

[0092] The second step involves inputting the 2D image patch into a linear projection layer to project it into a vector of length D, embedding tokens representing the category and location into the vector. First, linear projection is performed, flattening each patch vector... With learnable projection matrix Mapping to an embedding vector of length D:

[0093] ;

[0094] In the formula, D is the embedding dimension, which is used to control the representational power of the feature vector.

[0095] Next, a class token representing the category is added before the patch embedding sequence. This is used to represent the entire image patch sequence, storing the global features corresponding to all patches. Then, a learnable position embedding matrix is ​​used. Add corresponding location information to each token, as shown below:

[0096] ;

[0097] At this point, Z saves and aggregates the features corresponding to all patches, preparing for subsequent encoder processing.

[0098] The third step involves inputting the token sequence into the Vision Mamba Encoder to obtain the output. The Vision Mamba Encoder consists of multiple BasicLayers, each containing several MLLABlocks (Multi-Layer Linear Attention Blocks) for feature extraction. Each MLLABlock specifically includes the following components:

[0099] 1) Linear Attention: It captures the global dependencies between tokens through the linear attention mechanism, and efficiently models the temporal and spatial distribution features in the image.

[0100] (2) RoPE (Rotary Positional Embedding): Enhances the perception of positional information by rotating positional encoding, and the main network learns the spatial relationships in the image.

[0101] (3) LePE (Locality-enhanced Positional Encoding): Introduces locality through convolution operations to further extract spatial distribution features.

[0102] (4) MLP (Multilayer Perceptron): BasicLayer's MLP serves as the feed-forward network (FFN) of MLLABlock, performing nonlinear transformations and fusions on the extracted features to improve representation capabilities.

[0103] Multiple BasicLayers work together by using patch merging layers to reduce the spatial resolution of the feature maps while increasing the number of channels to extract higher-level features. The GCC-PHAT portion of the fused feature image contains temporal delay information, which is learned through a linear attention network to correlate these patterns with the sound source direction. The covariance matrix reflects the spatial correlation of the signal, and LePE and convolution operations in MLLABlock capture spatial patterns. After processing through multiple BasicLayers, the feature sequence is obtained. .

[0104] The fourth step is to normalize the output category labels and input them into the MLP of the classification head for mapping to obtain the final prediction. First, extract the output. The class token that gathers global characteristics .right LayerNorm normalization is performed to ensure feature stability. The normalized class token is then fed into the MLP and mapped to a 360-dimensional output vector. :

[0105] ;

[0106] Finally, the sigmoid activation function is used to... Convert to probability, prediction angle A correct prediction is represented by 1, and an incorrect prediction by 0. Therefore .

[0107] The direction-of-arrival (DOA) estimation method of this embodiment will be experimentally verified below, which includes two parts: studying the influence of signal-to-noise ratio on the accuracy of DOA estimation and the influence of the number of sound sources on the accuracy of DOA estimation.

[0108] The experiment was conducted on an Ubuntu 22.04 system, driver version 570.144, CUDA version 12.8, NVIDIA GeForce RTX 4090 graphics card, PyCharm 2025.1 development environment with Python 3.11.8, based on the PyTorch deep learning framework. The experiment compared and contrasted the modified Inception, DenseNet, ResNet50, Vision Transformer, and the main deep learning network of this invention, Vision Mamba. Graphics processing unit (GPU) acceleration was employed, which significantly improved the model training efficiency. During the simulated signal generation, the sound propagation speed of the target sound source was set to 343 m / s, and the carrier signal frequency was set to 1000 Hz.

[0109] The first part of the study investigates the impact of signal-to-noise ratio (SNR) on the accuracy of DOA estimation. The SNR in this experiment was set to -15 dB to 10 dB with a step size of 5 dB. For each number of sound sources, five fixed angle combinations were randomly generated for all SNRs. While the angle combinations were randomly generated, they were spaced 10° apart to prevent repetition. For each angle combination at each SNR, 500 samples were generated for each number of sound sources, for a total of 15,000 samples. These samples were then divided into a training set of 12,000 mixed samples, a validation set of 1,500 mixed samples, and a test set in an 8:1:1 ratio. DOA estimation experiments were conducted with all other parameters identical except for the SNR. By mixing data from different SNRs (including low SNRs) to train the model, the robustness of the model under various noise conditions can be better observed. Figure 7 As shown, experimental results indicate that the ViM network achieves relatively low accuracies of 56% and 65% at SNRs of -15 dB and -10 dB, respectively. However, the ViM network's accuracy is 5% and 7% higher than ResNet50 (51% and 58%), respectively; 9% and 16% higher than ViT (Vision Transformer); 9% and 12% higher than DenseNet; and 11% and 12% higher than Inception. Despite the overall poor performance of the ViM network, it still exhibits certain advantages compared to other networks, indicating relatively strong robustness in high-noise environments. However, under low SNR conditions, the model struggles to completely eliminate the interference of noise patterns, inhibiting its learning ability and limiting overall performance. As SNR increases, the impact of noise gradually weakens, and all models can more effectively learn feature information from the DOA angle, thus significantly improving accuracy. Specifically, when the SNR increased from -10 dB to -5 dB, the accuracy of the ViM network improved from 65% to 86%, an increase of 21%; while the accuracy of ResNet50, ViT, DenseNet, and Inception improved by 18%, 17%, 19%, and 13%, respectively. This indicates that with reduced noise, the ViM network can learn the time delay and spatial characteristics of angle combinations under different SNRs more quickly and effectively, and begins to effectively utilize signal features, giving it an advantage over other networks in signal feature utilization.

[0110] When SNR ≥ 0 dB, meaning the signal power equals or exceeds the noise power, the performance of all five networks shows a significant improvement. At SNR of 0 dB, the ViM network achieves an accuracy of 97%, approaching its performance limit. In comparison, ResNet50 achieves 92% and 97% accuracy at SNR of 0 dB and 5 dB, respectively. This means that the ViM network achieves the performance of ResNet50 at SNR = 5 dB at SNR = 0 dB, equivalent to a performance improvement of approximately 5 dB. Similarly, ViT achieves 87%, 94%, and 97% accuracy at SNR of 0 dB, 5 dB, and 10 dB, respectively. The ViM network's performance at SNR = 0 dB is close to that of ViT at SNR = 10 dB, representing a performance improvement of nearly 10 dB. The performance improvement is even greater for DenseNet. Furthermore, the ViM network's accuracy improves faster, approaching 100% at SNR = 5 dB, demonstrating a more significant performance advantage compared to other networks. This indicates that ViM networks are better at processing complex temporal and spatial information in fused feature images, thus exhibiting stronger adaptability when signal features are significant, and are more suitable for processing images containing rich and complex information, such as fused feature images.

[0111] Similarly, Figure 8 The F1 score curve also shows that the ViM network can more effectively learn and capture the temporal and spatial characteristics of the DOA angle when processing fused feature images. Since the F1 score combines precision and recall, its performance can be further understood by analyzing these two metrics. Figure 9 As shown, the ViM network achieves higher accuracy than other networks, indicating that it has a higher proportion of truly positive samples among those predicted as positive, meaning fewer false positives (FP). Meanwhile, Figure 10 The ViM network also exhibits high recall, indicating that it can detect most true positive samples with few false negatives (FN). In DOA estimation, the ViM network demonstrates both accurate prediction (high precision) and comprehensive detection (high recall), thus its F1 score and accuracy significantly outperform the other four networks. This characteristic may stem from the ViM network's sensitivity to key information in fused feature images, making it more competitive in complex data processing under different SNR conditions.

[0112] In summary, Figures 7 to 10This study demonstrates the trends of accuracy, precision, recall, and F1 score with SNR when the number of target sound sources is 2 to 6 and the DOA angle combinations are fixed. Experimental results show that model performance varies significantly with SNR. Since the DOA angle combinations are fixed, the noise patterns may also be relatively fixed. This could lead the model to overfit these fixed noise patterns at low SNR, rather than truly learning the ability to extract signal features from arbitrary angles. Therefore, the model performs poorly under low SNR conditions. Figure 7 and Figure 8 This is reflected in the accuracy and F1 score curves. It is evident that a fixed combination of angles may limit the model's ability to generalize signal features.

[0113] The second part investigates the impact of the number of sound sources on the accuracy of DOA estimation. In this experiment, the signal-to-noise ratio (SNR) was set from -15 dB to 10 dB with a step size of 5 dB. For each number of sound sources, five random angle combinations were generated at different SNRs. These angle combinations were also randomly generated, but with a 10° interval between angles to prevent repetition. Each angle combination generated 1000 samples, for a total of 30,000 samples. These samples were then divided into a training set of 24,000 mixed samples and a validation set of 3,000 mixed samples in an 8:1:1 ratio. DOA estimation experiments were conducted with all other parameters identical except for the SNR. By mixing data from different SNRs to train the model, the performance and robustness of the model under various numbers of sound sources and the influence of noise can be better observed. Figures 11 to 14 The experiment demonstrates the trends of accuracy, precision, recall, and F1 score as a function of SNR when the number of target sound sources is 2 to 6, and the DOA angle combinations are randomized with intervals of 10°. Experimental results show that the model performance varies significantly with the number of sound sources and SNR. Because the DOA angle combinations are randomized, the image samples contain diverse feature information, allowing the model to learn richer image feature information under different numbers of sound sources. Therefore, the ViM network exhibits a clear performance advantage.

[0114] like Figure 11As shown, when the number of sound sources is 2, the ViM network achieves an accuracy of 99%, significantly outperforming other networks by 3%, 8%, 7%, and 11% respectively compared to ResNet50 (96%), DenseNet (91%), Inception (92%), and ViT (88%). This indicates that the ViM network approaches its performance limit when the number of sound sources is small, demonstrating excellent feature learning capabilities. As the number of sound sources increases from 2 to 6, the complexity of the feature information contained in the image increases, making it more difficult for all networks to capture key features, resulting in a decline in performance. However, the accuracy of the ViM network decreases gradually, from 99% to 94%, a drop of only 5%, far less than the decreases of ResNet50 (14%), DenseNet (13%), Inception (19%), and ViT (19%). This gradual decline highlights the stability of the ViM network and its advantages in processing complex feature images. The increased number of sound sources leads to a higher probability of overlapping angle combinations and higher similarity between feature images, making it difficult for the network to learn general angle features. However, the ViM network, with its strong adaptability, effectively learns the temporal and spatial characteristics of angle combinations under different numbers of sound sources, maintaining relatively stable performance.

[0115] Similarly, Figure 12 The F1 score curve further validates the superior performance of the ViM network. As the number of sound sources increases from 2 to 6, the ViM network's F1 score only decreases from 99% to 96%, a drop of only 3%, significantly better than the larger drops seen in other networks such as ResNet50 (8%), DenseNet (11%), Inception (15%), and ViT (8%). The F1 score combines precision and recall performance, and therefore can be used to... Figure 13 and Figure 14 Further analysis is needed. Figure 13 This indicates that the ViM network has a higher accuracy than other networks, meaning that it has a higher proportion of correct samples among those predicted as positive and fewer false positives (FP). Figure 14 The results show that the ViM network also has a high recall rate, capable of detecting most true positive samples with few false negatives (FN). Overall, the ViM network demonstrates both accurate prediction (high precision) and comprehensive detection (high recall) in DOA estimation, resulting in significantly better F1 scores and accuracy compared to ResNet50, DenseNet, Inception, and ViT. This reflects its sensitivity to temporal and spatial information in fused feature images and its powerful feature capture capabilities.

[0116] In summary, the ViM network demonstrates stable performance under varying numbers of sound sources and different SNR conditions. Particularly when the number of sound sources increases, its performance degradation is significantly less than that of other networks. This advantage stems from its effective learning ability of key information in complex feature images, maintaining high robustness and adaptability even under challenges such as angle overlap and increased feature similarity. Therefore, in the DOA estimation task, the ViM network exhibits outstanding capabilities in handling multi-source scenes, providing a superior solution for image processing tasks involving the fusion of feature images, which contain rich and complex information.

[0117] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0118] Another embodiment of the present invention relates to a direction-of-arrival (DOA) estimation system. The implementation details of this DOA estimation system are described below. The following details are provided for ease of understanding and are not essential for implementing this solution. The DOA estimation system of this embodiment includes:

[0119] The sound source signal acquisition module is used to acquire multi-sound source signals in the target environment through a circular microphone array composed of multiple microphones;

[0120] The first signal conversion module is used to convert multi-source signals into a generalized cross-correlation-phase transform (GCC-PHAT) matrix, and then convert the GCC-PHAT matrix into a GCC-PHAT matrix image; wherein, the GCC-PHAT matrix image contains time delay information between signals acquired by each microphone.

[0121] The second signal conversion module is used to convert multi-source signals into a covariance matrix, and then convert the covariance matrix into a covariance matrix image; wherein, the covariance matrix image contains spatial distribution information of the signals acquired by each microphone;

[0122] The signal feature extraction module is used to fuse the GCC-PHAT matrix image and the covariance matrix image and input them into the Vision Mamba network to extract the time delay features and spatial distribution features of the multi-source signals from the fused GCC-PHAT matrix image and the covariance matrix image.

[0123] The direction-of-arrival (DOA) estimation module is used to estimate the DOA of multi-source signals based on time delay and spatial distribution characteristics.

[0124] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0125] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0126] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the XX methods in the above embodiments.

[0127] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0128] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0129] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0130] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method of direction of arrival estimation, characterized by, The method comprises: acquiring a multi-source signal in a target environment through a circular microphone array composed of multiple microphones; converting the multi-source signal into a generalized cross-correlation-phase transform (GCC-PHAT) matrix, and then converting the GCC-PHAT matrix into a GCC-PHAT matrix image; wherein the GCC-PHAT matrix image contains time delay information of signals acquired by each microphone; converting the multi-source signal into a covariance matrix, and then converting the covariance matrix into a covariance matrix image; wherein the covariance matrix image contains spatial distribution information of signals acquired by each microphone; inputting the fused GCC-PHAT matrix image and covariance matrix image into a Vision Mamba network, and extracting time delay features and spatial distribution features of the multi-source signal from the fused GCC-PHAT matrix image and covariance matrix image; performing direction of arrival estimation on the multi-source signal according to the time delay features and spatial distribution features.

2. The direction of arrival estimation method of claim 1, wherein, The conversion of the multi-source signal into the GCC-PHAT matrix comprises: taking any two microphones in the circular microphone array as a microphone pair, and for each microphone pair, calculating the cross-power spectrum after performing Fourier transform on the sound source signals obtained by the two microphones; removing the amplitude information in the cross-power spectrum through normalization; performing inverse Fourier transform on the normalized cross-power spectrum to obtain the generalized cross-correlation function of the microphone pair in the time domain; establishing the GCC-PHAT matrix according to the generalized cross-correlation functions of all microphone pairs.

3. The direction of arrival estimation method of claim 2, wherein, The conversion of the GCC-PHAT matrix into the GCC-PHAT matrix image comprises: obtaining the generalized cross-correlation function of a microphone pair composed of any one microphone and all other microphones in the GCC-PHAT matrix as a target generalized cross-correlation function; stacking multiple target generalized cross-correlation functions into a matrix, and converting the matrix into an image as the GCC-PHAT matrix image; wherein each row of the GCC-PHAT matrix image represents the time delay range of a microphone pair, and each column represents the correlation of all microphone pairs at a preset time delay.

4. The method of direction of arrival estimation according to claim 1, characterized in that, The conversion of the multi-source signal into the covariance matrix, and then the conversion of the covariance matrix into the covariance matrix image, comprises: segmenting the multi-source signal into multiple sub-source signals, and for each sub-source signal, performing inner product calculation on the transpose matrix of the sub-source signal and the sub-source signal itself to obtain a covariance matrix; obtaining an average covariance matrix through normalization of the covariance matrix; splicing the average covariance matrices of the multiple sub-source signals into a matrix according to columns, and then adjusting the matrix into an RGB image to obtain the covariance matrix image.

5. The method of direction of arrival estimation according to claim 1, characterized in that, The inputting of the fused GCC-PHAT matrix image and covariance matrix image into the Vision Mamba network, and the extraction of the time delay features and spatial distribution features of the multi-source signal from the fused GCC-PHAT matrix image and covariance matrix image, comprise: The fused GCC-PHAT matrix image and the covariance matrix image are taken as a fused image, the fused image is divided into a plurality of two-dimensional image blocks, and each two-dimensional image block is flattened into a one-dimensional vector; Each one-dimensional vector is mapped into an embedding vector of a preset length through a preset projection matrix, and each embedding vector is position encoded; A linear attention mechanism is used to capture global dependencies between all embedding vectors, and time delay features and spatial distribution features in the fused image are modeled.

6. A system for direction of arrival estimation, characterized by The system comprises: A sound source signal acquisition module configured to acquire multiple sound source signals in a target environment through a circular microphone array composed of multiple microphones; A first signal conversion module configured to convert the multiple sound source signals into a generalized cross-correlation-phase transform (GCC-PHAT) matrix, and then convert the GCC-PHAT matrix into a GCC-PHAT matrix image; wherein the GCC-PHAT matrix image contains time delay information of signals acquired by each microphone; A second signal conversion module configured to convert the multiple sound source signals into a covariance matrix, and then convert the covariance matrix into a covariance matrix image; wherein the covariance matrix image contains spatial distribution information of signals acquired by each microphone; A signal feature extraction module configured to input the fused GCC-PHAT matrix image and the covariance matrix image into a VisionMamba network, and extract time delay features and spatial distribution features of the multiple sound source signals from the fused GCC-PHAT matrix image and the covariance matrix image; A direction of arrival estimation module configured to estimate the direction of arrival of the multiple sound source signals according to the time delay features and the spatial distribution features.

7. A computer device, comprising: It comprises: At least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the direction of arrival estimation method of any one of claims 1 to 5.

8. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the direction of arrival estimation method of any one of claims 1 to 5. The computer program is executed by the processor to implement the direction of arrival estimation method of any one of claims 1 to 5.

Images