Microphone array sound source localization method and system based on cross-correlation-beam forming closed-loop optimization

Abstract

The invention relates to a microphone array sound source positioning method and system based on cross-correlation-beam forming closed-loop optimization, and belongs to the technical field of sound source positioning. The method comprises the following steps: collecting multichannel sound signals through a microphone array and preprocessing the multichannel sound signals to extract time-frequency features and suppress noise interference; time delay information among the microphones is estimated by adopting a generalized cross-correlation phase transformation algorithm, and an optimization strategy is introduced to improve estimation stability and anti-interference performance; enhancing the target sound source signal in combination with a minimum variance undistorted response beam forming algorithm and an adaptive Kalman filtering mechanism; constructing a closed-loop feedback optimization mechanism based on the beam output signal to realize feedback adjustment; and adopting a hybrid network architecture, taking the beam output signal amplitude spectrum as input, and outputting the frequency spectrum or mask of the obtained target sound source signal. The method has the advantages of high calculation efficiency, high positioning precision and strong anti-interference capability, and is suitable for real-time acoustic signal processing in a complex environment.

Description

Technical Field

[0001] This invention belongs to the field of sound source localization technology, and relates to a microphone array sound source localization method and system based on cross-correlation time delay estimation and adaptive beamforming closed-loop optimization. Background Technology

[0002] In recent years, with the rapid development of voice interaction technology, especially its widespread application in scenarios such as in-vehicle voice control, smart homes, and human-computer interaction systems, increasingly higher demands have been placed on the accuracy and robustness of sound source localization. Microphone arrays, as a crucial hardware foundation for sound source localization, directly impact the localization performance of the entire voice perception system.

[0003] Existing technologies for spatial localization of sound sources mainly fall into two categories: The first category is sound source localization methods based on time difference estimation. For example, algorithms such as generalized cross-correlation (GCC) and power spectrum phase cross-interference (PHAT) are used to estimate the time delay of sound waves arriving at each microphone, and then the spatial direction of the sound source is deduced. This type of method can achieve high localization accuracy in ideal environments, but it is prone to error accumulation in strong noise, multipath reverberation, or dynamic environments, leading to inaccurate localization.

[0004] The second category is sound source enhancement methods based on beamforming, such as the Minimum Variance Distortionless Response (MVDR) beamforming algorithm and the Delay Summation (DS) method. These methods effectively suppress sound from non-target directions by weighted combining signals collected from multiple microphones, thereby enhancing the sound source signal from a specific direction. However, because beamwidth and direction are highly dependent on array structure and algorithm parameters, existing systems have poor adaptability in complex and rapidly changing environments.

[0005] Furthermore, existing research attempts to fuse multiple algorithms, such as combining GCC-PHAT with beamforming results for multi-channel information fusion, or utilizing neural networks to assist in direction estimation. However, these methods still have the following problems: The system lacks an adaptive feedback mechanism and cannot dynamically adjust the time delay estimation and beam weight according to changes in the sound source location, noise fluctuations, or environmental reverberation, resulting in insufficient positioning accuracy and robustness. The computational complexity is high, making it difficult to meet real-time requirements, especially under multi-microphone arrays or high sampling rates; In complex and dynamic environments, such as inside vehicles, industrial plants, or conference settings, the system has limited ability to suppress noise interference and multipath echoes, resulting in significant performance fluctuations.

[0006] Therefore, how to implement a microphone array sound source localization method with low complexity, high accuracy, and adaptive adjustment capability in complex dynamic environments is an urgent problem to be solved. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a microphone array sound source localization method and system based on cross-correlation-beamforming closed-loop optimization, which aims to improve the real-time performance, accuracy and robustness of sound source localization in complex acoustic environments, and solve the problems of insufficient accuracy and poor adaptability of existing sound source localization methods in noise interference, reverberation scenarios and dynamic changing environments.

[0008] To achieve the above objectives, the present invention provides a microphone array sound source localization method based on cross-correlation-beamforming closed-loop optimization, the method comprising: Multi-channel acoustic signals are acquired using a microphone array and preprocessed to extract time-frequency features and suppress noise interference. The generalized cross-correlation phase transform algorithm is used to estimate the time delay information between microphones, and an optimization strategy is introduced to improve the stability and anti-interference ability of the generalized cross-correlation phase transform algorithm. Based on the estimated time delay information, the minimum variance distortionless response beamforming algorithm is used to calculate the beam weight and enhance the target sound source signal. During the calculation process, the beam weight is dynamically optimized by combining the adaptive Kalman filtering mechanism. Using the beam output signal as feedback input, a closed-loop feedback optimization mechanism is formed by combining a particle filter and evaluation index to adjust the time delay estimation and beam weight. A hybrid network of CNN-Transformer-GRU is employed, using the short-time Fourier transform amplitude spectrum of the beam output signal as input to the hybrid network, and the output yields the spectrum or mask of the target sound source signal.

[0009] Furthermore, for all input channel signals, the time spectrum of each frame is obtained through pre-emphasis and short-time Fourier transform:

[0010] In the formula, For the time series of the i-th channel, For window functions, For frame shift, Frame length; t This is a time frame identifier variable. f For frequency variables, n For discrete-time variables, bandpass filtering of the signal is achieved simultaneously through short-time Fourier transform.

[0011] Furthermore, based on the obtained time-spectrum data, the relative time delay between microphones is estimated using a generalized cross-correlation phase transform algorithm. For any pair of microphones... , their cross-correlation function In the frequency domain, it is calculated as follows:

[0012] In the formula, , microphones and The frequency domain representation of the signal, Indicates the inverse Fourier transform. To prevent small constants with a denominator of zero; Represented as The complex conjugate; An optimization strategy is introduced to improve the stability and anti-interference capability of the generalized cross-correlation phase transform algorithm. Specifically, the frequency domain weighting function is dynamically calculated based on the signal-to-noise ratio of different frequency bands in the microphone frequency domain signal. :

[0013] In the formula, frequency band f The corresponding signal-to-noise ratio; using a frequency domain weighting function Corrected cross-correlation function:

[0014] Subsequently, by extracting the maximum peak value from the cross-correlation output and combining it with spectral entropy calculation, false peaks caused by interference sources are eliminated.

[0015] Furthermore, based on the estimated time delay information, a minimum variance distortionless response beamforming algorithm is used to calculate beam weights and enhance the target sound source signal. The beam weight calculation process includes constructing an optimization problem: , Let be the covariance matrix of the input signal. This is the array direction response vector. For beam weighting, Represents the conjugate transpose; estimate the covariance matrix based on measured data. R :

[0016] In the formula, N Indicates the number of samples. Indicates the first n The signal vector of each sampled sample contains time-domain signal samples from all channels (microphones).

[0017] The optimal solution for beam weights, obtained from the estimated covariance matrix, is as follows: For the input signal Beam output is , This refers to the number of microphones.

[0018] Furthermore, the beam output signals of multiple frames and the preliminary sound source trajectory are input into the particle filter. The position of the subsequent frame is predicted by the previous frame in the particle filter to generate trajectory stability and delay deviation indices. These indices are used for closed-loop feedback to adjust the minimum variance distortionless response beamforming algorithm and adaptive Kalman filter parameters, as well as the short-time Fourier transform process of the acquired signal. The signal-to-noise ratio is calculated using the beam output signal as feedback input. SNR and reverberation time T 60 As an evaluation indicator, the pre-processing procedure is adjusted based on the evaluation indicator.

[0019] Furthermore, adjusting the pre-processing steps based on evaluation indicators includes: SNR <Signal-to-noise ratio threshold, or T 60 When the reverberation time threshold is reached, the number of particles in the particle filter is increased, the frequency band weighting parameters are adjusted, and the refresh frequency of the covariance matrix of MVDR is updated to enhance the update sensitivity of the Kalman filter gain.

[0020] On the other hand, the present invention provides a sound source localization system based on a microphone array, which includes a microphone array, a preprocessing module, a time delay estimation module, a beamforming module, a performance evaluation module, and a parameter feedback module.

[0021] The preprocessing module is used to pre-whiten and perform short-time Fourier transform processing on the acquired acoustic signals. The time delay estimation module uses a generalized cross-correlation phase transform algorithm to estimate the time delay between any microphone pair; The beamforming module performs beamforming operations based on the calculated time delay information to enhance the sound source in the target direction; The performance evaluation module is used to evaluate the performance indicators of the beam output signal. The parameter feedback module dynamically adjusts the beamforming parameters and generalized cross-correlation parameters based on the performance evaluation results, and feeds them back to the beamforming module and the time delay estimation module in a closed loop.

[0022] Furthermore, the performance evaluation module includes a signal-to-noise ratio (SNR) estimation unit, a speech clarity evaluation unit, and a beam directivity measurement unit. The SNR estimation unit calculates the SNR gain based on the beam output signal. The speech clarity evaluation unit uses metrics such as PESQ and STOI to evaluate the beam output quality. The beam directivity measurement unit is used to calculate the main lobe gain, side lobe suppression ratio, and angular resolution.

[0023] Furthermore, the parameter feedback module includes a state detection submodule, a feedback strategy selection submodule, and a parameter update submodule. The state detection submodule is used to monitor the changing trends of performance indicators in real time and construct a dynamic state vector. The feedback strategy selection submodule selects either a fast inner loop response or a slow outer loop optimization strategy based on the current state. The parameter update submodule adjusts parameters such as beam main lobe width, regularization coefficient, and time delay window length based on the stability constraint update algorithm of Lyapunov functions.

[0024] The beneficial effects of this invention are as follows: (1) This invention effectively combines the optimized generalized cross-correlation phase transform algorithm with the minimum variance distortionless response beamforming algorithm, which can accurately estimate time delay and enhance the target sound source signal in complex acoustic environments such as strong noise and multipath reverberation. At the same time, by introducing frequency domain weighting and false peak elimination mechanisms, the error accumulation caused by interference is reduced, thereby achieving more stable and reliable positioning performance and adapting to the needs of various dynamic scenarios such as vehicle-mounted and industrial plants.

[0025] (2) A closed-loop optimization mechanism with beam output signal as feedback is adopted, and the time delay estimation and beam weight parameters are dynamically adjusted by combining particle filters and evaluation indicators such as signal-to-noise ratio and reverberation time. This design can respond in real time to changes in sound source position, noise fluctuations and environmental reverberation, and improve the suppression effect on noise interference and multipath echo. Through the fast response of the inner loop and the slow optimization strategy of the outer loop, the processing process is continuously optimized to ensure high-precision positioning in dynamic environments.

[0026] (3) The beam weights are dynamically optimized by the adaptive Kalman filter mechanism and the spectral features are processed by the CNN-Transformer-GRU hybrid network. While ensuring positioning accuracy, the computational complexity can be effectively reduced. It can meet the real-time processing requirements of multi-microphone arrays and high sampling rates, and is suitable for application scenarios with high response speed requirements such as smart homes and human-computer interaction.

[0027] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a structural block diagram of a microphone array sound source localization method according to an embodiment of the present invention; Figure 2This is a schematic flowchart of the microphone array sound source localization method provided in an embodiment of the present invention. Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0031] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0032] Example 1 This embodiment uses an industrial production environment as a typical application scenario to demonstrate how the method proposed in this invention achieves high-precision sound source localization and fault sound identification under conditions of multiple noise interferences, dynamic changes in equipment operation, and complex reverberation. A 16-32 element microphone array can be used and deployed near key equipment or on the roof of the factory to capture operating sounds and abnormal sounds from different mechanical equipment.

[0033] Please see Figure 1 and Figure 2 In this embodiment, the sound source localization method is specifically described as follows: I. During equipment operation, the microphone array synchronously acquires multi-channel audio signals at a high sampling rate, and the following two preliminary processing steps are completed by the FPGA module: 1. Time-frequency transformation and preprocessing All input channel signals first undergo pre-emphasis and short-time Fourier transform (STFT) to obtain the time spectrum of each frame:

[0034] in, For the time series of the i-th channel, For window functions (such as Hamming windows). For frame shift, The frame length. t This is a time frame identifier variable. f For frequency variables, n These are discrete-time variables. Bandpass filtering of the signal is achieved simultaneously through short-time Fourier transform.

[0035] By using time-frequency transformation, the frequency domain characteristics of the acoustic signal can be extracted, and bandpass filtering (typically in the range of 100 Hz to 10 kHz) can be applied to suppress low-frequency vibration noise and high-frequency interference signals from the environment. Simultaneously, the high-pass filter built into the FPGA can remove DC drift and power supply coupling noise in real time, ensuring the accuracy of subsequent time delay estimation and beamforming processing.

[0036] 2. Data synchronization and flow control The transformed frequency domain data is pushed to the GPU within the FPGA via a DMA interface, and a double-buffered structure is used to ensure data processing continuity, thereby keeping the system latency within 5 ms. This mechanism is particularly suitable for multi-task parallel processing platforms in industrial equipment monitoring, such as heterogeneous computing platforms based on Xilinx Zynq UltraScale+ and NVIDIA Jetson series.

[0037] II. Estimating the time delay information between microphones based on cross-correlation analysis.

[0038] In this embodiment, the relative time delay between microphones is estimated based on the improved generalized cross-correlation phase transform (GCC-PHAT) algorithm, and parallel acceleration calculation is implemented in FPGA to cope with the interference from multiple noise sources and the real-time requirements in industrial sites.

[0039] For any pair of microphones , their cross-correlation function In the frequency domain, it is calculated using the following formula:

[0040] in: , microphones and The frequency domain representation of the signal, Indicates the inverse Fourier transform. To prevent small constants with a denominator of zero, express The complex conjugate of the equation. The core idea of ​​this formula is to retain only the phase information, thereby improving the positioning robustness in low signal-to-noise ratio environments (such as industrial machinery noise).

[0041] To further improve the stability and anti-interference capabilities of the GCC-PHAT algorithm, this embodiment introduces three optimization strategies: 1) Adaptive frequency band weighting: Dynamically calculates the frequency domain weighting function based on the signal-to-noise ratio of different frequency bands. Reduce the impact of high-noise frequency bands:

[0042] 2) Correct the cross-correlation function:

[0043] 3) Spatial spectral peak enhancement: By extracting the maximum peak value from the cross-correlation output and combining it with spectral entropy calculation, "false peaks" caused by interference sources are eliminated.

[0044] The FPGA can execute the aforementioned cross-correlation function using a parallel logic structure, supports 16-channel parallel input, and can output a delay matrix within 2ms with an accuracy of ±1 sampling point (e.g., 20.8 µs at a 48 kHz sampling rate).

[0045] Third, based on the time delay information, beamforming algorithm is used to calculate beam weights and enhance the target sound source signal.

[0046] Beamforming is performed based on the obtained multi-channel time delay estimation results to enhance the sound source signal from a specific direction while suppressing interference from other directions.

[0047] In this embodiment, the Minimum Variance Distortionless Response (MVDR) beamforming algorithm is adopted, and the beam weight is dynamically optimized by combining the adaptive Kalman filter (AKF) mechanism to improve the adaptability and robustness in complex noise environments with multiple sound sources in industrial equipment.

[0048] 1. MVDR Beam Weight Calculation Let the input signal be , Given the number of microphones, the beam output is:

[0049] in, For beam weighting, This indicates the conjugate transpose.

[0050] MVDR beam weights are obtained through the following optimization problem:

[0051] in, Let be the covariance matrix of the input signal; The array directional response vector is derived and constructed based on the time delay information obtained from the GCC-PHAT algorithm.

[0052] The covariance matrix estimated from the measured data is as follows:

[0053] The measured data is the array observation vector sequence formed by the multi-channel time-domain signals actually collected by the Kefeng array under the combined action of the target sound source and environmental noise, after preprocessing (STFT transformation, denoising, and filtering). Indicates the number of samples. Indicates the first n The signal vector of each sampled sample contains time-domain signal samples from all channels (microphones).

[0054] The optimal solution for MVDR beam weighting is: .

[0055] The updates to the covariance matrix and the array orientation response vector can be performed in parallel on the GPU, improving processing speed.

[0056] 2. Dynamically Optimized Beam Weights Using Kalman Filtering While the MVDR algorithm performs excellently in static scenes, it can cause directional drift and defocusing issues in industrial equipment where the sound source position may change due to mechanical movement. Therefore, this embodiment introduces Kalman filtering to dynamically track and optimize the beamforming weights.

[0057] Beam weight Modeled as a dynamic state vector The update and observation models are as follows:

[0058]

[0059] in, This is the state transition matrix (which can be an identity matrix). The observation matrix; , These are process noise and observation noise (Gaussian distribution), respectively.

[0060] The Kalman gain is calculated as follows:

[0061] And update the estimate accordingly:

[0062] in, This represents the observation noise covariance matrix, used to describe the statistical characteristics of noise during the measurement process.

[0063] By combining actual observed signal changes (such as SNR fluctuations and directional drift), the beam direction can be adaptively adjusted in real time, so that the target signal remains clear while interference noise is suppressed stably.

[0064] Fourth, the time delay estimation and beam weight are dynamically adjusted through a closed-loop feedback optimization mechanism to improve the accuracy and robustness of sound source localization.

[0065] 1. To improve the stability and trajectory continuity of the beamout, in the closed-loop feedback optimization mechanism, the time delay of each microphone pair is estimated using the GCC-PHAT algorithm after passing through the STFT and preprocessing of the microphone array signal in each frame. This time delay is then combined with the MVDR beamforming output to obtain the preliminary sound source trajectory for each frame. This trajectory serves as the input to the particle filter, predicting the position of the current frame based on the previous frame, as shown in the following formula:

[0066] in, Indicates the first Frame estimation results, To predict the increment, For Gaussian perturbation, σ This represents the standard deviation of process noise.

[0067] The trajectory stability and delay deviation of the particle filter output are used for closed-loop feedback. (1) Front-end STFT / preprocessing module: dynamically adjusts window length, frame shift and filtering parameters according to the indicators to reduce the impact of noise on time delay estimation; (2) MVDR+AKF module: Adjust the covariance matrix and filter noise parameters according to trajectory stability and delay deviation, thereby optimizing beam weight update, enhancing main lobe gain, suppressing side lobe interference, and improving the continuity of output signal and positioning accuracy.

[0068] Through FPGA parallel processing, this module can quickly output smooth trajectories and related quality indicators under multi-channel input, providing stable and reliable input signals for subsequent deep learning enhancement modules, while ensuring the real-time performance and robustness of sound source localization in industrial equipment environments.

[0069] 2. To improve robustness in dynamic scenarios, a closed-loop feedback mechanism can be introduced between time delay estimation and beamforming to sense signal quality in real time and adjust the estimation parameters accordingly. Specifically, the beam output signal is used as the feedback input, and the signal-to-noise ratio (SNR) is calculated... SNR ) and reverberation time ( T 60 Indicators such as these are used to control and optimize the pre-processing stage.

[0070] 1) Signal-to-noise ratio and reverberation time estimation Beam output signal The signal-to-noise ratio is estimated in the following form:

[0071] in, For signal energy estimation, For noise energy estimation (VAD separation via speech activity detection), the window length is 500 ms.

[0072] The reverberation time is obtained by fitting the energy decay curve, and is calculated as follows:

[0073] in, slope The slope is obtained by linear regression of the logarithmic energy decay curve.

[0074] 2) Dynamic feedback update strategy Will SNR and T 60 As a state variable, the following update strategy is adopted: when SNR <Signal-to-noise ratio threshold (e.g., 10 dB) or T 60 When the reverberation time threshold (e.g., 0.6 s) is reached, it indicates a decrease in signal quality. Increasing the number of particles in the particle filter and adjusting the frequency domain weighting function (e.g., if the target frequency band...) is considered. SNR If the frequency is below the threshold, the weight of that frequency band can be increased; if it is a sidelobe or noise frequency band... SNR If the reverberation time is too high, the corresponding weight can be reduced. If the reverberation time is too long, the low-frequency weight can be reduced or the high-frequency weight increased to reduce the impact of reverberation interference on the beam output. The adjustment process usually combines smoothing or normalization to avoid sudden weight changes that could lead to output discontinuities; at the same time, the covariance matrix refresh frequency of the MVDR and the Kalman filter gain are updated. Enhance update sensitivity.

[0075] The feedback path is as follows: beam output → signal-to-noise ratio / reverberation time analysis → parameter optimization (GCC window length, frequency domain weighting function, Kalman gain, etc.) → feedback to the front end.

[0076] This feedback mechanism forms an "adaptive loop" that can respond in real time to changes in the operation of industrial equipment, achieving high-precision, dynamically robust sound source tracking and voice enhancement.

[0077] V. To further improve the clarity and separability of the target sound source signal, a deep neural network is used to post-process the beam output signal.

[0078] Specifically, a hybrid network structure of CNN-Transformer-GRU can be run on a GPU to process mixed signals from multiple sound sources in industrial production environments and complex background noise generated by mechanical equipment, thereby achieving enhancement and robust tracking of the target sound source.

[0079] In the CNN-Transformer-GRU hybrid network, CNN serves as the front end to extract local structural features from the short-time Fourier transform (STFT) spectrogram; Transformer serves as the middle layer to model long-range dependencies in the spectrogram, which is suitable for far-field speech; and GRU serves as the tail end to enhance temporal continuity modeling and output a mask or clear spectrogram.

[0080] The input to the hybrid network is the STFT amplitude spectrum of the beam output signal, and the output is the spectrum or mask of the target sound source signal. During training, the loss function primarily uses SI-SNR (scale-invariant signal-to-noise ratio), supplemented by mean square error in the frequency domain (MSE).

[0081] in, These are weighting coefficients used to balance the SI-SNR loss. and MSE loss .

[0082] During network inference, let the beam output spectrum be... The estimated speech is :

[0083] in, DNN Represented as a deep neural network model, specifically a hybrid network structure of CNN-Transformer-GRU, it is used to process the beam output spectrum. Speech enhancement or source separation is performed to obtain target speech estimation. .

[0084] In addition, to meet the requirements of real-time processing, the above-mentioned hybrid network is compressed to about 3 M parameters through distillation and supports 16-bit quantization; TensorRT acceleration is used on the GPU, and the single-frame latency is controlled within 3 ms; the frame-level caching mechanism supports online streaming processing and achieves stable enhancement of continuous speech signals.

[0085] Example 2 This embodiment provides an application of the method proposed in this invention for acoustic monitoring during UAV flight, and its operation mainly includes: When performing inspection, environmental monitoring, or aerial photography missions, drones continuously collect surrounding acoustic signals via their onboard microphone array, including propeller noise, wind shearing, and potential target sound sources such as alarm sounds, mechanical anomalies, or other sudden acoustic events. When the instantaneous energy generated by a sound source exceeds a threshold dynamically adjusted in real-time based on ambient noise, an audio segment at that moment is immediately captured as the primary waveform. This waveform is used to capture subsequent related acoustic events and serves as a reference template for subsequent signal processing. This primary waveform can trigger modules such as cross-correlation analysis, beamforming, and dynamic enhancement, enabling high-precision identification and separation of sound sources with different properties.

[0086] In the signal processing, the acquired audio is first subjected to frequency domain analysis and preprocessing to suppress wind noise and high-frequency interference. At the same time, the relative direction of the sound source is quickly obtained through time delay estimation. In response to the Doppler effect generated by the UAV's own motion and the complex reverberation of the environment, adaptive enhancement and filtering mechanisms are used to enhance the signal of the target sound source and continuously track it, ensuring that key acoustic features are stably extracted during dynamic flight.

[0087] Environmental noise, such as propeller hum, wind shearing, or mechanical vibration, may briefly trigger the energy threshold. However, by analyzing the correlation and randomness of sound wave characteristics, these noises can be identified as background noise and a template can be established. In subsequent monitoring, any signals matching this template can be ignored, thus effectively suppressing steady-state environmental noise without affecting the ability to detect abnormal sound sources. For moving sound sources, state perception is achieved using energy envelope, time difference of arrival, and frequency characteristics. Based on this, enhancement strategies are dynamically adjusted to achieve continuous tracking of the sound source, signal clarification, and extraction of abnormal features.

[0088] By internalizing acoustic principles and environmental characteristics into signal processing logic, UAVs can achieve robust acoustic monitoring in complex, multi-source, and dynamic environments. Without relying on pre-set environmental models or large amounts of labeled data, they can complete real-time anomaly identification, sound source enhancement, and continuous tracking, ensuring the reliability and accuracy of monitoring tasks. This design maintains consistency with the industrial equipment monitoring example in Example 1 in terms of processing flow, signal capture logic, and closed-loop adaptation strategy, differing only in the scenario and sound source type, thus achieving the versatility and scalability of the same technical solution in different application environments.

[0089] Example 3 This embodiment provides a sound source localization system based on a microphone array, which includes: a microphone array, a preprocessing module, a time delay estimation module, a beamforming module, a performance evaluation module, and a parameter feedback module.

[0090] Among them, the microphone array is used to collect multi-channel sound signals from the target sound source, and a microphone array of 16 to 32 yuan can be used.

[0091] The preprocessing module is used to perform pre-whitening, noise reduction, bandpass filtering, and other processing on the acquired acoustic signals.

[0092] The time delay estimation module uses the environment-adaptive generalized cross-correlation function (GCC) to calculate the time delay between any microphone pair. It includes a signal window length adaptive unit, a frequency band weight allocation unit, and a cross-correlation function construction unit. The signal window length adaptive unit dynamically adjusts the frame length parameter based on the current signal-to-noise ratio and reverberation time of the acoustic signal; the frequency band weight allocation unit constructs a frequency domain weighting function based on the frequency energy distribution and noise spectral density; the cross-correlation function construction unit constructs a weighted GCC-PHAT function based on the correlation parameters and performs subsampling interpolation on the cross-correlation peaks to improve time delay resolution. The correlation parameters include: signal frame length... N and frame R Window function, frequency domain weighting function Noise spectral density Frequency energy distribution Regularization coefficient Cross-correlation peak subsampling interpolation step size.

[0093] The beamforming module performs beamforming operations based on calculated time delay information to enhance the sound source in the target direction. It includes an array response construction unit, a weight calculation unit, and an output generation unit. The array response construction unit constructs an array directional response vector based on the time delay matrix and the microphone array structure; the weight calculation unit calculates a beamforming weight vector based on the array response model; and the output generation unit performs a weighted summation of the multi-channel acoustic signals based on the beamforming weight vector to obtain the beam output signal. The array directional response vector is essentially an array directional response vector describing the relative response of the sound source in the target direction on each microphone channel. d Specifically, regarding frequency Next target direction ( , The array direction response vector can be expressed as: , It is the azimuth angle. The pitch angle; The time delay of the Mth microphone relative to the reference microphone can be obtained through the microphone array geometry and the time delay matrix.

[0094] The performance evaluation module is used to evaluate the performance metrics of the beam output signal, including but not limited to signal-to-noise ratio (SNR), intelligibility score, and peak power ratio (PSR). It includes an SNR estimation unit, a speech intelligibility evaluation unit, and a beam directivity measurement unit. The SNR estimation unit calculates the SNR gain based on the beam output signal. The speech intelligibility evaluation unit uses metrics such as Perceptual Speech Quality Assessment (PESQ) and Short-Time Objective Intelligibility (STOI) to evaluate the beam output quality. Specifically, it obtains an intelligibility score by comparing it with a reference signal, reflecting the intelligibility and perceived quality of speech after enhancement and noise suppression. The beam directivity measurement unit calculates parameters such as main lobe gain, side lobe suppression ratio, and angular resolution. The main lobe gain is based on the beamforming weight vector. With array direction response vector The formula is:

[0095] The sidelobe suppression ratio is calculated as the ratio of the output power in the non-target direction to the output power of the main lobe, using the following formula:

[0096] Angular resolution is determined by the main lobe half-power width (3 dB beamwidth):

[0097] in, , These represent the azimuth and elevation angles, respectively, and mainlobe represents the main lobe region in the beamforming output power distribution. This indicates the angle position corresponding to the 3 dB drop point on the right side of the main lobe. This indicates the angular position corresponding to the 3 dB drop point on the left side of the main lobe.

[0098] The above metrics are all based on beam weight vectors. Array directional response vector The beam output power distribution can be calculated at different frequency points and time frames, and a weighted average can be used to obtain the overall performance index. During the evaluation process, the performance index can be compared with preset thresholds or historical reference values ​​to determine whether the beam output quality meets the requirements. The evaluation results can be used for closed-loop feedback to dynamically adjust the beam weights, frequency domain weighting functions, and time delay estimation parameters, thereby optimizing the beam output and achieving adaptive sound source enhancement and robust localization.

[0099] The parameter feedback module dynamically adjusts beamforming and GCC parameters based on performance evaluation results and feeds them back to the beamforming and delay estimation modules in a closed loop, forming a closed-loop optimization mechanism. It includes a state detection submodule, a feedback strategy selection submodule, and a parameter update submodule. The state detection submodule monitors performance index trends in real time and constructs a dynamic state vector. The feedback strategy selection submodule selects either a fast inner-loop response or a slow outer-loop optimization strategy based on the current dynamic state vector. The parameter update submodule adjusts parameters such as beam main lobe width, regularization coefficient, and delay window length based on a Lyapunov function-based stability constraint update algorithm to ensure rapid convergence and stable operation of the system under varying environments.

[0100] Among them, the dynamic state vector Represented as: , , , The first k The rate of change of frame signal-to-noise ratio, PESQ, and sidelobe suppression ratio relative to the previous frame. These represent the changes in the beam main lobe direction and time delay estimate, respectively. This dynamic state vector reflects the dynamic evolution of the system's performance over time and is used to determine the stability and trend of the system's operating state.

[0101] The strategy selection mechanism of the feedback strategy selection submodule is as follows: when performance indicators show a rapid decline or abrupt change, the inner-loop fast response strategy is triggered, which achieves short-term adaptive correction by adjusting the weight regularization coefficient and window length parameter; when performance indicators show a shift trend over a longer time scale, the outer-loop slow optimization strategy is triggered, which adjusts the beam main lobe width and frequency domain weighting function through global search to optimize overall performance. The two strategies can run in parallel to balance fast response and long-term stability.

[0102] The parameter update submodule uses a stability-constrained update algorithm based on Lyapunov functions to update the parameters using gradients. The system's adjustable control parameters are denoted as: , Represents the weight regularization coefficient. Indicates the time delay window length. Indicates the main lobe width of the beam. This represents the parameters of the frequency domain weighting function.

[0103] There is a functional mapping relationship between performance status and control parameters, that is... Let the system stability function be:

[0104] When system stability variables If the system is in a stable state, then update the relevant parameters using gradient descent.

[0105] in, The learning rate is denoted by . By calculating the partial derivative of the state vector with respect to the control parameters, the system can adaptively adjust the regularization coefficient, time delay window length, beam main lobe width, and frequency domain weighting function parameters according to performance change trends. This update strategy based on Lyapunov stability can achieve adaptive closed-loop optimization control in complex industrial noise environments, enabling the sound source localization system to maintain stable convergence and improved robustness under dynamic interference conditions.

[0106] In addition, the system includes a hardware acceleration platform, which comprises an FPGA processing module and a GPU co-processing module. The FPGA processing module is used to implement parallel pipeline acceleration for preprocessing and cross-correlation function calculations; the GPU co-processing module is used to implement high-throughput parallel computation for beamforming matrix solving and interacts with the FPGA via a PCIe interface.

[0107] This system is compatible with various microphone array structures, including circular, ring, linear, and combined arrays. The array structure supports a dual-layer design, with each layer capable of being configured with different types of microphones to accommodate far-field and near-field sound sources. Furthermore, the array frame can be installed in a flexible vibration-damping structure and features electromagnetic shielding and thermal stability design to meet the needs of various scenarios such as automotive, drones, and industrial applications.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A microphone array sound source positioning method based on cross-correlation-beamforming closed-loop optimization, characterized in that, The multi-channel sound signals are collected by a microphone array and preprocessed to extract time-frequency features and suppress noise interference; A generalized cross-correlation phase transform algorithm is used to estimate the time delay information between microphones, and an optimization strategy is introduced to improve the stability and anti-interference of the generalized cross-correlation phase transform algorithm; According to the estimated time delay information, a minimum variance distortionless response beamforming algorithm is used to calculate the beam weight and enhance the target sound source signal. In the calculation process, the adaptive Kalman filtering mechanism is combined to dynamically optimize the beam weight; The beam output signal is used as the feedback input, and a closed-loop feedback optimization mechanism is formed by combining the particle filter and the evaluation index to adjust the time delay estimation and the beam weight; A hybrid network of CNN-Transformer-GRU is used, and the short-time Fourier transform amplitude spectrum of the beam output signal is used as the input of the hybrid network, and the output is obtained. The frequency spectrum or mask of the target sound source signal.

2. The method of claim 1, wherein, For all input channel signals, the time-frequency spectrum of each frame is obtained by pre-emphasis and short-time Fourier transform: wherein is a time series for the i-th channel, is a window function, is a frame shift, is a frame length; t is an identification variable for a time frame, f is a frequency variable, n is a discrete time variable, which simultaneously implements a band-pass filtering of the signal by means of a short-time Fourier transform.

3. The method of claim 2, wherein, According to the obtained time-frequency spectrum data, a generalized cross-correlation phase transform algorithm is used to estimate the relative time delay between the microphones, and for any pair of microphones , The cross-correlation function is calculated in the frequency domain as wherein , are the frequency domain representations of the signals of the microphones and , denotes the inverse Fourier transform, is a small constant to prevent division by zero; denotes the complex conjugate of ; An optimization strategy is introduced to improve the stability and anti-interference of the generalized cross-correlation phase transform algorithm, specifically, according to the signal-to-noise ratio of different frequency bands in the frequency domain signal of the microphone, a frequency domain weighting function is dynamically calculated : wherein is the frequency band f corresponding signal-to-noise ratio; a frequency domain weighting function is employed modified cross-correlation function: Subsequently, the maximum peak value in the cross-correlation output is extracted, and the spectral entropy calculation is combined to exclude false peaks caused by interference sources.

4. The method of claim 3, wherein, According to the estimated time delay information, a minimum variance distortionless response beamforming algorithm is used to calculate beam weights and enhance the target sound source signal, wherein the beam weight calculation process comprises: , is a covariance matrix of the input signal, is an array direction response vector, is a beam weight, denotes a conjugate transpose; the covariance matrix is estimated according to actual measurement data R : wherein N denotes the number of samples, denotes the signal vector of the n th sampled sample, containing the time-domain signal samples of all microphones; The optimal solution for beam weights from the estimated covariance matrix is ; then for an input signal , the beam output is , is the number of microphones.

5. The method of claim 4, wherein, The multi-frame beam output signal and the preliminary sound source trajectory are input into the particle filter, and the position of the subsequent frame is predicted by the previous frame in the particle filter to generate the trajectory stability and delay deviation index, which is used for closed-loop feedback adjustment of the minimum variance distortionless response beamforming algorithm and the adaptive Kalman filtering parameters, as well as the short-time Fourier transform process of the collected signal; calculating the signal-to-noise ratio using the beam output signal as feedback input SNR and reverberation time T 60 as an evaluation index, and adjusting the preceding process based on the evaluation index.

6. The method of claim 5, wherein, Adjusting the pre-stage processing according to the evaluation index includes, SNR < a signal-to-noise ratio threshold, or T 60 When the reverberation time threshold is reached, the number of particles in the particle filter is increased, the frequency band weighting parameter is adjusted, the covariance matrix of the MVDR is updated, the refresh frequency is increased, and the update sensitivity of the Kalman filter gain is enhanced.

7. A microphone array based sound source positioning system, characterized by The microphone array, the preprocessing module, the time delay estimation module, the beamforming module, the performance evaluation module, and the parameter feedback module are included. The preprocessing module is used for pre-whitening and short-time Fourier transform processing of the collected sound signals. The time delay estimation module estimates the time delay between any microphone pair using a generalized cross-correlation phase transform algorithm. The beamforming module performs beamforming operations based on the calculated time delay information to enhance the target direction sound source. The performance evaluation module is used to evaluate the performance indicators of the beam output signal, including signal-to-noise ratio, PESQ, STOI, main lobe gain, side lobe suppression ratio, and angle resolution capability. The parameter feedback module dynamically adjusts the beamforming parameters and the generalized cross-correlation parameters based on the performance evaluation results and feeds back to the beamforming module and the time delay estimation module.

8. The system of claim 7, wherein, The performance evaluation module includes a signal-to-noise ratio estimation unit, a speech intelligibility evaluation unit, and a beam directivity measurement unit. The signal-to-noise ratio estimation unit calculates the signal-to-noise ratio gain based on the beam output signal. The speech intelligibility evaluation unit evaluates the quality of the beam output using PESQ and STOI indicators, and obtains the intelligibility score by comparing with the reference signal, reflecting the intelligibility and perceptual quality of the speech after enhancement and noise suppression. The beam directivity measurement unit is used to calculate the main lobe gain, side lobe suppression ratio, and angle resolution capability.

9. The system of claim 7, wherein, The parameter feedback module includes a state detection submodule, a feedback strategy selection submodule, and a parameter update submodule. The state detection submodule is used to monitor the performance indicator change trend in real time and construct a dynamic state vector. The feedback strategy selection submodule selects an inner loop fast response or an outer loop slow optimization strategy according to a current dynamic state vector; The parameter updating submodule adjusts a beam main lobe width, a regularization coefficient and a time delay window length parameter based on a Lyapunov function stability constraint updating algorithm.

10. The system of claim 9, wherein, The dynamic state vector constructed by the state detection submodule is represented as follows: , , , The first k The rate of change of frame signal-to-noise ratio, PESQ, and sidelobe suppression ratio relative to the previous frame. These represent the changes in the beam main lobe direction and the time delay estimate, respectively. The feedback strategy selection submodule selects an inner loop fast response or an outer loop slow optimization strategy according to a current dynamic state vector; When a performance index appears to rapidly decrease or mutate, the inner loop fast response strategy is triggered to realize short-time self-adaptive correction by adjusting a weight regularization coefficient and a window length parameter; when a performance index exists in a long time scale, the outer loop slow optimization strategy is triggered to adjust a beam main lobe width and a frequency domain weighting function by global search; The parameter updating sub-module updates the parameters by gradient based on the stability constraint updating algorithm of Lyapunov function; the adjustable control parameters of the system are denoted as: , denotes a weight regularization coefficient, denotes a time delay window length, denotes a beam main lobe width, denotes a frequency domain weighting function parameter; there is a function mapping relationship between the performance state and the control parameters: ; the system stability function is: When the system stability variable is greater than 0, the system is in a stable state, otherwise the relevant parameters are updated by gradient descent: In the formula, is the learning rate.

Images