Sound acquisition and processing system based on cooperation of multiple microphone arrays

Abstract

The invention discloses a sound acquisition and processing system based on cooperation of multiple microphone arrays. The system comprises a sound acquisition module, a multi-channel signal preprocessing module, a sound signal feature extraction module, an abnormal sound recognition module, a sound source positioning module and an alarm module. A multi-channel mixed data signal is collected through a circularly-arranged multi-microphone array formed by a plurality of microphones, after echo cancellation, wave beam domain noise reduction and multi-sound-source separation, a single-sound-source feature vector is extracted, according to the single-sound-source feature vector, abnormal sound including explosion, screaming or glass breakage is recognized through a BiLSTM and an attention mechanism model, and the abnormal sound is recognized through an attention mechanism model. And the GCC-PHAT and MDS-MUSIC algorithms are combined to position abnormal sound production, and alarm information is generated. According to the invention, accurate identification, positioning and alarm of the abnormal sound can be realized, and the real-time performance, the accuracy and the multi-target processing capability of abnormal sound monitoring in a complex environment can be obviously improved.

Description

Technical Field

[0001] This invention relates to the field of sound acquisition and processing technology, and specifically to a sound acquisition and processing system based on multi-microphone array collaboration. Background Technology

[0002] With the ever-increasing demands in public safety, intelligent security, and emergency management, real-time monitoring, accurate identification, and rapid location of abnormal sounds have become key technologies for ensuring the safety of personnel and property. As an important carrier of environmental conditions, sound signal acquisition and processing technologies have always been a research hotspot in the field of acoustic monitoring.

[0003] Existing technologies for abnormal sound monitoring systems often employ single microphones or simple linear arrays, which have significant limitations: weak anti-interference capabilities, susceptibility to noise, echoes, and the superposition of multiple sound sources, resulting in low signal-to-noise ratios; insufficient multi-source processing capabilities, making it difficult to distinguish between multiple abnormal sounds occurring simultaneously or at different times, leading to confusion and missed detections; low recognition accuracy, relying on single features or traditional models, resulting in poor robustness in distinguishing similar sounds; limited positioning accuracy, making it difficult to achieve three-dimensional positioning; and poor adaptability between array layout and algorithm, leading to uneven spatial sampling and limiting the real-time performance and accuracy of multi-source positioning. Therefore, to address the needs for high signal-to-noise ratio acquisition, accurate identification, real-time positioning, and multi-target processing of multiple abnormal sounds in complex environments, there is an urgent need for a collaborative processing system integrating efficient anti-interference processing, advanced feature extraction, intelligent recognition models, and high-precision positioning algorithms to improve the reliability and practicality of abnormal sound monitoring. Summary of the Invention

[0004] The purpose of this invention is to provide a sound acquisition and processing system based on multi-microphone array collaboration to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A sound acquisition and processing system based on multi-microphone array collaboration includes a sound acquisition module, a multi-channel signal preprocessing module, a sound signal feature extraction module, an abnormal sound recognition module, a sound source localization module, and an alarm module;

[0007] The sound acquisition module is used to acquire multi-channel mixed data signals through a multi-microphone array;

[0008] The multi-channel signal preprocessing module is used to perform echo cancellation and beam domain noise reduction on the multi-channel mixed data signal to obtain a multi-channel clean data signal, and to separate the multi-channel clean data signal into multiple sound sources to obtain several single sound source signal streams.

[0009] The sound signal feature extraction module is used to extract the corresponding single sound source feature vector from the single sound source signal stream, and thereby obtain the single sound source feature vectors corresponding to all single sound source signal streams.

[0010] The abnormal sound recognition module is used to construct an abnormal sound recognition model and input the single sound source feature vectors corresponding to all single sound source signal streams into the abnormal sound recognition model for processing to obtain the abnormal sound recognition result.

[0011] The sound source localization module is used to perform multi-stage localization processing on multi-channel clean data signals based on the abnormal sound recognition results to obtain the location of the abnormal sound.

[0012] The alarm module is used to issue an alarm based on the abnormal sound recognition result and the location of the abnormal sound.

[0013] Preferably, in the sound acquisition module, the process of acquiring multi-channel mixed data signals through a multi-microphone array is as follows:

[0014] The peripheral microphone array consists of n microphones, where n is greater than 10. The acoustic centers of all microphones are located in the same horizontal plane and are evenly distributed on the circumference with the physical center of the array as the center. Meanwhile, the central microphone is deployed at the geometric center of the circumference, and its acoustic center is at the same height as the horizontal plane of the peripheral microphones, forming a circular multi-microphone array.

[0015] A reference clock signal is generated using a synchronous clock module, which serves as the time reference for the main control unit and the microphones. Based on this, the synchronous sampling technology enables the main control unit to trigger and control all microphones through the synchronous serial bus SPI. The sampled data is converted in parallel by a multi-channel analog-to-digital converter (ADC) to obtain a multi-channel mixed data signal.

[0016] Preferably, in the multi-channel signal preprocessing module, the process of echo cancellation and beam domain noise reduction is performed on the multi-channel mixed data signal:

[0017] For environmental noise contained in multi-channel mixed data signals, a beam domain denoising algorithm based on spatial correlation is used for beam domain denoising. The beam domain denoising algorithm based on spatial correlation utilizes the spatial correlation characteristics of signals collected by different microphones in a circular multi-microphone array. By constructing a spatial covariance matrix and performing eigenvalue decomposition, the signal subspace and noise subspace are separated. For the multi-channel signals collected by n peripheral microphones and one microphone at the center of the circle, the beam domain denoising algorithm based on spatial correlation retains the signal component from the direction of the target sound source while suppressing the noise component from the non-target direction, thereby achieving directional enhancement and noise suppression of the multi-channel mixed data signal, and finally obtaining the multi-channel mixed data signal after beam domain denoising.

[0018] The multi-channel mixed data signal after beam-domain denoising is processed using a multi-channel adaptive echo canceller based on the NLMS adaptive filtering algorithm. The NLMS-based multi-channel adaptive echo canceller uses the signal acquired by the central microphone as a reference signal. Through adaptive filtering, it estimates the transfer function of the echo path in real time, generating a cancellation signal with similar amplitude but opposite phase to the echo signal. This cancellation signal is then adaptively cancelled one by one with the beam-domain denoised signals acquired by n peripheral microphones to obtain a clean multi-channel data signal. ,in This refers to the clean data signal from the central microphone at the geometric center of the circle in a multi-microphone array. The signals are, in order, the multi-channel clean data signals from n peripheral microphones at the circumference of the multi-microphone array.

[0019] Preferably, in the multi-channel signal preprocessing module, the process of separating the multi-channel clean data signals from multiple sound sources is as follows:

[0020] The multi-channel clean data signals are sequentially processed using beamforming technology and the FastICA (Fast Independent Component Analysis) algorithm for multi-source separation. The beamforming technology utilizes a circular array of multiple microphones. Omnidirectional sensing characteristics, in An angular scan is performed within the azimuth range. During the scan, with... The step size is used to sequentially... The algorithm processes signals from 360 directions. By constructing a directional beamformer, it performs spatial filtering and energy enhancement on the sound source signals from each direction, while suppressing signal interference from other directions, thus obtaining a set of directional enhanced signals covering all directions. Based on this, the FastICA algorithm is based on the principle of statistical independence of signals. It assumes that multiple sound source signals are statistically independent and solves the separation matrix through an iterative optimization process, decomposing the mixed directional enhanced signals into several independent single sound source signal streams. Considering the spatial resolution of the circular multi-microphone array and the computational complexity of the algorithm, the maximum number of single sound source signal streams is limited to no more than 3 to ensure a balance between separation accuracy and real-time performance.

[0021] Preferably, in the sound signal feature extraction module, the process of obtaining the corresponding single-source feature vector from a single-source signal stream through feature extraction is as follows:

[0022] The single-source signal stream was framed with a frame length of 20ms and an overlap rate of 50%, resulting in... A single-source signal in a segmented frame Based on this, A single-source signal in a segmented frame The short-time energy was calculated using the short-time energy formula, the time-domain peak value formula, the zero-crossing rate formula, and the short-time average amplitude difference formula, respectively. Time-domain peak Zero crossing rate and short-term average amplitude difference ;

[0023] At the same time A single-source signal in a segmented frame All were processed using Fast Fourier Transform to obtain A single-source signal in a segmented frame Corresponding spectral amplitude The spectral centroid is calculated by substituting the formulas for spectral centroid, spectral flux, and spectral bandwidth into the formulas. Spectral flux and spectrum bandwidth Based on this, we can obtain information including short-term energy. Time-domain peak Zero crossing rate Short-term average amplitude difference Spectral centroid Spectral flux and spectrum bandwidth The feature vector of a single sound source;

[0024] The short-time energy formula is as follows: ;

[0025] Where N is the frame length, It is the sampling point index;

[0026] The formula for the centroid of the spectrum is: ;

[0027] in, It is the first The actual frequency value at each frequency point It is the number of points in the FFT. It is a frequency point index.

[0028] Preferably, in the abnormal sound recognition module, the abnormal sound recognition model is a model architecture integrating a bidirectional long short-term memory network and an attention mechanism, specifically including:

[0029] The input layer is used to convert the feature vector of a single sound source into... A 7-dimensional feature matrix, where The number of single-source signals per frame;

[0030] Feature embedding layer, used to... The 7-dimensional feature matrix is ​​used for feature space mapping and dimension adaptation to obtain... ×d-dimensional temporal embedding features, where d is greater than 7;

[0031] BiLSTM layers are used for... ×d-dimensional temporal embedding features are used to capture bidirectional long-short-term dependencies, resulting in The temporal dependency features are of the ×(2×h) dimension, where h is the dimension of the LSTM hidden layer;

[0032] Attention layer, used for Weights are assigned to the time-dependent features of dimension ×(2×h), and the weights are then used to determine the appropriate features. We perform a weighted summation of the 2×h-dimensional features to obtain a 1×(2×h)-dimensional global feature.

[0033] The fully connected layer is used to gradually map the 1×(2×h)-dimensional global features to a dimension that matches the number of abnormal sound recognition results through a fully connected network, resulting in 1×4-dimensional classification features.

[0034] The output layer is used to output the classification probability of the 1×4-dimensional classification features through the softmax activation function. The output layer outputs the probability distribution of glass breakage, screaming, explosion and no abnormality. The category with the highest probability is the abnormal sound recognition result, which is explosion, screaming, glass breakage or no abnormality.

[0035] Preferably, in the sound source localization module, the specific steps for multi-stage localization processing of multi-channel clean data signals are as follows:

[0036] S1. When the abnormal sound recognition module outputs an explosion, scream, or broken glass, the sound source localization module is triggered. Utilizing a dual-layer structure of a circular multi-microphone array—one microphone at the center and n evenly distributed microphones around the circumference—the module transmits the clean data signal from the central microphone. As a reference signal, it is compared with the clean data signals of n peripheral microphones respectively. One-to-one pairing, forming n microphone signal pairs. This pairing method utilizes the spatial centrality of the circular microphone to ensure that the time delay difference calculation benchmark of each microphone signal pair is consistent, reducing errors caused by the geometric asymmetry of the multi-microphone array;

[0037] S2. The GCC-PHAT algorithm (a generalized cross-correlation algorithm based on phase transformation) is used to calculate the microphone pair delay difference for each microphone signal pair. Specifically, for each microphone signal pair... First, the time-domain signal is converted to the frequency domain using a short-time Fourier transform. The cross-power spectrum is calculated and phase-transform weighting is applied. Then, the weighted cross-correlation function is obtained through an inverse Fourier transform. The peak position of this function represents the microphone pair delay difference when the two microphones receive the same sound signal. ;

[0038] Time delay difference of each microphone Substituting into the TDOA (Time Difference of Arrival) equation: ,in, These are the three-dimensional coordinates of the initial position estimate. These are the coordinates of the center microphone. It is the first The coordinates of the peripheral microphone, It is the speed of sound, and ultimately a system of TDOA equations containing n TDOA equations is constructed;

[0039] S3. The TDOA equations are solved using spherical interpolation. Based on the principle of spherical intersections, each microphone is considered as the center of a sphere, and a sphere is constructed with the speed of sound multiplied by the signal propagation time as its radius. Theoretically, the initial position estimate's three-dimensional coordinates are the intersection points of all spheres. Due to noise in actual measurements, the distance error from each sphere to the candidate point is minimized, and the Levenberg-Marquardt nonlinear optimization algorithm is used iteratively to obtain the initial position estimate. ;

[0040] S4. Using the initial position estimate as the center point, the MDS-MUSIC algorithm is used to search and improve positioning accuracy. The MDS-MUSIC algorithm (Multidimensional Scale Analysis Multi-Signal Classification Algorithm) combines the super-resolution characteristics of multi-signal classification with the spatial mapping capability of multidimensional scaling at the center point. Azimuth , Angle of elevation and distance Constructing a spatial spectral function within a defined range:

[0041] ,in It is an array manifold vector. It is the noise subspace matrix, and by searching for the peak positions of the spatial spectrum, the three-dimensional coordinates of the abnormal sound source are determined. This refers to locating the source of abnormal sounds.

[0042] Preferably, when multiple abnormal sounds appear in the abnormal sound recognition results, the difference in the spectral centroid of various abnormal sounds is used to compare each abnormal sound in the abnormal sound recognition results with the spectral centroid features of a single sound source signal stream. Based on this, each abnormal sound is associated with the corresponding single sound source signal stream. For each associated single sound source signal stream, multi-stage localization processing is performed on the multi-channel clean data signal in parallel, that is, S1, S2, S3 and S4 are executed in parallel to obtain the abnormal sound location corresponding to each abnormal sound. Among them, the spectral centroid of the explosion is less than 500Hz, the spectral centroid of the scream is 2 to 5kHz, and the spectral centroid of the glass breakage is 3 to 8kHz.

[0043] In the parallel execution of S1, S2, S3, and S4, the spatial spectrum function in S4 is adjusted to... ,in, These correspond to explosion, scream, and shattered glass, respectively. It is a weighting function for the frequency band corresponding to the abnormal sound;

[0044] pass The system enhances the weight of frequencies below 500Hz for explosions, enhances the weight of frequencies from 2 to 5kHz for screams, and enhances the weight of frequencies from 3 to 8kHz for glass breakage. This achieves spectral peak separation for each abnormal sound, with each abnormal sound corresponding to an independent peak in the spatial spectrum. The peak position is then extracted as the location of the abnormal sound for each sound.

[0045] Preferably, in the alarm module, the process of triggering an alarm based on the abnormal sound recognition result and the location of the abnormal sound is as follows:

[0046] Based on the abnormal sound recognition results and the location of the abnormal sound, an alarm message is generated and pushed to a preset mobile phone number or APP server via 4G / 5G cellular network.

[0047] The alarm information is constructed using a standardized data format, including the abnormal sound type, location coordinates, and timestamp. The abnormal sound type is an explosion, a scream, or broken glass, and the location coordinates are the location of the abnormal sound source converted to latitude and longitude format.

[0048] Due to the adoption of the above technical solution, the technical progress achieved by this invention compared to the prior art is as follows:

[0049] 1. This invention significantly improves the anti-interference capability and multi-target processing efficiency of abnormal sound monitoring in complex environments. By adopting a circular array layout of n peripheral microphones evenly distributed plus one central microphone, combined with a beam domain noise reduction algorithm based on spatial correlation and an NLMS multi-channel adaptive echo canceller, it achieves efficient suppression of noise and echo. At the same time, through beamforming technology and the two-stage separation strategy of the FastICA algorithm, it can accurately separate up to 3 single sound source signal streams, solving the confusion or missed detection problem caused by the superposition of multiple sound sources in the prior art. It supports parallel processing of multiple abnormal sounds at the same time and different locations, as well as at different times and different locations.

[0050] 2. This invention achieves breakthroughs in recognition accuracy and positioning performance. By extracting seven key features covering the time and frequency domains, and combining them with an abnormal sound recognition model based on BiLSTM and attention mechanisms, it captures cross-frame temporal sequences bidirectionally and weights key frames, thereby improving the accuracy of abnormal sound recognition. At the same time, it adopts a multi-stage positioning strategy of GCC-PHAT delay estimation, spherical interpolation initial positioning, and MDS-MUSIC fine search, and utilizes the symmetry of a circular multi-microphone array to optimize spatial spectrum estimation, achieving accurate output of the three-dimensional coordinates of the abnormal sound source. It also supports concurrent positioning of multiple abnormal sounds, providing accurate spatial information for emergency response and significantly enhancing the practical value of the system. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0052] Figure 1 This is a schematic diagram of the system functional modules of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Examples, such as Figure 1 The aforementioned sound acquisition and processing system based on multi-microphone array collaboration includes a sound acquisition module, a multi-channel signal preprocessing module, a sound signal feature extraction module, an abnormal sound recognition module, a sound source localization module, and an alarm module.

[0055] The sound acquisition module is used to acquire multi-channel mixed data signals through a multi-microphone array;

[0056] The multi-channel signal preprocessing module is used to perform echo cancellation and beam domain noise reduction on the multi-channel mixed data signal to obtain a multi-channel clean data signal, and to separate the multi-channel clean data signal into multiple sound sources to obtain several single sound source signal streams.

[0057] The sound signal feature extraction module is used to extract the corresponding single sound source feature vector from the single sound source signal stream, and thereby obtain the single sound source feature vectors corresponding to all single sound source signal streams.

[0058] The abnormal sound recognition module is used to construct an abnormal sound recognition model and input the single sound source feature vectors corresponding to all single sound source signal streams into the abnormal sound recognition model for processing to obtain the abnormal sound recognition result.

[0059] The sound source localization module is used to perform multi-stage localization processing on multi-channel clean data signals based on the abnormal sound recognition results to obtain the location of the abnormal sound.

[0060] The alarm module is used to issue an alarm based on the abnormal sound recognition result and the location of the abnormal sound.

[0061] Furthermore, the working principle of the present invention will be illustrated below through embodiments:

[0062] Take the scenario of abnormal sound monitoring in indoor public places as an example.

[0063] A circular multi-microphone array is employed, consisting of 16 peripheral microphones (meeting the design requirement of n>10). The acoustic centers of all peripheral microphones are located on the same horizontal plane, evenly distributed on a circle with a diameter of 30cm centered on the array's physical center. The angle between the centers of adjacent microphones is 360° / 16=22.5°, ensuring uniform spatial sampling across the 360° omnidirectional range and avoiding sensitivity attenuation in specific locations. A reference microphone of the same model is deployed at the center, with its acoustic center at the same horizontal plane as the peripheral microphones, ultimately forming a 17-channel acquisition array. MEMS microphones are selected, and high-performance microphones are used accordingly. The precision synchronization clock module generates a reference clock signal, which serves as a unified time reference for the main control unit and 17 microphones, effectively avoiding sound source localization time difference errors caused by asynchronous sampling. The synchronous sampling technology is implemented through the synchronous serial bus SPI. The main control unit sends a synchronization trigger signal to all microphones through the SPI bus to control the 17 microphones to simultaneously collect sound signals. The sampling parameters are set to a sampling rate of 48kHz and a sampling bit depth of 24bit. After the sampled data is converted in parallel by a multi-channel ADC, a multi-channel mixed data signal is generated, which includes ambient background sound, human voices, and possible abnormal sounds, such as the sound of breaking glass and screams.

[0064] For multi-channel mixed data signals, a beam domain noise reduction algorithm based on spatial correlation is first applied. Firstly, the spatial correlation characteristics of signals collected by 17 microphones in a circular multi-microphone array are used to construct a 17×17 spatial covariance matrix. This spatial covariance matrix is ​​then decomposed into signal and noise subspaces using eigenvalue decomposition. The target signal is reconstructed through the signal subspace, preserving the sound signal components in the mall's atrium area while suppressing environmental noise from non-target directions such as mall edges and corridors. Next, a multi-channel adaptive echo canceller based on the NLMS algorithm uses the signal collected by the central microphone as a reference signal. The NLMS adaptive filtering algorithm estimates the echo path transfer function in real time, including echo propagation patterns caused by wall and ceiling reflections. Based on the estimated transfer function, a cancellation signal with similar amplitude but opposite phase to the echo signal is generated. This cancellation signal is then adaptively cancelled one by one with the beam domain noise reduction signals collected by the 16 peripheral microphones to eliminate echo interference caused by building structure reflections, ultimately outputting a clean multi-channel data signal. ,in The signal is from the central microphone. to The signal is from the peripheral microphones. For the multi-channel clean data signal, beamforming technology is first used to utilize the 360° omnidirectional sensing characteristic of a circular multi-microphone array. The array scans angle by angle in 1° increments within the 0-360° azimuth range, processing 360 directions sequentially from 0°, 1°, 2°...359°. For each direction, a directional beamformer is constructed. By adjusting the delay and weight of each microphone signal, spatial filtering and energy enhancement are performed on the sound source signal in that direction, while suppressing signal interference from other directions. This ultimately yields a set of directional enhanced signals covering all directions. Based on the principle of statistical independence of signals, the directional enhanced signal set is input into the FastICA algorithm. The separation matrix is ​​solved through an iterative optimization process. The separation matrix is ​​used to decompose the mixed directional enhanced signals into independent single-source signal streams. Considering the array spatial resolution and algorithm computational complexity, the maximum number of single-source signal streams is limited to 3. In this embodiment, 2 single-source signal streams were successfully separated. Assuming that glass breaking and screaming are detected simultaneously, they are denoted as... (Broken glass) and (Screams).

[0065] For single-source signal stream and Frames are divided into two groups based on a 20ms frame length and a 50% overlap rate. Using a 48kHz sampling rate, each frame contains N = 48000Hz × 0.02s = 960 sampling points. A 50% overlap rate means that adjacent frames overlap by 480 sampling points, ensuring the continuity of features between frames. The final result is... Each frame is a single-source signal, and for each frame, it is a single-source signal. Using the short-time energy formula: Calculate short-time energy , Time-domain peak formula: Calculate peak , Zero-crossing rate formula: Calculate the zero-crossing rate , Short-time average amplitude difference formula: calculate Simultaneously, an FFT was performed on the single-source signal in each frame, with 512 points in the FFT, to obtain the spectral amplitude. , Substituting into the formula for the centroid of the spectrum: calculate (Broken glass) Concentrated at 3-8kHz, screaming Concentrated at 2-5kHz). Spectral flux formula: calculate Spectrum bandwidth formula: calculate This ultimately forms the single-source feature vector for each single-source signal stream, containing... .

[0066] Two single-source feature vectors are input into the abnormal sound recognition model for processing. The input layer receives a 50×7 dimensional feature matrix. The feature embedding layer maps the 50×7 dimensional feature matrix to a 50×64 dimensional temporal embedding feature. A BiLSTM layer (128 hidden units) bidirectionally captures temporal dependencies on the 50×64 dimensional temporal embedding feature, outputting a 50×256 dimensional feature. The attention layer weights keyframes, such as the impact frame of glass breaking, and outputs a 1×256 dimensional global feature. The fully connected layer maps the 1×256 dimensional global feature to a 1×4 dimensional feature. The output layer uses a softmax activation function to classify the 1×4 dimensional feature and output the classification probability, thus obtaining the abnormal sound recognition result. Because the glass broke. The abnormal sound recognition model is a pre-trained model. It uses 10,000 single-source feature markers randomly selected from an abnormal sound feature library to form a sample set. This sample set is divided into a 7:3 ratio (7000 markers) and a test set (3000 markers). The initialized abnormal sound recognition model is first trained using the training set. After training, the test set is input into the trained model to obtain the predicted abnormal sound recognition result for each single-source feature marker in the test set. In this embodiment, the number of samples where the predicted abnormal sound recognition result differs from the labeled abnormal sound recognition result is 58. This number accounts for less than 3% of the total number of samples in the test set, meeting the design accuracy requirements. The abnormal sound feature library stores single-source feature marker data, which are single-source feature vectors used to label the abnormal sound recognition results.

[0067] Based on the abnormal sound recognition results, multi-stage localization is performed in parallel on two single-source signal streams. A two-layer structure of a circular multi-microphone array—one microphone at the center and n uniformly distributed microphones around the circumference—is used to extract the clean data signal from the central microphone. As a reference signal for constructing the microphone signal pair, it was compared with the clean data signals of the 16 peripheral microphones. One-to-one pairing is used to form two groups of 16 microphone signal pairs each. This pairing method utilizes the spatial centrality of the circular microphone to ensure that the time delay difference calculation benchmark of each microphone signal pair is consistent, reducing errors caused by the geometric asymmetry of the multi-microphone array.

[0068] The time delay difference of each microphone signal pair was calculated using the GCC-PHAT algorithm. Specifically, for each group of microphone signals First, the time-domain signal is converted to the frequency domain using a short-time Fourier transform. The cross-power spectrum is calculated and phase-transform weighting is applied. Then, the weighted cross-correlation function is obtained through an inverse Fourier transform. The peak position of this function represents the time delay difference between the two microphones receiving the same sound signal. The time delay difference of each microphone signal pair Substitute into the TDOA equation: A system of 16 TDOA equations was constructed, and the initial location of glass breakage was estimated using spherical interpolation. and screaming initial position estimation Based on this, using the estimated initial locations of glass breakage and screams as the center points, the MDS-MUSIC algorithm is used to search for and improve positioning accuracy. The MDS-MUSIC algorithm combines the super-resolution characteristics of multi-signal classification with the spatial mapping capability of multi-dimensional scale analysis to construct a spatial spectrum function within ±15° azimuth angle, ±10° elevation angle, and ±1m distance. By searching for the peak position of the spatial spectrum, the (θ,φ,r) corresponding to the peak position of the spatial spectrum is the precise location and distance of the sound source. Based on this, the abnormal sound source location of the glass breaking (5.2m, 3.1m, 1.2m) and the abnormal sound source location of the scream (7.5m, 2.8m, 1.6m) are determined.

[0069] Since the identification results are two abnormal sounds, signal correlation is performed based on the difference in spectral centroids: the spectral centroid corresponding to an explosion is less than 500Hz, the spectral centroid corresponding to a scream is 2-5kHz, and the spectral centroid corresponding to broken glass is 3-8kHz. In this embodiment, the identified "broken glass" is correlated with the single-source signal stream. Related, "screaming" and Correlation; For the two correlated signal streams, multi-stage localization processing is performed in parallel to achieve concurrent and accurate localization of multiple abnormal sounds and avoid confusion in localization results.

[0070] Based on the abnormal sound recognition results (glass breakage, screaming) and the location of the abnormal sound, standardized alarm information is generated, including the abnormal sound type as glass breakage and the confidence level of screaming, the location coordinates as glass breakage (30°15'22"N, 120°05'10"E) and screaming (30°15'23"N, 120°05'09"E), and the timestamp 2024-06-10T14:30:15.236Z. The alarm information is then pushed to the security APP via 4G network and triggers on-site audible and visual alarms, enabling rapid response to abnormal events.

[0071] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A sound acquisition and processing system based on multi-microphone array collaborative operation, characterized in that, Includes the following modules; The sound acquisition module is used to acquire multi-channel mixed data signals through a multi-microphone array; The multi-channel signal preprocessing module is used to perform echo cancellation and beam domain noise reduction on multi-channel mixed data signals to obtain multi-channel clean data signals and perform multi-source separation to obtain several single-source signal streams. The sound signal feature extraction module is used to extract the corresponding single-source feature vector from the single-source signal stream, and thereby obtain the single-source feature vectors corresponding to all single-source signal streams. The abnormal sound recognition module is used to construct an abnormal sound recognition model and input the feature vectors of a single sound source into the abnormal sound recognition model for processing to obtain the abnormal sound recognition result. The sound source localization module performs multi-stage localization processing on multi-channel clean data signals based on the abnormal sound recognition results to obtain the location of the abnormal sound. The alarm module triggers an alarm based on the abnormal sound recognition results and the location of the abnormal sound.

2. The sound acquisition and processing system based on multi-microphone array collaborative processing according to claim 1, characterized in that, The sound acquisition module is implemented as follows: The peripheral microphone array consists of n microphones, where n is greater than 10. The acoustic centers of all microphones are located in the same horizontal plane and are evenly distributed on the circumference with the physical center of the array as the center. Meanwhile, the central microphone is deployed at the geometric center of the circumference, and its acoustic center is at the same height as the horizontal plane of the peripheral microphones, forming a circular multi-microphone array. A reference clock signal is generated using a synchronous clock module, which serves as the time reference for the main control unit and the microphones. Based on this, the synchronous sampling technology enables the main control unit to trigger and control all microphones through the synchronous serial bus SPI. The sampled data is converted in parallel by a multi-channel analog-to-digital converter (ADC) to obtain a multi-channel mixed data signal.

3. The sound acquisition and processing system based on multi-microphone array collaboration according to claim 1, characterized in that, In the multi-channel signal preprocessing module, echo cancellation and beam domain noise reduction of the multi-channel mixed data signal are specifically implemented as follows: For the environmental noise contained in the multi-channel mixed data signal, a beam domain denoising algorithm based on spatial correlation is used for beam domain denoising. The beam domain denoising algorithm based on spatial correlation utilizes the spatial correlation characteristics of the signals collected by different microphones in a circular multi-microphone array. By constructing a spatial covariance matrix and performing eigenvalue decomposition, the signal subspace and noise subspace are separated. For the multi-channel signals collected by all peripheral microphones evenly distributed on the circumference and one microphone at the center, the beam domain denoising algorithm based on spatial correlation retains the signal component from the direction of the target sound source while suppressing the noise component from the non-target direction, and finally obtains the multi-channel mixed data signal after beam domain denoising. The multi-channel mixed data signal after beam-domain denoising is processed using a multi-channel adaptive echo canceller based on the NLMS adaptive filtering algorithm. The NLMS-based multi-channel adaptive echo canceller uses the signal acquired by the central microphone as a reference signal. Through adaptive filtering, it estimates the transfer function of the echo path in real time, generating a cancellation signal with similar amplitude but opposite phase to the echo signal. This cancellation signal is then adaptively cancelled one by one with the beam-domain denoised signals acquired by the n peripheral microphones to obtain a clean multi-channel data signal. ,in This refers to the clean data signal from the central microphone at the geometric center of the circle in a multi-microphone array. The signals are, in order, the multi-channel clean data signals from n peripheral microphones at the circumference of the multi-microphone array.

4. The sound acquisition and processing system based on multi-microphone array collaborative processing according to claim 3, characterized in that, The multi-source separation is specifically implemented as follows: The multi-channel clean data signals are sequentially processed using beamforming technology and the FastICA algorithm for multi-source separation. The beamforming technology utilizes a circular array of multiple microphones. Omnidirectional sensing characteristics, in An angular scan is performed within the azimuth range. During the scan, with... The step size is used to sequentially... The signal is processed in 360 directions. By constructing a directional beamformer, spatial filtering and energy enhancement are performed on the sound source signal in each direction, while suppressing signal interference from other directions, to obtain a set of directional enhancement signals covering all directions. Based on this, the FastICA algorithm is based on the principle of statistical independence of signals. It assumes that multiple sound source signals are statistically independent and solves the separation matrix through an iterative optimization process, decomposing the mixed directional enhancement signal into several independent single sound source signal streams. Considering the spatial resolution of the circular multi-microphone array and the computational complexity of the algorithm, the maximum number of single sound source signal streams is limited to no more than 3.

5. The sound acquisition and processing system based on multi-microphone array collaborative processing according to claim 1, characterized in that, The audio signal feature extraction module is implemented as follows: The single-source signal stream is framed to obtain A single-source signal in a segmented frame Based on this, A single-source signal in a segmented frame The short-time energy was calculated using the short-time energy formula, the time-domain peak value formula, the zero-crossing rate formula, and the short-time average amplitude difference formula, respectively. Time-domain peak Zero crossing rate and short-term average amplitude difference ; At the same time A single-source signal in a segmented frame All were processed using Fast Fourier Transform to obtain A single-source signal in a segmented frame Corresponding spectral amplitude The spectral centroid is calculated by substituting the formulas for spectral centroid, spectral flux, and spectral bandwidth into the formulas. Spectral flux and spectrum bandwidth Based on this, we can obtain information including short-term energy. Time-domain peak Zero crossing rate Short-term average amplitude difference Spectral centroid Spectral flux and spectrum bandwidth The single-source feature vector.

6. The sound acquisition and processing system based on multi-microphone array collaborative processing according to claim 1, characterized in that, In the abnormal sound recognition module, the abnormal sound recognition model is a model architecture that integrates a bidirectional long short-term memory network and an attention mechanism. The specific implementation process is as follows: The input layer is used to convert the feature vectors of a single sound source into a feature matrix; The feature embedding layer maps the feature matrix to the feature space and adapts its dimensions to obtain temporal embedded features. The BiLSTM layer performs bidirectional long-short-term dependency capture on the temporal embedding features to obtain temporal dependency features. The attention layer assigns weights to temporally dependent features and performs a weighted summation of these features based on the weights to obtain the global features. Fully connected layers are used to progressively map global features to a dimension that matches the number of abnormal sound recognition results through a fully connected network, thereby obtaining classification features; The output layer is used to output the classification probability of the classification features through the softmax activation function. The category with the highest probability is the abnormal sound recognition result.

7. The sound acquisition and processing system based on multi-microphone array collaborative processing according to claim 1, characterized in that, The sound source localization module is implemented as follows: When the abnormal sound recognition module outputs an abnormal result, the sound source localization module is triggered. Utilizing a dual-layer structure of a circular multi-microphone array—one microphone at the center and n evenly distributed microphones around the circumference—the module detects the clean data signal from the central microphone. As a reference signal, it is paired with the clean data signals of n peripheral microphones to form n microphone signal pairs. ; The GCC-PHAT algorithm, a generalized cross-correlation algorithm based on phase transform, is used to calculate the microphone pair delay difference for each microphone signal pair. For each microphone signal pair, the time-domain signal is first converted to the frequency domain using a short-time Fourier transform, the cross-power spectrum is calculated and phase transform weighting is applied, and then the weighted cross-correlation function is obtained through inverse Fourier transform. The peak position of this function represents the microphone pair delay difference when the two microphones receive the same sound signal. ; the time delay difference of each microphone Substituting the Time Difference of Arrival (TDOA) equation into the equations, a system of TDOA equations containing n TDOA equations is constructed. The spherical interpolation method is used to solve the TDOA equation system. The spherical interpolation method is based on the principle of spherical intersections. Each microphone is regarded as the center of a sphere, and a sphere is constructed with the speed of sound multiplied by the signal propagation time as the radius. The initial position estimate is theoretically the intersection of all spheres. By minimizing the distance error from each sphere to the candidate point, the Levenberg-Marquardt nonlinear optimization algorithm is used to iteratively solve the problem to obtain the initial position estimate. Using the initial position estimate as the center point, the search is performed using the Multi-Scaling Analysis (MDS-MUSIC) algorithm, a multi-signal classification algorithm. The MDS-MUSIC algorithm combines the super-resolution characteristics of multi-signal classification with the spatial mapping capability of multi-dimensional scaling to find the center point. Azimuth , Angle of elevation and distance Within a defined range, a weighted spatial spectrum function is constructed. By searching for the peak position of the spatial spectrum, the three-dimensional coordinates of the abnormal sound source are determined, i.e., the location of the abnormal sound.

8. The sound acquisition and processing system based on multi-microphone array collaborative processing according to claim 7, characterized in that, When multiple abnormal sounds are detected in the abnormal sound recognition results, the difference in the spectral centroid of each abnormal sound is used to compare the spectral centroid features of each abnormal sound with those of a single sound source signal stream. Based on this, each abnormal sound is associated with the corresponding single sound source signal stream. For each associated single sound source signal stream, multi-stage localization processing is performed on the multi-channel clean data signal in parallel to obtain the abnormal sound location corresponding to each abnormal sound.

9. The sound acquisition and processing system based on multi-microphone array collaborative processing according to claim 1, characterized in that, The alarm module is specifically implemented as follows: it generates alarm information based on the abnormal sound recognition result and the location of the abnormal sound, and pushes the alarm information to a preset mobile phone number or APP server through a cellular network; the alarm information is constructed using a standardized data format, including the abnormal sound type, location coordinates and timestamp.

Images