Sound source positioning and detecting method and device

Abstract

The invention belongs to the technical field of sound source processing, and provides a sound source positioning and detecting method and device. The method comprises the following steps: acquiring delay estimation of microphones I and II and delay estimation of microphones I and III based on a three-path linear uniform microphone array; based on the delay estimation sum, respectively carrying out positioning calculation under near-field and far-field conditions; according to the sound source distance under the near-field condition, comparing the sound source distance with a distance judgment threshold value, and determining a far-field / near-field output sound source position; and carrying out feature extraction on the signals of any microphone array, and carrying out event classification based on a pre-trained convolutional neural network. According to the method, far / near field model selection is carried out according to the distance judgment threshold, large deviation generated by a single model in a critical region is avoided, continuous and stable positioning from short distance to long distance is achieved, time classification can be achieved while position calculation is carried out, and integrated output is achieved.

Description

Technical Field

[0001] This invention relates to the field of sound source processing technology, specifically to a sound source localization and detection method and apparatus. Background Technology

[0002] Sound event detection (SED) and localization refers to estimating the direction of arrival (DOA), origin, and offset of detected sound events, while classifying them into sound categories.

[0003] Convolutional Neural Networks (CNNs) are currently the mainstream architecture for SEDs. They are trained using log-Mel spectrograms as feature inputs. They are more effective than traditional classifiers in solving complex sound detection and classification problems. Moreover, compared to RNNs and LSTMs, they often achieve good detection results with fewer layers.

[0004] Sound source localization (DOAE) focuses on identifying the location of sound sources, primarily using parametric and learning-based methods. The most widely used parametric method is Time Difference of Arrival (TDOA) estimation. Since TDOA information is transmitted through the phase of the cross spectrum rather than the amplitude, the Gaussian Cross-Spectrum Transform (GCC-PHAT) was proposed. This method eliminates the influence of amplitude, retaining only the phase. Learning-based methods mainly utilize deep neural networks to enable the system to learn the relationship between input features and DOA. Currently, in addition to spectral features, GCC-based features (which effectively provide time difference information) are also used as input features. Learning-based methods have the advantage of good generalization ability under different reverberation and noise levels.

[0005] The latest research method is Solid Voice Event Localization and Detection (SELD), which combines sound event detection and localization. It uses amplitude and phase spectrograms as input features and jointly trains SED and DOAE targets. This not only estimates their respective spatial locations but also identifies the type and temporal information of the sounds. However, phase spectrograms are difficult for neural networks to learn, and the influence and potential relationship between SED and DOAE have not yet been revealed. Ordinary joint training of both degrades performance in various directions, and performance improvements are mainly achieved by modifying the training method. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a sound source localization and detection method and apparatus to solve the problem of low accuracy in the output results when performing integrated detection of sound event localization and classification.

[0007] In a first aspect, the present invention provides a method for sound source localization and detection, comprising: Based on a three-channel linear uniform microphone array, delay estimates for microphones I and II are obtained. Delay estimation with microphones I and III ; Based on delay estimation and Positioning calculations were performed for near-field and far-field scenarios respectively; in the far-field scenario, the calculation was based on time difference. Calculate the azimuth angle In the near field, based on time difference and The location information of the sound source was calculated; Based on the sound source distance in the near field, the distance determination threshold is compared to determine the far field / near field output sound source location; Feature extraction is performed on the signal from any microphone array, and event classification is performed based on a pre-trained convolutional neural network.

[0008] As can be seen from the above technical solution, the sound source localization and detection method provided by the present invention calculates two models, far-field and near-field, in parallel on the same observation frame and selects the model based on a physically interpretable distance threshold, avoiding large deviations caused by a single model in the critical region (near / far-field boundary), and achieving continuous and stable localization from near to far distance; after localization is completed, acoustic features (such as logarithmic Mel-filter spectrum, MFCC, time-frequency diagram) are extracted from any signal of the array, and bird sound source event classification is performed based on a pre-trained convolutional neural network, which can realize location localization and event classification in an integrated manner.

[0009] Optionally, before obtaining the delay estimate, preprocessing of the three-microphone array is also included, including: Pre-emphasis processing is performed on the audio signal; Perform frame segmentation processing on the audio signal; Hamming windowing is applied to the framed signal. Perform FFT transformation on the three microphone signals.

[0010] Optionally, the delay estimates of microphones I and II are obtained based on a three-line linear uniform microphone array. Delay estimation with microphones I and III ,include: Cross-power spectrum analysis was performed on microphones I and II and microphones I and III respectively; Weighting of cross-power spectra based on PHAT method; Perform inverse FFT to determine the cross-correlation function; Peak detection is performed on the cross-correlation function to determine the delay estimate.

[0011] Optionally, the weighting of the cross-power spectrum based on the PHAT method is specifically implemented using the phase method in the FPGA, including: Substituting into the CORDIC algorithm and Calculate the corresponding phase value ; Represents the cross power spectrum. , They represent the real part and the imaginary part, respectively. Using the CORDIC rotation mode based on the phase value The output (cosϕ, sinϕ) corresponds to the weighted, normalized real and imaginary parts of the complex number.

[0012] Optionally, the delay-based estimation and Positioning calculations are performed for near-field and far-field conditions, including: For the far-field case, delay estimation The calculated azimuth angle ;d represents the spacing between adjacent microphones; For the near-field case, the coordinate information of the sound source is represented as follows: The distance of the sound source is The distance to the origin; where microphones I, II, and III are located at (0,0), (0,d), and (0,2d) respectively. , c is the speed at which the sound source propagates.

[0013] Optionally, determining the far-field / near-field output sound source location by comparing the sound source distance in the near-field condition with a distance determination threshold includes: Based on the distance to the sound source location and the distance determination threshold A comparison is made to determine the location of the far-field / near-field output sound source; among which, the distance determination threshold is used. d is the spacing between the array elements, and n=3 is the number of arrays in a linear arrangement. The wavelength of the signal; When the distance to the sound source is greater than When it is far field, it is near field; otherwise, it is near field.

[0014] Optionally, the feature extraction of the signal from any microphone array includes: Feature extraction is performed on the preprocessed microphone array signal based on the Mel filter bank to obtain the processed power spectrum data; The processed power spectrum data is biased by a constant, and then a logarithmic transformation is performed to obtain the logarithmic result.

[0015] Optionally, the convolutional neural network includes: Convolutional layers generate new feature maps by performing weighted summation operations on each window of the feature map using a sliding window approach, thereby extracting local information. Pooling layers are used to downsample the output of convolutional layers; Fully connected layers are used to transform new feature maps into predicted classification results.

[0016] Secondly, the present invention provides a sound source localization and detection device, comprising: The delay estimation module is used to obtain delay estimates for microphones I and II based on a three-line linear uniform microphone array. Delay estimation with microphones I and III ; The positioning calculation module is used for delay estimation. and Positioning calculations were performed for near-field and far-field scenarios respectively; in the far-field scenario, the calculation was based on time difference. Calculate the azimuth angle In the near field, based on time difference and The location information of the sound source was calculated; The position output module is used to determine the position of the far-field / near-field output sound source by comparing the sound source distance in the near-field case with a distance determination threshold; The feature classification module is used to extract features from the signal of any microphone array and classify events based on a pre-trained convolutional neural network.

[0017] By adopting the above technical solution, this application has the following beneficial effects: (1) Far / near field adaptive positioning, covering the entire working range: By performing parallel calculations of both far-field and near-field models on the same observation frame and selecting the model based on a physically interpretable distance threshold, large deviations caused by a single model in the critical region (near / far-field boundary) are avoided, thus achieving continuous and stable positioning from near to far distance.

[0018] (2) Combining three arrays with two time differences improves solution stability: Based on a three-channel linear uniform microphone array, two independent time delay estimates (I–II, I–III) are obtained simultaneously for geometric constraints: In the near-field condition, the two TDOAs are combined into hyperboloid (or hyperbolic under the plane assumption) equations to improve the observability and stability of the position solution; compared with the two-microphone method, the robustness and noise resistance are significantly enhanced.

[0019] (3) Noise-robust delay estimation link: The standard acoustic front end adopts pre-emphasis-framing-Hamming window-FFT, and uses PHAT weighting in the frequency domain to suppress the influence of spectral amplitude fluctuations, retaining only phase information to participate in correlation, which significantly improves peak detectability in low signal-to-noise ratio and reverberant environments and reduces the interference of multipath on time difference estimation.

[0020] (4) Subsampling precision time delay refinement: By performing parabolic interpolation on the peak and its adjacent points on the generalized cross-correlation sequence obtained by IFFT, the latency resolution can be improved from "integer sampling points" to "subsampling level" without significantly increasing hardware resources, thereby reducing the variance of angle / distance calculation and improving the overall positioning accuracy.

[0021] (5) Real-time low latency, hardware-friendly, and low implementation cost: The entire chain can be implemented in a pipeline on FPGA: complex multiplication - PHAT phase method - IFFT - peak detection - interpolation - geometric solution; no need for high-cost multi-array or external synchronization beacons, small hardware resource consumption, low latency, easy field deployment and low power consumption operation.

[0022] (6) Integrated positioning and identification: After localization is completed, acoustic features (such as logarithmic Mel-filter spectrum, MFCC, and time-frequency graph) are extracted from any signal of the array, and bird sound source events are classified based on a pre-trained convolutional neural network: location localization and event classification can be achieved in one integrated manner.

[0023] (7) Good scalability and compatibility: The solution is compatible with array size and shape: the linear ternary array is the minimum configuration and can be expanded to longer apertures or multiple arrays in tandem; the classification module supports transfer learning and is compatible with public bird acoustic datasets and self-collected data, making it easy to quickly adapt to different regions and species.

[0024] In summary, compared with existing solutions that use only a single model or do not include event recognition, this invention achieves comprehensive improvements in distance coverage, positioning accuracy, noise resistance stability, real-time performance, and application completeness, and can be implemented in engineering with lower hardware complexity. Attached Figure Description

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0026] Figure 1 A flowchart of a sound source localization and detection method provided by an embodiment of the present invention is shown; Figure 2 A schematic diagram of the azimuth angle in the far field provided by an embodiment of the present invention is shown; Figure 3 A schematic diagram of sound source signal propagation in the far field situation provided by an embodiment of the present invention is shown; Figure 4 A schematic diagram of a ternary linear uniform microphone array provided in an embodiment of the present invention is shown; Figure 5 A schematic diagram of time delay calculation using the generalized cross-correlation function method provided in an embodiment of the present invention is shown; Figure 6 A schematic diagram of a convolutional neural network provided in an embodiment of the present invention is shown; Figure 7 This diagram illustrates the maximum pooling and average pooling provided in an embodiment of the present invention; Figure 8 One of the system structure diagrams provided in the embodiments of the present invention is shown; Figure 9 The second schematic diagram of the system structure provided in the embodiment of the present invention is shown. Detailed Implementation

[0027] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore merely examples, and should not be construed as limiting the scope of protection of the present invention. It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0028] In one embodiment, such as Figure 1 As shown, a method for sound source localization and detection is provided, including: S1. Based on a three-channel linear uniform microphone array, obtain delay estimates for microphones I and II. Delay estimation with microphones I and III .

[0029] The signal acquisition chip uses the ADAU1977 audio ADC, a 4-channel 24-bit synchronous sampling chip with a sampling rate set from 8 to 192 kHz. It can be designed in parallel with 4, 8, or 16 channels, and hardware expansion is possible depending on the selected microphone array elements. Bird speech frequencies can reach 8 kHz; based on the Nyquist sampling theorem, a sampling rate of 20 kHz is set.

[0030] The acquired sound signal suffers from aliasing, high-frequency and high-order harmonic distortion due to the sound-producing object itself and the equipment used to acquire the audio signal, resulting in reduced audio quality. After preprocessing operations such as pre-emphasis, framing, and windowing, the obtained signal is more uniform and smoother, allowing for the extraction of better signal parameters and more effective analysis and processing of specific sound signals, thus improving audio signal quality and processing efficiency. Therefore, before step S1, preprocessing of the three-channel microphone array is also included, including: A1. For audio signals, perform pre-emphasis processing: Taking bird sounds as an example, birds are usually concentrated in the 2kHz~8kHz frequency band. A first-order FIR high-pass digital filter can be used to filter low-frequency environmental noise for pre-emphasis processing.

[0031] A2. Perform frame segmentation on the audio signal; the audio signal is considered to be relatively stable within a short time (generally considered to be within 10 to 30 ms), so the audio signal can be segmented, each segment is called a "frame", and the frame length is usually 10 to 30 ms. Analyze the characteristic parameters of each frame, so as to analyze the time series of each characteristic parameter as a whole.

[0032] A3. Apply a Hamming window to the framed signal. The purpose of windowing is to enhance the audio waveform near sample n while weakening other waveforms. The three most commonly used window functions are the rectangular window, the Hamming window, and the Hanning window. Since the Hamming window can better preserve the spectral characteristics of the original audio signal compared to the other two window functions, the audio signal is processed using the Hamming window after short-time framing.

[0033] The Hamming window is defined by the following formula:

[0034] A4. Perform FFT transformation on the three microphone signals.

[0035] Step S1 includes: S110. Perform cross-power spectrum analysis on microphones I and II and microphones I and III respectively.

[0036] Taking microphones I and II as examples, the conjugate multiplication of the two FFT-transformed signals is performed to obtain their cross-power spectrum for the current frame. Assume the outputs of microphones I and II after FFT transformation are...

[0037]

[0038] Then its cross-power spectrum can be expressed as:

[0039] (1-1) S120. Weighting of power spectrum results based on PHAT method.

[0040] The purpose of frequency domain weighting after cross-power spectrum analysis is to whiten the signal and noise, enhancing the frequency components with higher signal-to-noise ratios, thereby suppressing noise. The PHAT method is chosen as the frequency domain weighting function, retaining only the phase and setting the amplitude to 1. The input to the next IFFT stage is then the unit complex number for each frequency point.

[0041] , In other words, the value retains its phase after the PHAT weighting function. In FPGA, the phase method can be used to implement PHAT weighting. The specific steps are as follows: (1) Substituted into the CORDIC algorithm and Calculate the corresponding phase value ; (2) Using CORDIC rotation mode based on phase value Output The real and imaginary parts of the complex number after weighting and normalizing are used as inputs for step S130.

[0042] S130. Perform an inverse Fourier transform to determine the cross-correlation function.

[0043] The inverse Fourier transform is performed on the signal after weighted sharpening of peaks to determine its cross-correlation function. The formula for the inverse Fourier transform can be expressed as: (1-2) In other words, the IFFT operation can be achieved by conjugating the Fourier transform output, calling the FFT module, conjugating again, and then taking 1 / N. In FPGAs, this can be implemented directly using the FFT IP core.

[0044] S140. Perform peak detection on the cross-correlation function to determine the delay estimate.

[0045] The overall process of this step can be understood as: the location of the FFT peak → discrete delay (sample) → continuous delay (seconds). The specific FPGA implementation process is as follows: S141. Peak Index → ​​Number of Discrete Delay Samples : Assuming the number of pixels per frame is N (1024 in this embodiment), the output after passing through standard GCC-PHAT and IFFT is... It is cyclically correlated, and the location of the maximum peak is the hysteresis sample point corresponding to the relative time delay between the two channels.

[0046] For the case where the IFFT output is direct and no FFT shift spectrum rearrangement is performed: Peak Index ; Converted to signed lag samples , (1-3) so .

[0047] S142. Discrete delay (samples) → Continuous delay (seconds): Known sampling rate : (1-4) To achieve subsampling precision, parabolic interpolation is performed on the maximum value and both sides thereof. During peak search, the maximum index n is cached. , , : (1-5) (1-6) Delay estimation and Based on formula (1-6) as described above, the delay estimate is determined. and Then, the angles of the far-field model and the angles and distances of the near-field model can be calculated based on this.

[0048] The GCC-PHAT method is explained below. Among positioning methods based on time difference of arrival, the Generalized Cross-Correlation (GCC) method, based on correlation, is suitable for embedded systems due to its relatively simple calculation and fast processing speed. The GCC method essentially weights the power spectrum of the received signal, highlighting frequency components with high signal-to-noise ratios, thereby reducing noise and sharpening peak values. The following section combines... Figure 5 Please provide an explanation.

[0049] In the presence of ambient noise, the signals received by the two microphones can be represented as: (1-7) in The time it takes for the sound source to reach different microphones. For sound source signal, It is a noise signal, and and None of them are relevant. The time difference between the sound signal transmission to the two microphones can be expressed as: (1-8) For delay estimation algorithms, the most important thing is estimation. Therefore, under reverberation-free conditions, when using the cross-correlation algorithm to estimate the delay, it is first necessary to perform cross-correlation on the signals received by the two microphones: (1-9) in This represents the mathematical expectation. Because... and , Since they are all unrelated, the mathematical expectation between the sound source and the noise is zero. Therefore, the above equation can be further expressed as: (1-10) In the absence of reverberation and with no correlation between the sound source and noise, the cross-correlation function between two microphones can be expressed as the autocorrelation of the sound source signal and the cross-correlation of the two noise signals. Generally, the energy of the sound source signal is much greater than that of the noise; that is, in environments with a high signal-to-noise ratio, the cross-correlation function can be estimated from the autocorrelation of the sound source. hour, Similarly, the cross-correlation function will also reach a maximum value when it reaches an extreme value. This extreme value corresponds to the following It's the time difference between the two microphones. .

[0050] In real-world environments, the signals received by the microphone are subject to spatial reverberation, and the noise signals are not entirely incoherent. To reduce the impact of noise and reverberation on the delay estimation algorithm, the transformed spectrum needs further processing to sharpen the peak values; this is the generalized cross-correlation method.

[0051] The generalized cross-correlation function method requires calculating the cross-power spectrum of the two signals. Transforming equation (1-9) yields: (1-11) in , and They are , and The cross-power spectrum. The cross-correlation function at this point can be expressed as: (1-12) The corresponding generalized cross-correlation function is: (1-13) In the formula This is the weighting function for GCC. Different weights will have different effects on the system under different environments. Among them, the Generalized Cross Correlation-Phase Transformation (GCC-PHAT) method is the most widely used. The PHAT weighting values ​​are as follows: (1-14) S2. Based on delay estimation and Positioning calculations were performed for near-field and far-field scenarios respectively; in the far-field scenario, the calculation was based on time difference. Calculate the azimuth angle In the near field, based on time difference and The location information of the sound source is calculated.

[0052] The time difference between sound arrival at the microphone can be obtained through delay estimation using the GCC algorithm. Localization estimation then converts this time difference into angle values ​​using geometric localization methods, thereby obtaining the sound source's location information. The far-field model only needs to obtain the angle information, while the near-field model can further calculate the distance information.

[0053] For far-field cases, such as Figure 2 As shown, delay estimation The calculated azimuth angle ; d represents the spacing between adjacent microphones. In the far field, the sound source signal can be considered to propagate outward in the form of a spherical wave, such as... Figure 3 As shown.

[0054] like Figure 4 The three-element linear uniform microphone array shown, for near-field conditions, , , These represent the sound source reaching the microphone. , , The distance. Then and , and The time difference between the received signals can be expressed as: (2-1) (2-2) According to the Law of Cosines: (2-3) (2-4) In a given microphone array system, the element spacing d is known. If it can be estimated that... and Then we can further solve the array. , , , , , By using parameters such as these, the location of the sound source can be determined.

[0055] The coordinates of microphones I, II, and III ( , )=(0,0), ( , )=(0,d), ( , Substituting (0, 2d) into equations (2-1) and transforming them, we can obtain: (2-5) (2-6) The coordinate information of the sound source is represented as The distance of the sound source is Distance to the origin; , c is the speed at which the sound source propagates.

[0056] S3. Based on the sound source distance in the near field, compare it with the distance judgment threshold to determine the far field / near field output sound source location.

[0057] Specifically, the distance determination threshold d represents the spacing between the array elements, and n=3 represents the number of microphones arranged linearly. The wavelength of the signal; When the distance to the sound source is greater than When it is far field, it is near field; otherwise, it is near field.

[0058] In a specific application scenario, taking the location of bird sounds as an example, considering the high-frequency range of bird calls, which typically reaches 8 kHz to 10 kHz, and taking the minimum wavelength corresponding to the highest frequency as 0.0343 m, a linearly arranged three-element microphone array is used with an element spacing d of 40 cm. Based on the formula for determining the distance threshold, the following can be calculated for this hardware configuration. Approximately 37.3m, and will be determined later based on... This allows you to choose whether to output the distance calculation results for the far-field or near-field model.

[0059] S4. Extract features from the signal of any microphone array and classify events based on a pre-trained convolutional neural network.

[0060] In step S4, feature extraction is performed on the signal from any microphone array, including: S411. Perform feature extraction based on the Mel filter bank to obtain the processed power spectrum data; S412. Apply a constant bias to the processed power spectrum data, then perform a logarithmic transformation to obtain the logarithmic operation result.

[0061] By performing a Fast Fourier Transform (FFT) on the preprocessed audio signal, the Mel spectrum features are extracted using the Log-Mel filtering method, and then the event is classified using a Convolutional Neural Network (CNN), thus achieving efficient and accurate sound event detection.

[0062] Log-Mel feature extraction leverages the principle that human auditory sensitivity varies with sound frequency. By applying a Mel filter to perform a Fourier transform on the audio signal, it nonlinearly maps energy onto the Mel scale in the frequency domain, extracting important frequency features from the sound. This effectively compresses the audio signal along the frequency axis, improving sound information resolution and extracting key information from the speech signal. This method reflects the characteristics of human acoustic perception, capturing deep features in speech data.

[0063] Calculating the power spectrum involves determining the power value at each frequency point of the speech signal after the Fast Fourier Transform (FFT). This step converts the complex signal into a real number. The specific mathematical expression is shown in Equation (4-1).

[0064] (4-1) In the formula, This represents the power spectral density, which reflects the energy of that frequency component in the signal; This is the complex frequency domain representation of the signal at discrete frequency k.

[0065] Mel filter banks extract features using a series of bandpass filters. The filter design maps the frequency axis of the audio signal from a linear scale to a Mel scale. A Mel filter bank consists of several Mel-scaled filters. When a Mel filter bank contains M bandpass triangular filters, its response frequency... The mathematical expression is shown in equation (4-2).

[0066] (4-2) In equation (4-2), Let be the response function of the m-th bandpass triangular filter; It is the center frequency of the m-th bandpass triangular filter.

[0067] The power spectrum data processed by the Mel filter bank undergoes a logarithmic transformation to reduce the difference between high-energy and low-energy components, making changes in the lower-energy region more prominent and thus helping to extract more information from subtle signal variations. To avoid the logarithmically transformed data values ​​being too small, which could lead to difficulties in numerical processing, a small constant bias is usually added to the input data before performing the logarithmic transformation; in this embodiment, a bias of 0.01 is added. Let the logarithmic result be... Its mathematical expression is shown in equation (4-3).

[0068] (4-3) In one specific embodiment, 64 Mel filters are used to convert the amplitude data of each frame into 64 Mel spectrum data. The number of features is a power of 2, which can ensure that the data size can be halved uniformly during the re-digestion process.

[0069] Among them, event classification is based on pre-trained convolutional neural networks.

[0070] A convolutional neural network (CNN) consists of a series of convolutional layers, pooling layers, and fully connected layers. Convolutional layers generate new feature maps by performing weighted summations on each window of the feature map using a sliding window approach, extracting local information. Pooling layers downsample the output of the convolutional layers. Fully connected layers convert the new feature maps into predicted classification results. The structure is as follows: Figure 6 As shown.

[0071] It should be noted that the feature map processed by the convolutional layer is the transformation result of the power spectrum after processing by S411-S412. It is a two-dimensional matrix, similar to image data, and therefore can be regarded as equivalent to feature map processing.

[0072] Convolutional layers generate a new feature map by performing a weighted summation operation on each window of the feature map using a sliding window approach, thereby extracting feature information.

[0073] (4-4) Y is the output feature map, X is the input feature map, W is the convolution kernel, k is the kernel size, and b is the bias.

[0074] After convolutional layer processing, activation functions are needed to introduce non-linearity into the network, such as ReLU, Sigmoid, and Tanh. ReLU is the simplest to implement in hardware, using a comparator to set values ​​less than 0 to 0 and outputting the original value for values ​​greater than 0. However, if the input value is always less than 0, the output will always be 0, making it impossible to learn the parameters. Sigmoid and Tanh functions can be implemented using lookup tables.

[0075] Pooling layers downsample the output of convolutional layers, often used to reduce the dimensionality of feature maps and improve feature robustness. They primarily include max pooling and average pooling, such as... Figure 7 As shown.

[0076] In one embodiment, the goal of event classification in step S4 is primarily to identify the category of bird sound sources. That is, by using steps S1-S4 of this embodiment, not only is the location coordinate information of the sound source output, but the sound source type is also intelligently determined to achieve automatic identification of bird species and event classification.

[0077] Data Sources and Construction: Training data comes from publicly available bird acoustic datasets (such as Xeno-Canto), and specific bird sound datasets (such as sparrows, crows, orioles, etc.) are selected. Alternatively, dedicated datasets can be constructed by recording sounds using microphone arrays in the field. The acquired signals undergo denoising, framing, and windowing during the preprocessing stage. A time-frequency spectrogram is obtained through FFT, which is further converted into Mel frequency cepstral coefficients (MFCC), logarithmic Mel filter spectra, or time-frequency images, serving as input features for classification.

[0078] Model Training: A pre-trained convolutional neural network (CNN) structure is used to extract and classify features from the acoustic feature maps. In the specific implementation, the bird sound source dataset is divided into training, validation, and test sets, and the network is fine-tuned to improve the model's classification accuracy in the bird sound source recognition task.

[0079] Classification Results: During operation, the classification module automatically identifies the input sound source signal as a specific bird species and outputs an event label. Combined with the location results, the system can achieve a comprehensive detection function of "a certain species of bird appeared at a certain time and location".

[0080] In one specific embodiment, the convolutional neural network architecture adopts the classic CNN structure, mainly consisting of 13 convolutional layers and 3 fully connected layers. The core of the FPGA design includes two-dimensional convolutional layers, activation functions, and pooling layers.

[0081] Convolutional layers employ a multiplication-accumulation module to slide weight kernels across feature map data. To conserve computational resources, multiple feature maps can reuse the same weighted convolutional kernel in parallel. This is achieved by using processing elements (PEs) to store the kernel weights in individual PE register sets, allowing for reuse during convolution computation.

[0082] The output of the convolutional layer uses an activation function, and the function is chosen to be Tanh (hyperbolic tangent). It can be achieved by using a lookup table with interpolation, which stores the values ​​related to the positive side of the function. The negative side function is the mirror image of the positive side, and the same value can be output in two's complement format.

[0083] The pooling layer uses max pooling, which selects the maximum value of the feature data after introducing nonlinearity as the output.

[0084] Therefore, a convolutional network module is constructed, and by selecting multiple layers of the network, the extracted features are finally converted into prediction results through fully connected layers.

[0085] Sound signal acquisition, feature extraction, and convolutional neural network recognition are all implemented on the PL (Plug-in) side. During the CNN detection algorithm's operation, a lot of intermediate data is generated. Due to the limited storage resources on the PL side, the PS (Power Switch) side needs to control the storage and transfer of intermediate data between the PL's RAM and DDR (Memory Transfer Server). The PS and PL are connected via AXI, and data transfer is controlled by DMA (Direct Memory Access) blocks. DMA is a hardware technology that allows peripheral devices to directly read and write data to system memory (DRAM). When a peripheral device needs to transfer data, the operating system is responsible for configuring the DMA controller and setting the necessary parameters to control the data transfer process. Specifically, the PS side is responsible for controlling the transfer of weight values ​​between DDR and PL and scheduling the modules. Figure 8 As shown.

[0086] The PS and PL convolutional modules can use the AXI stream master-slave interface for input / output transmission; the AXI LITE slave interface can be used to pass weights stored in configuration registers; the DMA controller can be configured, and necessary parameters can be set to control data transmission between memory and the PL, such as... Figure 9 As shown.

[0087] In one embodiment, a sound source localization and detection device is provided, comprising: The delay estimation module is used to obtain delay estimates for microphones I and II based on a three-line linear uniform microphone array. Delay estimation with microphones I and III ; The positioning calculation module is used for delay estimation. and Positioning calculations were performed for near-field and far-field scenarios respectively; in the far-field scenario, the calculation was based on time difference. Calculate the azimuth angle In the near field, based on time difference and The location information of the sound source was calculated; The position output module is used to determine the position of the far-field / near-field output sound source by comparing the sound source distance in the near-field case with a distance determination threshold; The feature classification module is used to extract features from the signal of any microphone array and classify events based on a pre-trained convolutional neural network.

[0088] The sound source localization and detection device provided in this application embodiment adopts the same inventive concept as the sound source localization and detection method described above, and can achieve the same beneficial effects, so it will not be described again here.

[0089] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method of acoustic source localization and detection, the method comprising: Comprise: Based on three linear uniform microphone array, obtain the delay estimation of microphone I, II The delay estimation of microphone I, III ; Based on delay estimation And , respectively, the positioning calculation in near field and far field cases; wherein, in far field case, based on time difference Solve the azimuth angle , in near field case, based on time difference And Solve the sound source position information; According to the distance of the sound source under the near field condition, compare with the distance determination threshold, determine the far field / near field output sound source position; The signal of any one microphone array is extracted, and the event classification is carried out based on the pre-trained convolutional neural network.

2. The method of claim 1, wherein, Before obtaining the delay estimation, the three microphone arrays are preprocessed, including: For audio signal, pre-emphasis processing is carried out; Frame processing is carried out on the audio signal; Hamming window processing is carried out on the signal after frame processing; FFT transform is carried out on the three microphone signals.

3. The method of claim 1, wherein, The three-way linear uniform microphone array is used to obtain the delay estimation of the microphones I and II The delay estimation of the microphones I and III is obtained , comprising: Interpower spectrum analysis is carried out on microphone I, II and microphone I, III respectively; The interpower spectrum is weighted based on PHAT method; The inverse FFT transform is carried out to determine the cross-correlation function; The peak value of the cross-correlation function is detected to determine the delay estimation.

4. The method of claim 3, wherein, The interpower spectrum is weighted based on PHAT method, which is realized by using phase method in FPGA, including: By substituting the CORDIC algorithm with to obtain the corresponding phase value ; denotes the cross power spectrum, , denotes the real and imaginary parts, respectively; using the CORDIC rotation mode according to the phase value output (cosϕ, sinϕ), the weighted amplitude-normalized complex real and imaginary parts as output.

5. The method of claim 4, wherein, The delay estimation based And Respectively, the positioning calculation in the near field and the far field includes: For far field cases, delay estimation , the azimuth angle ; d is the interval between adjacent microphones; For near field case, the coordinate information of sound source is expressed as , the distance of sound source position is to the distance of the origin; wherein the microphone I, II, III is located at (0, 0), (0, d), (0, 2d), , , c is the sound source propagation speed.

6. The method of claim 5, wherein, According to the distance of the sound source under the near field condition, compare with the distance determination threshold, determine the far field / near field output sound source position, including: According to the sound source position distance and the distance determination threshold The comparison is made to determine the far field / near field output sound source position; wherein the distance determination threshold , d is the distance between each array element, n=3 is the number of arrays in linear arrangement, is the signal wavelength; When the sound source position distance is greater than far field, otherwise near field.

7. The method of claim 4, wherein, The signal of any one microphone array is extracted, including: The preprocessed microphone array signal is extracted based on mel filter bank to obtain the processed power spectrum data; The processed power spectrum data is subjected to constant biasing, and then logarithmic transformation is carried out to obtain the logarithmic operation result.

8. The method of claim 7, wherein, The convolutional neural network comprises: Convolution layer, by sliding window method of convolution kernel on feature map, weighted sum operation is carried out on each window to generate new feature map of extracting local information; Pooling layer, used for reducing sampling of convolution layer output; Fully connected layer, used for converting new feature map into prediction classification result.

9. A sound source positioning and detection apparatus, characterized by, Comprise: The delay estimation module is configured to obtain delay estimation of the microphone I and the microphone II based on a three-way linear uniform microphone array delay estimation of the microphone I and the microphone III ; a positioning calculation module for performing positioning calculation in near field and far field cases respectively based on the delay estimation and ; wherein, in the far field case, the azimuth angle is solved based on the time difference ; in the near field case, the sound source position information is solved based on the time difference ; and and ​ Position output module, for comparing the distance of the sound source under the near field condition with the distance determination threshold to determine the far field / near field output sound source position; Feature classification module, for extracting the signal of any one microphone array, and carrying out event classification based on pre-trained convolutional neural network.

Images