Acoustic event positioning method, device and system and computer readable storage medium

Abstract

The invention provides an acoustic event positioning method, device and system and a computer readable storage medium, is applied to an acoustic event positioning system, and relates to the technical field of acoustic positioning. The method comprises the following steps: acquiring multi-modal fusion data by a high-order surround acoustic spherical array node; the high-order surround acoustic spherical array node compresses the multi-modal fusion data and then sends the multi-modal fusion data to the data processing subsystem; the data processing subsystem performs feature extraction and recognition on the multi-modal fusion data based on a pre-trained multi-source data fusion positioning model, and outputs the type of an acoustic event and the coordinate of the acoustic event; wherein the multi-source data fusion positioning model comprises a feature extraction module and a space-time fusion module; according to the method, accurate multi-modal fusion data is obtained, feature extraction and recognition are performed by using a positioning model with a feature extraction module and a space-time fusion module, the type and positioning of an acoustic event are accurately output, calculation is simplified, and the phenotype of the model in a complex environment is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sound localization, and in particular, to an acoustic event localization method, device, system, and computer-readable storage medium. Background Art

[0002] Acoustic event localization technology accurately determines the location where an event occurs by analyzing the characteristics and propagation mode of sound signals, and has broad application value.

[0003] In some existing sound localization technologies, signal aliasing is likely to occur in the time domain of the signal, resulting in ambiguous solutions in the event correlation matrix and making it difficult to output accurate time localization. In some other sound localization technologies, an acoustic field statistical model based on a Gaussian mixture model is constructed, and this model hypothetically assumes that the spatial distribution of sound sources has separability, which limits its application in complex scenarios. In still some other sound localization technologies, a distributed iterative algorithm is introduced, increasing the computational complexity and affecting the real-time performance. Summary of the Invention

[0004] The purpose of the present invention is to provide an acoustic event localization method, device, system, and computer-readable storage medium, which can obtain accurate multi-modal fusion data, and use a localization model with a feature extraction module and a spatio-temporal fusion module to perform feature extraction and recognition, accurately output the type and location of acoustic events, simplify the calculation, and significantly improve the performance of the model in complex environments.

[0005] In a first aspect, the present invention provides an acoustic event localization method, which is applied to an acoustic event localization system. The acoustic event localization system includes: a data processing subsystem and multiple high-order surround acoustic spherical array nodes; the method includes:

[0006] The high-order surround acoustic spherical array nodes obtain multi-modal fusion data; wherein, the multi-modal fusion data includes: the acoustic field information of the acoustic event, the time information of the acoustic event, the position information of the high-order surround acoustic spherical array nodes, the attitude information of the high-order surround acoustic spherical array nodes, and the inference type of the acoustic event.

[0007] The high-order surround acoustic spherical array nodes compress the multi-modal fusion data and then send it to the data processing subsystem.

[0008] The data processing subsystem performs feature extraction and recognition on the multi-modal fusion data based on a pre-trained multi-source data fusion localization model, and outputs the type of the acoustic event and the coordinates of the acoustic event; wherein, the multi-source data fusion localization model includes: a feature extraction module and a spatio-temporal fusion module; the feature extraction module includes: a time series branch and a spatial branch; the spatio-temporal fusion module is a deep learning model based on a multi-head attention mechanism.

[0009] In some preferred embodiments of the present invention, the high-order surrounding acoustic spherical array node includes: a spherical microphone array, a real-time kinematic (RTK) system, an attitude sensor, and an acoustic field information processor; the steps for the high-order surrounding acoustic spherical array node to obtain multi-modal fusion data include:

[0010] Obtaining the acoustic field information of the acoustic event based on the spherical microphone array;

[0011] Obtaining the position information of the high-order surrounding acoustic spherical array node based on the real-time kinematic (RTK) system;

[0012] Obtaining the attitude information of the high-order surrounding acoustic spherical array node based on the attitude sensor;

[0013] Determining the inference type of the acoustic event based on the acoustic field information processor.

[0014] In some preferred embodiments of the present invention, the acoustic event localization system further includes: a meteorological monitoring node; the method further includes:

[0015] The meteorological monitoring node obtains meteorological information; wherein, the meteorological information includes at least one of the following: wind direction, wind speed, air temperature, relative humidity, air pressure;

[0016] The meteorological monitoring node sends the meteorological information to the data processing subsystem.

[0017] In some preferred embodiments of the present invention, the method further includes:

[0018] The data processing subsystem corrects the acoustic field information based on the meteorological information.

[0019] In some preferred embodiments of the present invention, the high-order surrounding acoustic spherical array node sends the compressed multi-modal fusion data to the data processing subsystem through low-power wide-area transmission technology.

[0020] In some preferred embodiments of the present invention, the temporal branch captures the temporal correlation characteristics of the multi-modal fusion data through a bidirectional gated logic unit; the spatial branch uses a two-dimensional convolution kernel to extract the azimuth distribution characteristics of the multi-modal fusion data.

[0021] In some preferred embodiments of the present invention, the spatio-temporal fusion module splits the input vector into multiple sub-spaces, then splices the results and performs a linear transformation to obtain the output result; wherein, the attention pattern of each sub-space is different.

[0022] In a second aspect, the present invention provides an acoustic event localization device, which is applied to an acoustic event localization system. The acoustic event localization system includes: a data processing subsystem and a plurality of high-order surrounding acoustic spherical array nodes; the device includes:

[0023] A data acquisition module, which is used for a high-order surrounding acoustic spherical array node to acquire multi-modal fusion data; wherein, the multi-modal fusion data includes: the sound field information of an acoustic event, the time information of the acoustic event, the position information of the high-order surrounding acoustic spherical array node, the attitude information of the high-order surrounding acoustic spherical array node, and the inference type of the acoustic event;

[0024] A data transmission module, which is used for the high-order surrounding acoustic spherical array node to compress the multi-modal fusion data and then send it to the data processing subsystem;

[0025] A data analysis module, which is used for the data processing subsystem to perform feature extraction and recognition on the multi-modal fusion data based on a pre-trained multi-source data fusion and localization model, and output the type of the acoustic event and the coordinates of the acoustic event; wherein, the multi-source data fusion and localization model includes: a feature extraction module and a spatio-temporal fusion module; the feature extraction module includes: a temporal branch and a spatial branch; the spatio-temporal fusion module is a deep learning model based on a multi-head attention mechanism.

[0026] In a third aspect, the present invention provides an acoustic event localization system, which includes a processor and a memory. The memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the acoustic event localization method provided in the first aspect above.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to implement the acoustic event localization method provided in the first aspect above.

[0028] The present invention brings the following beneficial effects:

[0029] The present invention provides an acoustic event localization method, apparatus, system, and computer-readable storage medium, which are applied to an acoustic event localization system. The acoustic event localization system includes a data processing subsystem and a plurality of high-order surround acoustic spherical array nodes. The method includes: the high-order surround acoustic spherical array nodes acquire multimodal fusion data, where the multimodal fusion data includes the sound field information of the acoustic event, the time information of the acoustic event, the position information of the high-order surround acoustic spherical array nodes, the attitude information of the high-order surround acoustic spherical array nodes, and the inference type of the acoustic event; the high-order surround acoustic spherical array nodes compress the multimodal fusion data and then send it to the data processing subsystem; the data processing subsystem performs feature extraction and recognition on the multimodal fusion data based on a pre-trained multi-source data fusion localization model, and outputs the type of the acoustic event and the coordinates of the acoustic event. The multi-source data fusion localization model includes a feature extraction module and a spatio-temporal fusion module. The feature extraction module includes a temporal branch and a spatial branch. The spatio-temporal fusion module is a deep learning model based on a multi-head attention mechanism. By acquiring accurate multimodal fusion data and using a localization model with a feature extraction module and a spatio-temporal fusion module for feature extraction and recognition, the type and localization of the acoustic event are accurately output, the calculation is simplified, and the performance of the model in a complex environment is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0031] Figure 1 Schematic structural diagram of an acoustic event localization system provided by an embodiment of the present invention;

[0032] Figure 2 Flowchart of an acoustic event localization method provided by an embodiment of the present invention;

[0033] Figure 3 Block diagram of a high-order surround acoustic spherical array acoustic sensing system based on embedded AI inference provided by an embodiment of the present invention;

[0034] Figure 4 Block diagram of the composition of a meteorological node provided by an embodiment of the present invention;

[0035] Figure 5 Schematic structural diagram of a multi-source data fusion localization model provided by an embodiment of the present invention;

[0036] Figure 6Schematic diagram of the structure of an acoustic event localization device provided by an embodiment of the present invention;

[0037] Figure 7 Schematic diagram of the structure of an acoustic event localization system provided by an embodiment of the present invention.

[0038] Icons: 310 - Data acquisition module; 320 - Data transmission module; 330 - Data analysis module; 400 - Memory; 401 - Processor; 402 - Bus; 403 - Communication interface. Detailed implementation manners

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Generally, the components of the embodiments of the present invention described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0040] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0041] It should be noted that: Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0042] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present invention is usually placed during use. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used for descriptive distinction and cannot be construed as indicating or implying relative importance.

[0043] In addition, the terms "horizontal", "vertical", "hanging", etc. do not mean that the components are required to be absolutely horizontal or hanging, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0044] In the description of the present invention, it should also be noted that, unless otherwise clearly specified and defined, the terms "set", "install", "connect", and "couple" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0045] Acoustic event localization technology accurately determines the location where an event occurs by analyzing the characteristics and propagation patterns of sound signals, and has broad application value. In security monitoring, it can quickly locate sudden events (such as gunshots and explosions); in industrial inspections, it improves the accuracy of equipment fault diagnosis; in environmental monitoring and noise control, it helps to improve environmental quality. In addition, acoustic event localization technology has a rapid response and accurate positioning, and is widely used in multiple fields, demonstrating important practicality and social value.

[0046] In some existing technologies, a distributed single-channel microphone network architecture is adopted. Multi-node signal synchronous acquisition is achieved through asynchronous clock calibration. The acoustic events of each node are independently detected based on the time-domain energy detection algorithm and time-stamped. This system uses a cross-node event correlation algorithm to establish a spatio-temporal mapping relationship and combines the time difference of arrival (TDOA) estimation for sound source localization. This method has the advantages of a simple hardware architecture, only requiring a single-channel acquisition unit, and low cost. At the same time, based on the event-driven processing mechanism, it effectively reduces the data transmission bandwidth requirements. However, in the case of multiple events occurring concurrently, signal aliasing may occur in the time domain, resulting in ambiguous solutions in the event correlation matrix. In addition, the TDOA estimation has limited time delay measurement accuracy in a non-stationary noise environment, and a sound source physical model is not established, making it difficult to handle the multi-source separation problem in complex acoustic scenarios.

[0047] In some other existing technologies, an acoustic field statistical model based on the Gaussian mixture model (GMM) is constructed. The energy distribution of the sound source is iteratively estimated through the EM algorithm, and a robust mean consistency method is introduced to achieve collaborative parameter estimation of the distributed sensor network. This method improves the robustness to non-Gaussian noise through statistical modeling and effectively suppresses the influence of individual node measurement outliers using the consistency protocol. However, the GMM model assumes that the spatial distribution of the sound source is separable, which limits its application in reverberant scenarios. At the same time, this method does not consider the dispersion characteristics of sound wave propagation, resulting in a model mismatch problem during broadband sound source localization. In addition, the introduction of the distributed iterative algorithm increases the computational complexity and affects the real-time performance.

[0048] In summary, aiming at the problem of multi-source sound source localization in complex environments, an acoustic event localization method based on a distributed higher-order ambisonics (HOA) array sensor network is proposed. This method obtains acoustic signals through a distributed acoustic sensor network composed of multiple higher-order ambisonics arrays. Each node has a highly directional characteristic that is consistent in all spatial directions, effectively suppressing the multipath effect and clutter interference in complex environments. At the same time, each node is equipped with a high-precision Beidou positioning system and an accurate timestamp acquisition module, which can obtain the time and geographical information of acoustic signals in real time. An RK3588 embedded AI inference processor is locally deployed to support functions such as event type recognition, sound field distribution estimation, and event occurrence time inference. Finally, the acoustic event type, location, geographical, and time information detected by each node is transmitted to the network aggregation center node. Through multi-source data fusion based on a multi-modal neural network model, the multi-source time correlation problem is effectively solved, and the acoustic event localization accuracy is significantly improved. This method has high practical value and application prospects for sound source localization in complex environments.

[0049] The following will describe in detail some embodiments of the present invention with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0050] Embodiment 1

[0051] An embodiment of the present invention provides an acoustic event localization method, which is applied to an acoustic event localization system. Refer to Figure 1 the structural schematic diagram of an acoustic event localization system provided by an embodiment of the present invention as shown. The acoustic event localization system includes: a data processing subsystem and multiple higher-order ambisonic spherical array nodes; the data processing subsystem is used for multi-source information fusion processing.

[0052] Refer to Figure 2 the flowchart of an acoustic event localization method provided by an embodiment of the present invention as shown. The method includes:

[0053] Step S102, the higher-order ambisonic spherical array node obtains multi-modal fusion data; wherein, the multi-modal fusion data includes: the sound field information of the acoustic event, the time information of the acoustic event, the position information of the higher-order ambisonic spherical array node, the attitude information of the higher-order ambisonic spherical array node, and the inference type of the acoustic event.

[0054] Specifically, the multi-modal fusion data can be collected in real time or historical data can be obtained.

[0055] Further, in some preferred embodiments of the present invention, the high-order surround acoustic spherical array node includes: a spherical microphone array, a real-time kinematic (RTK) system, an attitude sensor, and an acoustic field information processor; the steps for the high-order surround acoustic spherical array node to obtain multi-modal fusion data include: obtaining the acoustic field information of an acoustic event based on the spherical microphone array; obtaining the position information of the high-order surround acoustic spherical array node based on the RTK system; obtaining the attitude information of the high-order surround acoustic spherical array node based on the attitude sensor; and determining the inference type of the acoustic event based on the acoustic field information processor.

[0056] Specifically, each high-order surround acoustic spherical array node integrates a spherical microphone array, a Beidou RTK system, an electronic compass, an attitude sensor, and a low-power wide-area transmission unit. The high-order surround acoustic spherical array nodes are evenly deployed around the monitoring area to form a coverage network.

[0057] The high-order surround acoustic spherical array node has consistent directivity in all directions, significantly improving the sound source localization accuracy and the ability to resist noise / multipath interference; the high-precision distributed synchronous acquisition system based on Beidou realizes high-precision time synchronization and positioning.

[0058] Further, the embodiments of the present invention provide a high-order surround acoustic spherical array acoustic sensing system based on embedded AI inference for local data processing, see Figure 3Block diagram of an acoustic sensing system based on embedded AI inference for a high-order surround acoustic spherical array provided by an embodiment of the present invention. This system adopts a high-order Ambisonics (HOA) acoustic sensing architecture and constructs a three-dimensional sound field monitoring platform by integrating new hardware modules and intelligent signal processing algorithms. The core sensing unit consists of a spherical array composed of 32 omnidirectional MEMS microphones, realizing fifth-order Ambisonics (a technology for recording, transmitting, and playing back three-dimensional spatial sound information) sound field reconstruction and enabling 360° spatial sound field sampling. The multi-modal data fusion acquisition system integrates a Beidou-3 RTK positioning module (positioning accuracy ±1 cm) and a Micro-Electro-Mechanical System (MEMS) inertial measurement unit (including a three-axis electronic compass and a six-degree-of-freedom attitude sensor). At the same time, the system uses a Xilinx Artix-7 FPGA to achieve nanosecond-level synchronous triggering of multi-channel ADCs and realizes 24-bit / 192 kHz audio digitization processing through a multi-channel Codec. The system also carries a Rockchip RK3588 NPU chip, supports edge computing processing, and has a dual-core Cortex-A76 processor and a 6 TOPS computing power NPU built-in. The system can locally implement real-time sound source detection and classification based on YOLO-Sound, as well as multi-channel beamforming and Ambisonics B-Format encoding.

[0059] Further, referring to Figure 1 , in some preferred embodiments of the present invention, the acoustic event positioning system further includes: a meteorological monitoring node; the method further includes: the meteorological monitoring node obtains meteorological information; wherein, the meteorological information includes at least one of the following: wind direction, wind speed, temperature, relative humidity, air pressure; the meteorological monitoring node sends the meteorological information to the data processing subsystem.

[0060] Specifically, the system further includes a meteorological monitoring node, a communication facility, a power supply device, and a convergence processing center. The meteorological monitoring node integrates various sensors such as a thermometer, a wind direction and speed meter, and a barometer; the communication system combines long-distance low-power wide-area transmission (LoRa) communication with signal relay nodes; the power supply system is jointly powered by a battery and a solar power generation device to ensure the long-term stable operation of the system; the convergence processing center is responsible for data fusion processing.

[0061] Specifically, the meteorological monitoring node aims to achieve all-round and real-time monitoring of key meteorological elements such as wind direction, wind speed, temperature, relative humidity, and air pressure. Refer to Figure 4Block diagram of a meteorological node provided by an embodiment of the present invention as shown. This node uses an advanced ARM processor to efficiently integrate data from multiple meteorological sensors and synchronously obtain accurate time and geographical location information provided by the Beidou Navigation System. All data is stably transmitted to the central control node through LoRa communication technology. The meteorological monitoring subsystem consists of a series of precision components, including professional meteorological sensors, micro meteorological data collectors, stable power supply systems, lightweight louvers, weather-resistant field protection boxes, and sturdy stainless steel brackets, etc. Among them, wind speed and wind direction sensors, etc., are all precision devices dedicated to the meteorological field.

[0062] Further, in some preferred embodiments of the present invention, the method further includes: the data processing subsystem corrects the sound field information based on the meteorological information.

[0063] Specifically, all collected meteorological data is based on the calculated sound speed and Doppler distortion, providing an important reference for the compensation and correction of subsequent sound propagation models.

[0064] Step S104, the high-order surround acoustic spherical array node compresses the multi-modal fusion data and then sends it to the data processing subsystem.

[0065] Specifically, the system uses data compression technology to compress the sound event type, occurrence time, and sound field information, effectively reducing the transmission burden, reducing the system cost, and improving the deployment flexibility.

[0066] Further, in some preferred embodiments of the present invention, the high-order surround acoustic spherical array node sends the compressed multi-modal fusion data to the data processing subsystem through low-power wide-area transmission technology.

[0067] Specifically, the low-power wide-area transmission system uses the SX1262 LoRa chipset, supports LDPC forward error correction and frequency hopping spread spectrum technology. The data transmission frame structure includes: Beidou spatio-temporal reference (UTC timestamp, WGS84 coordinates), node attitude parameters (pitch / roll / yaw angles), acoustic feature vectors (voiceprint information, sound field space information, event type). This embodiment of the invention combines spatial sound field perception technology with high-precision spatio-temporal information to realize edge extraction of acoustic features through embedded processing. Compared with traditional solutions, it greatly reduces the amount of raw data transmitted, while maintaining the complete spatial sound field reconstruction ability, providing a cost-effective positioning node solution for wide-area acoustic event monitoring networks.

[0068] Step S106, the data processing subsystem performs feature extraction and recognition on the multi-modal fusion data based on the pre-trained multi-source data fusion localization model, and outputs the type of the acoustic event and the coordinates of the acoustic event; wherein, the multi-source data fusion localization model includes: a feature extraction module and a spatio-temporal fusion module; the feature extraction module includes: a temporal branch and a spatial branch; the spatio-temporal fusion module is a deep learning model based on the multi-head attention mechanism.

[0069] Specifically, refer to Figure 5 The structural schematic diagram of a multi-source data fusion localization model provided by the embodiment of the present invention shown, the multi-source data fusion localization model based on the multi-modal neural network model, its core architecture is composed of a feature extraction module and a spatio-temporal fusion module.

[0070] Further, in some preferred embodiments of the present invention, the temporal branch captures the temporal correlation characteristics of the multi-modal fusion data through a bidirectional gated logic unit; the spatial branch extracts the azimuth distribution characteristics of the multi-modal fusion data by using a two-dimensional convolution kernel.

[0071] Further, in some preferred embodiments of the present invention, the spatio-temporal fusion module splits the input vector into multiple sub-spaces, then splices the results and performs a linear transformation to obtain an output result; wherein, the attention patterns of each sub-space are different.

[0072] Specifically, first, the multi-modal data collected by distributed sensing nodes is aggregated through the LoRa wireless communication network, including acoustic event type features, high-precision timestamp sequences, sound field information features, and geographical information metadata. To address the feature alignment problem of heterogeneous data, a two-stream feature extractor is constructed using a Convolutional Recurrent Neural Network (CRNN): the temporal branch captures the temporal correlation characteristics of the event sequence through a bidirectional GRU, and the spatial branch uses a two-dimensional convolutional kernel to extract the azimuth distribution features of the acoustic signal. Finally, the multi-source data is mapped to a unified 128-dimensional hidden space through a feature projection layer. In the feature fusion stage, a Transformer architecture based on the multi-head attention mechanism is innovatively introduced. The self-attention mechanism calculates the association weights between each element in the sequence and other elements (Query-Key-Value mechanism) to dynamically capture long-range dependencies. Moreover, its multi-head mechanism splits the input vector into multiple subspaces (heads), and each subspace independently learns different attention patterns (such as the structural / semantic focus of time series x frequency domain features). Finally, the results are concatenated and linearly transformed. This module converts the feature vector sequence of each node into a context representation containing spatio-temporal correlations through learnable spatio-temporal position encoding. Among them, the cross-modal attention layer can adaptively establish the semantic association between acoustic event types and spatial spectral features, while the hierarchical attention mechanism effectively integrates local node features and global topological information. Finally, the event localization coordinate estimation and type recognition tasks are jointly optimized through a fully connected network, and an end-to-end training is carried out using a weighted loss function of mean square error and cross entropy. This deep fusion framework provides a more accurate positioning technology for intelligent perception in the Internet of Things environment.

[0073] The present invention provides an acoustic event localization method, which is applied to an acoustic event localization system. The acoustic event localization system includes: a data processing subsystem and multiple high-order surround acoustic spherical array nodes; the method includes: the high-order surround acoustic spherical array nodes acquire multimodal fusion data; wherein, the multimodal fusion data includes: the sound field information of the acoustic event, the time information of the acoustic event, the position information of the high-order surround acoustic spherical array nodes, the attitude information of the high-order surround acoustic spherical array nodes, and the inference type of the acoustic event; the high-order surround acoustic spherical array nodes compress the multimodal fusion data and then send it to the data processing subsystem; the data processing subsystem performs feature extraction and recognition on the multimodal fusion data based on a pre-trained multi-source data fusion localization model, and outputs the type of the acoustic event and the coordinates of the acoustic event; wherein, the multi-source data fusion localization model includes: a feature extraction module and a spatio-temporal fusion module; the feature extraction module includes: a temporal branch and a spatial branch; the spatio-temporal fusion module is a deep learning model based on the multi-head attention mechanism; obtaining accurate multimodal fusion data, and using a localization model with a feature extraction module and a spatio-temporal fusion module to perform feature extraction and recognition, accurately outputting the type and location of the acoustic event, simplifies the calculation, and significantly improves the performance of the model in complex environments.

[0074] Embodiment 2

[0075] Based on the above embodiment, the embodiment of the present invention provides an acoustic event localization device, which is applied to an acoustic event localization system. The acoustic event localization system includes: a data processing subsystem and multiple high-order surround acoustic spherical array nodes; see Figure 6 the structural schematic diagram of an acoustic event localization device provided by the embodiment of the present invention shown in

[0076] A data acquisition module 310, configured to enable the high-order surround acoustic spherical array nodes to acquire multimodal fusion data; wherein, the multimodal fusion data includes: the sound field information of the acoustic event, the time information of the acoustic event, the position information of the high-order surround acoustic spherical array nodes, the attitude information of the high-order surround acoustic spherical array nodes, and the inference type of the acoustic event.

[0077] A data transmission module 320, configured to enable the high-order surround acoustic spherical array nodes to compress the multimodal fusion data and then send it to the data processing subsystem.

[0078] A data analysis module 330, configured to enable the data processing subsystem to perform feature extraction and recognition on the multimodal fusion data based on a pre-trained multi-source data fusion localization model, and output the type of the acoustic event and the coordinates of the acoustic event; wherein, the multi-source data fusion localization model includes: a feature extraction module and a spatio-temporal fusion module; the feature extraction module includes: a temporal branch and a spatial branch; the spatio-temporal fusion module is a deep learning model based on the multi-head attention mechanism.

[0079] Further, in some preferred embodiments of the present invention, the high-order surround acoustic spherical array node includes: a spherical microphone array, a real-time kinematic differential system, an attitude sensor, and an acoustic field information processor; a data acquisition module 310, configured to acquire the acoustic field information of an acoustic event based on the spherical microphone array; acquire the position information of the high-order surround acoustic spherical array node based on the real-time kinematic differential system; acquire the attitude information of the high-order surround acoustic spherical array node based on the attitude sensor; and determine the inference type of the acoustic event based on the acoustic field information processor.

[0080] Further, in some preferred embodiments of the present invention, the acoustic event positioning system further includes: a meteorological monitoring node; the device further includes: a meteorological data acquisition module, configured to acquire meteorological information from the meteorological monitoring node; wherein the meteorological information includes at least one of the following: wind direction, wind speed, temperature, relative humidity, and air pressure; the meteorological monitoring node sends the meteorological information to the data processing subsystem.

[0081] Further, in some preferred embodiments of the present invention, the device further includes: a meteorological data correction module, configured to correct the acoustic field information by the data processing subsystem based on the meteorological information.

[0082] Further, in some preferred embodiments of the present invention, the high-order surround acoustic spherical array node sends the compressed multi-modal fusion data to the data processing subsystem through a low-power wide-area transmission technology.

[0083] Further, in some preferred embodiments of the present invention, the temporal branch captures the temporal correlation characteristics of the multi-modal fusion data through a bidirectional gated logic unit; the spatial branch extracts the azimuth distribution characteristics of the multi-modal fusion data using a two-dimensional convolution kernel.

[0084] Further, in some preferred embodiments of the present invention, the spatio-temporal fusion module splits the input vector into multiple subspaces, then splices the results and performs a linear transformation to obtain an output result; wherein, the attention pattern of each subspace is different.

[0085] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described acoustic event positioning device can refer to the corresponding process in the embodiment of the foregoing acoustic event positioning method, and will not be elaborated herein.

[0086] Embodiment III

[0087] The embodiment of the present invention further provides an acoustic event positioning system for running the acoustic event positioning method; see Figure 7The structural schematic diagram of an acoustic event localization system provided by an embodiment of the present invention is shown. The acoustic event localization system includes a memory 400 and a processor 401. Among them, the memory 400 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 401 to implement the above-mentioned acoustic event localization method.

[0088] Furthermore, Figure 7 The shown acoustic event localization system further includes a bus 402 and a communication interface 403. The processor 401, the communication interface 403, and the memory 400 are connected through the bus 402.

[0089] Among them, the memory 400 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between this system network element and at least one other network element is realized through at least one communication interface 403 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 402 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 7 only a single bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0090] The processor 401 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 401 or the instructions in the form of software. The above-mentioned processor 401 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step and logic block diagram disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 400, and the processor 401 reads the information in the memory 400 and combines its hardware to complete the steps of the method in the foregoing embodiments.

[0091] The embodiments of the present invention also provide a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to implement the above acoustic event localization method. For the specific implementation, reference can be made to the method embodiments, and details are not described herein again.

[0092] The computer program product of the acoustic event localization method, device and acoustic event localization system provided by the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the foregoing method embodiments. For the specific implementation, reference can be made to the method embodiments, and details are not described herein again.

[0093] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems and / or devices can refer to the corresponding processes in the foregoing method embodiments, and details are not described herein again.

[0094] In addition, in the description of the embodiments of the present invention, unless otherwise clearly defined and limited, the terms "installation", "connection", and "coupling" shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be a direct connection or an indirect connection through an intermediate medium, and it may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0095] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present invention. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features. However, such modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An acoustic event localization method, characterized in that, Applied to an acoustic event localization system, the acoustic event localization system includes: a data processing subsystem and a plurality of high-order surround acoustic spherical array nodes; the method includes: The high-order surround acoustic spherical array nodes acquire multi-modal fusion data; wherein, the multi-modal fusion data includes: sound field information of the acoustic event, time information of the acoustic event, position information of the high-order surround acoustic spherical array nodes, attitude information of the high-order surround acoustic spherical array nodes, and inference type of the acoustic event; The high-order surround acoustic spherical array nodes perform compression processing on the multi-modal fusion data and then send it to the data processing subsystem; The data processing subsystem performs feature extraction and recognition on the multi-modal fusion data based on a pre-trained multi-source data fusion localization model, and outputs the type of the acoustic event and the coordinates of the acoustic event; wherein, the multi-source data fusion localization model includes: a feature extraction module and a spatio-temporal fusion module; the feature extraction module includes: a time series branch and a spatial branch; the spatio-temporal fusion module is a deep learning model based on a multi-head attention mechanism.

2. The acoustic event localization method according to claim 1, wherein The high-order surround acoustic spherical array nodes include: a spherical microphone array, a real-time kinematic (RTK) system, an attitude sensor, and a sound field information processor; the steps for the high-order surround acoustic spherical array nodes to acquire multi-modal fusion data include: Acquire the sound field information of the acoustic event based on the spherical microphone array; Acquire the position information of the high-order surround acoustic spherical array nodes based on the real-time kinematic (RTK) system; Acquire the attitude information of the high-order surround acoustic spherical array nodes based on the attitude sensor; Determine the inference type of the acoustic event based on the sound field information processor.

3. The acoustic event localization method according to claim 1, characterized in that, The acoustic event localization system further includes: a meteorological monitoring node; the method further includes: The meteorological monitoring node acquires meteorological information; wherein, the meteorological information includes at least one of the following: wind direction, wind speed, air temperature, relative humidity, air pressure; The meteorological monitoring node sends the meteorological information to the data processing subsystem.

4. The acoustic event localization method according to claim 3, wherein The method further includes: The data processing subsystem corrects the sound field information based on the meteorological information.

5. The acoustic event localization method according to claim 1, wherein The high-order surround acoustic spherical array nodes send the compressed multi-modal fusion data to the data processing subsystem through low-power wide-area transmission technology.

6. The acoustic event localization method according to claim 1, wherein, The time series branch captures the time-domain correlation characteristics of the multi-modal fusion data through a bidirectional gated logic unit; the spatial branch extracts the azimuth distribution characteristics of the multi-modal fusion data using a two-dimensional convolution kernel.

7. The acoustic event localization method according to claim 1, wherein The spatio-temporal fusion module splits the input vector into multiple sub-spaces, then splices the results and performs a linear transformation to obtain the output result; wherein, the attention pattern of each sub-space is different.

8. An acoustic event positioning device, characterized in that, Applied to an acoustic event localization system, the acoustic event localization system includes: a data processing subsystem and a plurality of high-order surround acoustic spherical array nodes; the device includes: A data acquisition module, configured to acquire multi-modal fusion data for the high-order surround acoustic spherical array node; wherein, the multi-modal fusion data includes: sound field information of an acoustic event, time information of the acoustic event, position information of the high-order surround acoustic spherical array node, attitude information of the high-order surround acoustic spherical array node, and inference type of the acoustic event. A data transmission module, configured to compress the multi-modal fusion data by the high-order surround acoustic spherical array node and then send it to the data processing subsystem. A data analysis module, configured to perform feature extraction and recognition on the multi-modal fusion data by the data processing subsystem based on a pre-trained multi-source data fusion and localization model, and output the type of the acoustic event and the coordinates of the acoustic event; wherein, the multi-source data fusion and localization model includes: a feature extraction module and a spatio-temporal fusion module; the feature extraction module includes: a temporal branch and a spatial branch; the spatio-temporal fusion module is a deep learning model based on a multi-head attention mechanism.

9. An acoustic event localization system, characterized in that, It includes a processor and a memory, the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the acoustic event localization method according to any one of claims 1 to 7 above.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to implement the acoustic event localization method according to any one of claims 1 to 7.

Images