Ambient sound event detection method based on multi-modal data fusion

Abstract

The invention relates to the technical field of environmental sound event detection, in particular to an environmental sound event detection method based on multi-modal data fusion, which utilizes a microphone array, a camera and a sensor to synchronously acquire audio, video and environmental data to form a multi-modal data source. And the collected multi-modal data is preprocessed, so that the data quality and the accuracy of subsequent analysis are improved. And key features are extracted from the processed data, and the features are fused to form multi-modal features. And a deep learning model is constructed, and multi-modal features are used for training, so that a model capable of identifying an environmental sound event can be obtained. And performing optimization such as pruning, quantification and knowledge distillation on the trained model, and deploying the model on edge equipment to realize real-time preliminary feature extraction and event detection. The problems that an environment sound event detection method in the prior art is poor in robustness, low in real-time performance and limited in generalization ability are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental sound event detection, and in particular to an environmental sound event detection method based on multimodal data fusion. Background Art

[0002] Ambient sound event detection is a key technology for identifying and classifying specific sound events from ambient audio data. In recent years, with the rapid development of the Internet of Things (IoT), smart homes, smart cities and other fields, the application scope of ambient sound event detection technology has been greatly expanded, including but not limited to security monitoring, disaster warning, smart homes, health monitoring and autonomous driving.

[0003] Traditional environmental sound event detection methods mainly rely on single-mode data input, such as analysis based only on audio signals. The main drawbacks of this method include:

[0004] Poor robustness: In noisy environments or multi-source mixed sound scenes, a single audio signal is difficult to accurately capture and distinguish different environmental sound events, resulting in high false alarm and missed alarm rates.

[0005] Poor real-time performance: The processing of single-modal audio data is complex and time-consuming, making it difficult to meet real-time requirements, especially in a dynamically changing environment.

[0006] Strong data dependence: Single audio data lacks diversity and cannot fully express different types of events in the environment, and it is highly dependent on the size and quality of the training dataset.

[0007] Limited generalization ability: Models based on single-modal data are difficult to maintain consistent detection performance when facing different application scenarios, and their generalization ability is insufficient. Summary of the invention

[0008] The purpose of the present invention is to provide an environmental sound event detection method based on multimodal data fusion, which solves the problems of poor robustness, low real-time performance and limited generalization ability of the environmental sound event detection method in the prior art.

[0009] To achieve the above object, the present invention provides an environmental sound event detection method based on multimodal data fusion, comprising the following steps:

[0010] Use microphone arrays, cameras and sensors to synchronously collect audio, video and environmental data to obtain multimodal data;

[0011] Processing of multimodal data;

[0012] Perform feature extraction and feature fusion on the processed multimodal data to obtain multimodal features;

[0013] Build a deep learning model and use multimodal features for training to obtain a trained deep learning model;

[0014] Deploy the deep learning model after pruning, quantization and knowledge distillation on the edge device, implement preliminary feature extraction and event detection through edge computing, and respond based on the detection results;

[0015] Continuously monitor and analyze test results, promptly identify errors or false positives, and make corresponding corrections and adjustments.

[0016] The microphone array, camera and sensor are used to synchronously collect audio, video and environmental data to obtain multimodal data. The steps also include:

[0017] Use a microphone array to collect ambient audio signals and use short-time Fourier transform to convert the audio signals into a representation in the time-frequency domain;

[0018] Use high frame rate cameras to capture visual information in the scene, including the movement trajectory of objects, changes in lighting, and dynamic changes in the background environment, and use visual SLAM technology to generate a 3D reconstruction of the video data;

[0019] The use of multiple sensors can capture multi-dimensional information of environmental changes, and the data from different sensors can be integrated through multi-sensor fusion technology to form an environmental information representation.

[0020] Wherein, the multimodal data is processed, and the step further includes:

[0021] Use adaptive filters and noise suppression algorithms to reduce the noise of audio data, use bilateral filtering and convolutional neural networks to denoise video frames, and use sliding window filtering and Kalman filtering to smooth sensor data, eliminate burst noise in sensor readings, and obtain pre-processed data of each modality;

[0022] Apply dynamic time warping techniques to align cross-modal data;

[0023] Data enhancement technology is used to enhance the data of each modality.

[0024] The processed multimodal data is subjected to feature extraction and feature fusion to obtain multimodal features, and the step further includes:

[0025] Apply short-time Fourier transform to convert the audio signal into time-frequency domain representation and generate a spectrogram. Then, use deep convolutional neural network to extract features from the spectrogram and extract the time-frequency domain features in the audio signal.

[0026] A convolutional neural network is used to extract the spatial features of each frame of the image, and then the time series features are processed through a long short-term memory network to extract dynamic features that reflect the movement and changes of the object;

[0027] Use multi-layer perceptron or convolutional network to extract sensor data features.

[0028] Wherein, a deep learning model after pruning, quantization and knowledge distillation is deployed on the edge device, preliminary feature extraction and event detection are realized through edge computing, and a response is made according to the detection result. The steps also include:

[0029] When the edge device detects a specific ambient sound event, it classifies the event according to the preset priority rules and responds quickly. For high-priority events, the early warning mechanism is immediately activated. For medium and low priority events, logs are recorded or reminder notifications are issued. The edge device is linked with the cloud server to upload complex events or multimodal feature fusion tasks to the cloud for further processing and analysis.

[0030] The present invention discloses an environmental sound event detection method based on multimodal data fusion, which significantly improves the robustness of environmental sound event detection by fusing multimodal data such as audio, video and sensors. In complex real-world scenarios, such as traffic noise, crowd noise and multi-source mixed sound, single-modal data is easily interfered with and causes false alarms or missed alarms. The multimodal fusion method of the present invention can comprehensively utilize the data features of each modality to accurately capture and identify different environmental sound events. Through the combination of attention mechanism and deep fusion network, the advantages of different modalities are complementary, showing extremely high anti-interference ability and robustness when dealing with complex environments. A series of advanced model compression technologies, such as model pruning, quantization and knowledge distillation, are adopted, combined with edge computing architecture, to achieve real-time detection and response of environmental sound events. On resource-constrained edge devices, the deep learning model after pruning and quantization processing greatly reduces the computing overhead and significantly improves the detection speed. At the same time, the introduction of edge computing effectively reduces the delay of data transmission, enabling responses to be made within milliseconds to adapt to dynamically changing application scenarios, such as smart cities and disaster warnings. Through event priority classification and interrupt-driven mechanisms, high-priority events are quickly processed, and an efficient closed loop from detection to response is achieved. Through online learning and adaptive learning mechanisms, the generalization ability and adaptability in diverse application scenarios are improved. When facing different environments and changing sound patterns, model parameters and detection strategies can be adjusted in real time to ensure that efficient detection performance is always maintained. Whether in noisy urban streets or in quiet residential areas, detection thresholds and strategies can be adaptively adjusted to reduce false alarm rates and improve detection accuracy. In addition, through data enhancement and incremental update technology, the model can continuously learn new event types and sound features, with strong generalization capabilities to adapt to various emerging application needs. It has a highly flexible design and can be modularly customized and configured according to actual application needs. Users can select data sources of different modalities (such as adding new sensor types) according to specific scenarios, as well as adjust fusion strategies and detection models. This flexibility not only enables it to adapt to diverse detection needs, but also enables it to maintain excellent performance in different application scenarios. The modular design also greatly facilitates maintenance and upgrades. Users can easily integrate new functional modules according to needs to meet changing technical and market needs. The present invention solves the problems of poor robustness, low real-time performance and limited generalization ability in the ambient sound event detection method in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0032] Figure 1It is a flowchart of the steps of the environmental sound event detection method based on multimodal data fusion according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0033] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0034] The first embodiment of the present application is:

[0035] See also Figure 1 ,in, Figure 1 It is a flowchart of the steps of the environmental sound event detection method based on multimodal data fusion according to the first embodiment of the present invention.

[0036] The present invention provides an environmental sound event detection method based on multimodal data fusion, which comprises the following steps:

[0037] S100: Use microphone arrays, cameras and sensors to synchronously collect audio, video and environmental data to obtain multimodal data;

[0038] Specifically, audio data: A high-sensitivity microphone array is used to collect environmental audio signals. The microphone array can not only collect a variety of sound types, such as human voice, mechanical noise, natural sound, etc., but also enhance the sound signal in a specific direction through beamforming technology to reduce the impact of environmental noise. In addition, the short-time Fourier transform (STFT) is used to convert the audio signal into a representation in the time-frequency domain, providing a basis for subsequent feature extraction. Video data: A high-frame rate camera is used to capture visual information in the scene, including the movement trajectory of objects, changes in lighting, and dynamic changes in the background environment. Multi-angle acquisition of video data is achieved through a multi-lens system, thereby enriching the dimensions of visual information. At the same time, the visual SLAM (Simultaneous Localization and Mapping) technology is used to generate a three-dimensional reconstruction of the video data to further improve the understanding of complex scenes. Sensor data: Sensor data comes from multiple environmental sensors, such as accelerometers, temperature and humidity sensors, light sensors, and air pressure sensors. These sensors can capture multi-dimensional information closely related to environmental changes, such as vibration frequency, ambient temperature and humidity, light intensity, and air pressure fluctuations. In order to improve the accuracy of sensor data collection, Kalman Filter is used for noise suppression, and multi-sensor fusion technology is used to integrate data from different sensors to form a more stable and accurate representation of environmental information.

[0039] S200: processing multimodal data;

[0040] Specifically, audio preprocessing: Noise reduction and filtering: Use adaptive filters (Adaptive Filtering) and noise suppression algorithms, such as Wiener filters, to perform noise reduction on audio data to reduce the interference of background noise. Furthermore, use a bandpass filter (Bandpass Filter) to retain the frequency components related to the target event. Feature extraction: Use Mel-frequency cepstral coefficients (MFCC), Chroma features, Spectral Contrast and other methods to extract audio features. In order to enhance the robustness of the features, a convolutional autoencoder (ConvolutionalAutoencoder) can be used to perform automated deep feature extraction of audio features. Video preprocessing:

[0041] Image denoising: Use bilateral filtering and convolutional neural networks (CNN) to denoise video frames and enhance image clarity, especially in low-light scenes. Motion detection and keyframe extraction: Use optical flow to detect motion information in the video, and combine spatio-temporal interest points (STIP) technology to extract keyframes to ensure that the extracted visual features are representative for sound event detection. Spatial-temporal feature extraction: Combine 3D convolutional neural networks (3D-CNN) to extract spatial-temporal features from video sequences to capture the motion patterns and behavioral characteristics of objects.

[0042] Sensor data preprocessing: Signal smoothing: Smoothing the sensor data through sliding window filtering (MovingAverage Filter) and Kalman filtering to eliminate burst noise in sensor readings and ensure signal stability. Feature extraction: Extract corresponding features according to different sensor types, such as the spectrum characteristics of accelerometer data (such as FFT transform), the periodic change characteristics of temperature and humidity data, and the brightness distribution characteristics of light sensors. For composite sensor data, use principal component analysis (PCA) to reduce the dimension and retain the most representative feature vectors.

[0043] Time synchronization: In order to ensure the consistency of multimodal data, the global clock synchronization technology is used, and each data sampling point is marked with a timestamp to ensure the time alignment between audio, video and sensor data. Cross-modal alignment: Dynamic time warping (DTW) technology is applied to align cross-modal data. Especially in the case of inconsistent sampling rates, this method can effectively align the feature sequences of different modalities to ensure the consistency of multimodal data fusion. Data augmentation: In order to improve the robustness of the model, data augmentation technology is used, such as time stretching (TimeStretching) and pitch shifting (Pitch Shifting) of audio data and adding background noise, and rotating, cropping, flipping and other operations on video data to generate more sample data. Normalization: The feature data of each modality is normalized, such as zero mean unit variance (Z-score normalization) and Min-Max normalization, to map different feature values ​​to a unified scale range, reducing the impact of differences between modalities on the fusion results.

[0044] S300: performing feature extraction and feature fusion on the processed multimodal data to obtain multimodal features;

[0045] Specifically, in a multimodal ambient sound event detection system, data of different modes carry rich information. In order to make full use of this information, it is necessary to use advanced feature extraction methods to extract high-dimensional feature vectors of each modal data.

[0046] Audio feature extraction: For audio signals, first apply short-time Fourier transform (STFT) to convert them into time-frequency domain representation to generate a spectrogram. Then, use deep convolutional neural network (CNN) to extract features from the spectrogram. CNN can effectively extract time-frequency domain features from audio signals, such as frequency change patterns and harmonic structures, through convolutional layers and pooling layers. Specifically, let the input spectrogram be Where T is the number of time frames and F is the number of frequency bins. CNN generates feature representation after L layers of convolution and pooling. Where C is the number of channels, T′ and F′ are the reduced time and frequency dimensions respectively.

[0047] Video feature extraction: For video sequences, we first use a convolutional neural network (CNN) to extract the spatial features of each frame, and then use a long short-term memory network (LSTM) to process the temporal features. The LSTM model can capture the temporal dependencies between video frames, thereby extracting dynamic features that reflect the movement and changes of objects. Specifically, let the video sequence be V = {V1, V2, …, V T},in Represents the tth frame image, H and W are the image height and width respectively. CNN extracts spatial features F for each frame image t =CNN(V t ), LSTM then performs {F1,F2,…,F T} to generate video feature representation H video =LSTM(F).

[0048] Sensor feature extraction: Sensor data contains physical information such as acceleration, temperature, humidity, and light. To this end, multi-layer perceptron (MLP) or convolutional network is used to extract features for different sensor data types. Suppose the sensor data is S = {S1, S2, ..., S N}, where S i Represents the output signal of the i-th sensor. The sensor data is extracted through MLP to generate a feature vector H sensor =MLP(S), these features capture the physical information of environmental changes, such as temperature fluctuations, vibration patterns, etc.

[0049] The key to multimodal feature fusion is to effectively combine the features of different modalities to form a unified feature representation. To this end, a variety of advanced feature fusion strategies are adopted to ensure that the fused features can reflect more comprehensive environmental information.

[0050] Attention mechanism fusion: The attention mechanism plays a vital role in feature fusion. By introducing the attention mechanism, different weights can be assigned to different modal features according to their importance, ensuring that the most critical information is processed first. Suppose that the audio, video and sensor features are H audio , H video , H sensor . Introducing attention weight α audio , α video , α sensor , these weights are generated by the attention network:

[0051]

[0052] The final fusion feature H fusion It can be expressed as:

[0053] H fusion =α audio ·H audio +α video ·H video +α sensor ·H sensor

[0054] This weighted fusion method can dynamically adjust the influence of each modality, making the system more adaptable when dealing with different scenarios.

[0055] Deep Fusion Network: The deep fusion network gradually fuses multimodal features through multiple layers of nonlinear transformations to capture the complex relationship between modalities. In specific implementation, a Transformer-based multimodal fusion architecture is adopted to achieve information interaction between modalities through a multi-head self-attention mechanism. Transformer first maps the features of each modality to a unified vector space:

[0056]

[0057] Then the features of each modality are fused through the self-attention mechanism:

[0058]

[0059] Finally, after a multi-layer feed-forward network (FFN) and normalization layer (LayerNormalization), a high-level fusion feature H is output. final This deep fusion strategy not only captures the interactive relationship between modalities, but also extracts feature representations with a global perspective, greatly improving the detection performance of the system in complex environments.

[0060] S400: construct a deep learning model, and use multimodal features for training to obtain a trained deep learning model;

[0061] Specifically, the core of environmental sound event detection is to build a deep learning model that can fully utilize multimodal data. To achieve this goal, a Transformer-based deep learning architecture is adopted and optimized to ensure efficient and accurate event detection in various complex scenarios.

[0062] 1. Selection of deep learning model:

[0063] 1.1 Transformer-based multimodal fusion model:

[0064] The Transformer model was originally applied in the field of natural language processing (NLP), but its powerful self-attention mechanism also makes it perform well in processing long sequence data and multimodal data fusion. The following is a specific application of the model in environmental sound event detection:

[0065] Multi-head Self-Attention mechanism, which allows the model to capture the correlation between different modal features independently on different "attention heads". Let the input feature matrix be X = [H audio ,H video ,H sensor ], where each eigenvector H i Represents the features of a modality. The multi-head self-attention is calculated as follows:

[0066]

[0067] Among them, Q, K, and V are query, key, and value matrices obtained through different linear transformations, respectively. k is the dimension of the key. Through the parallel computation of multiple attention heads, the model can extract global context information from different modal features and finally fuse this information into a global feature representation.

[0068] Positional Encoding, since Transformer lacks a built-in sequential processing mechanism, positional encoding is introduced to preserve sequence information. For time series data (such as audio and video), positional encoding generated using sine and cosine functions can effectively help the model understand the order of time steps:

[0069]

[0070] Among them, pos represents the position of the time step, i represents the dimension index, and d is the feature dimension.

[0071] Cross-modal attention, in addition to the conventional self-attention mechanism, cross-modal attention allows the model to better handle the interaction between modalities. For example, the correlation between audio and video features can be modeled through a cross-modal attention layer, the specific formula is:

[0072] H fusion =Cross-Attention(H audio ,H video )

[0073] This mechanism further improves the effect of multimodal feature fusion by learning the interactive relationship between different modal features.

[0074] 1.2 Integration of timing models:

[0075] In order to process audio and video data with a long span, the present invention also integrates a combined model of a bidirectional long short-term memory network (BiLSTM) and a Transformer. BiLSTM can capture the dependencies between sequences, while Transformer is good at extracting global features. This combined model structure can better understand the complex temporal information of environmental sound events, thereby improving the accuracy of detection.

[0076] 2. Model training:

[0077] 2.1 Dataset preparation:

[0078] Construction of multimodal datasets:

[0079] During the model training process, a large-scale dataset covering a variety of scenarios and events is required. This dataset contains multi-modal annotated data, including audio, video, and sensor data. Each piece of data is annotated in detail to indicate the specific type of environmental sound event and the time of occurrence.

[0080] Data Augmentation: In order to improve the generalization ability of the model, especially in the case of limited data, data augmentation technology is applied. Audio data enhancement techniques include adding noise, time stretching, pitch shifting, etc. Video data generates more samples through enhancement methods such as rotation, cropping, and color transformation.

[0081] 2.2 Loss function optimization:

[0082] Cross-Entropy Loss: Cross-Entropy loss is a commonly used loss function in classification problems, which is used to measure the difference between the probability distribution predicted by the model and the actual label distribution:

[0083]

[0084] Among them, y c is the distribution of true labels, is the probability distribution predicted by the model and C is the number of categories.

[0085] Focal Loss: In order to solve the problem of class imbalance, especially the long-tail distribution problem commonly seen in environmental sound event detection, Focal Loss is used. This loss function reduces the weight of easy-to-classify samples and focuses on difficult-to-classify samples by introducing a conditioning factor:

[0086]

[0087] Among them, γ is a tuning parameter used to control the model's attention to difficult-to-classify samples.

[0088] 2.3 Model optimization and tuning:

[0089] Hyperparameter Optimization: Bayesian Optimization is used to tune the model's hyperparameters, such as learning rate, batch size, number of Transformer layers, number of attention heads, etc. Bayesian Optimization gradually updates the probability model to find the optimal parameter combination with a smaller number of experiments.

[0090] Model pruning and quantization: In order to achieve real-time detection on edge devices, model pruning and quantization techniques are used to reduce model complexity. Pruning technology reduces the amount of computation by removing redundant parameters, while quantization reduces storage requirements and inference time by reducing the bit width of weights and activation values.

[0091] Knowledge Distillation: In order to reduce the model size while maintaining model performance, knowledge distillation technology is used. By letting a small student model learn the output distribution of a large teacher model, the small model can significantly reduce the number of parameters while maintaining performance:

[0092]

[0093] Among them, α is the weight coefficient, which adjusts the ratio of the loss of the teacher model to the loss of the student model.

[0094] S500: Deploy the deep learning model after pruning, quantization and knowledge distillation on the edge device, implement preliminary feature extraction and event detection through edge computing, and respond based on the detection results;

[0095] Specifically, the real-time detection and response mechanism is a key component to ensure that the environmental sound event detection system has efficient, accurate and rapid response capabilities in practical applications. In order to achieve efficient detection and response in a resource-constrained environment, the present invention proposes a complete solution that combines edge computing, model compression and online learning to meet complex environmental sound detection needs.

[0096] 1. Edge computing and model compression:

[0097] 1.1 Introduction of edge computing:

[0098] With the development of the Internet of Things (IoT), edge computing technology has gradually become an important means of processing real-time data. By performing some calculations on edge devices close to the data source, data transmission delays can be significantly reduced and the burden on cloud servers can be reduced. Specifically, the present invention applies edge computing in the ambient sound event detection system and designs a distributed detection architecture:

[0099] Layered computing architecture: The system divides computing tasks into edge and cloud layers. The edge layer is responsible for preliminary feature extraction and event detection, while the cloud layer performs complex multimodal feature fusion and in-depth analysis. In this way, edge devices can complete real-time detection of simple ambient sound events locally, while complex events will be transmitted to the cloud for further processing.

[0100] Edge Inference Engine: To achieve real-time performance, edge devices are equipped with a lightweight inference engine that runs on pruned deep learning models and can efficiently perform tasks such as fast feature extraction and preliminary classification of audio on resource-limited devices.

[0101] 1.2 Model pruning, quantization and distillation:

[0102] In order to efficiently run complex deep learning models on edge devices, the present invention adopts a series of model compression techniques, including model pruning, quantization, and knowledge distillation.

[0103] Model Pruning: Model pruning reduces the complexity and computational complexity of the model by removing unimportant weights or neurons in the network. The pruned model can significantly speed up the inference speed with almost no impact on accuracy. Specific methods include global pruning and layer pruning. Global pruning prunes according to the importance of the weights of the entire model, while layer pruning optimizes specific layers.

[0104] The pruned model M′ can be expressed as:

[0105] M′=Prune(M,θ)

[0106] Among them, M is the original model and θ is the pruning threshold.

[0107] Model Quantization: By quantizing the model's weights and activation values ​​from 32-bit floating point numbers to 8-bit or lower integer values, the model's memory usage and computational complexity can be greatly reduced. Common quantization techniques include dynamic quantization, static quantization, and mixed precision quantization.

[0108] The calculation process after quantizing the model is:

[0109]

[0110] Among them, Q(·) is the quantization function, W is the model weight, and x is the input data.

[0111] Knowledge Distillation: By allowing a lightweight student model to learn the knowledge of a large teacher model, the student model can significantly reduce the size of the model while retaining key features. The distillation process not only optimizes the prediction results of the student model during the training phase, but also retains the generalization ability of the teacher model to a certain extent.

[0112] The distillation loss function is:

[0113]

[0114] in, is the cross entropy loss function, y T is the output of the teacher model.

[0115] 2. Real-time response:

[0116] 2.1 Event detection and priority processing:

[0117] After detecting a specific ambient sound event, the system responds quickly through preset priority rules. This rule assigns different urgency levels based on different types of events to ensure that the system can immediately handle high-priority events, such as fire alarms, broken windows, etc.:

[0118] Priority classification: Ambient sound events are classified into three priority levels: high, medium, and low. When a high-priority event is triggered, the system immediately activates the early warning mechanism, such as sending an alarm, triggering emergency linkage equipment, etc. Medium and low priority events may only record logs or issue reminder notifications.

[0119] Response time optimization: To reduce response delays, the system replaces the traditional polling mechanism with an event-driven interrupt mechanism. This can significantly improve response speed, especially when high-priority events are triggered.

[0120] 2.2 Linkage Mechanism:

[0121] In order to achieve a full range of event responses, the system supports linkage operations with other smart devices or systems. For example, when a fire alarm is detected, the system can automatically trigger the smoke alarm, close the gas valve, and make an emergency call. This linkage mechanism is implemented through standardized protocols (such as MQTT, HTTP, etc.) to ensure seamless collaboration between different devices.

[0122] S600: Continuously monitor and analyze test results, identify errors or false alarms in a timely manner, and make corresponding corrections and adjustments.

[0123] Specifically, in order to continuously improve the accuracy and adaptability of the system, a real-time data feedback mechanism is integrated into the detection system. By continuously monitoring and analyzing the results of system detection, errors or false alarms can be identified in a timely manner, and corresponding corrections and adjustments can be made:

[0124] Feedback loop: When the system detection results do not match the actual events, users can report errors through the feedback interface. These feedback data will be recorded and used for subsequent optimization of the model.

[0125] 3.2 Online learning and model updating:

[0126] The system uses online learning technology to adapt to environmental changes by updating model parameters in real time. Online learning can continuously adjust the model weights based on new data to ensure that the system maintains efficient detection performance after long-term operation:

[0127] Incremental Update: Using incremental learning algorithms, the system can adapt to new data distributions by updating some parameters without retraining the entire model. For example, using stochastic gradient descent (SGD) to fine-tune the new data to ensure the continuity and stability of the model.

[0128] The updating process can be expressed as:

[0129]

[0130] Where η is the learning rate, w t is the weight of the model at time step t, x t and t are the input data and corresponding labels.

[0131] 3.3 Adaptive learning of the model:

[0132] As time goes by and the environment changes, the system needs to be able to adapt. Through adaptive learning mechanisms, the system can detect and respond to dynamic changes in the environment. For example, when the characteristics of a certain type of event are detected to change, the system will automatically adjust its detection strategy to maintain efficient detection performance:

[0133] Adaptive threshold adjustment: The system can dynamically adjust the detection threshold according to the changes in the ambient noise level, thereby reducing the false alarm rate. For example, in a quiet environment at night, the noise detection threshold is lowered to increase sensitivity.

[0134] System application and scalability of the environmental sound event detection method based on multimodal data fusion:

[0135] 1. Wide range of application scenarios:

[0136] The multimodal ambient sound event detection system of the present invention has a wide range of application potentials and is suitable for security monitoring, disaster warning, smart home, traffic monitoring, health monitoring and other fields. In the construction of smart cities, the system can effectively improve public safety and emergency response capabilities.

[0137] 2. Scalability:

[0138] The system design has good scalability, and the detection capability can be enhanced by adding new sensors or data sources. For example, in smart home applications, the data of household appliances can be combined to further improve the detection accuracy of abnormal events.

[0139] 3. Customized and modular design:

[0140] The system supports customized configuration, and users can choose different modal data sources and detection models according to specific needs. In addition, the modular design makes system maintenance and upgrades more convenient, and users can easily integrate new functional modules to meet changing application needs.

[0141] 4. Advantages:

[0142] High robustness brought by multimodal fusion: The present invention significantly improves the robustness of environmental sound event detection by fusing multimodal data such as audio, video and sensors. In complex real-world scenarios, such as traffic noise, crowd noise and multi-source mixed sound environments, single-modal data is easily interfered with and causes false alarms or missed alarms. The multimodal fusion method of the present invention can comprehensively utilize the data features of each modality to accurately capture and identify different environmental sound events. Through the combination of attention mechanism and deep fusion network, the advantages of different modalities are complementary, and the system exhibits extremely high anti-interference ability and robustness when dealing with complex environments. High efficiency of real-time detection and response: The present invention adopts a series of advanced model compression technologies, such as model pruning, quantization and knowledge distillation, combined with edge computing architecture, to achieve real-time detection and response of environmental sound events. On resource-constrained edge devices, the deep learning model after pruning and quantization greatly reduces the computing overhead and significantly improves the detection speed. At the same time, the introduction of edge computing effectively reduces the delay of data transmission, enabling the system to respond within milliseconds and adapt to dynamically changing application scenarios, such as smart cities and disaster warning systems. Through event priority classification and interrupt-driven mechanism, the system ensures rapid processing of high-priority events and realizes an efficient closed loop from detection to response. Strong generalization and adaptability: The present invention improves the generalization and adaptability of the system in diversified application scenarios through online learning and adaptive learning mechanisms. When facing different environments and changing sound patterns, the system can adjust model parameters and detection strategies in real time to ensure that efficient detection performance is always maintained. Whether in high-noise urban streets or in quiet residential areas, the system can adaptively adjust detection thresholds and strategies to reduce false alarm rates and improve detection accuracy. In addition, the system uses data enhancement and incremental update technology to enable the model to continuously learn new event types and sound features, and has strong generalization capabilities to adapt to various emerging application requirements. Highly flexible modular design: The present invention has a highly flexible system design that can be modularly customized and configured according to actual application requirements. Users can select data sources of different modalities (such as adding new sensor types) according to specific scenarios, as well as adjust fusion strategies and detection models. This flexibility not only enables the system to adapt to diverse detection needs, but also enables it to maintain excellent performance in different application scenarios. The modular design also greatly facilitates the maintenance and upgrade of the system. Users can easily integrate new functional modules as needed to meet the ever-changing technology and market requirements.

[0143] What is disclosed above is only one or more preferred embodiments of the present application, and cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for detecting environmental sound events based on multimodal data fusion, characterized in that: The following steps are involved: Use microphone arrays, cameras and sensors to synchronously collect audio, video and environmental data to obtain multimodal data; Processing of multimodal data; Perform feature extraction and feature fusion on the processed multimodal data to obtain multimodal features; Build a deep learning model and use multimodal features for training to obtain a trained deep learning model; Deploy the deep learning model after pruning, quantization and knowledge distillation on the edge device, implement preliminary feature extraction and event detection through edge computing, and respond based on the detection results; Continuously monitor and analyze test results, promptly identify errors or false positives, and make corresponding corrections and adjustments.

2. The method for detecting environmental sound events based on multimodal data fusion according to claim 1, characterized in that: The microphone array, camera and sensor are used to synchronously collect audio, video and environmental data to obtain multimodal data. The steps also include: Use a microphone array to collect ambient audio signals and use short-time Fourier transform to convert the audio signals into a representation in the time-frequency domain; Use high frame rate cameras to capture visual information in the scene, including the movement trajectory of objects, changes in lighting, and dynamic changes in the background environment, and use visual SLAM technology to generate a 3D reconstruction of the video data; The use of multiple sensors can capture multi-dimensional information of environmental changes, and the data from different sensors can be integrated through multi-sensor fusion technology to form an environmental information representation.

3. The method for detecting environmental sound events based on multimodal data fusion according to claim 2, characterized in that: Processing the multimodal data, the step further includes: Use adaptive filters and noise suppression algorithms to reduce the noise of audio data, use bilateral filtering and convolutional neural networks to denoise video frames, and use sliding window filtering and Kalman filtering to smooth sensor data, eliminate burst noise in sensor readings, and obtain pre-processed data of each modality; Apply dynamic time warping techniques to align cross-modal data; Data enhancement technology is used to enhance the data of each modality.

4. The method for detecting environmental sound events based on multimodal data fusion according to claim 3, characterized in that: Performing feature extraction and feature fusion on the processed multimodal data to obtain multimodal features, the step further includes: Apply short-time Fourier transform to convert the audio signal into time-frequency domain representation and generate a spectrogram. Then, use deep convolutional neural network to extract features from the spectrogram and extract the time-frequency domain features in the audio signal. A convolutional neural network is used to extract the spatial features of each frame of the image, and then the time series features are processed through a long short-term memory network to extract dynamic features that reflect the movement and changes of the object; Use multi-layer perceptron or convolutional network to extract sensor data features.

5. The method for detecting environmental sound events based on multimodal data fusion according to claim 4, characterized in that: Deploy the deep learning model after pruning, quantization and knowledge distillation on the edge device, implement preliminary feature extraction and event detection through edge computing, and respond according to the detection results. The steps also include: When the edge device detects a specific ambient sound event, it classifies the event according to the preset priority rules and responds quickly. For high-priority events, the early warning mechanism is immediately activated. For medium and low priority events, logs are recorded or reminder notifications are issued. The edge device is linked with the cloud server to upload complex events or multimodal feature fusion tasks to the cloud for further processing and analysis.

Images