Noise identification and detection method and system based on multi-modal fusion

Abstract

The invention relates to a noise identification and detection method and system based on multi-modal fusion. The system comprises an acoustic acquisition and analysis module, a visual acquisition and detection module, a multi-modal feature fusion module and a decision and output module. The acoustic acquisition and analysis module comprises a microphone array and is responsible for acquiring environmental sound signals and performing feature extraction; the visual acquisition and detection module comprises a camera and is used for identifying potential sound source objects in a picture based on a target detection algorithm; the multi-modal feature fusion module is connected with the acoustics acquisition and analysis module and the visual acquisition and detection module and is used for performing time alignment and semantic fusion on the acoustics modal features of the acoustics acquisition and analysis module and the visual modal features of the visual acquisition and detection module; the decision and output module is connected with the multi-modal feature fusion module and outputs a noise type, a sound source position and a confidence result; the device is reasonable in design, compact in structure and convenient to use.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and environmental noise monitoring technology, specifically relating to a multimodal noise recognition and detection method and system based on a combination of audio recognition and computer vision target detection, and to a noise recognition and detection method and system based on multimodal fusion. Background Technology

[0002] Currently, environmental noise monitoring and noise source identification mainly rely on single-modal identification techniques based on acoustic features. These methods determine the noise type or event category by performing spectral analysis, feature extraction, and classification modeling on audio signals. Common algorithms include traditional machine learning methods based on MFCC (Mel-frequency cepstral coefficients), as well as deep learning models based on convolutional neural networks (CNNs) or recurrent neural networks (RNNs).

[0003] However, single-modal acoustic recognition has the following significant limitations:

[0004] 1. Sensitive to environmental interference: In complex sound fields, sound is affected by background noise, echoes, and the superposition of multiple sources, which leads to a decrease in recognition accuracy;

[0005] 2. Unable to determine the location and target of the sound source: The acoustic signal lacks spatial information, making it impossible to pinpoint the specific target (such as a vehicle, crowd, or machinery) from which the noise originates.

[0006] 3. Insufficient interpretability: The identification results are based solely on acoustic features and lack visual confirmation, making the results difficult to use directly in noise monitoring and evidence collection scenarios;

[0007] 4. Difficulty in identifying multiple overlapping sound sources: When multiple sound sources emit sound simultaneously, traditional acoustic models struggle to separate and match them.

[0008] Therefore, existing technologies generally face problems such as difficulty in positioning, unstable identification, and lack of visual traceability, which limits their practical application in the fields of urban intelligent monitoring and public safety. Summary of the Invention

[0009] To address the shortcomings of existing acoustic recognition technologies, this invention proposes a noise recognition and detection method and system based on multimodal fusion. By combining an audio signal recognition model with a computer vision detection model, it achieves accurate identification of noise events and localization of sound sources, thereby achieving the detection goal of "hearing the sound and seeing the person".

[0010] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0011] A noise recognition and detection system based on multimodal fusion, comprising an acoustic acquisition and analysis module, a visual acquisition and detection module, a multimodal feature fusion module, and a decision and output module;

[0012] The acoustic acquisition and analysis module, including a microphone array, is responsible for acquiring ambient sound signals and extracting features.

[0013] The visual acquisition and detection module includes a camera, which identifies potential sound source objects in the image based on a target detection algorithm;

[0014] The multimodal feature fusion module connects the acoustic acquisition and analysis module and the visual acquisition and detection module, and is used to perform temporal alignment and semantic fusion of the acoustic modal features of the acoustic acquisition and analysis module and the visual modal features of the visual acquisition and detection module.

[0015] The decision and output module connects to the multimodal feature fusion module and outputs the noise type, sound source location, and confidence level results.

[0016] Furthermore, the acoustic acquisition and analysis module uses a microphone array to acquire ambient audio signals. After endpoint detection, noise reduction, and normalization, the ambient audio signal is... Converted to a time-domain signal, the time-domain signal is then subjected to a Short-Time Fourier Transform (STFT) to obtain its time spectrum:

[0017] Formula (1);

[0018] Will Mapping to the Mel scale yields the Mel spectrum:

[0019] Formula (2);

[0020] in, For window functions, Frame length;

[0021] Discrete-time signals;

[0022] Intra-frame sample coordinates;

[0023] Frame index, time frame;

[0024] Frequency index;

[0025] Imaginary unit;

[0026] Where M is the Mel spectrum, i.e., the Mel filter;

[0027] m is the first The index of a Mel filter, (1 ≤ m ≤ M). For the m-th Mel filter;

[0028] After logarithmic transformation and normalization of formula (2), it is input into the lightweight convolutional neural network YAMNet to obtain the acoustic feature tensor F. a :

[0029] Formula (3);

[0030] Where 'a' stands for 'audio', and it is a subscript for the feature, indicating an acoustic feature.

[0031] YAMNet consists of multiple convolutional layers, batch normalization, and pooling, and outputs embedded feature vectors through a final fully connected layer. ;

[0032] in, Indicates the number of time frames. Represents the dimensions of acoustic features;

[0033] Meanwhile, based on formula (3), the preliminary acoustic classification results are as follows: Represented as:

[0034] Formula (4);

[0035] in, This indicates a global average pooling operation. and These are the classification layer weights and biases, respectively.

[0036] Visual Acquisition and Detection Module: The visual acquisition module acquires video frame sequences through a synchronous camera. And input YOLOv13n to detect the network;

[0037] The YOLO network structure is described as follows:

[0038] Formula (5);

[0039] F V For visual feature tensors; v is video;

[0040] in: It serves as the backbone network for extracting multi-scale features; The neck network is used to achieve feature fusion; This is the detection head, used to output the detection results of the target;

[0041] The target detection output D is:

[0042] Formula (6);

[0043] n represents the number of targets detected by the visual detection module in a single frame of video image;

[0044] in, For the first The coordinates of each detection box. For category labels, To detect confidence levels, the feature tensor of the previous layer of the Head is extracted. .

[0045] Furthermore, the multimodal feature fusion module achieves temporal and semantic alignment between acoustic information from the acoustic acquisition and analysis module and visual information from the visual acquisition and detection module.

[0046] The multimodal feature fusion module adopts a lightweight Transformer structure based on cross-attention;

[0047] In the multimodal feature fusion module,

[0048] First, the acoustic feature tensor With visual feature tensor Projected onto a space of the same dimension:

[0049] Formula (7);

[0050] in, These are the trainable mapping matrices;

[0051] Where Q, Query, represents the semantic information that the current acoustic modality hopes to extract from the visual modality; K, Key, represents the key description of the visual features; and V, Value, represents the content information in the visual features that can be used for fusion.

[0052] Based on formula (7), Cross-Attention is calculated as follows:

[0053] Formula (8);

[0054] d K It is the dimension of the key vector;

[0055] K T It is the transpose of K;

[0056] The output features after fusion are:

[0057] Formula (9);

[0058] f It is finally;

[0059] in, This represents a feedforward fully connected network, consisting of two linear layers and ReLU activations:

[0060] Formula (10);

[0061] x is the input feature vector; It is the first layer weight matrix; It is the first-level bias term; It is the second layer weight matrix; It is the second-level bias term;

[0062] Decision and Output Module: Output Characteristics of the Fusion Formula (9)

[0063] After global average pooling and fully connected layers, the final noise class probability distribution P is output. f :

[0064] Formula (11);

[0065] in, For global average pooling, This is the classification weight matrix. For bias;

[0066] c, the index of the category or a category-related parameter dimension;

[0067] A weighted confidence fusion strategy is adopted, based on the target detection output, to integrate the acoustic classification results. With visual detection confidence

[0068] Comprehensive calculation:

[0069] Formula (12);

[0070] in, c represents the weighting coefficient. i P represents the i-th noise category; final ( ) indicates that the target belongs to the category. The final confidence probability;

[0071] The final output includes noise category, sound source object, and final confidence level P. final ;

[0072] Acoustic classification results Including noise category and sound source object; Pa It represents the probability of the acoustic classification result;

[0073] The system is powered by the Jetson Orin Nano Super platform.

[0074] A noise identification and detection method based on multimodal fusion, based on the above-mentioned system;

[0075] Perform the following steps;

[0076] S1: Acquire ambient sound signals and extract acoustic features through the acoustic acquisition and analysis module;

[0077] S2, through the visual acquisition and detection module, identifies potential sound source targets in the image and obtains visual semantic features;

[0078] S3, through a multimodal feature fusion module, achieves temporal alignment and semantic fusion of acoustic and visual information;

[0079] S4: Through a comprehensive decision and output module, calculate the noise category probability, sound source location, and confidence level.

[0080] Furthermore, S1 specifically includes the following steps:

[0081] S1.1, Signal Acquisition: Acquire ambient audio signals synchronously through a microphone array, and perform endpoint detection and bandpass filtering on the input signals;

[0082] S1.2, Preprocessing: Noise reduction and signal amplitude standardization are performed using spectral subtraction and normalization techniques;

[0083] S1.3, Feature Transformation: Converting the time-domain signal into a Mel-spectrum.

[0084] In step S1.3, the window length is set to 25 ms, the frame shift to 10 ms, and the number of Mel filters is set to 64.

[0085] S1.4, Feature Extraction: Input to the lightweight acoustic event recognition network YAMNet;

[0086] The YAMNet network is based on the MobileNetV1 architecture;

[0087] In step S1.4, there are 13 depthwise separable convolutional layers and 3 pooling layers; each convolutional layer uses ReLU activation and batch normalization; finally, 1024-dimensional acoustic embedding features are obtained through global average pooling (GAP).

[0088] S1,5, Feature compression mapping: The feature is mapped to a 256-dimensional vector space via a fully connected layer (Dense(256)) to align with the visual features in terms of dimension. The output is a 256-dimensional acoustic feature, denoted as: .

[0089] Furthermore, in step S2, the process is as follows:

[0090] S2.1, Image Acquisition: Acquire video frame feature sequences through a camera;

[0091] S2.2, Object Detection: Input video frames into the YOLOv13n object detection network for object recognition;

[0092] The network consists of a Backbone, a Neck, and a Detection Head. Features are extracted through the Backbone, fused and enhanced through the Neck, and the final detection result is generated through the Detection Head.

[0093] S2.3, Semantic Feature Extraction: Extract global visual embeddings from the intermediate feature maps of the layer preceding the Detection Head. Obtain 256-dimensional visual features through 1×1 convolution and global average pooling, thus obtaining the visual feature tensor. .

[0094] Furthermore, in step S3, the following steps are performed:

[0095] S3.1, Feature Input: Input the acoustic feature tensor As a query, visual feature tensor As Key and Value;

[0096] S3.2, Time Alignment: Synchronization matching based on frame index ensures the acoustic feature tensor With visual feature tensor The two modal features correspond to the same time window;

[0097] S3.3, Cross-Attention Fusion: Cross-Attention structure is used to achieve information exchange between modalities. Its core calculation is as follows:

[0098] Formula (8);

[0099] in, , These are the learnable mapping matrices;

[0100] S3.4, Fusion Output: After passing through self-attention and the feedforward network, the output is a fused feature:

[0101] Formula (9);

[0102] in For self-attention mechanism, It is a feedforward fully connected network;

[0103] S3.5, Semantic Compression: Mapped to a final fused semantic vector through a Dense(256) layer, used for noise category determination and sound source matching.

[0104] Furthermore, in step S4, the following steps are performed:

[0105] S4.1, Acoustic Classification: Fusion Features Perform global average pooling and feed it into the classification head to obtain the predicted probabilities:

[0106] Formula (11);

[0107] in For global average pooling, This is the classification weight matrix. For bias;

[0108] S4.2, Visual Confidence Fusion: Jointly calculate the confidence scores from acoustic classification results and YOLO outputs.

[0109] Formula (12);

[0110] in The value is 0.7;

[0111] S4.3, Output Results: The output includes noise category, sound source object, overall confidence level, and visualization results.

[0112] Furthermore, S5: performs system operation and deployment;

[0113] The system's model was deployed on the Jetson Orin Nano Super platform, using a microphone array to collect audio data and a camera to collect video data. TensorRT was used to accelerate the model on the video stream.

[0114] Furthermore, the audio sampling rate is 48 kHz, and the video frame rate is 30 fps.

[0115] Compared with existing technologies, the multimodal noise recognition and detection method and system proposed in this invention, by integrating two perceptual modalities—acoustic recognition and computer vision detection—has the following significant advantages and beneficial effects:

[0116] (1) Precise identification and visual positioning:

[0117] This invention utilizes the complementary characteristics of acoustic and visual models to improve recognition stability in complex environments.

[0118] (2) Enhanced interpretability:

[0119] By synchronizing sound events with the temporal characteristics of video frames, this invention can visualize the individual sources of noise, enabling the system not only to "hear the noise" but also to "see who is making the noise," thereby improving the interpretability and practical value of noise monitoring, incident evidence collection, and environmental monitoring.

[0120] (3) Strong anti-interference and robustness

[0121] In multi-sound-source environments, this invention uses visual information as an auxiliary constraint to correct acoustic judgments when visual targets and sound source signals do not match, thereby reducing false detection and false negative rates.

[0122] (4) Possesses real-time processing capability

[0123] This invention employs a lightweight deep neural network structure and a parallel inference mechanism, supporting real-time operation on Jetson edge computing devices. Attached Figure Description

[0124] Figure 1 This is a system overall structure block diagram of the present invention.

[0125] Figure 2 This is the model network structure of the present invention.

[0126] Figure 3 This is a system schematic diagram of the present invention. Detailed Implementation

[0127] like Figure 1-3 The system of the present invention includes an acoustic acquisition and analysis module, a visual acquisition and detection module, a multimodal feature fusion module, and a decision and output module;

[0128] The acoustic acquisition and analysis module, including a microphone array, is responsible for acquiring ambient sound signals and extracting features.

[0129] The visual acquisition and detection module includes a camera, which identifies potential sound source objects in the image based on a target detection algorithm;

[0130] The multimodal feature fusion module connects the acoustic acquisition and analysis module and the visual acquisition and detection module, and is used to perform temporal alignment and semantic fusion of features from the two modalities of the acoustic acquisition and analysis module and the visual acquisition and detection module.

[0131] The decision and output module outputs the noise type, sound source location, and confidence level results.

[0132] Specifically,

[0133] The acoustic acquisition and analysis module uses a microphone array to acquire ambient audio signals. After endpoint detection, noise reduction, and normalization, the ambient audio signal is... The signal is converted to a time-domain signal, and the time-domain signal is then subjected to a short-time Fourier transform (STFT) to obtain its time spectrum.

[0134] Formula (1);

[0135] To reduce frequency domain redundancy and improve sensing consistency, Mapping to the Mel scale yields the Mel spectrum:

[0136] Formula (2);

[0137] in, Window function Frame length; Discrete-time signals; Intra-frame sample coordinates;

[0138] : Frame index, which is a time frame; Frequency index; Imaginary unit;

[0139] M: Mel spectrum, i.e., Mel filter;

[0140] m: the The index of a Mel filter, (1 ≤ m ≤ M);

[0141] : The m-th Mel filter.

[0142] After logarithmic transformation and normalization of formula (2), the result is input into the lightweight convolutional neural network YAMNet to obtain the acoustic feature sequence F. a :

[0143] Formula (3);

[0144] Where 'a' stands for 'audio', and it serves as a subscript for the feature, indicating an acoustic feature.

[0145] YAMNet consists of multiple layers of convolutional vectors (Conv), batch normalization (BN), and pooling, and outputs embedded feature vectors through a final fully connected layer. ;

[0146] in Time frame count, Acoustic feature dimension.

[0147] Meanwhile, based on formula (3), the preliminary acoustic classification results are as follows: Represented as:

[0148] Formula (4);

[0149] in, Global average pooling operation. and These are the classification layer weights and biases, respectively.

[0150] Visual Acquisition and Detection Module: The visual acquisition module acquires video frame sequences through a synchronous camera. And input YOLOv13n to detect the network;

[0151] The YOLO network structure can be described as follows:

[0152] Formula (5);

[0153] F V : Visual feature tensor; v: video;

[0154] in: : Backbone network, extracting multi-scale features; Neck network, enabling feature fusion; : Detection head, outputs the detection results of the target.

[0155] The target detection output is:

[0156] Formula (6);

[0157] Where, n: the number of targets detected by the visual detection module in a frame of video image;

[0158] : No. The coordinates of each detection box; Category tags; : Detect confidence level, and simultaneously extract the feature tensor of the previous layer of the Head. This is used as input for subsequent multimodal fusion.

[0159] Multimodal feature fusion module: To achieve temporal and semantic alignment of acoustic and visual information, this invention adopts a lightweight Transformer structure based on cross-attention;

[0160] First, the acoustic feature tensor With visual feature tensor Projected onto a space of the same dimension:

[0161] Formula (7);

[0162] in Here, Q stands for Trainable Mapping Matrix; Q: Query, representing the semantic information that the current acoustic modality hopes to extract from the visual modality; K: Key, representing the key description of the visual features; V: Value, representing the content information in the visual features that can be used for fusion.

[0163] Based on formula (7), Cross-Attention is calculated as follows:

[0164] Formula (8);

[0165] d K : The dimension of the key vector;

[0166] K T : The transpose of Key;

[0167] The output features after fusion are:

[0168] Formula (9);

[0169] in, f finally; A feedforward fully connected network, consisting of two linear layers and ReLU activation:

[0170] Formula (10);

[0171] x: Input feature vector; First layer weight matrix; First-level bias term; Second layer weight matrix; Second-level bias term;

[0172] In formula (10), the structure realizes the dynamic focusing of the region in the visual features associated with the sound source by using the sound feature tensor as a guide, thereby realizing the feature aggregation of "sound-driven vision".

[0173] Decision and Output Module: Output Features of the Fusion Formula (9) After global average pooling and fully connected layers, the final noise class probability distribution P is output. f :

[0174] Formula (11);

[0175] in, For global average pooling, This is the classification weight matrix. For bias; c: index of the class or category-related parameter dimension;

[0176] To improve system robustness, this invention employs a weighted confidence fusion strategy to integrate acoustic classification results. With visual detection confidence Comprehensive calculation:

[0177] Formula (12);

[0178] in, Weighting coefficient; c i : The i-th noise category; P final ( ) indicates that the target belongs to the category. The final confidence probability;

[0179] The final output includes noise category, sound source object, and final confidence level P. final This enables real-time identification and visualization of environmental noise. The final confidence level P... final Based on confidence level get.

[0180] Acoustic classification results Including noise category and sound source object; P a It represents the probability of the acoustic classification result;

[0181] The system is powered by the Jetson Orin Nano Super platform.

[0182] like Figure 1 As shown, the present invention provides a noise recognition and detection system based on multimodal fusion, comprising four parts: an acoustic acquisition and analysis module, a visual acquisition and detection module, a multimodal feature fusion module, and a decision and output module. Figure 2 This refers to the model's network structure.

[0183] The specific steps of this invention are as follows:

[0184] S1: Acoustic Acquisition and Analysis Module, used to acquire ambient sound signals and extract acoustic features.

[0185] S1 specifically includes the following steps:

[0186] S1.1, Signal Acquisition: Ambient audio signals are synchronously acquired through a microphone array, and endpoint detection and bandpass filtering are performed on the input signals to remove silence and low-frequency interference components;

[0187] S1.2, Preprocessing: Noise reduction and signal amplitude standardization are performed using spectral subtraction and normalization techniques;

[0188] S1.3, Feature Conversion: Convert the time-domain signal into a Mel spectrogram, wherein the preferred settings are a window length of 25 ms, a frame shift of 10 ms, and a number of Mel filters of 64;

[0189] S1.4, Feature Extraction: Input to the lightweight acoustic event recognition network YAMNet, which is based on the MobileNetV1 architecture and can contain 13 depthwise separable convolutional layers and 3 pooling layers; each convolutional layer uses ReLU activation and batch normalization; finally, 1024-dimensional acoustic embedding features are obtained through global average pooling (GAP);

[0190] S1,5, Feature compression mapping: The feature is mapped to a 256-dimensional vector space via a fully connected layer (Dense(256)) to align with the visual features. The output 256-dimensional acoustic features are denoted as: .

[0191] S2, the visual acquisition and detection module, is used to identify potential sound source targets in the image and obtain visual semantic features; its process is as follows:

[0192] S2.1, Image Acquisition: Acquire video frame sequences through a camera;

[0193] S2.2, Target Detection: Input video frames into the YOLOv13n target detection network for target recognition; the network includes a Backbone, a Neck, and a Detection Head; the Backbone extracts features, the Neck performs feature fusion and enhancement, and the Detection Head generates the final detection result.

[0194] S2.3, Semantic Feature Extraction: Extract global visual embeddings from the intermediate feature map of the layer preceding the Detection Head. Obtain 256-dimensional visual features through 1×1 convolution and global average pooling. Visual feature tensor .

[0195] S3, the multimodal feature fusion module, is used to achieve temporal alignment and semantic fusion of acoustic and visual information, and mainly includes the following steps:

[0196] S3.1, Feature Input: Input the acoustic feature tensor As a query, visual feature tensor As Key and Value;

[0197] S3.2, Time Alignment: Synchronization matching based on frame index ensures the acoustic feature tensor With visual feature tensor The two modal features correspond to the same time window;

[0198] S3.3, Cross-Attention Fusion: Cross-Attention structure is used to achieve information exchange between modalities. Its core calculation is as follows:

[0199] Formula (8);

[0200] in, , These are the learnable mapping matrices.

[0201] S3.4, Fusion Output: After passing through self-attention and the feedforward network, the output is a fused feature:

[0202] Formula (9);

[0203] in For self-attention mechanism, It is a feedforward fully connected network.

[0204] S3.5, Semantic Compression: Mapped to a final fused semantic vector through a Dense(256) layer, used for noise category determination and sound source matching.

[0205] S4: The integrated decision and output module is used to calculate the noise category probability, sound source location, and confidence level. The steps are as follows:

[0206] S4.1, Acoustic Classification: Fusion Features Perform global average pooling and feed it into the classification head to obtain the predicted probabilities:

[0207] Formula (11);

[0208] in For global average pooling, This is the classification weight matrix. For bias;

[0209] S4.2, Visual Confidence Fusion: Jointly calculate the confidence scores from acoustic classification results and YOLO outputs.

[0210] Formula (12);

[0211] in The value is 0.7.

[0212] S4.3, Output Results: The output includes noise category, sound source object, overall confidence level, and visualization results.

[0213] The overall confidence level is a statistical indicator used to measure the reliability of estimation results. It is usually used in conjunction with the confidence interval and represents the probability that the population parameter falls within the confidence interval.

[0214] S5: System operation and deployment. Figure 3 This diagram illustrates the system, which deploys the model to the Jetson Orin NanoSuper platform, uses a microphone array to collect audio data, a camera to collect video data, and uses TensorRT to accelerate the model on the video stream.

[0215] The audio sampling rate is 48 kHz, and the video frame rate is 30 fps.

[0216] Experimental results show that, under typical urban construction and traffic conditions, the average recognition accuracy of the system of this invention is significantly improved compared to single-modal audio recognition, with an edge inference speed of approximately 80 fps, and it can output noise type and sound source visualization objects in real time.

[0217] The present invention has been described in detail for the purpose of making the disclosure clearer, and the prior art will not be listed in detail.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. It is obvious to those skilled in the art that multiple technical solutions of the present invention can be combined. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All technical contents not described in detail in the present invention are well-known technologies.

Claims

1. A noise identification and detection system based on multimodal fusion, characterized in that: The system includes an acoustic acquisition and analysis module, a visual acquisition and detection module, a multimodal feature fusion module, and a decision and output module; The acoustic acquisition and analysis module, including a microphone array, is responsible for acquiring ambient sound signals and extracting features. The visual acquisition and detection module includes a camera, which identifies potential sound source objects in the image based on a target detection algorithm; The multimodal feature fusion module connects the acoustic acquisition and analysis module and the visual acquisition and detection module, and is used to perform temporal alignment and semantic fusion of the acoustic modal features of the acoustic acquisition and analysis module and the visual modal features of the visual acquisition and detection module. The decision and output module connects to the multimodal feature fusion module and outputs the noise type, sound source location, and confidence level results.

2. The noise identification and detection system based on multimodal fusion according to claim 1, characterized in that: The acoustic acquisition and analysis module uses a microphone array to acquire ambient audio signals. After endpoint detection, noise reduction, and normalization, the ambient audio signal is... Converted to a time-domain signal, the time-domain signal is then subjected to a Short-Time Fourier Transform (STFT) to obtain its time spectrum: Formula (1); Will Mapping to the Mel scale yields the Mel spectrum: Formula (2); in, Window function Frame length; Discrete-time signals; Intra-frame sample coordinates; : Frame index, which is a time frame; Frequency index; Imaginary unit; M: Mel spectrum, i.e., Mel filter; m: the The index of a Mel filter, (1 ≤ m ≤ M); : The m-th Mel filter; After logarithmic transformation and normalization of formula (2), it is input into the lightweight convolutional neural network YAMNet to obtain the acoustic feature tensor F. a : Formula (3); Where 'a' stands for 'audio', and it serves as a subscript for the feature, indicating an acoustic feature. YAMNet consists of multiple convolutional layers, batch normalization, and pooling, and outputs embedded feature vectors through a final fully connected layer. ; in, : Time frame count; Acoustic feature dimensions; Meanwhile, based on formula (3), the preliminary acoustic classification results are as follows: Represented as: Formula (4); in, Global average pooling operation. and These are the classification layer weights and biases, respectively. Visual Acquisition and Detection Module: The visual acquisition module acquires video frame sequences through a synchronous camera. And input YOLOv13n to detect the network; The YOLO network structure is described as follows: Formula (5); F V : Visual feature tensor; v: video; in: The backbone network is used to extract multi-scale features. Neck network, used to achieve feature fusion; : Detection head, used to output the detection results of the target; The target detection output D is: Formula (6): Where, n: the number of targets detected by the visual detection module in a frame of video image; : No. The coordinates of each detection box; Category tags; : Detect confidence level, and simultaneously extract the feature tensor of the previous layer of the Head. .

3. The noise identification and detection system based on multimodal fusion according to claim 2, characterized in that: Multimodal feature fusion module: realizes temporal and semantic alignment of acoustic information from the acoustic acquisition and analysis module and visual information from the visual acquisition and detection module; The multimodal feature fusion module adopts a lightweight Transformer structure based on cross-attention; In the multimodal feature fusion module, First, the acoustic feature tensor With visual feature tensor Projected onto a space of the same dimension: Formula (7): in, Here, Q stands for Trainable Mapping Matrix; Q: Query, representing the semantic information that the current acoustic modality hopes to extract from the visual modality; K: Key, representing the key description of the visual features; V: Value, representing the content information in the visual features that can be used for fusion. Based on formula (7), Cross-Attention is calculated as follows: Formula (8): d K : The dimension of the key vector; K T : The transpose of Key; The output features after fusion are: Formula (9): in, f finally; A feedforward fully connected network, consisting of two linear layers and ReLU activation: Formula (10); x: Input feature vector; First layer weight matrix; First-level bias term; Second layer weight matrix; Second-level bias term; Decision and Output Module: Output Characteristics of the Fusion Formula (9) After global average pooling and fully connected layers, the final noise class probability distribution P is output. f : Formula (11); in, For global average pooling, This is the classification weight matrix. For bias; c: index of the class or category-related parameter dimension; A weighted confidence fusion strategy is adopted, based on the target detection output, to integrate the acoustic classification results. With visual detection confidence Comprehensive calculation: Formula (12); in, Weighting coefficient; c i : The i-th noise category; P final ( ) indicates that the target belongs to the category. The final confidence probability; The final output includes noise category, sound source object, and final confidence level P. final ; Acoustic classification results Including noise category and sound source object; P a It represents the probability of the acoustic classification result; The system is powered by the Jetson Orin Nano Super platform.

4. A noise identification and detection method based on multimodal fusion, characterized in that: Based on the system described in claim 1; Perform the following steps; S1: Acquire ambient sound signals and extract acoustic features through the acoustic acquisition and analysis module; S2, through the visual acquisition and detection module, identifies potential sound source targets in the image and obtains visual semantic features; S3, through a multimodal feature fusion module, achieves temporal alignment and semantic fusion of acoustic and visual information; S4: Through a comprehensive decision and output module, calculate the noise category probability, sound source location, and confidence level.

5. The noise identification and detection method based on multimodal fusion according to claim 4, characterized in that: S1 specifically includes the following steps: S1.1, Signal Acquisition: Acquire ambient audio signals synchronously through a microphone array, and perform endpoint detection and bandpass filtering on the input signals; S1.2, Preprocessing: Noise reduction and signal amplitude standardization are performed using spectral subtraction and normalization techniques; S1.3, Feature Transformation: Converting the time-domain signal into a Mel-spectrum. In step S1.3, the window length is set to 25 ms, the frame shift to 10 ms, and the number of Mel filters is set to 64. S1.4, Feature Extraction: Input to the lightweight acoustic event recognition network YAMNet; The YAMNet network is based on the MobileNetV1 architecture; In step S1.4, there are 13 depthwise separable convolutional layers and 3 pooling layers; each convolutional layer uses ReLU activation and batch normalization; finally, 1024-dimensional acoustic embedding features are obtained through global average pooling (GAP). S1,5, Feature compression mapping: The feature is mapped to a 256-dimensional vector space via a fully connected layer (Dense(256)) to align with the visual features in terms of dimension. The output is a 256-dimensional acoustic feature, denoted as: .

6. The noise identification and detection method based on multimodal fusion according to claim 5, characterized in that: In step S2, the process is as follows: S2.1, Image Acquisition: Acquire video frame feature sequences through a camera; S2.2, Object Detection: Input video frames into the YOLOv13n object detection network for object recognition; The network consists of a Backbone, a Neck, and a Detection Head. Features are extracted through the Backbone, fused and enhanced through the Neck, and the final detection result is generated through the Detection Head. S2.3, Semantic Feature Extraction: Extract global visual embeddings from the intermediate feature maps of the layer preceding the Detection Head. Obtain 256-dimensional visual features through 1×1 convolution and global average pooling, thus obtaining the visual feature tensor. .

7. The noise identification and detection method based on multimodal fusion according to claim 6, characterized in that: In step S3, the following steps are performed: S3.1, Feature Input: Input the acoustic feature tensor As a query, visual feature tensor As Key and Value; S3.2, Time Alignment: Synchronization matching based on frame index ensures the acoustic feature tensor With visual feature tensor The two modal features correspond to the same time window; S3.3, Cross-Attention Fusion: Cross-Attention structure is used to achieve information exchange between modalities. Its core calculation is as follows: Formula (8): in, , These are the learnable mapping matrices; S3.4, Fusion Output: After passing through self-attention and the feedforward network, the output is a fused feature: Formula (9): in For self-attention mechanism, It is a feedforward fully connected network; S3.5, Semantic Compression: Mapped to a final fused semantic vector through a Dense(256) layer, used for noise category determination and sound source matching.

8. The noise identification and detection method based on multimodal fusion according to claim 7, characterized in that: In step S4, the following steps are taken: S4.1, Acoustic Classification: Fusion Features Perform global average pooling and feed it into the classification head to obtain the predicted probabilities: Formula (11); in For global average pooling, This is the classification weight matrix. For bias; S4.2, Visual Confidence Fusion: Jointly calculate the confidence scores from acoustic classification results and YOLO outputs. Formula (12); in The value is 0.7; S4.3, Output Results: The output includes noise category, sound source object, overall confidence level, and visualization results.

9. The noise identification and detection method based on multimodal fusion according to claim 8, characterized in that: S5: Perform system operation and deployment; The system's model was deployed on the Jetson Orin Nano Super platform, using a microphone array to collect audio data and a camera to collect video data. TensorRT was used to accelerate the model on the video stream.

10. The noise identification and detection method based on multimodal fusion according to claim 9, characterized in that: The audio sampling rate is 48 kHz, and the video frame rate is 30 fps.

Images