Urban environment noise source positioning, identification and classification method based on visual and acoustic information fusion

Abstract

The invention discloses an urban environment noise source positioning, identification and classification method based on visual and acoustic information fusion. The method comprises the following steps: positioning a coordinate, an azimuth angle, a pitch angle and a distance of a noise source relative to an array center by using an acoustic array; adjusting the angle of the camera to align to the noise source position for video shooting; noise targets are classified by using a noise source target identification method based on fusion of acoustic data and video data, and the types of noise sources are determined; positioning the coordinate, azimuth angle, pitch angle and distance of the noise source relative to the camera by combining the visual data with the noise source classification result, and adjusting the result into the coordinate, azimuth angle, pitch angle and distance relative to the array center; and fusing the two groups of coordinates, azimuth angles, pitch angles and distances relative to the center of the array, and calculating a final result. Not only can the noise source target position be accurately positioned, but also the noise source target type can be accurately identified and classified.

Description

Technical Field

[0001] This invention belongs to the field of environmental perception and intelligent monitoring technology, specifically relating to a method for locating, identifying and classifying urban environmental noise sources based on the fusion of visual and acoustic information. Background Technology

[0002] Noise pollution is an important monitoring indicator in urban environmental monitoring. Traditional methods for noise source localization and identification classification usually rely on single acoustic data, which cannot comprehensively and accurately locate and identify noise sources, and suffer from problems such as low accuracy, poor real-time performance, and poor environmental adaptability.

[0003] In the field of urban environmental monitoring and management, noise pollution has become a key factor affecting residents' quality of life and the healthy development of cities. Accurate location, identification, and classification of noise sources are prerequisites for effective noise control.

[0004] Currently, mainstream noise monitoring technologies primarily rely on acoustic sensor arrays. By analyzing acoustic signals collected by microphone arrays, algorithms such as beamforming and time difference of arrival (TDOA) are used to spatially locate noise sources. However, these purely acoustic methods have inherent limitations: First, their positioning accuracy is easily affected by environmental reflections, background noise, and weather conditions; in complex urban reverberation environments, the positioning results often have significant deviations. Second, acoustic methods struggle to identify the physical properties of noise sources, failing to distinguish whether the noise originates from vehicles, construction machinery, or crowds, thus limiting the decision support value of the monitoring data.

[0005] On the other hand, while computer vision-based target detection and recognition technologies can effectively identify and locate specific targets in visual scenes, they lack the ability to "hear" and cannot accurately associate sound with the target source. Furthermore, visual technologies are greatly affected by conditions such as lighting, weather, and occlusion.

[0006] Therefore, in existing technologies, acoustic localization and visual recognition are often independent systems with limited functions and fragmented information. Simply combining the two technologies cannot automatically achieve accurate association and identification of noise sources. How to deeply integrate the wide-area, non-line-of-sight detection capabilities of acoustic information with the precise and intuitive recognition capabilities of visual information to establish an integrated system that can work collaboratively and complement each other, thereby achieving the leap from "perceiving noise" to "accurately identifying and locating noise sources," has become a pressing technical challenge in this field.

[0007] Therefore, this invention proposes a fusion method combining visual and acoustic information, aiming to improve the accuracy and reliability of noise source localization and identification. Summary of the Invention

[0008] To address the aforementioned problems in existing technologies, this invention proposes a method for locating, identifying, and classifying urban environmental noise sources based on the fusion of visual and acoustic information. The method is rationally designed, overcomes the shortcomings of existing technologies, and has good results.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A method for locating, identifying, and classifying urban environmental noise sources based on the fusion of visual and acoustic information includes the following steps: Step 1: When the noise intensity is detected to exceed the threshold within a specific time period, the acoustic array is used to locate the coordinates, azimuth, elevation angle and distance of the noise source relative to the array center; Step 2: Based on the acoustic positioning coordinates and its own coordinates, the camera adjusts its angle to aim at the noise source and capture video. Step 3: Classify noise targets and determine the types of noise sources using a noise source target identification method that fuses acoustic data and video data; Step 4: Using visual data combined with noise source classification results, locate the coordinates, azimuth, pitch and distance of the noise source relative to the camera, and adjust the results to the coordinates, azimuth, pitch and distance relative to the center of the acoustic array; Step 5: Combine the two sets of coordinates, azimuth, elevation, and distance relative to the center of the acoustic array to calculate the final coordinates, azimuth, elevation, and distance of the noise source relative to the center of the acoustic array; Step 6: Output the final classification and localization results.

[0010] Further, step 1 specifically includes: An acoustic array is constructed, which is a regular icosahedron structure, and 12 microphones are respectively placed at the 12 vertices of the regular icosahedron to collect sound signals from noise sources. In the presence of multiple noise sources, the noise source with the highest sound pressure level is determined as the main noise source based on the sound signals collected by the acoustic array. Let the coordinates of the main noise source be... The coordinates of the center position of the acoustic array are , No. The location of each microphone is The speed of sound in air is The arrival time difference is ,in , These are the sound source to the... The and the first The distance between the microphones is expressed as: ; ; A system of equations was established using the time difference of arrival to solve for the position of the main noise source relative to the center of the acoustic array. ; The distance from the main noise source to the center of the acoustic array, determined based on acoustic data, is... azimuth Pitch angle .

[0011] Furthermore, step 2 specifically includes: Let the camera coordinates be... Because of the height difference between the camera and the center of the acoustic array, the camera rotation method needs to be compensated. First, calculate the coordinates of the main noise source relative to the camera. , , , ; Compensated azimuth Compensated pitch angle ; Rotate the camera to , The direction is to capture images of the area where the main noise source is located, and to obtain video frame data.

[0012] Furthermore, in step 3, a noise target classification model is constructed, including an input layer, an acoustic data feature extraction layer, a video frame preprocessing layer, a first optical flow feature extraction layer, an image feature extraction layer, a second optical flow feature extraction layer, a feature fusion layer, a temporal modeling layer, a fully connected layer, and an output layer. The input layer is used to input acoustic data and video frame data. The acoustic data is collected by 12 microphones distributed at the vertices of the icosahedral structure. The signal-to-noise ratio (SNR) of each of the 12 audio signals is measured, and the audio signal with the highest SNR is selected as the acoustic input signal. The acoustic data size is [size missing]. ,in The number of sampling points for the audio signal is [number], and the video frame data size is [size]. ,in For time frames, Image height, Image width, Number of channels; The acoustic data feature extraction layer performs multi-layer convolution and pooling operations on the acoustic data to extract its time-frequency features. The convolutional layers include several convolutional kernels, each 3*3 in size, and employ a non-linear activation function. The residual block consists of several convolutional and pooling layers to enhance the stability and robustness of feature extraction. The acoustic data feature extraction layer sequentially includes convolutional layer 1, pooling layer 1, convolutional layer 2, pooling layer 2, convolutional layer 3, pooling layer 3, residual block 1, and residual block 2. Convolutional layers 1, 2, and 3 are all 1D convolutional layers with a kernel size of 3, and the activation function is... All functions are ReLU. Convolutional layer 1 includes 64 filters, convolutional layer 2 includes 128 filters, and convolutional layer 3 includes 256 filters. Pooling layers 1, 2, and 3 are all max pooling layers with a pooling size of 2. Residual block 1 includes convolutional layers 4 and 5, both 1D convolutional layers with 512 filters each, a kernel size of 3, and ReLU activation function. Residual block 2 includes convolutional layers 6 and 7, both 1D convolutional layers with 1024 filters each, a kernel size of 3, and ReLU activation function. The video frame preprocessing layer is used to normalize each input frame image and convert the RGB image into a grayscale image. The first optical flow feature extraction layer is used to calculate the optical flow information between adjacent frames. The Farneback method is used to calculate the inter-frame optical flow field and convert it into an optical flow image. The image feature extraction layer is used to extract the static features of the video frame images. A pre-trained convolutional neural network model is used to extract features from each frame image and output the image features through a fully connected layer. The second optical flow feature extraction layer is used to extract features from the optical flow image. It uses the same convolutional neural network model as the image feature extraction layer to obtain optical flow image features. The feature fusion layer is used to fuse acoustic features, image features and optical flow image features, and a multi-head attention mechanism is used to further enhance the interaction between features of different modalities. The temporal modeling layer employs a recurrent neural network structure or a network structure based on a self-attention mechanism to perform time-series modeling of fused features, thereby capturing the temporal dependencies in audio and video. The recurrent neural network structure uses a multi-layer bidirectional LSTM layer, including three layers of bidirectional LSTM, with the first layer containing 256 units, the second layer containing 512 units, and the third layer containing 1024 units. The Transformer model includes multiple attention heads and six encoding layers. The fully connected layer is used to receive the features output by the temporal modeling layer and perform further feature learning and classification on the features; the fully connected layer includes a four-layer fully connected layer structure, with a Dropout layer set after each fully connected layer. The output layer is the last fully connected layer, and the number of its output neurons is equal to the number of classification categories. The activation function is Softmax.

[0013] Furthermore, in step 4, the actual size of the target is defined as height based on the classification results of the main noise sources. Hekuan Let the image height of the target in the video frame be... and width The position coordinates in the image are The focal length in the camera parameters is The image sensor size is width and height The camera coordinates are ; First, calculate the actual size of the object in the camera's image sensor, and then convert the pixel dimensions in the video frame to the actual size on the image sensor: ; ; in, and It is the resolution of the video frame; The distance between the target and the camera is estimated using both height and width. and and take the average value. The expression is: ; ; ; Based on the target's position in the video frame The coordinates of the target relative to the camera are calculated as follows: ,in , , ; The target's coordinates relative to the center of the acoustic array are ,in ; Distance from the noise source target to the center of the acoustic array, calculated based on visual data. azimuth Pitch angle .

[0014] Furthermore, in step 5, the coordinates of the two noise sources relative to the center of the acoustic array, obtained based on acoustic and visual localization, are weighted and averaged to obtain the final coordinates of the noise sources relative to the center of the acoustic array. Azimuth Pitch angle and distance ; Let the weights be determined based on the confidence levels of acoustic localization and visual localization, respectively. and ,in: ; ; ; ; ; .

[0015] The beneficial technical effects of this invention are as follows: This invention not only accurately locates the position of noise sources, but also accurately identifies and classifies the types of noise sources. Furthermore, it can fully utilize acoustic and visual data, combined with different practical situations and hardware configurations, to flexibly realize the functions of noise source localization and classification. Attached Figure Description

[0016] Figure 1 A flowchart of a method for locating, identifying, and classifying urban environmental noise sources based on the fusion of visual and acoustic information; Detailed Implementation

[0017] The specific embodiments of the present invention will be further described below with reference to specific examples: Urban environmental noise source localization, identification, and classification methods based on the fusion of visual and acoustic information, such as Figure 1 As shown, it includes the following steps: Step 1: When the noise intensity is detected to exceed a specific threshold within a specific time period, the acoustic array is used to locate the coordinates, azimuth, elevation angle and distance of the noise source relative to the array center; Step 1 is as follows: This embodiment is for monitoring noise around residential areas. First, based on feedback from residents around the residential area, specific values ​​of noise intensity are set according to different time periods. An acoustic array is constructed, which is a regular icosahedron structure. Twelve microphones are respectively placed at the twelve vertices of the regular icosahedron to collect sound signals from noise sources. In the case of multiple noise sources, the noise source with the highest sound pressure level is determined as the main noise source based on the sound signals collected by the acoustic array. A system of equations is established using the Time Difference of Arrival (TDOA) to solve for the location of the main noise source. Let the coordinates of the main noise source location be... The coordinates of the center position of the acoustic array are , No. The location of each microphone is Then the sound source reaches the first The distance between the microphones is: ; No. The signal received by the microphone for: ; in, The attenuation coefficient is... , The distance between the sound source signal and the microphone; The propagation delay of the source signal After arriving at the The signal from one microphone, , For the speed of sound, For the first Noise from one microphone; First, select a microphone as the reference microphone, and denote the reference microphone as the [number]. The microphone, calculate the... Cross-correlation function of the signals received by each microphone and the reference microphone for: ; The time delay corresponding to the maximum value of the cross-correlation function is the relative time difference. The expression is: ; Construct a time difference matrix, where each microphone forms a microphone pair with a reference microphone, and collect the time differences of all microphone pairs to form the time difference matrix. ; Time difference and distance difference The relationship between them is: ; For each microphone pair, the following equation is established: ; in, Construct a system of nonlinear equations; Finally, a linearization method was used for position estimation. For the far-field approximation, it was assumed that the distance to the sound source was much larger than the diameter of the spherical acoustic array structure. The nonlinear equations were linearized, and the least squares method was used to solve for the position of the main noise source relative to the center of the acoustic array. ; The distance from the main noise source to the center of the acoustic array, determined based on acoustic data, is... azimuth Pitch angle .

[0018] This step determines the position coordinates of each microphone, collects sound signals from each microphone, and calculates the time difference of arrival between each pair of microphones. The sound source location is then determined using nonlinear least squares or other optimization methods, and the final result is calculated using the formula described above.

[0019] Step 2: Based on the acoustic positioning of the noise source target coordinates and its own coordinates, the camera adjusts its angle to aim at the noise source location and captures video. Step 2 is as follows: In this embodiment, a noise monitoring station is set up near the residential area, and a camera is installed on the station.

[0020] Let the camera coordinates be... Because of the height difference between the camera and the center of the acoustic array, the camera rotation method needs to be compensated. First, calculate the coordinates of the main noise source relative to the camera. , , , ; Compensated azimuth Compensated pitch angle ; Rotate the camera to , The direction is to capture images of the area where the main noise source is located, and to obtain video frame data.

[0021] Step 3: Classify noise targets and determine the types of noise sources by using a noise source target identification method that fuses acoustic data and video frame data; Step 3 specifically involves: A noise target classification model is constructed, including an input layer, an acoustic data feature extraction layer, a video frame preprocessing layer, a first optical flow feature extraction layer, an image feature extraction layer, a second optical flow feature extraction layer, a feature fusion layer, a temporal modeling layer, a fully connected layer, and an output layer. The input layer is used to input acoustic data and video frame data. The acoustic data is collected by 12 microphones distributed at the vertices of an icosahedral structure. The signal-to-noise ratio (SNR) of each of the 12 audio signals is measured, and the audio signal with the highest SNR is selected as the acoustic input signal. The acoustic data size is [size missing]. ,in The number of sampling points for the audio signal is [number], and the video frame data size is [size]. ,in For time frames, Image height, Image width, Number of channels (RGB three channels); The acoustic data feature extraction layer performs multi-layer convolution and pooling operations on the acoustic data to extract its time-frequency features. The convolutional layers consist of several 3x3 convolutional kernels and employ non-linear activation functions (such as ReLU). The residual blocks are composed of several convolutional and pooling layers to enhance the stability and robustness of feature extraction. The acoustic data feature extraction layer sequentially includes convolutional layer 1, pooling layer 1, convolutional layer 2, pooling layer 2, convolutional layer 3, pooling layer 3, residual block 1, and residual block 2. Convolutional layers 1, 2, and 3 are all 1D convolutional layers with a kernel size of 3 and a ReLU activation function. Convolutional layer 1 contains 64 filters, convolutional layer 2 contains 128 filters, and convolutional layer 3 contains 256 filters. Pooling layers 1, 2, and 3 are all max pooling layers with a pooling size of 2. Residual block 1 includes convolutional layer 4 and convolutional layer 5, both of which are 1D convolutional layers, each containing 512 filters, with a kernel size of 3 and an activation function of ReLU. Residual block 2 includes convolutional layer 6 and convolutional layer 7, both of which are 1D convolutional layers, each containing 1024 filters, with a kernel size of 3 and an activation function of ReLU. The video frame preprocessing layer is used to normalize each frame of the input image and convert the RGB image to a grayscale image; The first optical flow feature extraction layer is used to calculate the optical flow information between adjacent frames. The Farneback method is used to calculate the inter-frame optical flow field and convert it into an optical flow image. The image feature extraction layer is used to extract static features of video frame images. It uses a pre-trained convolutional neural network model (such as EfficientNet) to extract features from each frame image and outputs image features through a fully connected layer. The second optical flow feature extraction layer is used to extract features from the optical flow image. It uses the same convolutional neural network model as the image feature extraction layer to obtain optical flow image features. The feature fusion layer is used to fuse acoustic features, image features and optical flow image features, and a multi-head attention mechanism is used to further enhance the interaction between features of different modalities; The temporal modeling layer employs a recurrent neural network structure (such as an LSTM layer) or a network structure based on a self-attention mechanism (such as a Transformer model) to perform time-series modeling of fused features, thereby capturing the temporal dependencies in audio and video. The recurrent neural network structure uses a multi-layer bidirectional LSTM layer, including three bidirectional LSTM layers: the first layer contains 256 units, the second layer contains 512 units, and the third layer contains 1024 units. The Transformer model includes multiple attention heads and six encoding layers. The fully connected layer receives the features output by the temporal modeling layer and performs further feature learning and classification on these features. The fully connected layer consists of four fully connected layers, with a Dropout layer after each fully connected layer, and the Dropout ratio is 0.5. The activation function of the fully connected layer is ReLU. The number of neurons in the four fully connected layers are 2048, 1024, 512, and 256, respectively. The output layer is the last fully connected layer, with the number of output neurons equal to the number of classification categories, and the activation function is Softmax.

[0022] Model training includes: 1) acquiring environmental noise data and video frame data and their corresponding labels, and constructing training sets, validation sets and test sets respectively; 2) using them for model training, observing changes in model performance during training, and testing model performance.

[0023] In this embodiment, the constructed noise target classification model is used to identify and classify urban noise sources. The main noise source categories include cars, vans, trucks, motorcycles, buses, pedestrians, and loudspeakers. Step 4: Using visual data combined with noise source classification results, locate the coordinates, azimuth, pitch and distance of the noise source relative to the camera, and adjust the results to the coordinates, azimuth, pitch and distance relative to the center of the acoustic array; Step 4 is as follows: Based on the classification results of the main noise sources, the actual size of the target is defined as height. Hekuan Let the image height of the target in the video frame be... and width The position coordinates in the image are The focal length in the camera parameters is The image sensor size is width and height The camera coordinates are ; First, calculate the actual size of the object in the camera's image sensor, and then convert the pixel dimensions in the video frame to the actual size on the image sensor: ; ; in, and It is the resolution (number of pixels) of the video frame. The distance between the target and the camera is estimated using both height and width. and To calculate the distance more accurately, the average value is taken. The expression is: ; ; ; Based on the target's position in the video frame The coordinates of the target relative to the camera are calculated as follows: ,in , , ; The target's coordinates relative to the center of the acoustic array are ,in ; Distance from the noise source target to the center of the acoustic array, calculated based on visual data. azimuth Pitch angle .

[0024] Step 5: Combine the two sets of coordinates, azimuth, elevation, and distance relative to the center of the acoustic array to calculate the final coordinates, azimuth, elevation, and distance of the noise source relative to the center of the acoustic array.

[0025] Step 5 specifically involves: Based on the coordinates, azimuth, elevation, and distance of the two noise sources relative to the center of the acoustic array obtained from acoustic and visual localization, a weighted average is used to obtain the final coordinates of the noise sources relative to the center of the acoustic array. Azimuth Pitch angle and distance ; Let the weights be determined based on the confidence levels of acoustic localization and visual localization, respectively. and The final coordinates, azimuth, elevation, and distance of the noise source relative to the center point of the acoustic array are as follows: ; ; ; ; ; .

[0026] Step 6: Output the final classification and localization results.

[0027] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for locating, identifying, and classifying urban environmental noise sources based on the fusion of visual and acoustic information, characterized in that, Includes the following steps: Step 1: When the noise intensity is detected to exceed the threshold within a specific time period, the acoustic array is used to locate the coordinates, azimuth, elevation angle and distance of the noise source relative to the array center; Step 2: Based on the acoustic positioning coordinates and its own coordinates, the camera adjusts its angle to aim at the noise source and capture video. Step 3: Classify noise targets and determine the types of noise sources using a noise source target identification method that fuses acoustic data and video data; Step 4: Using visual data combined with noise source classification results, locate the coordinates, azimuth, pitch and distance of the noise source relative to the camera, and adjust the results to the coordinates, azimuth, pitch and distance relative to the center of the acoustic array; Step 5: Combine the two sets of coordinates, azimuth, elevation, and distance relative to the center of the acoustic array to calculate the final coordinates, azimuth, elevation, and distance of the noise source relative to the center of the acoustic array; Step 6: Output the final classification and localization results.

2. The method for locating and classifying urban environmental noise sources based on the fusion of visual and acoustic information according to claim 1, characterized in that, Step 1 specifically involves: An acoustic array is constructed, which is a regular icosahedron structure, and 12 microphones are respectively placed at the 12 vertices of the regular icosahedron to collect sound signals from noise sources. In the presence of multiple noise sources, the noise source with the highest sound pressure level is determined as the main noise source based on the sound signals collected by the acoustic array. Let the coordinates of the main noise source be... The coordinates of the center position of the acoustic array are , No. The location of each microphone is The speed of sound in air is The time difference of arrival is ,in , These are the sound source to the first The and the first The distance between the microphones is expressed as: ； ； A system of equations was established using the time difference of arrival to solve for the position of the main noise source relative to the center of the acoustic array. ; The distance from the main noise source to the center of the acoustic array, determined based on acoustic data, is... azimuth Pitch angle .

3. The method for locating and classifying urban environmental noise sources based on the fusion of visual and acoustic information according to claim 2, characterized in that, Step 2 specifically involves: Let the camera coordinates be... Because of the height difference between the camera and the center of the acoustic array, the camera rotation method needs to be compensated. First, calculate the coordinates of the main noise source relative to the camera. , , , ; Compensated azimuth Compensated pitch angle ; Rotate the camera to , The direction is to capture images of the area where the main noise source is located, and to obtain video frame data.

4. The method for locating, identifying, and classifying urban environmental noise sources based on the fusion of visual and acoustic information according to claim 3, characterized in that, In step 3, a noise target classification model is constructed, including an input layer, an acoustic data feature extraction layer, a video frame preprocessing layer, a first optical flow feature extraction layer, an image feature extraction layer, a second optical flow feature extraction layer, a feature fusion layer, a temporal modeling layer, a fully connected layer, and an output layer. The input layer is used to input acoustic data and video frame data. The acoustic data is collected by 12 microphones distributed at the vertices of the icosahedral structure. The signal-to-noise ratio (SNR) of each of the 12 audio signals is measured, and the audio signal with the highest SNR is selected as the acoustic input signal. The acoustic data size is [size missing]. ,in The number of sampling points for the audio signal is [number], and the video frame data size is [size]. ,in For time frames, Image height, Image width, Number of channels; The acoustic data feature extraction layer is used to perform multi-layer convolution and pooling operations on the acoustic data to extract its time-frequency features. The convolutional layer consists of several convolutional kernels, each 3x3 in size, and employs a non-linear activation function. The residual block comprises several convolutional and pooling layers to enhance the stability and robustness of feature extraction. The acoustic data feature extraction layer sequentially includes convolutional layer 1, pooling layer 1, convolutional layer 2, pooling layer 2, convolutional layer 3, pooling layer 3, residual block 1, and residual block 2. Convolutional layers 1, 2, and 3 are all 1D convolutional layers with a kernel size of 3 and ReLU activation function. Convolutional layer 1 includes 64 filters. The filter consists of two layers: convolutional layer 2 with 128 filters and convolutional layer 3 with 256 filters; pooling layers 1, 2, and 3 are all max-pooling layers with a pooling size of 2; residual block 1 includes convolutional layers 4 and 5, both 1D convolutional layers with 512 filters, a kernel size of 3, and ReLU activation function; residual block 2 includes convolutional layers 6 and 7, both 1D convolutional layers with 1024 filters, a kernel size of 3, and ReLU activation function. The video frame preprocessing layer is used to normalize each input frame image and convert the RGB image into a grayscale image. The first optical flow feature extraction layer is used to calculate the optical flow information between adjacent frames. The Farneback method is used to calculate the inter-frame optical flow field and convert it into an optical flow image. The image feature extraction layer is used to extract the static features of the video frame images. A pre-trained convolutional neural network model is used to extract features from each frame image and output the image features through a fully connected layer. The second optical flow feature extraction layer is used to extract features from the optical flow image. It uses the same convolutional neural network model as the image feature extraction layer to obtain optical flow image features. The feature fusion layer is used to fuse acoustic features, image features and optical flow image features, and a multi-head attention mechanism is used to further enhance the interaction between features of different modalities. The temporal modeling layer employs a recurrent neural network structure or a network structure based on a self-attention mechanism to perform time-series modeling of fused features, thereby capturing the temporal dependencies in audio and video. The recurrent neural network structure employs a multi-layer bidirectional LSTM structure, including three bidirectional LSTM layers. The first layer contains 256 units, the second layer contains 512 units, and the third layer contains 1024 units. The Transformer model includes multiple attention heads and six encoding layers. The fully connected layer is used to receive the features output by the temporal modeling layer and perform further feature learning and classification on the features; the fully connected layer includes a four-layer fully connected layer structure, with a Dropout layer set after each fully connected layer. The output layer is the last fully connected layer, and the number of its output neurons is equal to the number of classification categories. The activation function is Softmax.

5. The method for locating and classifying urban environmental noise sources based on the fusion of visual and acoustic information according to claim 4, characterized in that, In step 4, the actual size of the target is defined as height based on the classification results of the main noise sources. Hekuan Let the image height of the target in the video frame be... and width The position coordinates in the image are The focal length in the camera parameters is The image sensor size is width and height The camera coordinates are ; First, calculate the actual size of the object in the camera's image sensor, and then convert the pixel dimensions in the video frame to the actual size on the image sensor: ； ； in, and It is the resolution of the video frame; The distance between the target and the camera is estimated using both height and width. and and take the average value. The expression is: ； ； ； Based on the target's position in the video frame The coordinates of the target relative to the camera are calculated as follows: ,in , , ; The target's coordinates relative to the center of the acoustic array are ,in ; Distance from the noise source target to the center of the acoustic array, calculated based on visual data. azimuth Pitch angle .

6. The method for locating and classifying urban environmental noise sources based on the fusion of visual and acoustic information according to claim 5, characterized in that, In step 5, the coordinates of the two noise sources relative to the center of the acoustic array, obtained from acoustic and visual localization, are weighted and averaged to obtain the final coordinates of the noise sources relative to the center of the acoustic array. Azimuth Pitch angle and distance ; Let the weights be determined based on the confidence levels of acoustic localization and visual localization, respectively. and ,in: ； ； ； ； ； 。

Images