Noise source detection classification and positioning method based on visual information under cross-domain perception

Abstract

The invention discloses a noise source detection classification and positioning method based on visual information under cross-domain perception, and the method comprises the steps: obtaining a video, carrying out the frame decomposition, and calculating a pixel difference value between two adjacent frames; calculating an inter-frame difference accumulated value or average value, and setting a threshold value of frame variation to judge whether the noise exceeds the standard or not; noise target recognition and classification are carried out by adopting a target detection and classification network, and the category and probability value of each moving target in each frame of image are acquired; calculating the sum of probability values of all noise categories appearing in all the image frames, wherein the category corresponding to the maximum sum of the probability values is a main noise source category; a target with a main noise source located at the center of an image is selected for positioning, and an image frame, a target bounding box and a category of the noise source target in the middle of the image, and coordinates, a distance, an azimuth angle and a pitch angle of the noise source relative to a camera are output. According to the method, the video frame variable quantity is used as the judgment basis of the noise level, and the comprehensive analysis of the noise source is realized by combining the target identification and positioning technology.

Description

Technical Field

[0001] The present invention belongs to the field of environmental noise monitoring, and specifically relates to a noise source detection, classification and positioning method based on visual information under cross-domain perception. Background Art

[0002] In the identification, classification and location of noise sources, the acoustic method is a common traditional method, which mainly relies on the feature extraction and analysis of sound wave signals. However, the acoustic method has significant shortcomings in the following aspects:

[0003] 1. Feature fuzziness: When the spectral features of different target noise sources are highly similar, it is difficult to accurately distinguish the target categories by relying solely on acoustic methods. For example, the engine sounds of large passenger vehicles and medium-to-large freight vehicles are similar, but these two types of vehicles belong to different types of noise sources.

[0004] 2. Significant environmental interference: Aliasing, multipath propagation and reflection between noise sources in complex environments will significantly reduce the recognition, classification and positioning accuracy of acoustic methods.

[0005] 3. Spatial positioning error: The positioning technology based on sound waves is highly dependent on the external environment. For example, environmental noise, sensor layout and spatial distribution of targets may affect the positioning results.

[0006] 4. Poor adaptability to dynamic scenes: Acoustic methods have low robustness in identifying and locating dynamic noise sources, especially when the target noise source moves or the ambient noise changes dynamically, it is difficult to ensure the accuracy of classification and positioning.

[0007] 5. In environmental noise monitoring applications, acoustic signals are more difficult to use as intuitive evidence than visual signals. Summary of the invention

[0008] In view of the above problems existing in the prior art, the present invention proposes a noise source detection, classification and positioning method based on visual information under cross-domain perception, which has a reasonable design, solves the shortcomings of the prior art and has good effects.

[0009] The technical solution of the present invention is as follows:

[0010] The noise source detection, classification and positioning method based on visual information under cross-domain perception includes the following steps:

[0011] S1, obtain the video and perform frame decomposition to obtain an image frame sequence;

[0012] S2, performing inter-frame difference on continuous image frames, and calculating the pixel difference between two adjacent frames;

[0013] S3, calculating the inter-frame difference accumulation value or the difference average value within a preset time window;

[0014] S4, setting a threshold value of the frame change amount within the time window to determine whether the noise exceeds the standard;

[0015] S5, preprocessing each frame of the image within the time window determined to have excessive noise, using a target detection and classification network to perform noise target recognition and classification, obtaining the category and probability value of the moving target in each frame of the image, and calculating the sum of the probability values ​​of each noise category appearing in all image frames within the time window, and the category corresponding to the maximum sum of probability values ​​is the category of the main noise source in the time window;

[0016] S6. Select the target whose main noise source is located at the center of the image within the time window as the object of noise source positioning, and finally output the image frame of the noise source target in the middle of the image, the target bounding box, the noise source category, the coordinates, distance, azimuth and pitch angle of the noise source relative to the camera.

[0017] Furthermore, in S1, a fixed camera is used to monitor the target area, a video stream is acquired in real time, a continuous video stream is decomposed into single-frame images, and an image frame sequence is acquired at a fixed frequency.

[0018] Further, in S2, it is assumed that I t (x, y) is the pixel value at (x, y) in the t-th frame image, I t-1 (x, y) is the pixel value at (x, y) in the t-1th frame image, and the inter-frame difference formula is:

[0019] D t (x,y)=|I t (x,y)-I t-1 (x,y)|;

[0020] Among them, D t (x, y) represents the absolute difference between the pixel position (x, y) in the t-th frame image and the t-1-th frame image;

[0021] The difference values ​​of all pixel positions in adjacent frame images are accumulated to obtain the total pixel difference S between the t-th frame image and the t-1-th frame image. t :

[0022]

[0023] Where W and H are the width and height of the image respectively.

[0024] Furthermore, in S3, a time window including N frames is set, and the inter-frame difference accumulation value or the difference average value within this time period is calculated, wherein the difference accumulation formula is:

[0025]

[0026] Among them, S total Represents the differential accumulation value from the t-N+1th frame to the tth frame;

[0027] The difference average formula is:

[0028]

[0029] Among them, S avg Represents the differential average from the t-N+1th frame to the tth frame.

[0030] Furthermore, in S4, a frame variation threshold T1 or T2 is set to determine whether the noise level exceeds the standard. total >T1 or S avg >T2, it means the noise level exceeds the standard, otherwise it means the noise level is normal;

[0031] The threshold value can be set as follows: a fixed value for the whole day or different time thresholds can be set in different time ranges according to the actual situation.

[0032] Furthermore, in S5, the preprocessing refers to marking the motion area according to the pixel difference between the input image and the previous frame image, and the expression is:

[0033] M t (x,y)=1,D t (x,y)>T motion ;

[0034] M t (x,y)=0,D t (x,y)≤T motion ;

[0035] Among them, M t (x, y) is the binary mask of the motion area, T motion is the differential value threshold of the moving target;

[0036] The generated binary mask image is morphologically processed to eliminate noise and connect the broken target areas. The expression is:

[0037] M′ t (x,y)=Morph(M t (x,y));

[0038] Where M′ t (x, y) refers to the motion mask after morphological processing optimization, Morph() refers to the morphological processing operation;

[0039] By M' t(x, y) extracts the moving area. When there are multiple moving targets, multiple moving targets are merged into one mask, and the stationary target and background pixels are shielded. The expression is:

[0040]

[0041] in, Refers to images containing only motion areas, I t (x,y) refers to the original input image;

[0042] Then will The size is adjusted to W*H*C for input into the target detection and classification network, where W and H are the width and height of the image respectively, and C is the number of channels.

[0043] Furthermore, the target detection and classification network includes an input layer, a feature extraction module, a multi-scale feature fusion module, a target detection and classification head, and an output layer;

[0044] The feature extraction module is used to extract multi-scale image features, which includes 4 convolution blocks, each of which includes a convolution layer, a batch normalization layer, a ReLU activation function layer, and a maximum pooling layer; the convolution kernel size of the convolution layer in convolution block 1 is 3×3, the number of output channels is 32, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 2 is 3×3, the number of output channels is 64, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 3 is 3×3, the number of output channels is 128, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 4 is 3×3, the number of output channels is 256, the step size is 1, and the padding size is 1; the pooling kernel size of the maximum pooling layer in the four convolution blocks is 2×2, and the step size is 2;

[0045] The multi-scale feature fusion module uses upsampling to improve the resolution of deep features, reduce the size difference with shallow features, and combine medium and deep features to enhance feature expression capabilities; deep features refer to the output of convolution block 4, medium and deep features refer to the output of convolution block 3, and shallow features include the output of convolution block 1 and convolution block 2; the upsampled deep features and medium and deep features are combined with shallow features through lateral connections to form a multi-scale fusion feature map for target detection and classification;

[0046] The detection head of the target detection and classification head performs bounding box regression on the feature map to predict the position and size of the target, and the classification head performs target classification on the feature map to predict the noise source category of the target;

[0047] The detection head includes a regression layer and a bounding box coordinate conversion module. The regression layer includes a convolution layer 5 and a linear activation function layer. The convolution kernel size of the convolution layer 5 is 3×3, and the number of output channels is 4, which correspond to the four parameters (x, y, w, h) of the bounding box respectively. The bounding box coordinate conversion module calculates the coordinates of the four vertices of the bounding box (x, y, w, h) according to the parameters output by the regression layer. min ,y min )、(x max ,y min )、(x min ,y max )、(x max ,y max ), where x and y are the center coordinates of the bounding box, w and h are the width and height, w=x max -x mon , h=y max -y min ;

[0048] The input of the classification head is the fused feature map, and the output is the probability P of each noise source category. c ,c=1,2,…,C,C is the total number of categories; the classification head includes convolution layer 6 and Softmax activation function layer, the convolution kernel size of convolution layer 6 is 3×3, and the number of output channels is C;

[0049] The output layer outputs the predicted bounding box parameters, the coordinates of the four vertices of the bounding box, and the probability P of each category c .

[0050] Furthermore, for the identified noise source categories, the cumulative probability value of consecutive frames of each category in the exceeding time period T is calculated, and the expression is:

[0051]

[0052] in, is the probability that the i-th moving target in the t-th frame image belongs to category c, N t is the number of targets detected in the tth frame;

[0053] The category with the highest cumulative probability value is selected as the main noise source category during the exceeding period.

[0054] Furthermore, in S6, an image frame with the highest cumulative probability value of the main noise source located at the center of the image is selected as the object of noise source positioning, and the known parameters include: the height and width of the target in the image are the height h and width w of the bounding box of the target in the frame image respectively; the equivalent focal length f of the camera, the size of the photosensitive chip and the resolution W×H of the image frame, and the size of the photosensitive chip includes the horizontal size w s and the vertical dimension hs , calculate the physical size of a single pixel as:

[0055]

[0056] Among them, p x is the physical horizontal size of a single pixel on the sensor, p y is the physical vertical size of a single pixel on the sensor;

[0057] According to the target category, the actual height H of the target is pre-set r , use similar triangles to calculate the distance D between the target and the camera along the line of sight:

[0058]

[0059] Among them, h p is the physical height of the target in the image, expressed as:

[0060] h p =h*p y ;

[0061] The vertical viewing angle of the camera is θ v and horizontal viewing angle θ h Calculated by the following formula:

[0062]

[0063] Calculate the position offset of the target in the image, the horizontal position offset Δx is:

[0064] Δx=(x min +x max ) / 2-W / 2;

[0065] The vertical position offset Δy is:

[0066] Δy=H / 2-(y min +y max ) / 2;

[0067] Calculate the angle offset, knowing that the horizontal viewing angle of the camera is θ h , then the angle corresponding to each pixel is The horizontal angle deviation for:

[0068]

[0069] The vertical angle offset θ is:

[0070] θ=Δy*α y ;

[0071] Among them, α yis the angular interval corresponding to each pixel in the image in the vertical direction, θ v is the vertical viewing angle of the camera;

[0072] Calculate the actual coordinates of the noise source relative to the camera: Given the camera installation position (X c ,Y c ,Z c ) and the camera's pitch angle θ c and azimuth Convert polar coordinates to Cartesian coordinates (X,Y,Z):

[0073]

[0074] Z=Z c +D*sin(θ c +θ);

[0075] The pitch angle θ of the noise source relative to the camera r and azimuth They are;

[0076] θ r =θ c +θ;

[0077]

[0078] The final output is an image frame with the noise source target in the middle of the image, the target bounding box, the noise source category, the coordinates, distance, azimuth and pitch angle of the noise source relative to the camera.

[0079] Beneficial technical effects brought by the present invention:

[0080] In view of the limitations of acoustic methods, the noise source identification, classification and positioning method based on visual information can effectively make up for the above shortcomings. Its main advantages include:

[0081] Rich feature dimensions: Visual methods can accurately distinguish different noise sources with similar spectral features by capturing multi-dimensional information such as the target's appearance, shape, and motion state.

[0082] Small impact of environmental interference: Visual methods rely on image and video data and are not affected by acoustic multipath propagation and reflection in the environment. They can ensure stable recognition, classification and positioning effects even in complex acoustic environments.

[0083] Accurate spatial positioning capability: The method proposed in the present invention can achieve high-precision spatial positioning of the noise source. In addition, the relevant images and videos can also be used as visual evidence in the process of noise law enforcement and monitoring.

[0084] Multi-target processing capability: Vision technology can detect and classify multiple target noise sources simultaneously without relying on complex separation algorithms, effectively improving system processing efficiency.

[0085] In summary, compared with traditional acoustic recognition, classification and positioning methods, visual methods can provide more comprehensive target features and higher environmental adaptability. In complex scenes where acoustic methods cannot accurately classify or locate, visual technology not only improves the classification accuracy of noise sources by analyzing the appearance and spatial information of the target, but also achieves accurate positioning of the target. Therefore, vision-based noise source recognition and classification technology shows significant technical advantages in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a flow chart of the noise source detection, classification and positioning method based on visual information under cross-domain perception in the present invention;

[0087] Figure 2 is a flow chart of the noise level determination process in the present invention;

[0088] Figure 3 This is a flow chart of the noise source identification and classification process in the present invention;

[0089] Figure 4 This is a flow chart of the noise source positioning process in the present invention; DETAILED DESCRIPTION

[0090] The specific implementation of the present invention is further described below in conjunction with specific embodiments:

[0091] Noise source detection, classification and positioning methods based on visual information under cross-domain perception, such as Figure 1 As shown, the following steps are included:

[0092] S1, obtain the video and perform frame decomposition to obtain an image frame sequence;

[0093] A fixed camera is used to monitor the target area, obtain video stream in real time, decompose the continuous video stream into single-frame images, and obtain image frame sequences at a fixed frequency.

[0094] For example, when detecting road traffic noise, traditional acoustic recognition has difficulty in accurately distinguishing different types of vehicles, especially in urban environments with complex noise backgrounds. However, the cross-domain use of visual information for traffic noise detection, identification, classification and positioning methods can accurately distinguish different types of vehicles (such as cars, buses, trucks, motorcycles, etc.). And by efficiently utilizing the existing camera resources of the urban environmental management / traffic management departments, noise detection, identification, classification and positioning can be achieved without the need to redeploy acoustic base stations. In addition, compared to acoustic data, videos or key images can be used as key evidence for urban noise environment monitoring and traffic management department law enforcement.

[0095] S2, performing inter-frame difference on continuous image frames, and calculating the pixel difference between two adjacent frames;

[0096] Only targets in motion are monitored to reduce false detections caused by stationary objects, such as parked vehicles and roadblocks when detecting traffic noise, and improve the accuracy of noise monitoring. False alarms caused by stationary or slow-moving objects are avoided to ensure the reliability of monitoring data.

[0097] Let I t (x, y) is the pixel value at (x, y) in the t-th frame image, I t-1 (x, y) is the pixel value at (x, y) in the t-1th frame image, and the inter-frame difference formula is:

[0098] D t (x,y)=|I t (x,y)-I t-1 (x,y)|;

[0099] Among them, D t (x, y) represents the absolute difference between the pixel position (x, y) in the t-th frame image and the t-1-th frame image;

[0100] The difference values ​​of all pixel positions in adjacent frame images are accumulated to obtain the total pixel difference S between the t-th frame image and the t-1-th frame image. t :

[0101]

[0102] Where W and H are the width and height of the image respectively.

[0103] S3, calculating the inter-frame difference accumulation value or the difference average value within a preset time window;

[0104] Set a time window containing N frames, and calculate the inter-frame difference accumulation value or difference average value within this period of time, where the difference accumulation formula is:

[0105]

[0106] Among them, S total Represents the differential accumulation value from the t-N+1th frame to the tth frame;

[0107] The difference average formula is:

[0108]

[0109] Among them, S avg Represents the differential average from the t-N+1th frame to the tth frame.

[0110] S4, such as Figure 2 As shown, a threshold of the frame variation within the time window is set to determine whether the noise exceeds the standard;

[0111] According to experiments, experience or feedback from the people around the monitoring area, set the threshold of frame change as the criterion for judging whether the noise exceeds the standard. If the frame change exceeds the threshold, it is judged that the noise may exceed the standard in this time period. If the frame change is lower than the threshold, it is judged that the noise level is normal. The threshold can be set as a fixed value for the whole day, or different time thresholds can be set in different time ranges according to actual conditions.

[0112] For example, since large changes in the video (such as large transport trucks passing quickly) are usually accompanied by high noise levels, the frame change amount can be used to estimate whether the noise exceeds the standard. For example, people are less sensitive to noise during the day than at night, so the daytime threshold can be set higher than the nighttime threshold.

[0113] Set the frame change threshold T1 or T2 to determine whether the noise level exceeds the standard. total >T1 or S avg >T2, it means the noise level exceeds the standard, otherwise it means the noise level is normal;

[0114] The threshold value can be set as follows: a fixed value for the whole day or different time thresholds can be set in different time ranges according to the actual situation.

[0115] S5, such as Figure 3 As shown, each frame of the image in the time window judged as exceeding the noise standard is preprocessed, and the target detection and classification network is used to identify and classify the noise target, obtain the category and probability value of the moving target in each frame of the image, and calculate the sum of the probability values ​​of each noise category appearing in all image frames in the time window. The category corresponding to the maximum sum of probability values ​​is the category of the main noise source in the time window;

[0116] Preprocessing refers to marking the motion area based on the pixel difference between the input image and the previous frame image. The expression is:

[0117] Mt (x,y)=1,D t (x,y)>T motion ;

[0118] M t (x,y)=0,D t (x,y)≤T motion ;

[0119] Among them, M t (x, y) is the binary mask of the motion area, T motion is the differential value threshold of the moving target;

[0120] The generated binary mask image is morphologically processed to eliminate noise and connect the broken target areas. The expression is:

[0121] M′ t (x,y)=Morph(M t (x,y));

[0122] Among them, M′ t (x, y) refers to the motion mask after morphological processing optimization, Morph() refers to the morphological processing operation;

[0123] By M' t (x, y) extracts the moving area. When there are multiple moving targets, they are merged into one mask, and the stationary targets and background pixels are shielded. The expression is:

[0124]

[0125] in, Refers to images containing only motion areas, I t (x,y) refers to the original input image;

[0126] Then will The size is adjusted to W*H*C for input into the target detection and classification network, where W and H are the width and height of the image respectively, and C is the number of channels.

[0127] The object detection and classification network includes an input layer, a feature extraction module, a multi-scale feature fusion module, an object detection and classification head, and an output layer;

[0128] The feature extraction module is used to extract multi-scale image features. It includes 4 convolution blocks, each of which includes a convolution layer, a batch normalization layer, a ReLU activation function layer, and a maximum pooling layer; the convolution kernel size of the convolution layer in convolution block 1 is 3×3, the number of output channels is 32, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 2 is 3×3, the number of output channels is 64, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 3 is 3×3, the number of output channels is 128, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 4 is 3×3, the number of output channels is 256, the step size is 1, and the padding size is 1; the pooling kernel size of the maximum pooling layer in the four convolution blocks is 2×2, and the step size is 2;

[0129] The multi-scale feature fusion module fuses features of different scales to meet the detection requirements of targets of different sizes. The feature pyramid structure (including upsampling in the top-down path and lateral connections) is used to enhance the detection capability of small targets. The multi-scale feature fusion module uses upsampling to increase the resolution of deep features, reduce the size difference with shallow features, and combine medium and deep features to enhance feature expression capabilities; deep features refer to the output of convolution block 4, medium and deep features refer to the output of convolution block 3, and shallow features include the output of convolution block 1 and convolution block 2; the upsampled deep features and medium and deep features are combined with shallow features through lateral connections to form a multi-scale fusion feature map for target detection and classification.

[0130] The detection head of the target detection and classification head performs bounding box regression on the feature map to predict the position and size of the target, and the classification head performs target classification on the feature map to predict the noise source category of the target;

[0131] The detection head includes a regression layer and a bounding box coordinate conversion module. The regression layer includes a convolution layer 5 and a linear activation function layer. The convolution kernel size of the convolution layer 5 is 3×3 and the number of output channels is 4, which correspond to the four parameters of the bounding box (x, y, w, h). The bounding box coordinate conversion module calculates the coordinates of the four vertices of the bounding box (x, y, w, h) according to the parameters output by the regression layer. min ,y min )、(x max ,y min )、(x min ,y max )、(x max ,y max ), where x and y are the center coordinates of the bounding box, w and h are the width and height, w=x max -x min , h=y max -y min ;;

[0132] The input of the classification head is the fused feature map, and the output is the probability P of each noise source category. c ,c=1,2,…,C,C is the total number of categories; the classification head includes convolution layer 6 and Softmax activation function layer, the convolution kernel size of convolution layer 6 is 3×3, and the number of output channels is C;

[0133] The output layer outputs the predicted bounding box parameters, the coordinates of the four vertices of the bounding box, and the probability P of each category. c .

[0134] For the identified noise source categories, the cumulative probability value of each category in the continuous frames within the exceeding time period T is calculated, and the expression is:

[0135]

[0136] in, is the probability that the i-th moving target in the t-th frame image belongs to category c, N t is the number of targets detected in the tth frame;

[0137] The category with the highest cumulative probability value is selected as the main noise source category during the exceeding period. For example, if the cumulative probability value of the "large vehicle" category is the highest, the noise source is determined to be a large vehicle in the traffic noise.

[0138] S6, such as Figure 4 As shown, the target with the main noise source located at the center of the image within the time window is selected as the object of noise source positioning, and the final output is the image frame of the noise source target in the middle of the image, the target bounding box, the noise source category, the coordinates, distance, azimuth and pitch angle of the noise source relative to the camera.

[0139] The image frame with the highest cumulative probability value of the main noise source located at the center of the image (or closest to the center) is selected as the object of noise source positioning, because the target located at the center of the image is usually close to the optical axis of the camera, and the positioning calculation error is small. Assume that the known parameters include: the height and width of the target in the image are the height h and width w of the bounding box of the target in the frame image; the equivalent focal length f of the camera, the size of the photosensitive chip, and the resolution W×H of the image frame. The size of the photosensitive chip includes the horizontal size w s and the vertical dimension h s , calculate the physical size of a single pixel as:

[0140]

[0141] Among them, p x is the physical horizontal size of a single pixel on the sensor, p y is the physical vertical size of a single pixel on the sensor;

[0142] According to the target category, the actual height H of the target is pre-set r , use similar triangles to calculate the distance D between the target and the camera along the line of sight:

[0143]

[0144] Among them, h p is the physical height of the target in the image, expressed as:

[0145] h p =h*p y ;

[0146] The vertical viewing angle of the camera is θ v and horizontal viewing angle θ h Calculated by the following formula:

[0147]

[0148] Calculate the position offset of the target in the image, the horizontal position offset Δx is:

[0149] Δx=(x min +x max ) / 2-W / 2;

[0150] The vertical position offset Δy is:

[0151] Δy=H / 2-(y min +y max ) / 2;

[0152] Calculate the angle offset, knowing that the horizontal viewing angle of the camera is θ h , then the angle corresponding to each pixel is The horizontal angle deviation for:

[0153]

[0154] The vertical angle offset θ is:

[0155] θ=Δy*α y ;

[0156] Among them, α y is the angular interval corresponding to each pixel in the image in the vertical direction, θ v is the vertical viewing angle of the camera;

[0157] Calculate the actual coordinates of the noise source relative to the camera: Given the camera installation position (X c ,Y c ,Z c ) and the camera's pitch angle θc and azimuth Convert polar coordinates to Cartesian coordinates (X,Y,Z):

[0158]

[0159] Z=Z c +D*sin(θ c +θ);

[0160] The pitch angle θ of the noise source relative to the camera r and azimuth They are;

[0161] θ r =θ c +θ;

[0162]

[0163] The final output is an image frame with the noise source target in the middle of the image, the target bounding box, the noise source category, the coordinates, distance, azimuth and pitch angle of the noise source relative to the camera.

[0164] The present invention innovatively uses the change in video frames as the basis for determining the noise level, and combines target recognition and positioning technology to achieve a comprehensive analysis of the noise source. The method has the advantages of low implementation cost, wide application range, strong real-time performance, etc., and has good application prospects.

[0165] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A noise source detection, classification and positioning method based on visual information under cross-domain perception, characterized in that: The following steps are involved: S1, obtain the video and perform frame decomposition to obtain an image frame sequence; S2, performing inter-frame difference on continuous image frames, and calculating the pixel difference between two adjacent frames; S3, calculating the inter-frame difference accumulation value or the difference average value within a preset time window; S4, setting a threshold value of the frame change amount within the time window to determine whether the noise exceeds the standard; S5, preprocessing each frame of the image within the time window determined to have excessive noise, using a target detection and classification network to perform noise target recognition and classification, obtaining the category and probability value of the moving target in each frame of the image, and calculating the sum of the probability values ​​of each noise category appearing in all image frames within the time window, and the category corresponding to the maximum sum of probability values ​​is the category of the main noise source in the time window; S6. Select the target whose main noise source is located at the center of the image within the time window as the object of noise source positioning, and finally output the image frame of the noise source target in the middle of the image, the target bounding box, the noise source category, the coordinates, distance, azimuth and pitch angle of the noise source relative to the camera.

2. According to claim 1, the noise source detection, classification and positioning method based on visual information under cross-domain perception is characterized in that: In S1, a fixed camera is used to monitor the target area, a video stream is acquired in real time, a continuous video stream is decomposed into single-frame images, and an image frame sequence is acquired at a fixed frequency.

3. The noise source detection, classification and positioning method based on visual information under cross-domain perception according to claim 2 is characterized in that: In S2, let I t (x, y) is the pixel value at (x, y) in the t-th frame image, I t-1 (x, y) is the pixel value at (x, y) in the t-1th frame image, and the frame difference formula is: D t (x,y)=|I t (x,y)-I t-1 (x,y)|; Among them, D t (x, y) represents the absolute difference between the pixel position (x, y) in the t-th frame image and the t-1-th frame image; The difference values ​​of all pixel positions in adjacent frame images are accumulated to obtain the total pixel difference S between the t-th frame image and the t-1-th frame image. t : Where W and H are the width and height of the image respectively.

4. The noise source detection, classification and positioning method based on visual information under cross-domain perception according to claim 3 is characterized in that: In S3, a time window including N frames is set, and the inter-frame difference accumulation value or the difference average value within this time period is calculated, wherein the difference accumulation formula is: Among them, S total Represents the differential accumulation value from the t-N+1th frame to the tth frame; The difference average formula is: Among them, S avg Represents the differential average from the t-N+1th frame to the tth frame.

5. According to claim 4, the noise source detection, classification and positioning method based on visual information under cross-domain perception is characterized in that: In S4, the frame change threshold T1 or T2 is set to determine whether the noise level exceeds the standard. total >T1 or S avg >T2, it means the noise level exceeds the standard, otherwise it means the noise level is normal; The threshold value can be set as follows: a fixed value for the whole day or different time thresholds can be set in different time ranges according to the actual situation.

6. The noise source detection, classification and positioning method based on visual information under cross-domain perception according to claim 5 is characterized in that: In S5, preprocessing refers to marking the motion area according to the pixel difference between the input image and the previous frame image, and the expression is: M t (x,y)=1,D t (x,y)>T motion ; M t (x,y)=0,D t (x,y)≤T motion ; Among them, M t (x, y) is the binary mask of the motion area, T motion is the differential value threshold of the moving target; The generated binary mask image is morphologically processed to eliminate noise and connect the broken target areas. The expression is: M′ t (x,y)=Morph(M t (x,y)); Where M′ t (x, y) refers to the motion mask after morphological processing optimization, Morph () refers to the morphological processing operation; By M' t (x, y) extracts the moving area. When there are multiple moving targets, they are merged into one mask, and the stationary target and background pixels are shielded. The expression is: in, Refers to images containing only motion areas, I t (x, y) refers to the original input image; Then will The size is adjusted to W*H*C for input into the target detection and classification network, where W and H are the width and height of the image respectively, and C is the number of channels.

7. The noise source detection, classification and positioning method based on visual information under cross-domain perception according to claim 6 is characterized in that: The target detection and classification network includes an input layer, a feature extraction module, a multi-scale feature fusion module, a target detection and classification head, and an output layer; The feature extraction module is used to extract multi-scale image features, which includes 4 convolution blocks, each of which includes a convolution layer, a batch normalization layer, a ReLU activation function layer, and a maximum pooling layer; the convolution kernel size of the convolution layer in convolution block 1 is 3×3, the number of output channels is 32, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 2 is 3×3, the number of output channels is 64, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 3 is 3×3, the number of output channels is 128, the step size is 1, and the padding size is 1; the convolution kernel size of the convolution layer in convolution block 4 is 3×3, the number of output channels is 256, the step size is 1, and the padding size is 1; the pooling kernel size of the maximum pooling layer in the four convolution blocks is 2×2, and the step size is 2; The multi-scale feature fusion module uses upsampling to improve the resolution of deep features, reduce the size difference with shallow features, and combine medium and deep features to enhance feature expression capabilities; deep features refer to the output of convolution block 4, medium and deep features refer to the output of convolution block 3, and shallow features include the output of convolution block 1 and convolution block 2; the upsampled deep features and medium and deep features are combined with shallow features through lateral connections to form a multi-scale fusion feature map for target detection and classification; The detection head of the target detection and classification head performs bounding box regression on the feature map to predict the position and size of the target, and the classification head performs target classification on the feature map to predict the noise source category of the target; The detection head includes a regression layer and a bounding box coordinate conversion module. The regression layer includes a convolution layer 5 and a linear activation function layer. The convolution kernel size of the convolution layer 5 is 3×3, and the number of output channels is 4, which correspond to the four parameters (x, y, w, h) of the bounding box respectively. The bounding box coordinate conversion module calculates the coordinates of the four vertices of the bounding box (x, y, w, h) according to the parameters output by the regression layer. min ,y min )、(x max ,y min )、(x min ,y max )、(x max ,y max ), where x and y are the center coordinates of the bounding box, w and h are the width and height, w=x max -x min , h=y max -y min ; The input of the classification head is the fused feature map, and the output is the probability P of each noise source category. c , c=1,2,...,C, C is the total number of categories; the classification head includes convolution layer 6 and Softmax activation function layer, the convolution kernel size of convolution layer 6 is 3×3, and the number of output channels is C; The output layer outputs the predicted bounding box parameters, the coordinates of the four vertices of the bounding box, and the probability P of each category c .

8. The noise source detection, classification and positioning method based on visual information under cross-domain perception according to claim 7 is characterized in that: For the identified noise source categories, the cumulative probability value of each category in the continuous frames within the exceeding time period T is calculated, and the expression is: in, is the probability that the i-th moving target in the t-th frame image belongs to category c, N t is the number of targets detected in the tth frame; The category with the highest cumulative probability value is selected as the main noise source category during the exceeding period.

9. The noise source detection, classification and positioning method based on visual information under cross-domain perception according to claim 8 is characterized in that: In S6, an image frame in which the main noise source with the highest cumulative probability value is located at the center of the image is selected as the object of noise source positioning, and the known parameters include: the height and width of the target in the image are the height h and width w of the bounding box of the target in the frame image respectively; the equivalent focal length f of the camera, the size of the photosensitive chip and the resolution W×H of the image frame, and the size of the photosensitive chip includes the horizontal size w s and the vertical dimension h s , calculate the physical size of a single pixel as: Among them, p x is the physical horizontal size of a single pixel on the sensor, p y is the physical vertical size of a single pixel on the sensor; According to the target category, the actual height H of the target is pre-set r , use similar triangles to calculate the distance D between the target and the camera along the line of sight: Among them, h p is the physical height of the target in the image, expressed as: h p =h*p y ; The vertical viewing angle of the camera is θ v and horizontal viewing angle θ h Calculated by the following formula: Calculate the position offset of the target in the image, the horizontal position offset Δx is: Δx=(x min +x max ) / 2-W / 2; The vertical position offset Δy is: Δy=H / 2-(y min +y max ) / 2; Calculate the angle offset, knowing that the horizontal viewing angle of the camera is θ h , then the angle corresponding to each pixel is The horizontal angle deviation for: The vertical angle offset θ is: θ= Δ y*αy; Among them, α y is the angular interval corresponding to each pixel in the image in the vertical direction, θ v is the vertical viewing angle of the camera; Calculate the actual coordinates of the noise source relative to the camera: Given the camera installation position (X c , Y c , Z c ) and the camera's pitch angle θ c and azimuth Convert polar coordinates to Cartesian coordinates (X, Y, Z): Z=Z c +D*sin(θ c +θ); The pitch angle θ of the noise source relative to the camera r and azimuth They are; i r =θ c +θ; The final output is an image frame with the noise source target in the middle of the image, the target bounding box, the noise source category, the coordinates, distance, azimuth and pitch angle of the noise source relative to the camera.

Images