A multi-modal image change detection and prior generation method for city inspection

Abstract

This invention relates to the fields of computer vision and artificial intelligence, and discloses a multimodal image change detection and prior generation method for urban inspection. The method involves acquiring historical problem-free images, registering and fusing them to generate a reference image, and registering this reference image with the current image to form a comparable image pair. Grayscale, texture, and semantic differences are calculated between the reference image and the current image, and weighted and fused to form a comprehensive difference map. The comprehensive difference map is binarized to obtain a change mask. A high threshold is used for judgment in flat texture areas, and a low threshold is used for judgment in complex texture areas. Candidate regions are extracted from the change mask, region features are calculated, and the Top-K significant change regions are selected based on the comprehensive score, and their significance is evaluated. A multimodal large model is input, and the final judgment result is output. Compared with existing technologies, this invention improves the effectiveness of detecting changes in images of locations requiring identification compared to historical normal images, while also improving the efficiency of identifying key change regions.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, particularly to image change detection, multimodal data fusion, and multimodal large model application technologies. Specifically, it relates to a multimodal image change detection and prior generation method for urban inspection, which can be applied to scenarios such as UAV urban inspection, environmental monitoring, and security monitoring. Background Technology

[0002] Currently, drones are used in scenarios such as urban patrol, environmental monitoring, and security surveillance. Drones are pre-programmed with flight paths in the city. Under fixed flight paths and set shooting parameters (such as position, altitude, and angle), drones periodically collect image data (such as once a month). The data is then compared with historical images by the back-end personnel or algorithms to determine if there are any changes in problems, such as illegal construction, piles of debris, damage to greenery, or road damage, thereby providing support for urban management decisions.

[0003] However, in practical applications, existing problem identification methods have obvious performance limitations.

[0004] 1. Manual review method: Manually comparing images from each inspection to identify environmental changes and problems is a huge and time-consuming task, and is easily affected by subjective factors. Due to the large number of urban inspection images (one image per location per month, and thousands of images per flight route), relying on manual review of each image is not only inefficient, but also heavily dependent on human resources, making it difficult to meet the needs of large-scale deployment.

[0005] 2. Lightweight model recognition (e.g., YOLO): While automatic recognition using object detection models (e.g., YOLO) can partially replace manual work, it faces several challenges: First, it requires a large amount of localized training data: customized datasets must be built for specific urban scenarios (e.g., damaged greenery, piled-up debris, etc.), resulting in high data collection and annotation costs; second, it has poor training transferability: recognition accuracy may drop sharply in different cities or seasons, requiring retraining; third, the model's expressive power is limited: lightweight models struggle to capture abstract problems (e.g., "street vending" or "abnormal walls"), are only applicable to a limited number of fixed targets, and lack versatility.

[0006] 3. Directly using multimodal large models (such as Qwen, GPT, Gemini) for review: The current industry trend is to use multimodal large models to directly analyze the change areas of inspection images, but this strategy also has major bottlenecks: First, it lacks accurate change guidance areas, and the model needs to reason within the entire image, which leads to attention diversion problems; second, it is prone to "illusions": large models may make incorrect conclusions based on vague details (such as misjudging shadows as piles of objects), lacking image change context, resulting in a high risk of false positives and false negatives; third, directly inputting the entire image into the large model results in high token consumption, leading to high costs.

[0007] In summary, even with automation, existing processes still face challenges such as difficulty in training, poor generalization, and weak interpretability. Summary of the Invention

[0008] Purpose of the invention: To address the problems existing in the prior art, this invention proposes a multimodal image change detection and prior generation method for urban inspection. By fusing the difference information of grayscale, texture and high-level semantic features, and introducing an adaptive threshold and region sorting mechanism based on local texture intensity, the method improves the effectiveness of detecting changes in the image to be identified at the point compared with historical normal images, while also improving the efficiency of identifying key change areas.

[0009] Technical Solution: This invention provides a multimodal image change detection and prior generation method for urban patrol inspection, comprising the following steps:

[0010] Step 1: Obtain historical images without problems and perform registration and fusion to generate a reference image. Register the reference image with the current image to form a comparable image pair. Perform brightness normalization processing on the reference image and the current image.

[0011] Step 2: Calculate grayscale, texture, and semantic differences between the reference image and the current image, and then weight and fuse them to form a comprehensive difference map;

[0012] Step 3: The adaptive thresholding algorithm based on local texture characteristics is used to binarize the composite difference map to obtain a change mask. A high threshold is used to judge in flat texture areas and a low threshold is used to judge in complex texture areas.

[0013] Step 4: Extract candidate regions from the change mask, calculate region features, and select the Top-K significant change regions based on the comprehensive score and evaluate their significance;

[0014] Step 5: Use the generated spatially significant change regions, the Top-K ROI detail patches, and the scene task prompts as input to the multimodal large model, or use the selected Top-K ROI detail patches as input to the multimodal large model, and output the final judgment result.

[0015] Furthermore, obtaining comparable image pairs in step 1 specifically involves:

[0016] Collect multiple images of normal, problem-free conditions from historical moments. and the image to be detected at the current time point ;

[0017] Select a clear image from historical images that represents a normal environment as the baseline image, and then compare the other historical images. Geometric alignment is performed to the baseline coordinate system, and finally, all aligned images are pixel-level fused to generate a reference image. ;

[0018] The current image to be detected Align to reference image In the coordinate system, obtain the aligned current image. Make it similar to the reference image Pixel-level correspondence.

[0019] Furthermore, the brightness normalization in step 1 specifically involves:

[0020] The brightness of the reference image and the current image is adjusted by converting the image from RGB color space to LAB color space, extracting the brightness channel L, and applying adaptive histogram equalization to the L channel; then the adjusted LAB is converted back to RGB to obtain the reference image and the current image after brightness normalization.

[0021] Furthermore, the specific process for obtaining the comprehensive difference map in step 2 is as follows:

[0022] The normalized current image and reference image are converted to grayscale images. The absolute value of the difference is calculated pixel by pixel and normalized to the range of 0 to 1 to obtain the grayscale difference image. ;

[0023] A deep convolutional neural network is introduced to extract semantic features from the reference image and the current image. After the extracted feature maps are normalized, cosine similarity is calculated pixel by pixel, and then converted to cosine distance to obtain semantic feature difference maps. ;

[0024] The gradient magnitude is used to characterize the image texture intensity of the reference image and the current image, and the gradient difference between the two images is calculated to obtain the texture difference map. ;

[0025] A weighted fusion strategy is used to construct the comprehensive difference graph D, and the fusion formula is as follows:

[0026] ;

[0027] Among them, weight parameters , and satisfy x represents the column position of the pixel in the corresponding image, that is, the number of pixels from the left edge of the image; y represents the row position of the pixel in the corresponding image, that is, the number of pixels from the top edge of the image.

[0028] Furthermore, when weighted fusion is used to form a comprehensive difference map, stable regions are first detected and then suppressed to reduce noise in the comprehensive difference map. Specifically:

[0029] Set low thresholds for grayscale difference, texture difference, and feature difference to... Perform pixel-by-pixel comparison: simultaneously satisfy , and Pixels identified as "stable backgrounds" are used to generate stable mask regions. These stable region pixels are then forcibly set to 0 in the composite difference map using the stable region detection mask, resulting in the final composite difference map. .

[0030] Furthermore, the adaptive thresholding algorithm for local texture characteristics in step 3 is specifically as follows:

[0031] Step 3.1: Divide the image into several sub-blocks of a fixed block size. For each sub-block, calculate the mean and standard deviation of the comprehensive difference values ​​within the block, and simultaneously calculate the texture difference map within the block. The average value;

[0032] Step 3.2: Set the texture intensity discrimination threshold ,like If the sub-block is considered to belong to a low-texture region, then... This indicates a high-texture area;

[0033] Step 3.3: Set two threshold adjustment coefficients: low texture block coefficient and high texture block coefficient Calculate the threshold for each sub-block: The value of k is selected based on the current sub-block texture condition, choosing a low texture block coefficient. and high texture block coefficient ;

[0034] Step 3.4: Compare all pixels within the sub-block with the current sub-block threshold to generate a sub-block binary mask.

[0035] Step 3.5: On the generated binary mask, detect stable regions and use the stable region mask to force all stable region pixels to zero.

[0036] Furthermore, in step 4, the Top-K regions of significant change are selected, specifically as follows:

[0037] Step 4.1: Extract candidate Regions of Interest (ROIs) using connected component analysis;

[0038] Step 4.2: For each retained connected component, calculate its bounding rectangle and area, and simultaneously collect the mean of the comprehensive difference of the current region;

[0039] Step 4.3: When selecting ROIs, perform statistical analysis on the historical frequency change data within the selected ROI region, calculate the mean of historical frequency change for that region (mean_hist), and determine the historical freshness based on the mean of historical frequency change (mean_hist). : ;

[0040] Step 4.3: Based on the area of ​​ROI, the comprehensive difference mean, and historical freshness, ROIs are comprehensively scored and ranked to select the Top-K areas of significant change.

[0041] Furthermore, the comprehensive scoring and ranking of ROI in step 4.3 specifically involves:

[0042] Step 4.3.1: Perform min-max normalization on several features of all ROIs, including the area of ​​the ROI, the mean of the overall difference, and the historical freshness.

[0043] Step 4.3.2: Set the weights of each feature and obtain the basic change score of the ROI through linear combination. ;

[0044] Step 4.3.3: Bidirectional Illumination Consistency Verification Mechanism Based on HSV Color Space: For each candidate ROI, extract its color features in the current image and the reference image, and calculate the brightness difference. Hue difference ;

[0045] Step 4.3.4: Based on hue differences Perform hue consistency verification;

[0046] Step 4.3.5: Based on the difference in brightness Perform bidirectional illumination detection: simultaneously monitor both darkening and brightening conditions, when When the color darkens, it is checked whether the condition of "stable hue and saturation fluctuation within the tolerable range" is met, thereby determining the probability of shadow; when When the light brightens, it is checked whether the "hue stability" is met, and the tolerance for saturation reduction is relaxed to determine the probability of strong light.

[0047] Step 4.3.6: Based on the combined characteristics of changes in hue, lightness, and saturation, calculate the probability that the region belongs to a pure illumination change. Set the lighting penalty coefficient. The final score calculation formula is revised as follows:

[0048]

[0049] in, This is the light penalty factor.

[0050] Furthermore, the determination of shadow probability and strong light probability in step 4.3.5 is specifically as follows:

[0051] Calculate normalized hue differences Difference in brightness Saturation difference ;

[0052] Take min(abs( () * 2.0, 1) represents the brightness variation, taking the value of max(0, 1.0-). ×4.0) represents hue stability;

[0053] When it gets darker, take the value of max(0, 1.0-). ×2.0) represents saturation stability; when brightening, it takes the value max(0, 1.0 - 2.0). ×1.0) represents saturation stability;

[0054] = Brightness variation × Hue stability × Saturation stability

[0055] Furthermore, in step 4, a spatially significant variation region, i.e., a spatial saliency map, is generated using border markings, specifically as follows:

[0056] Based on the comprehensive score ranking, the Top-K most significant candidate regions of change are selected. On a copy of the current image to be detected, bounding boxes are drawn for these K regions, and the boxes are labeled with serial numbers or confidence scores. The image with clear bounding boxes is the spatial saliency map. At the same time, these Top-K ROI regions are cropped from the original image and saved as independent detail patches.

[0057] Beneficial effects:

[0058] (1) End-to-end closed-loop design, highly adaptable to the engineering needs of urban inspection. This invention proposes for the first time a complete end-to-end system architecture suitable for fixed-track UAV inspection, combining visual change detection, large-scale model analysis and structured output, covering the following complete chain: upstream: spatiotemporally aligned multi-frame background modeling (historical image geometric registration and fusion), midstream: cross-coordinate system multimodal change detection (brightness + texture + semantic feature differences), downstream: multimodal large-scale model triple attention recognition mechanism, output end: structured output of problem type, location, and anomaly description. The whole forms a closed-loop system with front-end data perception, intermediate change modeling, and back-end intelligent analysis, significantly improving the level of intelligence in urban inspection.

[0059] (2) Multimodal joint change difference modeling. Compared with existing technologies that rely solely on single grayscale difference or inter-frame difference, this invention fuses multiple visual modalities to enhance the robustness of change detection, including: 1) Brightness difference: absolute difference in grayscale values ​​(sensitive to primary brightness changes); 2) Texture difference: gradient magnitude change based on the Sobel operator (able to distinguish between illumination changes and structural changes); 3) Semantic feature difference: extracting feature maps from intermediate layers (such as layer 4) of backbone networks such as ResNet, VGG, EfficientNet, ViT (Vision Transformer), and constructing semantic difference (sensitive to newly added / disappeared objects) through cosine distance. After unified normalization, the difference maps are fused to form a multimodal fusion difference map, reflecting the comprehensive perception ability of different modalities to changes. It can accurately distinguish between "structural real changes" (such as illegal constructions, piles of objects) and "unstructured environmental noise" (such as swaying trees, water ripples) from two dimensions: physical attributes (grayscale / texture) and semantic attributes (depth features).

[0060] (3) Variation modeling based on multimodal fusion and illumination consistency verification. Addressing the performance degradation of traditional methods in low-light, strong light reflection, or shadow movement scenarios during early morning and late evening, this invention innovatively introduces a bidirectional illumination consistency verification mechanism based on the HSV color space, building upon multimodal feature fusion. Unlike traditional grayscale / gradient difference methods, this mechanism analyzes the bidirectional fluctuations (brightening or darkening) of hue stability and brightness in changing regions to obtain the probability of illumination changes. This mechanism can accurately distinguish between "structural changes caused by object movement" and "non-structural illumination noise caused by shadows, reflections, and overexposure," enabling the system to adapt to complex illumination environments across multiple time periods and weather conditions without retraining or manual threshold adjustment, significantly improving its generalization ability and robustness in complex urban lighting environments.

[0061] (4) ROI Screening Module. To avoid processing redundant regions in a large model and improve the efficiency of identifying effective problems, the ROI (Regional Area of ​​Change) screening mechanism proposed in this invention uses a multi-factor importance scoring and ranking mechanism to calculate and score multiple indicators for each candidate ROI, including: region area, comprehensive differential intensity, historical freshness, etc. Through a bidirectional illumination consistency verification mechanism based on the HSV color space, and by setting an illumination penalty factor, regions of change caused by shadows or strong light are penalized to reduce the risk of false alarms. The system ultimately retains only the K most important regions of change and outputs them as ROI tiles, serving as the main processing unit for input into the large model. This process significantly reduces the computational burden of the LMM while improving the model's ability to perceive core problems.

[0062] (5) In the screening of ROI (candidate region of change), this invention also performs fragment merging of change regions. Actual changes are often divided into multiple small blocks, which can easily lead to misidentification or duplicate identification. To this end, this method adopts a multi-round iterative merging strategy based on the spatial distance and boundary overlap between ROIs to merge adjacent or overlapping change regions into a complete structure.

[0063] (6) The visual-language dual-guided attention model input scheme of this invention emphasizes local-global consistency: visual guidance: saliency maps provide key attention locations; detailed region guidance: top-ranked ROI regions (regions of interest); and textual guidance: prompts set according to specific scenes and locations are input into the multimodal large model. The large model is guided to focus on areas of significant change through spatial saliency map overlay; the large model is guided to focus on specific issues through scene-specific prompts; and the ROI region is used as a local detail reference to guide the large model to focus on suspicious areas. The input of multi-channel information enables the multimodal large model to identify issues from text and images, and from the global to the overall perspective. Attached Figure Description

[0064] Figure 1 This is a flowchart of the overall method of the present invention;

[0065] Figure 2 This is a schematic diagram of the comprehensive differential steps of the present invention;

[0066] Figure 3 This is a schematic diagram of the ROI screening steps of the present invention;

[0067] Figure 4 A block diagram of the multimodal large model guidance mechanism;

[0068] Figure 5 This is a schematic diagram of the significant features of the present invention;

[0069] Figure 6 This is a reference figure for the present invention. Detailed Implementation

[0070] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0071] This invention provides a multimodal image change detection and prior generation method for urban inspection, the overall process of which is as follows: Figure 1 As shown, the main steps include:

[0072] Step 1: Acquire historical images without problems and perform registration and fusion to generate a reference image, while registering the current image to form a comparable image pair.

[0073] For each inspection point, collect multiple historical images showing the normal, problem-free condition. and the image to be detected at this point at the current time. .

[0074] First, select a clear image from historical images that represents a normal environment as a baseline image (e.g., a recently taken image without any anomalies). Then, use other historical images... Geometric alignment to the reference image coordinate system is performed. The steps of geometric alignment are as follows: feature points are extracted for each historical image and the reference image using the ORB algorithm; initial feature pairs are obtained through Hamming distance matching; and the homography transformation matrix is ​​estimated using the RANSAC algorithm. ,Will Projecting the image onto the reference image yields the aligned image. (Alternatively, SIFT, SURF, AKAZE, or deep learning-based alignment methods can be used.) Finally, all aligned images are pixel-wise fused to generate a reference image. The fusion method employs a pixel-wise median method (taking the median value of N aligned images for each pixel position) to suppress the influence of temporarily appearing foreground objects on the background, thereby obtaining a stable reference image.

[0075] The current image is geometrically aligned with the reference image, and the current image to be detected is... Align to reference image In the coordinate system. For and Perform ORB feature matching and calculate the homography matrix from the current image to the reference image using RANSAC. . use right Perform a transformation to obtain the aligned current image. Make it similar to the reference image Pixel-level alignment (SIFT, SURF, AKAZE, or deep learning-based alignment methods can also be used). Precise image alignment removes differences in viewpoint and shooting position, ensuring that subsequent pixel differences have physical meaning.

[0076] Step 2: Perform brightness normalization processing on the reference background image and the current alignment image.

[0077] When drones are used for surveillance and filming, variations in lighting angle and intensity can lead to differences in image brightness at the same location. To reduce brightness differences caused by different shooting times, this invention modifies the reference image... Alignment with current image Adjust the brightness.

[0078] Specifically, the image is converted from the RGB color space to the LAB color space, the luminance channel L is extracted, and adaptive histogram equalization is applied to the L channel. This processed luminance channel effectively reduces overall brightness deviations caused by factors such as exposure and whether the environment is sunny or cloudy, improving brightness consistency. Then, the adjusted LAB image is converted back to RGB to obtain a luminance-normalized reference image and the current image. and This preprocessing ensures that subsequent differences primarily reflect structural changes rather than changes in light intensity.

[0079] Step 3: Calculate the grayscale, texture, and semantic differences and then weight and fuse them to form a comprehensive difference map.

[0080] On the preprocessed image, this invention simultaneously calculates three types of difference information: grayscale, texture, and semantics, in order to comprehensively capture change information:

[0081] 1) Gray-scale difference image The process involves converting the normalized current and reference images into grayscale images, calculating the absolute value of the difference pixel by pixel, and normalizing it to the range of 0-1. Grayscale difference reflects changes in image brightness and coarse texture; when a new object appears or an existing object disappears, a noticeable grayscale difference will occur in the corresponding area. It's important to note that changes in lighting are also reflected in the grayscale difference. It is unstable to illumination, but as part of multimodal fusion, it provides basic sensitivity to change.

[0082] 2) Semantic feature difference map Considering that high-level semantic changes (such as the addition of new object types or changes in appearance) cannot be fully captured by grayscale alone, this invention introduces semantic features extracted by deep convolutional neural networks to detect changes.

[0083] In this embodiment, a pre-trained ResNet network is used to extract intermediate layers (layer 4 of ResNet-18 is used in this invention; VGG, EfficientNet, ViT (Vision Transformer), or other feature extraction backbone networks can also be used) as the feature extractor. The reference image and the current image are input into the feature extractor to obtain the corresponding feature map tensors. and The specific steps are as follows:

[0084] First, the input image is proportionally reduced (long side no more than 1024 pixels) to balance computational complexity and feature resolution. The extracted feature maps are then normalized, and cosine similarity is calculated pixel-by-pixel, which is then converted to cosine distance. Obtained in this way This reflects differences in high-level semantics. Finally, regarding... Create a difference map, upsample and interpolate back to the original image size, and perform Z-score normalization to linearly map it to the range of 0 to 1. It is not sensitive to pure overall changes in brightness; for example, a brightening or darkening of the entire image has little impact on depth features. However, if a completely new object appears in the scene or existing objects undergo significant changes, the depth feature difference will increase significantly in that region. Therefore, It provides a robust semantic change detection method, which makes up for the shortcomings of gray-level difference detection which is susceptible to low-level perturbations.

[0085] 3) Texture difference map To capture changes in detailed texture, this invention uses gradient magnitude to characterize image texture intensity and calculates the gradient difference between the previous and next images.

[0086] In practical implementation, the Sobel operator is applied to the grayscale images of the image to be detected and the reference image to calculate the gradients in the x and y directions, respectively, and the gradient magnitude map is calculated. First, the average gradient magnitudes of the reference image and the current image are taken to obtain the texture intensity map, which is then normalized to 0~1 to represent the local texture richness of each pixel. Simultaneously, the gradient magnitude difference is calculated and normalized to obtain a texture difference map of 0~1. If a new texture appears in a certain region (e.g., the edge of a newly added object), the gradient magnitude difference map is then calculated. The region will have a higher response; conversely, changes in pure brightness (such as dimming or brightening of the light) will not significantly alter the gradient amplitude. It is less sensitive to this. Therefore, texture difference helps to distinguish between real structural changes and lighting changes.

[0087] 4) Multimodal difference fusion: After obtaining the difference maps of each modality, the present invention uses a weighted fusion strategy to construct a comprehensive difference map D.

[0088] Considering that grayscale difference is sensitive to most changes while deep feature difference is more reliable for semantic changes, weight parameters are set in the embodiments. , and (satisfy (This is used for fusion.) The fusion formula is:

[0089]

[0090] Where x represents the column position of the pixel in the image, that is, the number of pixels from the left edge of the image. y represents the row position of the pixel in the image, that is, the number of pixels from the top edge of the image. For example, D gray (5, 6) refers to the grayscale difference image D. gray In the image, the value of the pixel located in the 5th row and 6th column.

[0091] The result of fusion Then, normalization is performed: the minimum and maximum values ​​are calculated, and D is linearly stretched and mapped to the full range of 0 to 1 to increase contrast. Subsequently, a stabilization mask from the stable region detection is used to force the pixels in the stable regions of the composite difference map to be set to 0, thus obtaining the final composite difference map. The comprehensive difference map is saved in grayscale format, which can intuitively show which areas have changed relative to the historical background in the current map. The higher the grayscale value, the greater the magnitude of the change.

[0092] In obtaining the comprehensive difference map The invention detects stable regions and suppresses them in the composite difference map to reduce noise. Since some areas in the urban environment remain unchanged or change very little (e.g., fixed texture of road surfaces, background buildings, etc.), this invention detects stable regions before fusing the difference map to avoid noise interference.

[0093] Specifically, low thresholds are set for grayscale difference, texture difference, and feature difference. The aforementioned... Perform pixel-by-pixel comparison: all that simultaneously satisfy , and Pixels that are not stable are identified as "stable backgrounds," and stable masks are generated to mark these regions. Stable regions typically correspond to backgrounds that are unchanged and noise-free; for these pixels, we subsequently force them to be considered "unchanged." This significantly reduces background noise in the differential fusion results: even if some stable background pixels have slight fluctuations in a certain modality (e.g., slight changes in grayscale but almost unchanged texture and features), as long as they are sufficiently stable overall, we exclude them from the change candidate list. Stable region detection improves the reliability of the results, making truly changing regions more prominent.

[0094] Step 4: Binarize the composite difference map based on the adaptive threshold of local texture characteristics to obtain the variation mask.

[0095] To transform a continuous difference map into a discrete region of variation, this invention designs a local texture-aware adaptive thresholding algorithm. The data is divided into foreground and background. Unlike a globally fixed threshold or a simple OTSU method, standard algorithms may only consider local mean / variance. The adaptive thresholding method of this invention combines local statistical characteristics and texture information, enabling dynamic adjustment of sensitivity for different regions. Its core idea is to use a higher threshold for strict judgment in regions with flat textures, and a lower threshold for improved sensitivity in regions with complex textures. The implementation steps are as follows:

[0096] Step 4.1: Divide the image into fixed block sizes (in this embodiment of the invention, ...). The pixel is divided into several sub-blocks. For each sub-block, the mean of the comprehensive difference value within that block is calculated. and standard deviation Simultaneously, the average value of the texture intensity map within this block is calculated. Here, the texture intensity is taken from the aforementioned texture difference map. (Reflects the richness of local texture).

[0097] Step 4.2: Set the texture intensity discrimination threshold .like If the sub-block is considered to belong to a low-texture region (such as a shadowed area on the ground or a large flat area), then a pure brightness change in such a region may be a pseudo-change caused by lighting, and therefore a higher threshold is selected to determine the change. This is a high-texture area (a region rich in detail, such as weeds and gravel), and even a slight change may mean a change in the real object. Therefore, a lower threshold is used to increase detection sensitivity.

[0098] Step 4.3: Set two threshold adjustment coefficients: low texture block coefficient and high texture block coefficient Calculate the threshold for each sub-block: The value of k is selected based on the texture characteristics of the block, choosing a low texture block coefficient. and high texture block coefficient Finally, all pixels within a sub-block are compared with the threshold of that block to generate a sub-block binary mask.

[0099] Step 4.4: On the generated binary mask, the stable region mask is used again to force all stable region pixels to zero, ensuring that these regions are not falsely detected as changes. In addition, as mentioned above, image alignment may introduce edge artifacts, so the mask values ​​within a certain pixel width of the image edge can be selectively cleared to zero to remove false changes at alignment edges.

[0100] Compared to the global threshold method, this method adjusts the standard based on local characteristics, significantly reducing the false alarm rate under illumination changes. This strategy effectively distinguishes illumination artifacts from actual structural changes and has proven to be one of the key aspects of this invention's performance improvement.

[0101] Step 5: Extract candidate regions from the change mask, calculate region features, and filter Top-K change regions based on the comprehensive score. After obtaining the binary change mask, it is necessary to locate the specific change regions and evaluate their significance.

[0102] Step 5.1: Extract candidate ROIs (Regions of Interest) using connected component analysis. Specifically, connected component labeling (8-neighborhood connectivity) is performed on the binary mask to obtain several connected components, each corresponding to a candidate ROI. Noise regions with excessively small areas (connected components with an area smaller than a threshold, such as 200 pixels, will be ignored) are filtered out to remove isolated noise points.

[0103] Step 5.2: For each retained connected component, calculate its bounding rectangle and area, and collect the corresponding values ​​on each difference plot for subsequent analysis. Specifically, calculate the following within each ROI region:

[0104] ① Overall Difference Mean: Calculate the mean of the ROI region in the overall difference map. The average difference value reflects the overall degree of change and the most significant change value in the region.

[0105] ② Historical Freshness: Based on a pixel-level or fine-grained grid, this is a historical change frequency probability map used to record the cumulative number of times each location in the scene is identified as changing within a set time window (e.g., the past 30 days). When selecting ROIs, the historical change frequency map data within the selected ROI area is statistically analyzed to calculate the mean historical change frequency (mean_hist) for that area. The mean historical change frequency reflects the normality of changes in the area: a higher mean indicates that the area changes frequently (e.g., areas with swaying trees or water surfaces), and is more likely to be part of the environmental background noise; a lower mean indicates that the area changes very rarely, and when it does change, it has high "freshness" and attention value. In this embodiment, the historical freshness is calculated using the following function:

[0106]

[0107] This scoring system is used to weight and improve the ranking of areas that appear for the first time or undergo rare changes. The location and perspective captured by the drone are a fixed area. The parts of this area that are usually in motion (such as trees and water ripples) are unimportant information and are given low scores; while the parts that have never changed before but have suddenly changed today (such as the sudden appearance of illegal buildings) are more important information and are given high scores.

[0108] In particular, when the system lacks historical data for a location (such as during a cold start or at a new location), the default mean_hist will be 0 (or the history_score will be set to the default maximum value of 1.0) to ensure that the system can accurately capture all potential anomalies during the initial operation phase and avoid missed detections due to the lack of prior information.

[0109] Step 5.3: Change ordering and spatial saliency map construction.

[0110] This invention performs a comprehensive scoring and ranking of ROIs, identifying regions of significant variation (e.g., the Top-K). The scoring mechanism primarily considers the area of ​​the ROI, the overall difference intensity, and the influence of lighting factors. The specific steps are as follows:

[0111] 1) Normalization: Min-max normalization is performed on several features of all ROIs, including the area of ​​the ROI, the mean of the overall difference, and historical freshness. Normalization transforms features of different dimensions into the 0-1 range, which facilitates weighted fusion.

[0112] 2) Calculate the base score: Weights for each feature are set empirically, and a linear combination is used to obtain the base change score of the ROI. For example, the area of ​​the ROI and the strength of the comprehensive difference can be given higher weights (indicating that areas with large ranges and amplitudes of change are more important), followed by texture change, with semantic prior and historical freshness as auxiliary factors. In this embodiment, the weights used are: ROI area 0.4, comprehensive difference mean 0.4, and historical freshness 0.2.

[0113] 3) Illumination Consistency Judgment and Two-Way Penalty Mechanism: Addressing the complex illumination variations in urban environments (such as shadow casting, strong light reflection, and differences in automatic exposure), this invention abandons the traditional method relying solely on grayscale and texture ratios, and proposes an illumination consistency probability model based on the HSV color space. Specifically, for each candidate ROI, its color features in the current image and the reference image are extracted, and the brightness difference is calculated (…). ) and hue difference ( ).

[0114] Hue consistency verification: According to the principles of physical optics, changes in light intensity (brightening or darkening) usually do not change the inherent hue of an object's surface. If a certain area experiences a significant change in brightness but a very small difference in hue, then that area is highly likely to be a pseudo-change caused by lighting.

[0115] Two-way illumination detection: This invention simultaneously monitors both darkening (shadow) and brightening (highlight / reflection).

[0116] when hour( (i.e., darkening), and check whether it meets the condition of "stable hue and saturation fluctuation within the tolerance range" to determine the probability of shadow.

[0117] when hour( (Brightening) checks whether the "hue stability" condition is met, while relaxing the tolerance for saturation reduction (highlights diluting colors) to determine the probability of strong light.

[0118] Calculate the probability of illumination Calculate the probability that the change in hue, lightness, and saturation in this area is due to a pure change in illumination. (0~1).

[0119] Specifically, it means the probability that only the illumination changes within the ROI region. Shadow probability, on the other hand, is the probability under the condition of "darkening". The probability of strong light is under the condition of "brightening". The calculation process is the calculation itself. The two branches (darkening) It refers to the probability of shadows becoming brighter. It refers to the probability of strong light.

[0120] The specific determination process is as follows:

[0121] Calculate the normalized hue difference ( ), brightness difference ( ), saturation difference ( )

[0122] Brightness variation level: take min(abs( ) * 2.0, 1);

[0123] Hue stability: Take max(0, 1.0-) ×4.0);

[0124] Saturation stability: When darkening, take the maximum value (0, 1.0-). ×2.0), when brightening, take max(0, 1.0 - ×1.0).

[0125] = Degree of change in brightness × Hue stability × Saturation stability

[0126] Adaptive penalty: Sets the lighting penalty coefficient. (e.g., 0.8). The final score calculation formula is revised as follows:

[0127]

[0128] The final ROI scoring formula is as follows:

[0129] = 0.4 * area + 0.4 * average difference + 0.2 * historical freshness.

[0130] = (0.4 * Area + 0.4 * Average of Overall Differences + 0.2 * Historical Freshness) * .

[0131] Through this mechanism, whether it is the shadow cast by trees or the strong reflection of glass curtain walls, as long as it does not change the essential color of the ground features, its score will be greatly suppressed, thereby effectively filtering ambient light noise.

[0132] 4) ROI Ranking and Filtering: To provide accurate hard attention guidance for large multimodal models, this invention uses bounding box labeling to generate spatial saliency maps. The specific steps are as follows:

[0133] Based on the comprehensive scoring ranking, the Top-K (e.g., the top 10) most salient candidate regions of change are selected. On a copy of the current image to be detected, prominent bounding boxes are drawn for these K regions, with optional annotations of serial numbers or confidence scores next to the boxes. This image with clearly marked bounding boxes is the spatial saliency map. Simultaneously, the system crops these Top-K ROI regions from the original image and saves them as independent detail patches. For example, a large model can further perform target recognition or category determination on each ROI patch to determine the specific nature of the change (e.g., identifying "a new bicycle has been added" or "a pothole has appeared on the road surface," etc.). Through the saliency map and ROI output, this invention provides change detection results in a form easily usable by both humans and machines: the saliency map provides an intuitive reference for the overall location distribution, and the ROI patches provide detailed local images of each region for analysis. These prior results significantly reduce the burden on backend algorithms or manual inspection, allowing them to focus their attention on a small number of high-probability anomalous regions.

[0134] Here, when selecting ROIs, multiple rounds of iterative merging are performed using boundary overlap (IoU) and the Euclidean distance between the centers of two ROIs. The steps are as follows:

[0135] 1. Calculate the boundary overlap ratio (IoU), which is the ratio of the area of ​​the intersection of two ROI regions to the area of ​​their union.

[0136] 2. Calculate the center distance between the two ROI regions.

[0137] 3. If the IoU is greater than a threshold or the center distance between the two regions is less than a fixed value, the two rectangles are merged into one large rectangle (all candidate regions for change are rectangles). The coordinates of the new rectangle are determined by the minimum upper-left corner coordinates and the maximum lower-right corner coordinates of the original two regions, thus completely covering the original fragments.

[0138] This process is repeated until no regions need to be merged in a complete traversal, ensuring that all overlapping regions are aggregated.

[0139] Step 6: Multimodal large model change identification.

[0140] Method 1: Multimodal large model change recognition mode guided by saliency map: The generated spatial saliency map (full image + red box marking), Top-K ROI detail map and scene task prompt words are sent as input to the multimodal large model.

[0141] 1) Spatial saliency map: Provides a global view and spatial location reference for large models. The red box forces the model's visual attention to focus on high confidence areas, preventing the model from diverging on irrelevant backgrounds.

[0142] 2) ROI detail tiles: Provide uncompressed, high-resolution local details, enabling the model to see tiny objects (such as road cracks and illegally parked vehicles).

[0143] 3) Joint reasoning: The large model combines the prompt words with the location information of the global saliency map and the texture details of the local tiles to output the final judgment result.

[0144] Method 2: Lightweight Inference Mode Based on Pure Local Input of ROI: Considering that in actual engineering deployments, there may be situations such as limited network bandwidth, API token cost sensitivity, or insufficient computing power at the edge, in this mode, the system no longer transmits the "spatial saliency map" containing a large amount of redundant background information to the multimodal large model, but only uses the selected Top-K ROI detail patches as input. The specific process is as follows:

[0145] 1) Batch aggregation of ROI: serialize and encode the K high-resolution ROI patches that have been cropped.

[0146] 2) Lightweight input: Send the ROI sequence and task prompts to the large model.

[0147] 3) Result mapping: The large model classifies or describes each ROI patch independently. After receiving the returned results, the system maps them back to their original locations based on the ROI's metadata (original coordinate ID).

[0148] In this embodiment, multimodal fusion significantly improves detection accuracy and effectively reduces the false alarm rate. In real-world urban scenario testing, this invention increases the detection rate of changing areas to 85%~90%; especially in complex, unstructured scenarios, the false alarm rate decreases by an average of over 12%, greatly enhancing the practicality and data reliability of the automated inspection system.

[0149] In summary, the technical solution of this invention achieves high-precision change detection prior map construction for urban inspection images through steps such as multimodal image differencing, fusion, threshold segmentation, connected component analysis, and ranking. Those skilled in the art can understand and implement the solution of this invention without creative effort after reading the above content. It should be noted that this invention is not limited to the specific parameters and algorithm selections mentioned above. For example, the feature extraction network can be replaced with other deep learning models, the feature levels and weights can be adjusted, and the block size and coefficients of the adaptive threshold can be optimized according to the scenario. These variations all fall within the protection scope of this invention.

[0150] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent transformations or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multimodal image change detection and prior generation method for urban patrol inspection, characterized in that, Includes the following steps: Step 1: Obtain historical images without problems and perform registration and fusion to generate a reference image. Register the reference image with the current image to form a comparable image pair. Perform brightness normalization processing on the reference image and the current image. Step 2: Calculate the grayscale, texture, and semantic differences between the reference image and the current image, and then weight and fuse them to form a comprehensive difference map; Step 3: The adaptive thresholding algorithm based on local texture characteristics is used to binarize the composite difference map to obtain a change mask. A high threshold is used to judge in flat texture areas and a low threshold is used to judge in complex texture areas. Step 4: Extract candidate regions from the change mask, calculate region features, and select the Top-K significant change regions based on the comprehensive score and evaluate their significance; Step 5: Use the generated spatially significant change regions, the Top-K ROI detail patches, and the scene task prompts as input to the multimodal large model, or use the selected Top-K ROI detail patches as input to the multimodal large model, and output the final judgment result.

2. The multimodal image change detection and prior generation method for urban inspection according to claim 1, characterized in that, The specific steps for obtaining comparable image pairs in step 1 are as follows: Collect multiple images of normal, problem-free conditions from historical moments. and the image to be detected at the current time point ; Select a clear image from historical images that represents a normal environment as the baseline image, and then compare the other historical images. Geometric alignment is performed to the baseline coordinate system, and finally, all aligned images are pixel-level fused to generate a reference image. ; The current image to be detected Align to reference image In the coordinate system, obtain the aligned current image. Make it similar to the reference image Pixel-level correspondence.

3. The multimodal image change detection and prior generation method for urban inspection according to claim 2, characterized in that, The brightness normalization in step 1 specifically involves: The brightness of the reference image and the current image is adjusted by converting the image from RGB color space to LAB color space, extracting the brightness channel L, and applying adaptive histogram equalization to the L channel; then the adjusted LAB is converted back to RGB to obtain the reference image and the current image after brightness normalization.

4. The multimodal image change detection and prior generation method for urban inspection according to claim 1, characterized in that, The specific process for obtaining the comprehensive difference map in step 2 is as follows: The normalized current image and reference image are converted to grayscale images. The absolute value of the difference is calculated pixel by pixel and normalized to the range of 0 to 1 to obtain the grayscale difference image. ; A deep convolutional neural network is introduced to extract semantic features from the reference image and the current image. After the extracted feature maps are normalized, cosine similarity is calculated pixel by pixel, and then converted to cosine distance to obtain semantic feature difference maps. ; The gradient magnitude is used to characterize the image texture intensity for the reference image and the current image, and the gradient difference between the two images is calculated to obtain the texture difference map. ; A weighted fusion strategy is used to construct the comprehensive difference map D, and the fusion formula is as follows: ； Among them, weight parameters , and satisfy x represents the column position of the pixel in the corresponding image, that is, the number of pixels from the left edge of the image; y represents the row position of the pixel in the corresponding image, that is, the number of pixels from the top edge of the image.

5. The multimodal image change detection and prior generation method for urban inspection according to claim 4, characterized in that, When weighted fusion is used to form a comprehensive difference map, stable regions are first detected and then suppressed to reduce noise in the comprehensive difference map. Specifically: Set low thresholds for grayscale difference, texture difference, and feature difference to... Perform pixel-by-pixel comparison: simultaneously satisfy , and Pixels identified as "stable background" are used to generate stable mask regions. These stable region pixels are then forcibly set to 0 in the composite difference map using the stable region detection mask, resulting in the final composite difference map. .

6. The multimodal image change detection and prior generation method for urban inspection according to claim 1, characterized in that, The adaptive threshold algorithm for local texture characteristics in step 3 is as follows: Step 3.1: Divide the image into several sub-blocks of a fixed block size. For each sub-block, calculate the mean and standard deviation of the comprehensive difference values ​​within the block, and simultaneously calculate the texture difference map within the block. The average value; Step 3.2: Set the texture intensity discrimination threshold ,like If the sub-block is considered to belong to a low-texture region, then... This indicates a high-texture area; Step 3.3: Set two threshold adjustment coefficients: low texture block coefficient and high texture block coefficient Calculate the threshold for each sub-block: The value of k is selected based on the current sub-block texture condition, choosing a low texture block coefficient. and high texture block coefficient ; Step 3.4: Compare all pixels within the sub-block with the current sub-block threshold to generate a sub-block binary mask. Step 3.5: On the generated binary mask, detect stable regions and use the stable region mask to force all stable region pixels to zero.

7. The multimodal image change detection and prior generation method for urban inspection according to claim 1, characterized in that, In step 4, the Top-K regions of significant change are selected, specifically as follows: Step 4.1: Extract candidate Regions of Interest (ROIs) using connected component analysis; Step 4.2: For each retained connected component, calculate its bounding rectangle and area, and simultaneously collect the mean of the comprehensive difference of the current region; Step 4.3: When selecting ROIs, perform statistical analysis on the historical frequency change data within the selected ROI region, calculate the mean of historical frequency change for that region (mean_hist), and determine the historical freshness based on the mean of historical frequency change (mean_hist). : ; Step 4.3: Based on the area of ​​ROI, the comprehensive difference mean, and historical freshness, ROIs are comprehensively scored and ranked to select the Top-K areas of significant change.

8. The multimodal image change detection and prior generation method for urban inspection according to claim 7, characterized in that, The specific steps for ranking and scoring ROI in step 4.3 are as follows: Step 4.3.1: Perform min-max normalization on several features of all ROIs, including the area of ​​the ROI, the mean of the overall difference, and the historical freshness. Step 4.3.2: Set the weights of each feature and obtain the basic change score of the ROI through linear combination. ; Step 4.3.3: Bidirectional Illumination Consistency Verification Mechanism Based on HSV Color Space: For each candidate ROI, extract its color features in the current image and the reference image, and calculate the brightness difference. Hue difference ; Step 4.3.4: Based on hue differences Perform hue consistency verification; Step 4.3.5: Based on the difference in brightness Perform bidirectional illumination detection: simultaneously monitor both darkening and brightening conditions, when When the color darkens, it is checked whether the condition of "stable hue and saturation fluctuation within the tolerable range" is met, thus determining the probability of shadow; when When the light brightens, it is checked whether the "hue stability" condition is met, and the tolerance for saturation reduction is relaxed to determine the probability of strong light. Step 4.3.6: Based on the combined characteristics of changes in hue, lightness, and saturation, calculate the probability that the region belongs to a pure illumination change. Set the lighting penalty coefficient. The final score calculation formula is revised as follows: in, This is the light penalty factor.

9. The multimodal image change detection and prior generation method for urban inspection according to claim 8, characterized in that, The determination of shadow probability and strong light probability in step 4.3.5 is as follows: Calculate normalized hue differences Difference in brightness Saturation difference ; Take min(abs( () * 2.0, 1) represents the brightness variation, taking the value of max(0, 1.0-). ×4.0) represents hue stability; When it gets darker, take the value of max(0, 1.0-). ×2.0) represents saturation stability; when brightening, it takes the value max(0, 1.0 - 2.0). ×1.0) represents saturation stability; = Brightness variation × Hue stability × Saturation stability 10. A multimodal image change detection and prior generation method for urban inspection according to claim 8, characterized in that, In step 4, a spatially significant variation region, i.e., a spatial saliency map, is generated using border markings. Specifically: Based on the comprehensive score ranking, the Top-K most significant candidate regions of change are selected. On a copy of the current image to be detected, bounding boxes are drawn for these K regions, and the boxes are labeled with serial numbers or confidence scores. The image with clear bounding boxes is the spatial saliency map. At the same time, these Top-K ROI regions are cropped from the original image and saved as independent detail patches.

Images