Computer vision-based image change detection method for ancient buildings

Abstract

This invention provides a computer vision-based method for detecting changes in ancient building images, applicable to fields such as computer vision and image processing. The method includes: generating a difference heatmap based on the differences between features of a reference image and features of the image to be detected; extracting multiple geometric cue points based on the pixel values ​​of multiple pixels in the difference heatmap; encoding the geometric coordinates and semantic labels of the geometric cue points respectively, and fusing the encoded coordinates and encoded labels to obtain cue features; using a twin decoding mechanism, simultaneously segmenting the reference image and the image to be detected in the same physical coordinate system based on the cue features, reference image features, and features of the image to be detected, obtaining a reference segmentation mask and its corresponding confidence level, and a test segmentation mask and its corresponding confidence level; and determining the detection result corresponding to the image to be detected based on the reference segmentation mask and its corresponding confidence level, and the test segmentation mask and its corresponding confidence level for each of the multiple geometric cue points.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image processing, and other technical fields, and specifically to a method for detecting changes in ancient building images based on computer vision. Background Technology

[0002] Image change detection of ancient buildings is a key technology for automated monitoring of defects and structural safety assessment of ancient buildings. Existing general-purpose visual recognition models are adapted to conventional natural image scenes, but they are unable to cope with the complex textures, lighting, occlusion, and other interferences in ancient building images. Therefore, they cannot be effectively applied to this specific detection task, and it is even more difficult to achieve accurate qualitative and quantitative analysis of changes in ancient building components.

[0003] Furthermore, the lack of high-quality, large-scale dedicated datasets in this field fails to provide sufficient training support for deep learning models. The combined effect of insufficient model adaptability and a scarcity of datasets results in low accuracy in detecting changes in ancient building images, making it difficult to accurately identify and locate component defects and structural changes, and hindering precise qualitative analysis of these changes. Consequently, it fails to meet the practical needs of refined preservation of ancient buildings. Summary of the Invention

[0004] In view of this, the present invention provides a method for detecting changes in ancient building images based on computer vision.

[0005] This invention provides a computer vision-based method for detecting changes in ancient building images, comprising: generating a difference heatmap based on the differences between reference image features and the target image features; the reference image and the target image are images of the target ancient building at different time points, with the reference image corresponding to a time point earlier than the target image; extracting multiple geometric cue points based on the pixel values ​​of multiple pixels in the difference heatmap, each geometric cue point including geometric coordinates and a semantic label, the semantic label being used to distinguish whether the geometric cue point represents changes related to the ancient building or changes due to environmental interference; encoding the geometric coordinates and semantic labels of the geometric cue points respectively, and fusing the encoded coordinates and encoded labels to obtain cue features; using a twin decoding mechanism, simultaneously segmenting the reference image and the target image in the same physical coordinate system based on the cue features, reference image features, and target image features to obtain a reference segmentation mask and its corresponding confidence level, and a target segmentation mask and its corresponding confidence level; and determining the detection result corresponding to the target image based on the reference segmentation mask and its corresponding confidence level, and the target segmentation mask and its corresponding confidence level for each of the multiple geometric cue points.

[0006] According to an embodiment of the present invention, the method for detecting changes in ancient building images based on computer vision further includes: performing registration processing on a reference image and a detection image to obtain a registered reference image and a registered detection image; performing field-of-view normalization processing on the registered reference image and the registered detection image to obtain a normalized reference image and a normalized detection image; and extracting features unaffected by illumination changes from the normalized reference image and the normalized detection image to obtain reference image features and detection image features, respectively.

[0007] According to an embodiment of the present invention, features unaffected by illumination changes are extracted from a normalized reference image and a normalized image to be detected, respectively, to obtain reference image features and image to be detected features. This includes: dividing the normalized reference image or the normalized image to be detected into image blocks of a predetermined size; marking the position information of each image block to preserve the spatial structure of the normalized reference image or the normalized image to be detected, resulting in multiple processed image blocks; and inputting the multiple processed image blocks into a visual encoder to capture the dependencies between the processed image blocks and then outputting the reference image features or image to be detected features.

[0008] According to an embodiment of the present invention, field-of-view normalization processing is performed on the registered reference image and the registered image to be detected to obtain a normalized reference image and a normalized image to be detected. This includes: cropping the center region of the target area including the target ancient building in the registered reference image and the registered image to be detected, respectively, to obtain a cropped reference image and a cropped image to be detected; and performing dimensional normalization and numerical calibration processing on the cropped reference image and the cropped image to be detected to ensure that the spatial dimension and numerical distribution of the normalized reference image and the normalized image to be detected are consistent.

[0009] According to an embodiment of the present invention, the geometric positioning coordinates include sub-pixel level floating-point coordinates. Based on the pixel values ​​of multiple pixels in the difference heatmap, multiple geometric cue points are extracted, including: obtaining a pixel distribution histogram based on the pixel values ​​of multiple pixels in the difference heatmap, and determining a segmentation threshold based on the pixel distribution histogram; selecting target pixels with pixel values ​​greater than the segmentation threshold from the multiple pixels, and determining the target response region of the difference heatmap based on the target pixels; performing maximum value filtering on the target response region to extract the geometric cue points; determining a set of neighboring pixels within a preset range from the target response region with the geometric cue points as the center; and performing weighted calculation on the integer-level pixel coordinates of each target pixel in the neighboring pixel set using the target pixel value of each target pixel in the neighboring pixel set as the weight to obtain the sub-pixel level floating-point coordinates corresponding to each geometric cue point.

[0010] According to an embodiment of the present invention, the twin decoding mechanism includes two mask decoders with identical structures and shared parameters. Utilizing the twin decoding mechanism, in the same physical coordinate system, based on cue features, reference image features, and the image to be detected, the reference image and the image to be detected are simultaneously segmented to obtain a reference segmentation mask and its corresponding confidence level, and a segmentation mask and its corresponding confidence level. This includes: in the same physical coordinate system, concatenating the cue features with learnable output labels to obtain fused cue features; performing self-attention aggregation on the fused cue features, and using the fused cue features as the query to map the reference image features or the image to be detected features as keys and values, performing cue-image cross-addition. The intentional operation yields the interactive image features. The output labels from the fused cue features after self-attention aggregation are input into a multilayer perceptron and an intersection-union prediction multilayer perceptron, respectively, to output mask weight vectors and mask confidence scores. The interactive image features are then reshaped, and spatial resolution is upsampled through convolutional upsampling to obtain mask spatial features. The mask weight vector and mask spatial features are then multiplied by a dot product to generate multiple candidate mask features. Bilinear interpolation is used to restore the resolution of the multiple candidate mask features, and a filtering process is performed based on the mask confidence scores to obtain a reference segmentation mask and its corresponding confidence score, or a segmentation mask to be tested and its corresponding confidence score.

[0011] According to an embodiment of the present invention, the mask decoder is obtained from a trained semantic segmentation model. The semantic segmentation model includes an image encoder, a cue encoder, and a mask decoder. The training steps of the semantic segmentation model include: constructing a training dataset, which includes multiple sample images and a ground truth mask corresponding to each sample image. The sample images contain ancient architectural components that have changed due to disease and ancient architectural components that have not been affected by disease; generating multiple types of cue samples based on multiple ground truth masks, the cue samples including foreground cue points and background cue points; and performing feature extraction and standardization processing on the sample images based on the image encoder to obtain... The process involves several steps: First, inputting the cue samples into the cue encoder for encoding and transformation to obtain cue features that match the feature dimensions of the sample image. Then, concatenating these cue features with learnable sample output labels in the same physical coordinate system to obtain fused cue features. Next, inputting the fused cue features and sample image features into the mask decoder, performing feature interaction, transformation, and upsampling processes sequentially. This outputs a pixel class probability map matching the size of the sample image, where the value of each pixel indicates the confidence level of its classification into the target segmentation category. Finally, generating a predicted segmentation mask based on the pixel class probability map. Finally, determining the loss value based on the predicted segmentation mask, the ground truth mask, and the pixel class probability map, and performing backpropagation to update the model parameters of the image encoder, cue encoder, and mask decoder, resulting in a trained semantic segmentation model.

[0012] According to an embodiment of the present invention, constructing a training dataset includes: collecting multiple images of ancient building components that have changed due to disease and those that have not been affected by disease under multiple temporal phases, multiple perspectives, and different lighting conditions; performing pixel-level segmentation and annotation on the multiple ancient building images to obtain ground truth masks corresponding to the ancient building images; superimposing noise textures of different frequencies, performing color space transformation, or changing the image brightness on any one of the multiple ancient building images to simulate the physical aging characteristics or lighting disturbances of the building surface to obtain multiple sample images; and mapping the multiple sample images one-to-one with the ground truth masks to obtain a training dataset.

[0013] According to an embodiment of the present invention, determining a loss value based on a predicted segmentation mask, a ground truth mask, and a pixel category probability map includes: determining a focal loss value for calculating pixel classification error based on the pixel category probability map; determining a Dessian loss value for calculating geometric shape similarity based on the intersection area of ​​the predicted segmentation mask and the ground truth mask and the relative ratio of the intersection area to the total area of ​​the predicted segmentation mask and the ground truth mask; performing an Euclidean distance transformation on the boundary of the ground truth mask to generate a distance field of pixel points from the boundary; mapping the distance field to a penalty weight, and weighting it with the confidence level corresponding to each pixel point in the pixel category probability map to obtain a boundary loss value for calculating contour fitting accuracy; and obtaining the loss value based on the focal loss value, the Dessian loss value, and the boundary loss value.

[0014] According to an embodiment of the present invention, the detection result corresponding to the image to be detected is determined based on the reference segmentation mask and its corresponding confidence level corresponding to each of the multiple geometric cue points, and the segmentation mask to be tested and its corresponding confidence level. This includes: determining the intersection-over-union ratio (IoU) based on the reference segmentation mask and the segmentation mask to be tested corresponding to each of the multiple geometric cue points; the IoU characterizes the spatial overlap between the reference segmentation mask and the segmentation mask to be tested in the same physical coordinate system; determining the area change rate of the segmentation mask to be tested relative to the reference segmentation mask based on the area of ​​the reference segmentation mask and the area of ​​the segmentation mask to be tested; and determining the detection result corresponding to the image to be detected based on the IoU, the area change rate, the confidence levels corresponding to the reference segmentation masks and the segmentation mask to be tested corresponding to each of the multiple geometric cue points, and a preset judgment rule.

[0015] According to embodiments of the present invention, by analyzing the differences between the features of the reference image and the image to be detected, low-level environmental changes can be weakened, while the semantic and structural information of the target itself can be strengthened. This helps to suppress the interference of environmental factors such as illumination and viewing angle on change detection. Combining the precise location of geometric cue points with semantic classification, it is possible to distinguish between changes related to ancient buildings and environmental interference. Utilizing a twin decoding mechanism for synchronous segmentation ensures the consistency and comparability of temporal image segmentation results and provides quantified confidence. Therefore, it is possible to more accurately identify the true changes in ancient building components, reduce the false alarm rate, and improve the accuracy and reliability of ancient building defect detection. Attached Figure Description

[0016] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings.

[0017] Figure 1 A flowchart of a computer vision-based method for detecting changes in images of ancient buildings according to an embodiment of the present invention is shown.

[0018] Figure 2 A schematic diagram of ancient building image change detection according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic diagram of the network structure of an image encoder for a cue-driven semantic segmentation model according to an embodiment of the present invention is shown.

[0020] Figure 4 A schematic diagram of the network structure of the cue encoder and mask decoder of the cue-driven semantic segmentation model according to an embodiment of the present invention is shown.

[0021] Figure 5 A block diagram of a computer vision-based ancient building image change detection device according to an embodiment of the present invention is shown. Detailed Implementation

[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] The following will be through Figures 1-3 The method for detecting changes in ancient building images based on computer vision according to embodiments of the present invention will be described in detail.

[0027] Figure 1 A flowchart of a computer vision-based method for detecting changes in images of ancient buildings according to an embodiment of the present invention is shown.

[0028] like Figure 1 As shown, the computer vision-based method for detecting changes in ancient building images includes steps S110 to S150.

[0029] In step S110, a difference heatmap is generated based on the differences between the reference image features of the reference image and the detection image features of the image to be detected.

[0030] In step S120, multiple geometric cue points are extracted based on the pixel values ​​of multiple pixels in the differential heat map.

[0031] In step S130, the geometric coordinates and semantic labels of the geometric prompt points are encoded respectively, and the encoded coordinates and encoded labels are fused to obtain the prompt features.

[0032] In step S140, using the twin decoding mechanism, the reference image and the image to be detected are simultaneously segmented in the same physical coordinate system based on the cue features, reference image features, and image to be detected features, to obtain the reference segmentation mask and its corresponding confidence, and the image to be detected segmentation mask and its corresponding confidence.

[0033] In step S150, based on the reference segmentation mask and corresponding confidence level of each of the multiple geometric cue points, and the segmentation mask to be tested and corresponding confidence level, the detection result corresponding to the image to be detected is determined.

[0034] In this embodiment, the reference image and the image to be detected can be images of the target ancient building at different time points. The time point corresponding to the reference image is earlier than that of the image to be detected, and it can be used as a benchmark for change detection.

[0035] The features of the reference image and the features of the image to be detected can be high-dimensional information that can characterize the image content, extracted from the corresponding reference image and the image to be detected. The image content can reflect the structure and state of the ancient building itself, such as information about the local texture, edges, or color distribution of the image. For example, the features of the reference image and the features of the image to be detected can be obtained from the reference image and the image to be detected, respectively, by a pre-trained general feature extraction network.

[0036] In a difference heatmap, the pixel value of each point reflects the degree of difference between two image features at the corresponding location. For example, the difference can be quantified by calculating the Euclidean distance or absolute difference between the reference image features and the features to be detected at corresponding pixel locations, and mapping these difference values ​​to image space generates a difference heatmap. Areas with higher pixel values ​​can be considered as potential areas of change.

[0037] Each geometric cue point can include geometric coordinates and a semantic label. The semantic label can be used to distinguish whether the geometric cue point represents changes related to the ancient building or changes caused by environmental disturbances. Changes related to the ancient building may include, but are not limited to, defects or structural damage. Changes caused by environmental disturbances may include, but are not limited to, changes in lighting or shadow movement. For example, a geometric cue point can be a key point extracted from a difference heatmap that indicates the location of potential changes.

[0038] Cue features can combine location precision and semantic discriminability to guide image segmentation tasks. For example, geometric coordinates can be directly used as numerical values, or they can be normalized and mapped to a specific range to obtain coded coordinates. Semantic labels can be converted into numerical vectors using one-hot encoding to obtain coded labels. Coded coordinates and coded labels are then fused through concatenation or element-wise addition to obtain cue features.

[0039] A twin decoding mechanism is an image processing architecture that can include two structurally identical decoders with shared parameters. It can be used to synchronously process a reference image and a target image in the same physical coordinate system to ensure the consistency and comparability of segmentation results. In the same physical coordinate system, the pixels of the reference image and the target image correspond spatially. For example, each decoder can use convolutional layers, upsampling layers, and activation functions to convert the features of the reference image or the target image, as well as cue features, into a segmentation mask that matches the size of the original image, obtaining either a reference segmentation mask or a target segmentation mask. Furthermore, it can obtain the probability value of each pixel belonging to the target category and determine this probability value as the confidence level, obtaining the corresponding confidence level for each segmentation mask.

[0040] For example, for each geometric cue point, changes can be detected based on its state in the reference segmentation mask and the segmentation mask to be tested, combined with the corresponding confidence level. For instance, if a geometric cue point is not segmented into the target region and has a low confidence level in the reference segmentation mask, but is segmented into the target region and has a high confidence level in the segmentation mask to be tested, a sub-detection result representing a newly added object can be obtained. By detecting all geometric cue points and combining them with semantic labels, the specific type and location of changes in the image to be detected can be determined.

[0041] According to embodiments of the present invention, by analyzing the differences between the features of the reference image and the image to be detected, low-level environmental changes can be weakened, while the semantic and structural information of the target itself can be strengthened. This helps to suppress the interference of environmental factors such as illumination and viewing angle on change detection. Combining the precise location of geometric cue points with semantic classification, it is possible to distinguish between changes related to ancient buildings and environmental interference. Utilizing a twin decoding mechanism for synchronous segmentation ensures the consistency and comparability of temporal image segmentation results and provides quantified confidence. Therefore, it is possible to more accurately identify the true changes in ancient building components, reduce the false alarm rate, and improve the accuracy and reliability of ancient building defect detection.

[0042] In the process of implementing this invention, it was discovered that the lack of registration between the reference image and the image to be detected leads to misalignment and false differences. Furthermore, inconsistent field of view and changes in illumination affect the accuracy of feature extraction, thereby reducing the precision and reliability of change detection.

[0043] In view of this, in one embodiment of the present invention, the ancient building image change detection method based on computer vision includes, in addition to the above-mentioned... Figure 1In addition to steps S110 to S150, the method may further include: performing registration processing on the reference image and the image to be detected to obtain a registered reference image and a registered image to be detected; performing field-of-view normalization processing on the registered reference image and the registered image to be detected to obtain a normalized reference image and a normalized image to be detected; and extracting features unaffected by illumination changes from the normalized reference image and the normalized image to be detected, respectively, to obtain reference image features and image to be detected features.

[0044] Registration processing can be used to precisely align the reference image and the image to be detected in the same spatial coordinate system, thereby eliminating image deviations caused by factors other than changes in the ancient building itself, such as camera displacement and shooting angle changes. These factors may include, but are not limited to, translation, rotation, scaling, and perspective distortion.

[0045] For example, registration algorithms can be used to perform feature matching and geometric transformation on the reference image and the image to be detected.

[0046] Field of view normalization can be used to eliminate inconsistencies in the field of view that may occur during image capture, ensuring that the reference image and the image to be detected are consistent in spatial dimensions, such as size and resolution, and in numerical distribution, such as brightness and contrast, thus avoiding spurious changes introduced by differences in the field of view.

[0047] For example, a higher-order interpolation algorithm can be used to resample the registered reference image and the registered image to be detected, respectively, to preserve fine textures. Then, a joint strategy of center cropping and matrix correction can be executed to obtain the normalized reference image and the normalized image to be detected.

[0048] Features unaffected by changes in lighting can include, for example, intrinsic structural and textural features that are insensitive to lighting conditions.

[0049] For example, self-supervised learning or contrastive learning can be used to enable the model to learn similar feature representations between images of the same scene under different lighting conditions. Alternatively, images can be converted to a color space that is insensitive to lighting, and texture features can be extracted from this space.

[0050] According to embodiments of the present invention, since the calculation of the difference heatmap depends on pixel-level correspondences, and the twin decoding mechanism needs to drive model prediction under the same coordinates, registration can establish a unified spatial coordinate benchmark, ensuring that the subsequently calculated difference values ​​originate from the physical changes on the surface of the ancient building, rather than the registration residuals caused by camera displacement. Furthermore, extracting features unaffected by changes in illumination ensures that subsequent difference analysis accurately reflects the physical changes of the ancient building itself, rather than fluctuations in ambient light.

[0051] For example, performing field-of-view normalization on the registered reference image and the registered image to be detected to obtain a normalized reference image and a normalized image to be detected may include: cropping the center region of the target area including the target ancient building in the registered reference image and the registered image to be detected, respectively, to obtain a cropped reference image and a cropped image to be detected; and performing dimensional normalization and numerical calibration on the cropped reference image and the cropped image to be detected to ensure that the spatial dimension and numerical distribution of the normalized reference image and the normalized image to be detected are consistent.

[0052] The registered reference image and the registered image to be detected contain invalid regions or stretched distortion bands, such as black borders, due to registration. Center region cropping can remove these black borders, such as cropping a predetermined number of pixels (e.g., 100 pixels) of black borders vertically, horizontally, and vertically, while retaining the target region including the target ancient building, resulting in a cropped reference image and a cropped image to be detected. Dimensional normalization can scale both the cropped reference image and the cropped image to a predetermined resolution. Numerical calibration can normalize pixel values ​​to the [0,1] range, etc.

[0053] According to an embodiment of the present invention, dimension normalization and numerical calibration are performed simultaneously with physical image cropping, which solves the coordinate mapping deviation problem caused by edge distortion after registration and helps to improve the geometric accuracy of lesion location.

[0054] In another example, a translation matrix can be constructed based on the coordinate offset of the position difference between the top left corner of the cropped reference image or the top left corner of the cropped image and the corresponding reference image or the top left corner of the image to be detected. This matrix is ​​then used to synchronously correct the original homography matrix H, which represents the mapping relationship between the planes of the reference image and the image to be detected. This ensures that the correspondence between the cropped pixels in the mathematical space remains valid, thus supporting the subsequent accurate mapping of the detected lesion coordinates back to the image to be detected.

[0055] For example, extracting features unaffected by illumination changes from the normalized reference image and the normalized image to be detected, respectively, to obtain reference image features and image to be detected features, may include: dividing the normalized reference image or the normalized image to be detected into image blocks of a predetermined size; marking the position information of each image block to preserve the spatial structure of the normalized reference image or the normalized image to be detected, to obtain multiple processed image blocks; inputting the multiple processed image blocks into a visual encoder to capture the dependencies between the processed image blocks and then outputting reference image features or image to be detected features.

[0056] After dividing an image into independent image blocks, the spatial relative positions between these blocks are lost. By labeling the positional information of each image block, the subsequent visual encoder can perceive their spatial layout in the original image when processing these blocks, thereby reconstructing and utilizing the overall spatial structure of the image.

[0057] Visual encoders can be used to extract high-level semantic features from multiple processed image patches. Through their internal mechanisms, such as self-attention or convolutional operations, visual encoders learn and capture the interdependencies between different image patches, thereby generating reference image features or features to be detected that are globally context-aware.

[0058] This invention, by segmenting image blocks, helps reduce the computational complexity of subsequent feature extraction and enables the visual encoder to capture local texture and structural information more precisely, thus effectively addressing potential local lighting unevenness or subtle changes in images of ancient buildings. Labeling the location information of each image block avoids the loss of global spatial information caused by local feature extraction. The visual encoder then captures the deep dependencies between image blocks, thereby extracting robust features that are not only unaffected by lighting changes but also rich in spatial context information.

[0059] In the process of implementing the embodiments of the present invention, it was found that integer pixel coordinates may lead to insufficient geometric positioning accuracy, affecting the accuracy of subsequent segmentation.

[0060] Based on this, in one embodiment of the present invention, the geometric positioning coordinates may include sub-pixel level floating-point coordinates. Multiple geometric cue points are extracted based on the pixel values ​​of multiple pixels in the difference heatmap, including: obtaining a pixel distribution histogram based on the pixel values ​​of multiple pixels in the difference heatmap, and determining a segmentation threshold based on the pixel distribution histogram; selecting target pixels with pixel values ​​greater than the segmentation threshold from the multiple pixels, and determining the target response region of the difference heatmap based on the target pixels; performing maximum value filtering on the target response region to extract the geometric cue points; determining a set of neighboring pixels within a preset range from the target response region, centered on the geometric cue points; and performing weighted calculations on the integer-level pixel coordinates of each target pixel in the neighboring pixel set, using the target pixel value of each target pixel in the neighboring pixel set as a weight, to obtain the sub-pixel level floating-point coordinates corresponding to each geometric cue point.

[0061] For example, the pixel value corresponding to a predetermined percentile of the pixel distribution histogram can be used as the segmentation threshold. The predetermined percentile can be a high percentile selected based on the cumulative distribution of the pixel distribution histogram, such as 90%. Alternatively, it can be determined based on the objective of minimizing the intra-class variance of the difference regions and maximizing the inter-class variance, in order to achieve adaptive identification of difference regions and avoid the limitations that may arise from fixed thresholds under different lighting, materials, or types of changes.

[0062] Maximum value filtering selects the maximum value of a pixel within a local window as the new value of the center pixel, highlighting local peaks within the region. For example, by sliding a filter window of a preset size, such as 3x3 or 5x5, within the target response region and selecting the maximum pixel value within each window as the local peak, the local peak point can be used as a geometric cue point to characterize the central location where the change is most significant.

[0063] The preset range can be a square or circular area of ​​fixed size, such as a 5x5 pixel window centered on the geometric cue point, or an area dynamically determined based on the distribution of local difference intensity.

[0064] This invention, through a combination of maximum value filtering and weighted centrifugal techniques, not only robustly extracts representative geometric cue points but also improves the coordinate precision of these cue points from the integer level to the sub-pixel level by utilizing the difference in intensity information of neighboring pixels. These high-precision geometric cue points can more accurately guide the twin decoding mechanism for synchronous segmentation, resulting in more refined and accurate generation of the reference segmentation mask and the segmentation mask to be tested.

[0065] According to an embodiment of the present invention, the twin decoding mechanism includes two mask decoders with identical structures and shared parameters, ensuring that the reference image and the image to be detected are affected by the same processing logic and parameters during the decoding process, thereby guaranteeing the consistency and comparability of the segmentation results in the physical coordinate system.

[0066] Using a twin decoding mechanism, under the same physical coordinate system, based on cue features, reference image features, and the image to be detected, the reference image and the image to be detected are simultaneously segmented to obtain a reference segmentation mask and its corresponding confidence score, and a test segmentation mask and its corresponding confidence score. This includes: concatenating cue features with learnable output labels under the same physical coordinate system to obtain fused cue features; performing self-attention aggregation on the fused cue features, and using the fused cue features as the query, mapping the reference image features or the image to be detected features as keys and values, performing a cue-image cross-attention operation to obtain the interacted image features; and then... The output labels in the fusion cue features after row-attention aggregation are input into the multilayer perceptron and the cross-union prediction multilayer perceptron, respectively, and output mask weight vectors and mask confidence scores. The image features after interaction are reshaped, and spatial resolution is upsampled through convolutional upsampling to obtain mask space features. The mask weight vector and mask space features are then multiplied by a dot product to generate multiple candidate mask features. Bilinear interpolation is used to restore the resolution of the multiple candidate mask features, and after filtering by combining the mask confidence scores, a reference segmentation mask and its corresponding confidence score, or a segmentation mask to be tested and its corresponding confidence score, are obtained.

[0067] This invention does not specify the particular method of splicing, such as channel splicing, etc.

[0068] Self-attention aggregation can employ a multi-head self-attention mechanism, where each head independently calculates attention weights, and then the results are concatenated. Cue-image cross-attention can be designed so that the query comes from fused cue features, and the keys and values ​​come from features in the reference image or the image to be detected, and vice versa.

[0069] Filtering processes could include applying a preset confidence threshold for binarization, or using morphological operations to smooth mask edges and remove small-area noise.

[0070] This invention can improve the accuracy and consistency of synchronous segmentation in the detection of changes in ancient building images, and effectively solve the problems of insufficient feature fusion and errors introduced by resolution restoration.

[0071] Figure 2 A schematic diagram of ancient building image change detection according to an embodiment of the present invention is shown.

[0072] like Figure 2As shown, in this embodiment, the reference image and the image to be detected are registered to obtain a registered reference image and a registered image to be detected. The registered reference image and the registered image to be detected are then cropped to obtain image blocks corresponding to the image to be detected and image blocks corresponding to the reference image. The image blocks corresponding to the image to be detected and the image blocks corresponding to the reference image are input into an image encoder, respectively, to output features of the image to be detected and features of the reference image. The cosine similarity between the features of the image to be detected and the features of the reference image can be calculated pixel-by-pixel based on the feature dimension, and a difference heatmap can be constructed. Based on the pixel values ​​of multiple pixels in the difference heatmap, multiple geometric cue points are extracted. These geometric cue points are input into a cue encoder, and after encoding and conversion processing, they are input into a mask decoder along with the features of the image to be detected and the features of the reference image, respectively. After feature interaction, transformation, and upsampling processing, the following outputs are given: a segmentation mask to be tested and its corresponding confidence score, and a segmentation mask to be tested and its corresponding confidence score. The detection result corresponding to the image to be detected is determined based on the segmentation mask to be tested and its corresponding confidence score, and the segmentation mask to be tested and its corresponding confidence score.

[0073] For example, the normalized reference image can be used. and the normalized image to be detected Input the trained visual encoder into each image and extract reference image features that are unaffected by changes in illumination. and features of the image to be detected Calculate the reference image features pixel by pixel in the feature dimension. and features of the image to be detected The cosine similarity between them is calculated, and a difference heatmap D is constructed as shown in equation (1):

[0074] (1),

[0075] in, For pixel coordinates, Represents the vector dot product. The L2 norm is represented; the directional consistency of the eigenvectors is used to eliminate the shadow displacement interference caused by the difference in the angle of sunlight.

[0076] According to an embodiment of the present invention, the mask decoder may be obtained from a trained semantic segmentation model, which includes an image encoder, a cue encoder, and a mask decoder.

[0077] The training steps of the semantic segmentation model include: constructing a training dataset; generating multi-type cue samples based on multiple ground truth masks; performing feature extraction and standardization on the sample images based on an image encoder to obtain sample image features; inputting the cue samples into the cue encoder for encoding transformation to obtain sample cue features matching the feature dimensions of the sample images; concatenating the sample cue features with learnable sample output labels in the same physical coordinate system to obtain sample fusion cue features; inputting the sample fusion cue features and sample image features into a mask decoder for sequential feature interaction, transformation, and upsampling processing to output a pixel category probability map matching the size of the sample images; generating a predicted segmentation mask based on the pixel category probability map; determining the loss value based on the predicted segmentation mask, ground truth mask, and pixel category probability map, and performing backpropagation based on the loss value to update the model parameters of the image encoder, cue encoder, and mask decoder to obtain the trained semantic segmentation model. In this embodiment, the training dataset may include multiple sample images and a ground truth mask corresponding to each sample image. The sample images include ancient building components that have changed due to disease and ancient building components that have not been affected by disease. Ancient architectural components can include at least three of the following: roof tiles, eaves brackets, wooden beams and columns, brick and stone walls, painted murals, and stone carvings. Types of damage can include at least three of the following: structural cracks, surface peeling, missing materials, color fading, biological adhesion, and water stains.

[0078] The cue sample can include foreground cue points and background cue points. Foreground cue points can be pixels located in the diseased area. Background cue points can be pixels located in the background area.

[0079] Image encoders can transform raw pixel data into more abstract, semantically rich high-dimensional features. They can include deep convolutional neural networks, such as ResNet and VGG, which extract image features step by step through multiple convolution and pooling operations.

[0080] A cue encoder can convert discrete cue information, such as coordinates and semantic labels, into continuous, high-dimensional feature representations that are consistent in dimension with the image features output by the image encoder. It can include multilayer perceptrons or Transformer networks.

[0081] Mask decoders can include architectures such as U-Net, which fuse features from different levels of the encoder with the decoder through skip connections and use deconvolution or bilinear interpolation for upsampling.

[0082] The value of each pixel in the pixel category probability map indicates the confidence level of whether the pixel belongs to the category of the segmentation target.

[0083] In the process of updating the model parameters of the image encoder, cue encoder, and mask decoder, it is necessary to minimize the loss value to obtain the trained semantic segmentation model.

[0084] Figure 3 A schematic diagram of the network structure of an image encoder for a cue-driven semantic segmentation model according to an embodiment of the present invention is shown.

[0085] like Figure 3 As shown, the input image, also known as the sample image, has a size of R. 3×1024×1024 The input image can be divided into 16×16 image blocks by an image encoder. Each image block is then mapped to a 1280-dimensional vector using a convolutional layer (Conv2d, in_channels=3, out_channels=1280, kernel_size=(16,16), stride=(16,16)). The vector is then reshaped to obtain R. 4096×1280 The block features correspond to 64×64 tokens. A learnable positional code is added to each image block to preserve the spatial location information of the image, resulting in a dimensional R... 4096×1280 The block features with location information are used. Long-range dependencies in the image are modeled progressively through a 32-layer Transformer module (including layer normalization, multi-head attention, global feedforward network, and residual connections), with an output dimension of R. 4096×1280 High-level semantic features. The feature dimension is reduced to kernel_size=(1,1)R through convolutional layers (Conv2d, in_channels=1280, out_channels=256, kernel_size=(2,2)) and layer normalization. 256×64×64 After smoothing, the final feature dimension is R. 256×64×64 The features of the sample images.

[0086] Figure 4 A schematic diagram of the network structure of the cue encoder and mask decoder of a cue-driven semantic segmentation model according to an embodiment of the present invention is shown. Figure 4 As shown, the sample cue points (N points, including point coordinates (x, y) and point labels, such as foreground / background) of the cue samples can be input into the cue encoder, and R... N×2 The point coordinates (x, y) are used for Fourier feature encoding, which maps them to a dimension R. N×256 The coordinate features; mapping the foreground / background labels of the point labels to dimensions R through a lookup table (Embedding layer). N×256 The label features are obtained by adding the coordinate features to the label features element by element, resulting in a dimension R. N×256 The sample cue features are aligned with the dimensions of the sample image features.

[0087] like Figure 4 As shown, the sample image features output by the image encoder, the sample cue features output by the cue encoder, and the feature dimension R are... 4×256 The learnable sample output token is input to the mask decoder, which concatenates the sample cue features with the learnable sample output token to obtain a result of dimension R. (N+4)×256 The sample fusion cue features are used; self-attention aggregation is performed on the sample fusion cue features to model the dependencies within the cue information; using the fusion cue features as the query and the sample image features as the key and value, a cue-image cross-attention operation is performed to guide and focus the cue information on the sample image features; and a fully connected layer (upgrading the dimension to R) is used to achieve this. (N+4)×2048 ), activation function (GELU), fully connected layer (dimensionality reduced to R) (N+4)×256 The features are subjected to nonlinear transformation; using sample image features as queries and sample fusion cue features as keys and values, an image-cue cross-attention operation is performed to achieve feedback from image features to cue features; the sample fusion cue features with dimension R after self-attention aggregation are then processed. 4×256 The sample output tokens are input into the multilayer perceptron and the prediction multilayer perceptron, respectively, with corresponding output dimensions of R. 4×32 The sample mask weight vector and multiple sample mask confidences are used. The interactive sample image features are reshaped to obtain a dimension R. 256×64×64 The features are then subjected to spatial resolution upsampling through convolutional upsampling (Conv2d, kernel_size=2, stride=2) to obtain a dimension of R. 32×256×256 The sample mask space features. The dimension R 32×256×256 The sample mask space features and dimension are R 4×32 Perform a dot product operation on the sample mask weight vectors to generate four dimensions R. 256×256 Masks, such as mask 1, mask 2, mask 3, and mask 4. Masks 1, 2, 3, and 4 are upsampled to dimension R using bilinear interpolation. 1024×1024 After thresholding and filtering, the output image has the same dimension as the input image, which is R. 1×1024×1024 The pixel category probability map is used to generate a predictive segmentation mask.

[0088] This invention optimizes the segmentation process in the twin decoding mechanism by introducing a specially trained semantic segmentation model. During the training phase, this model utilizes sample images containing damaged components of ancient buildings and precise ground truth masks, enabling it to learn the unique change patterns of ancient buildings, rather than just general image features. By generating foreground and background cues, the model can more effectively focus on the target area during training, distinguishing between changes related to the ancient building and environmental interference, avoiding false detections of environmental interference or missed detections of subtle defects. It ensures that, under the same physical coordinate system, the segmentation results of the reference image and the image to be detected accurately reflect the real changes in the ancient building.

[0089] In implementing the embodiments of the present invention, it was discovered that images of ancient buildings contain numerous textures and interferences. Existing large-scale visual recognition models are difficult to apply effectively, and the lack of datasets in this field makes it difficult to train the corresponding networks using deep learning methods.

[0090] Based on this, in one embodiment of the present invention, constructing a training dataset may include: collecting multiple images of ancient building components that have changed due to disease and those that have not been affected by disease, under multiple temporal phases, multiple perspectives, and different lighting conditions; performing pixel-level segmentation and annotation on the multiple ancient building images to obtain ground truth masks corresponding to the ancient building images; superimposing noise textures of different frequencies, performing color space transformation, or changing the image brightness on any one of the multiple ancient building images to simulate the physical aging characteristics or lighting disturbances of the building surface to obtain multiple sample images; and mapping the multiple sample images one-to-one with the ground truth masks to obtain a training dataset.

[0091] For example, aerial photography by drones and high-precision ground-based cameras can be used to collect images of ancient buildings at different historical moments or during different monitoring periods. This allows for the construction of a rich training database for deep learning models, enabling the models to learn how to identify and differentiate between physical and environmental changes brought about by the passage of time. Physical changes include weathering and crack propagation, while environmental changes include lighting and shadows. By collecting images from multiple time periods, from multiple perspectives, and under different lighting conditions, the training dataset can be ensured to cover the performance of ancient buildings in various real-world scenarios, thereby improving the model's robustness to environmental changes.

[0092] This system can classify each pixel in multiple images of ancient buildings, identifying whether it belongs to an architectural component, a damaged area, or the background, thereby generating a binary mask or a multi-class ground truth mask with the same dimensions as the original image. Architectural components can include, but are not limited to, walls, beams, columns, and brackets. Damage can include, but is not limited to, cracks, dampness, and mold.

[0093] Superimposing noise textures of different frequencies can simulate surface defects such as weathering, peeling, water stains, and mold. This improves the model's resilience to variations in the surface texture of ancient buildings. For example, noise can be directly superimposed on the pixel values ​​of an image of an ancient building while keeping the ground truth mask label unchanged.

[0094] Performing color space transformations can include, but is not limited to, adjusting hue, saturation, brightness, or performing grayscale processing. This can simulate material discoloration, uneven lighting, and other situations to improve the model's ability to recognize the same object under different appearance conditions, without involving changes in the object's shape or position.

[0095] Changing image brightness can include, but is not limited to, non-linear grayscale correction and overlaying custom-shaped shadow masks. Non-linear grayscale correction can simulate different sunlight intensities or shadow occlusion, thereby simulating the non-linear changes in the intensity of reflected light from an object's surface due to varying sunlight intensities in a real physical environment. Overlaying custom-shaped shadow masks can simulate dynamic shadows cast by trees or buildings.

[0096] This invention effectively simulates the various physical aging characteristics and environmental disturbances of ancient buildings under long-term weathering, erosion, and complex lighting conditions by randomly superimposing noise textures, performing color space transformations, or changing image brightness. This expands the diversity of training samples, enabling the trained semantic segmentation model to better adapt to the complex and ever-changing images of ancient buildings in the real world, and improving the model's generalization ability and robustness to unknown scenarios.

[0097] In implementing the embodiments of the present invention, it was found that a single loss function cannot simultaneously optimize pixel classification, region fitting, and boundary alignment, resulting in insufficient segmentation accuracy of the model, such as difficulty in accurately capturing subtle changes in the defects of ancient buildings.

[0098] Based on this, in one embodiment of the present invention, determining the loss value based on the predicted segmentation mask, the ground truth mask, and the pixel category probability map may include: determining a focal loss value for calculating pixel classification error based on the pixel category probability map; determining a Dessian loss value for calculating geometric similarity based on the intersection area of ​​the predicted segmentation mask and the ground truth mask and the relative ratio of the intersection area to the total area of ​​the predicted segmentation mask and the ground truth mask; performing an Euclidean distance transformation on the boundary of the ground truth mask to generate a distance field of pixel points from the boundary; mapping the distance field to a penalty weight, and weighting it with the confidence level corresponding to each pixel point in the pixel category probability map to obtain a boundary loss value for calculating contour fitting accuracy; and obtaining the loss value based on the focal loss value, the Dessian loss value, and the boundary loss value.

[0099] The predicted segmentation mask can be an image output by a semantic segmentation model. Each pixel in the image is assigned a specific category, such as foreground or background, representing the model's prediction of the shape and location of the target ancient building component or lesion.

[0100] For example, based on the cross-entropy loss function, a modulation factor related to the prediction probability is introduced. By reducing the loss weight of easily distinguishable samples with high prediction probabilities, the model training process focuses on difficult samples with low prediction probabilities, resulting in the focus loss function as shown in equation (2) below:

[0101] (2),

[0102] in, Indicates the focal loss value. This represents the total number of pixels in the sample image. For pixels; Indicates the model predicts the pixel point The probability value of belonging to the category of ancient building defects; A balancing factor to balance the weights of positive and negative samples; This is a focusing parameter used to adjust the decay rate of the weights of easily distinguishable samples. That is, the modulation factor, when When the value approaches 1, the factor approaches 0, thus reducing the contribution of easily separable samples to the total loss.

[0103] Based on the principle of set similarity, the ratio of the intersection area of ​​the predicted segmentation mask and the ground truth mask to the total area of ​​the two is calculated. The geometric overlap is quantified into a loss value to constrain the overall shape of the ancient building components. The Dess loss function is shown in the following equation (3):

[0104] (3),

[0105] in, Indicates the Des loss value. The corresponding truth mask label (value is 0 or 1); This represents an approximate calculation of the intersection between the predicted region and the true value region. To prevent smoothing terms with zero denominator, equation (3) converts the regional overlap into a loss value, forcing the prediction result to approximate the true value geometrically.

[0106] The boundary of the ground truth mask is subjected to Euclidean distance transformation, the distance field from the pixel to the boundary contour of the nearest ground truth mask is calculated, and this distance field is mapped as a penalty weight. The boundary regions with prediction errors are weighted to improve the alignment accuracy of the segmentation edges.

[0107] Set of boundary contours for truth masks Perform Euclidean distance transformation to generate a distance field. As shown in equation (4):

[0108] (4),

[0109] in, For pixels spatial coordinates, The spatial coordinates of points on the boundary contour of the truth mask. This represents the Euclidean distance.

[0110] This distance field The boundary loss function is calculated by weighting the penalty weight with the predicted probability, as shown in equation (5) below:

[0111] (5),

[0112] in, This represents the boundary loss value, for the boundary contour far from the ground truth mask (i.e. Larger) and incorrectly predicted as a prospect (i.e. For larger pixels, equation (5) will produce a huge loss value, thus forcing the model to converge to high accuracy.

[0113] The total loss function is defined as shown in equation (6) below:

[0114] (6),

[0115] in, , and These are hyperparameters used to dynamically balance the contributions of pixel classification, region fitting, and boundary alignment. This represents the loss value.

[0116] This invention introduces boundary loss in addition to conventional focus loss and Descein loss to specifically address the problem of blurred edges and difficulty in precise segmentation of defects in ancient buildings. Based on the focus loss value, Descein loss value, and boundary loss value, the loss values ​​are obtained, comprehensively balancing the optimization requirements of pixel classification, region fitting, and boundary alignment. This achieves a comprehensive improvement in model training, enabling more accurate and precise identification and localization of defects in ancient building image change detection, providing more reliable technical support for the protection and restoration of ancient buildings.

[0117] According to embodiments of the present invention, determining the detection result corresponding to the image to be detected based on the reference segmentation mask and its corresponding confidence level corresponding to each of the multiple geometric cue points, and the segmentation mask to be tested and its corresponding confidence level, may include: determining the intersection-over-union ratio (IoU) based on the reference segmentation mask and the segmentation mask to be tested corresponding to each of the multiple geometric cue points; determining the area change rate of the segmentation mask to be tested relative to the reference segmentation mask based on the area of ​​the reference segmentation mask and the area of ​​the segmentation mask to be tested; and determining the detection result corresponding to the image to be detected based on the IoU, the area change rate, the confidence levels corresponding to the reference segmentation masks and the segmentation mask to be tested corresponding to each of the multiple geometric cue points, and a preset judgment rule.

[0118] In this embodiment, the intersection-union ratio (IUU) characterizes the degree of spatial overlap between the reference segmentation mask and the segmentation mask to be tested in the same physical coordinate system.

[0119] For example, based on the reference segmentation mask With the segmentation mask to be tested The determined Intersection over Union (IoU) is shown in Equation (7) below. Based on the area of ​​the reference segmentation mask and the area of ​​the segmentation mask to be tested, the rate of change of the area of ​​the segmentation mask to be tested relative to the reference segmentation mask is determined. As shown in equation (8):

[0120] (7),

[0121] (8).

[0122] Based on the intersection-over-union ratio (IoU), area change rate, confidence scores corresponding to reference segmentation masks for each of the multiple geometric cue points, confidence scores corresponding to the segmentation mask to be tested, and preset judgment rules, the detection result for the image to be detected can include: if the IoU is less than a preset low correlation threshold... ′ (i.e., IoU< ′), and the rate of change of area It does not meet the characteristics of significant positive growth or significant negative growth (i.e. , The threshold for significant negative growth. (A significant positive growth threshold) is used to determine whether the change is a random perturbation of the background texture or a registration residual, and it is automatically marked as an invalid detection and discarded.

[0123] If the intersection-union ratio (IoU) is greater than the preset low correlation threshold ′ (i.e., IoU> (′), the following situation exists:

[0124] If the confidence level corresponds to the reference segmentation mask Less than the preset threshold (Right now And the confidence level corresponding to the segmentation mask to be tested. Greater than or equal to a preset threshold (Right now ( ), which is determined to be completely new, corresponding to new diseases or artificial additives.

[0125] If the confidence level corresponds to the reference segmentation mask Greater than or equal to a preset threshold (Right now And the confidence level corresponding to the segmentation mask to be tested. Less than the preset threshold (Right now If the object is identified as missing, the corresponding component is missing or the attached material is removed.

[0126] If the confidence level corresponds to the reference segmentation mask Confidence level corresponding to the segmentation mask to be tested All are greater than or equal to the preset existence threshold. (Right now and ), and area change rate Greater than the preset significant positive growth threshold (Right now This indicates disease development, corresponding to the extension of cracks or the spread of mold.

[0127] If the confidence level corresponds to the reference segmentation mask Confidence level corresponding to the segmentation mask to be tested All are greater than or equal to the preset existence threshold. (Right now and ), and area change rate Less than the preset significant negative growth threshold (Right now The result indicates that the object has shrunk, which corresponds to the drying up of water or the pruning of plants.

[0128] Figure 5 A block diagram of a computer vision-based ancient building image change detection device according to an embodiment of the present invention is shown.

[0129] like Figure 5 As shown, the ancient building image change detection device 500 based on computer vision includes a generation module 510, an extraction module 520, an encoding and fusion module 530, a segmentation module 540, and a determination module 550.

[0130] The generation module 510 is used to generate a difference heatmap based on the differences between the reference image features of the reference image and the detection image features of the detection image. The reference image and the detection image are images of the target ancient building at different time points, and the time point corresponding to the reference image is earlier than that of the detection image.

[0131] The extraction module 520 is used to extract multiple geometric cue points based on the pixel values ​​of multiple pixels in the difference heat map. Each geometric cue point includes geometric positioning coordinates and semantic labels. The semantic labels are used to distinguish whether the geometric cue point is related to changes in ancient buildings or changes caused by environmental disturbances.

[0132] The encoding and fusion module 530 is used to encode the geometric positioning coordinates and semantic labels of the geometric cue points respectively, and fuse the encoded coordinates and encoded labels to obtain the cue features.

[0133] The segmentation module 540 is used to simultaneously segment the reference image and the image to be detected in the same physical coordinate system based on cue features, reference image features, and image to be detected features, to obtain a reference segmentation mask and its corresponding confidence level, and an image to be detected segmentation mask and its corresponding confidence level.

[0134] The determination module 550 is used to determine the detection result corresponding to the image to be detected based on the reference segmentation mask and corresponding confidence score corresponding to each of the multiple geometric cue points, the segmentation mask to be tested and corresponding confidence score.

[0135] According to embodiments of the present invention, the ancient building image change detection device based on computer vision further includes: a registration module, a normalization module, and a feature extraction module. The registration module performs registration processing on a reference image and a target image to obtain a registered reference image and a registered target image. The normalization module performs field-of-view normalization processing on the registered reference image and the registered target image to obtain a normalized reference image and a normalized target image. The feature extraction module extracts features unaffected by illumination changes from the normalized reference image and the normalized target image, respectively, to obtain reference image features and target image features.

[0136] According to embodiments of this disclosure, any multiple modules among the generation module 510, extraction module 520, encoding and fusion module 530, segmentation module 540, and determination module 550 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the generation module 510, extraction module 520, encoding and fusion module 530, segmentation module 540, and determination module 550 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the generation module 510, extraction module 520, encoding and fusion module 530, segmentation module 540 and determination module 550 may be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0137] It should be noted that the ancient building image change detection device based on computer vision in the embodiments of the present invention corresponds to the ancient building image change detection method based on computer vision in the embodiments of the present invention. For a detailed description of the ancient building image change detection device based on computer vision, please refer to the ancient building image change detection method based on computer vision, which will not be repeated here.

[0138] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0139] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for detecting changes in ancient building images based on computer vision, characterized in that, The method includes: A difference heatmap is generated based on the differences between the reference image features and the detection image features of the target ancient building. The reference image and the detection image are images of the target ancient building at different time points, and the time point corresponding to the reference image is earlier than that of the detection image. Based on the pixel values ​​of multiple pixels in the difference heatmap, multiple geometric cue points are extracted. Each geometric cue point includes geometric positioning coordinates and a semantic label. The semantic label is used to distinguish whether the geometric cue point is related to changes in ancient buildings or changes caused by environmental disturbances. The geometric coordinates and semantic labels of the geometric prompt points are encoded respectively, and the encoded coordinates and encoded labels are fused to obtain the prompt features; Using the twin decoding mechanism, under the same physical coordinate system, the reference image and the image to be detected are simultaneously segmented based on the prompt features, the reference image features, and the image to be detected to obtain a reference segmentation mask and its corresponding confidence, and a segmentation mask to be detected and its corresponding confidence. Based on the reference segmentation mask and corresponding confidence level corresponding to each of the multiple geometric cue points, and the segmentation mask to be tested and corresponding confidence level, the detection result corresponding to the image to be detected is determined.

2. The method according to claim 1, characterized in that, The method further includes: The reference image and the image to be detected are registered to obtain the registered reference image and the registered image to be detected. The registered reference image and the registered image to be detected are subjected to field-of-view normalization processing to obtain the normalized reference image and the normalized image to be detected. Features unaffected by illumination changes are extracted from the normalized reference image and the normalized image to be detected, respectively, to obtain the reference image features and the image to be detected features.

3. The method according to claim 2, characterized in that, The step of extracting features unaffected by illumination changes from the normalized reference image and the normalized image to be detected, respectively, to obtain the reference image features and the image to be detected features, includes: The normalized reference image or the normalized image to be detected is divided into image blocks of a predetermined size; Each image block is marked with location information to preserve the spatial structure of the normalized reference image or the normalized image to be detected, resulting in multiple processed image blocks; Multiple processed image blocks are input into a visual encoder to capture the dependencies between the processed image blocks and then output the reference image features or the image features to be detected.

4. The method according to claim 3, characterized in that, The step of performing field-of-view normalization processing on the registered reference image and the registered image to be detected to obtain normalized reference image and normalized image to be detected includes: The target regions containing the target ancient building in the registered reference image and the registered image to be detected are respectively cropped at the center region to obtain the cropped reference image and the cropped image to be detected. The cropped reference image and the cropped image to be detected are subjected to dimension normalization and numerical calibration to ensure that the spatial dimension and numerical distribution of the normalized reference image and the normalized image to be detected are consistent.

5. The method according to any one of claims 1 to 4, characterized in that, The geometric positioning coordinates include sub-pixel level floating-point coordinates. The extraction of multiple geometric cue points based on the pixel values ​​of multiple pixels in the difference heatmap includes: Based on the pixel values ​​of multiple pixels in the differential heatmap, a pixel distribution histogram is obtained, and a segmentation threshold is determined based on the pixel distribution histogram. Target pixels with pixel values ​​greater than the segmentation threshold are selected from the plurality of pixels, and the target response region of the difference heatmap is determined based on the target pixels; The target response region is subjected to maximum value filtering to extract the geometric cue points; Centered on the geometric prompt point, a set of neighboring pixels within a preset range is determined from the target response area; Using the target pixel value of each target pixel in the neighborhood pixel set as a weight, the integer pixel coordinates of each target pixel in the neighborhood pixel set are weighted and calculated to obtain the sub-pixel level floating-point coordinates corresponding to each geometric prompt point.

6. The method according to claim 1, characterized in that, The twin decoding mechanism consists of two mask decoders with identical structures and shared parameters; The method of utilizing twin decoding, under the same physical coordinate system, synchronously segments the reference image and the image to be detected based on the prompt features, the reference image features, and the image to be detected, to obtain a reference segmentation mask and its corresponding confidence level, and a segmentation mask to be detected and its corresponding confidence level, including: In the same physical coordinate system, the prompt features are concatenated with the learnable output labels to obtain fused prompt features; Perform self-attention aggregation on the fused prompt features, and use the fused prompt features as the query to map the reference image features or the image features to be detected as keys and values, and perform a prompt-image cross-attention operation to obtain the interactive image features; The output labels in the fused cue features after self-attention aggregation are input into the multilayer perceptron and the cross-union prediction multilayer perceptron, respectively, and the corresponding outputs are mask weight vector and mask confidence. The image features after the interaction are reshaped and the spatial resolution is upsampled by convolution upsampling operation to obtain the mask space features. The dot product operation is performed between the mask weight vector and the mask space features to generate multiple candidate mask features; Bilinear interpolation is used to restore the resolution of multiple candidate mask features, and after filtering by combining the mask confidence, the reference segmentation mask and its corresponding confidence, or the segmentation mask to be tested and its corresponding confidence, are obtained.

7. The method according to claim 6, characterized in that, The mask decoder is obtained from a trained semantic segmentation model, which includes an image encoder, a cue encoder, and a mask decoder. The training steps of the semantic segmentation model include: A training dataset is constructed, which includes multiple sample images and a ground truth mask corresponding to each sample image. The sample images contain ancient building components that have changed due to disease and ancient building components that have not been affected by disease. Based on multiple truth masks, multiple types of cue samples are generated, including foreground cue points and background cue points; Based on the image encoder, feature extraction and standardization are performed on the sample image to obtain sample image features; The prompt sample is input into the prompt encoder for encoding conversion processing to obtain sample prompt features that match the feature dimensions of the sample image; In the same physical coordinate system, the sample cue features are concatenated with the learnable sample output labels to obtain the sample fusion cue features; The sample fusion prompt features and the sample image features are input into the mask decoder, and feature interaction, transformation and upsampling processing are performed in sequence to output a pixel category probability map that matches the size of the sample image. The value of each pixel in the pixel category probability map indicates the confidence that the pixel belongs to the segmentation target category. Generate a predicted segmentation mask based on the pixel category probability map; Based on the predicted segmentation mask, the ground truth mask, and the pixel category probability map, a loss value is determined, and backpropagation is performed based on the loss value to update the model parameters of the image encoder, the cue encoder, and the mask decoder, thereby obtaining the trained semantic segmentation model.

8. The method according to claim 7, characterized in that, The construction of the training dataset includes: Collect multiple images of ancient building components that have changed due to disease and those that have not been affected by disease, from multiple time periods, multiple perspectives and under different lighting conditions; Pixel-level segmentation and annotation are performed on multiple images of ancient buildings to obtain the ground truth mask corresponding to the images of ancient buildings; By superimposing noise textures of different frequencies, performing color space transformation, or changing the image brightness on any one of the multiple ancient building images, multiple sample images are obtained to simulate the physical aging characteristics or light disturbances of the building surface. The training dataset is obtained by mapping multiple sample images one-to-one with the ground truth mask.

9. The method according to claim 7, characterized in that, The step of determining the loss value based on the predicted segmentation mask, the ground truth mask, and the pixel category probability map includes: Based on the pixel category probability map, a focal loss value is determined for calculating the pixel classification error; Based on the intersection area of ​​the predicted segmentation mask and the ground truth mask, and the relative ratio of the intersection area to the total area of ​​the predicted segmentation mask and the ground truth mask, the Dessian loss value used to calculate the geometric similarity is determined. Perform Euclidean distance transformation on the boundary of the truth mask to generate a distance field between the pixel and the boundary; The distance field is mapped to a penalty weight, which is then weighted with the confidence level corresponding to each pixel in the pixel category probability map to obtain the boundary loss value used to calculate the contour fitting accuracy. The loss value is obtained based on the focus loss value, the Dess loss value, and the boundary loss value.

10. The method according to claim 1, characterized in that, The step of determining the detection result corresponding to the image to be detected based on the reference segmentation mask and corresponding confidence score corresponding to each of the multiple geometric cue points, and the segmentation mask to be tested and corresponding confidence score, includes: Based on the reference segmentation mask and the segmentation mask to be tested corresponding to each of the multiple geometric cue points, the intersection-union ratio is determined. The intersection-union ratio characterizes the spatial overlap between the reference segmentation mask and the segmentation mask to be tested in the same physical coordinate system. Based on the area of ​​the reference segmentation mask and the area of ​​the segmentation mask to be tested, determine the rate of change of the area of ​​the segmentation mask to be tested relative to the reference segmentation mask; Based on the intersection-union ratio, the area change rate, the confidence level of the reference segmentation mask corresponding to each of the multiple geometric cue points, the confidence level of the segmentation mask to be tested, and the preset judgment rules, the detection result corresponding to the image to be detected is determined.

Images