An Instance-Level Change Detection Method and System with Collaborative Optimization of Profile Accuracy and Positioning Accuracy

Abstract

The present invention proposes an instance-level change detection method and system for collaborative optimization of contour accuracy and positioning accuracy. Among them, the method includes: making change detection labels for preprocessed satellite data; training a change detection model using a joint loss composed of pixel-level loss and instance-level loss with the significance of positioning accuracy constraint; adapting the instance-level evaluation index AP10 to the evaluation of change detection results; evaluating the trained change detection model using the adapted evaluation index AP10; and inputting the initial change patches and dual-temporal images obtained by the trained change detection model into the SAM change detection algorithm to obtain change patches. The solution proposed by the present invention can meet the dual requirements of high positioning accuracy and high contour accuracy, improve the integrity of large patches, reduce the false detection and missed detection of small patches, enhance the boundary fit degree between the detected area and the real change area, and improve the contour accuracy and positioning accuracy of change detection results in complex scenes.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing change detection, and particularly relates to an instance-level change detection method and system for collaborative optimization of contour accuracy and positioning accuracy. Background Art

[0002] Remote sensing change detection can help us understand ground objects at the geospatial scale and time scale, and explore the dynamic development process or future trend of ground objects under the influence of natural or human factors. It is widely used in fields such as land use change investigation (Li Qimei, 2022), disaster damage assessment after disasters (Ma Jianwen et al., 2004), and urban development planning (Peng Shunxi, 2007). Currently, a large number of remote sensing change detection applications (such as illegal construction inspection, ecological reserve monitoring, etc.) require the algorithm to be able to accurately detect changes and obtain fine change patches, which means that change detection algorithms with high positioning accuracy and contour accuracy are important requirements for the current algorithm to be applied. Among them, the contour accuracy represents the degree of fit between the patch contour and the boundary of the actual change area, that is, to evaluate the fineness of the contour. The positioning accuracy represents the degree of intersection between the patch and the real change area in space, that is, to evaluate the false detection and missed detection of the patch. Figure 2 Shows some evaluation scenarios of positioning accuracy and contour accuracy, where the blue patches are the outputs of the change detection algorithm, and the red patches are the change labels.

[0003] Change detection has gone through a long development process. With the advent of the big data era in remote sensing (Zhang Bing, 2017, Zhang Bing et al., 2022), deep learning change detection algorithms have made great progress. A large number of research results show that in the change detection task of large-scale high-resolution remote sensing images, the performance of deep learning methods is better than traditional pixel-based and object-based change detection algorithms (Wang Yihao, 2024, Hou et al., 2021, Lukang Wang et al., 2024). Traditional methods (such as vector analysis methods) are difficult to process complex texture information in high-resolution images, and excessive manual intervention results in insufficient generalization ability of the model on a large scale. In contrast, data-driven deep learning change detection algorithms bring less manual intervention, higher accuracy, stronger robustness, and better generalization ability, and gradually become a huge breakthrough for change detection to carry out application services. In deep learning change detection algorithms, according to the granularity of the detection results, the algorithms can be divided into three types: scene-level change detection (SLCD), object-level change detection (OLCD), and pixel-level change detection (PLCD). Schematic diagrams of different detection results are shown in Figure 3As shown. Among them, SLCD and OLCD lack contour accuracy in the detection granularity, and it is difficult to detect the changes of complex structures (such as roads, rivers, construction sites, etc.), so it is difficult to be applied to high-precision change detection in large-scale high-resolution scenarios. PLCD judges whether each pixel changes or not, and at the same time realizes the positioning of the changed area and the extraction of fine contours, and has achieved relatively good results. However, there is currently a problem of lack of positioning accuracy constraints in PLCD. Even if high contour accuracy can be obtained, the problems of missed detection and false detection of small patches are still serious. As Figure 2 shown, although PLCD has obtained fine detection results for obvious large changed patches, it has missed a large number of small patches, resulting in low positioning accuracy. The lack of positioning accuracy constraints in PLCD is largely due to the lack of attention to small patches in the loss calculation and evaluation indicators. PLCD mainly constrains model training through pixel-level loss functions (such as binary cross-entropy loss, dice loss), and evaluates model accuracy with pixel-level evaluation indicators (such as kappa coefficient, F1 score). Among them, all loss calculations and accuracy evaluations are completed pixel by pixel. Whether it is a large patch pixel or a small patch pixel, they have the same attention during the calculation process. This also leads to a greater optimization force for larger patches with more obvious and more pixels, while small patches are often ignored. In addition, the high pixel-level accuracy evaluation indicators cannot effectively indicate the detection accuracy of the changed area (i.e., positioning accuracy). For example, when a large number of small patches are missed or misdetected, the current pixel-level accuracy evaluation indicators will still show high accuracy values due to the influence of larger patches with more pixels, which will lead to a gradual deviation of the optimization direction during the model training process.

[0004] Disadvantages of the prior art

[0005] Currently, there is still a lack of a change detection model that combines high contour accuracy and positioning accuracy. The research intends to carry out research on remote sensing image change detection algorithms that couple instance-level constraints and vision large models. Based on the mechanism of the problem, this project intends to construct a change detection loss function from the scale of patch instances to improve the patch positioning ability of the change detection algorithm, and further improve the integrity and accuracy of patches based on the vision large model, and jointly optimize the positioning accuracy and contour accuracy of the changed patches. Finally, this project will form a set of change detection algorithms that combine high contour accuracy and positioning accuracy to meet the actual needs of current large-scale change detection tasks and improve the application implementation ability of deep learning change detection algorithms. Summary of the invention

[0006] To solve the above technical problems, the present invention proposes a technical solution for an instance-level change detection method that co-optimizes contour accuracy and positioning accuracy. To solve the above technical problems, the concept of "instance" is introduced to construct an instance-level change detection algorithm. The comparison of instance-level change detection results with those of SLCD, OLCD, and PLCD is as Figure 3 shown. The concept of "instance" comes from the field of computer vision and represents the set of all pixels belonging to the same ground object. Introducing this concept into change detection, different change patches can be regarded as different instances, and each instance is an equal individual with a clear position and fine contour (such as Figure 3 in the 6th column, pixels of different colors belong to different instances). The missed detection and misdetection of large and small patches can be regarded as the missed detection and misdetection of an instance. Instance-level change detection is beneficial for paying attention to the detection of small patches, and thus better meets the requirements of high positioning accuracy and high contour accuracy in current remote sensing change detection applications. The instance-level change detection algorithm is based on PLCD. Aiming at the problem that small patches are lacking attention in the calculation of the loss function and accuracy evaluation in PLCD, an instance-level loss function and an instance-level accuracy evaluation index are constructed to increase the penalty for missed detection and misdetection of small patches and improve the positioning accuracy of the change detection result patches. At the same time, to avoid over-optimizing the positioning accuracy while ignoring the contour accuracy, the project plans to migrate the large deep learning segmentation model SAM (Kirillov et al., 2023) to the change detection task. By combining the change patch hints with high positioning accuracy and the low-cost fine contour acquisition ability of SAM, a bridge for co-optimizing positioning accuracy and contour accuracy is built to further improve the accuracy of the instance-level change detection results.

[0007] The first aspect of the present invention discloses an instance-level change detection method for co-optimizing contour accuracy and positioning accuracy, and the method includes:

[0008] Step S1: Collect high-resolution satellite data, and preprocess the satellite data; make change detection labels for the preprocessed satellite data; divide the satellite data and the corresponding labels into a training set and a validation set;

[0009] Step S2: Apply the training set and train a change detection model using a joint loss composed of pixel-level loss and instance-level loss with the significance of positioning accuracy constraint;

[0010] Step S3: Adapt the instance-level evaluation index AP10 to the evaluation of the change detection results; apply the adapted evaluation index AP10 to evaluate the trained change detection model using the validation set;

[0011] Step S4: Input the initial change patches obtained by the trained change detection model and the bi-temporal images into the SAM change detection algorithm to obtain change patches.

[0012] According to the method of the first aspect of the present invention, in the step S1, the operation of generating change detection labels for the preprocessed satellite data includes:

[0013] Combining the front and back phase images, using ArcGIS as a sample marking tool, distinguishing and outlining the changed patches; taking the changed patches as samples with change detection labels; the marked changed patches target the land cover changes of cultivated land, forest land, orchard land, grassland, water bodies, roads, residential areas, greenhouse greenhouses, and construction areas outside the built-up area; the samples require marked changed patches with an area greater than 150 square meters.

[0014] According to the method of the first aspect of the present invention, in the step S2, the combined loss composed of the pixel-level loss and the instance-level loss with the significance of positioning accuracy constraint is:

[0015] loss = l changt + l inst

[0016]

[0017] where loss represents the combined loss; l change represents the pixel-level loss function calculated based on dice; l inst represents the instance loss function; L inst,i represents the prediction result; P inst,i represents the label of the sample; dice() represents the dice calculation; N represents the total number of instances.

[0018] According to the method of the first aspect of the present invention, in the step S3, the adaptation of the instance-level evaluation index AP10 to the evaluation of the change detection result includes:

[0019] AP10 distributes the accuracy evaluation of the entire image to each changed patch instance, calculates the IoU between the predicted patch and the true patch with the instance as the unit, regards the patch with IoU greater than 0.1 as a correctly predicted patch, that is, TP; regards the patch with IoU less than 0.1 as a misdetected patch, that is, FP; regards the true patch not detected as a missed detection patch, that is, FN; if there are multiple predicted patches satisfying IoU greater than 0.1 for the same true value, then the one with the highest confidence is TP, and the others are all FP. [[ID=�6]]

[0020] According to the method of the first aspect of the present invention, in the step S4, the operation of inputting the initial change map and the dual-temporal images obtained by the trained change detection model into the SAM change detection algorithm to obtain the change map includes:

[0021] Based on the trained change detection model, an initial change pattern is obtained, and prompt points of the change area are generated according to the initial change pattern; the prompt points pass through the SAM prompt encoder to obtain prompt features with change area location information; the dual-phase image is input into the SAM image encoder and encoded separately to obtain dual-phase features; the dual-phase features are fused through the feature fusion module to obtain change features; the change features and prompt features are input into the SAM decoder together to output the change pattern.

[0022] According to the method of the first aspect of the present invention, in step S4, the feature fusion module fuses the bi-temporal features based on the cross-layer cross attention module to obtain the change feature;

[0023] The cross-layer cross attention module inputs the previous phase features and the subsequent phase features into two cross-layer self-attention modules respectively, takes the previous phase features as the query vector, and the subsequent phase features as the key vector and the value vector, and sends them to the first cross-layer self-attention module. The output result vector of the first cross-layer self-attention module and the previous phase feature vector are combined and sent to the first residual connection layer normalization. The output result of the first residual connection layer normalization is sent to the first feedforward neural network, and then the output result of the first feedforward neural network and the output result of the first residual connection layer normalization are combined and input into the second residual connection normalization. , the output result of the second residual link normalization is used as the query vector, the posterior phase feature is used as the key vector and the value vector combination and input into the second cross-layer self-attention module, the output result of the second residual link normalization and the output result of the second cross-layer self-attention module are combined and input into the third residual connection normalization, the output result of the third residual connection normalization is sent to the second feedforward neural network, the output result of the second feedforward neural network and the output result of the third residual link layer normalization are combined and input into the fourth residual connection layer normalization, and the output result of the fourth residual connection layer normalization is the obtained change feature.

[0024] According to the method of the first aspect of the present invention, in step S4, generating prompt points of the change area according to the initial change pattern includes:

[0025] For the initial change patch, different instance patch regions are obtained by image connected domain calculation;

[0026] For each of the example pattern spots, dividing the example pattern spots into regions using a grid of a predefined size;

[0027] The central pixel of each grid is selected as a cue point, and the foreground / background category to which the central pixel belongs represents the category of the cue point.

[0028] The second aspect of the present invention discloses an instance-level change detection system for collaborative optimization of contour accuracy and positioning accuracy. The system includes:

[0029] A first processing module configured to collect high-resolution satellite data, preprocess the satellite data, generate change detection labels for the preprocessed satellite data, and divide the satellite data and corresponding labels into a training set and a validation set.

[0030] A second processing module configured to apply the training set to train a change detection model using a joint loss composed of a pixel-level loss and an instance-level loss with the significance of positioning accuracy constraint.

[0031] A third processing module configured to adapt the instance-level evaluation metric AP10 to the evaluation of change detection results, and use the adapted evaluation metric AP10 to evaluate the trained change detection model using the validation set.

[0032] A fourth processing module configured to input the initial change patches and the bi-temporal images obtained by the trained change detection model into the SAM change detection algorithm to obtain change patches.

[0033] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor. When the processor executes the computer program stored in the memory, the steps in an instance-level change detection method for collaborative optimization of contour accuracy and positioning accuracy according to any one of the first aspects of the present disclosure are implemented.

[0034] The fourth aspect of the present invention discloses a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps in an instance-level change detection method for collaborative optimization of contour accuracy and positioning accuracy according to any one of the first aspects of the present disclosure are implemented.

[0035] In summary, the solution proposed by the present invention can meet the dual requirements of high positioning accuracy and high contour accuracy, improve the integrity of large patches, reduce false detections and missed detections of small patches, enhance the boundary fit between the detected area and the true change area, and improve the contour accuracy and positioning accuracy of change detection results in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1Flowchart of an instance-level change detection method for collaborative optimization of contour accuracy and positioning accuracy according to an embodiment of the present invention;

[0038] Figure 2 Partial positioning accuracy and contour accuracy evaluation scenarios according to the background art of the present invention;

[0039] Figure 3 Schematic diagram of change detection results with different granularities according to the background art of the present invention;

[0040] Figure 4 Technical roadmap according to an embodiment of the present invention;

[0041] Figure 5 Schematic diagram of the image processing flow according to an embodiment of the present invention;

[0042] Figure 6 Schematic diagram of instance-level loss function calculation according to an embodiment of the present invention;

[0043] Figure 7 AP10 evaluation schematic diagram according to an embodiment of the present invention;

[0044] Figure 8 SAM change detection framework integrating remote sensing cues according to an embodiment of the present invention;

[0045] Figure 9 Change feature acquisition module based on cross-layer cross-attention according to an embodiment of the present invention;

[0046] Figure 10 Schematic diagram of a change detection cue generation strategy for SAM according to an embodiment of the present invention;

[0047] Figure 11 Structure diagram of an instance-level change detection system for collaborative optimization of contour accuracy and positioning accuracy according to an embodiment of the present invention;

[0048] Figure 12 Structure diagram of an electronic device according to an embodiment of the present invention. Detailed implementation manners

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0050] The first aspect of the present invention discloses an instance-level change detection method for collaborative optimization of contour accuracy and positioning accuracy. Figure 1 FIG. is a flowchart of an instance-level change detection method for collaborative optimization of contour accuracy and positioning accuracy according to an embodiment of the present invention, as Figure 1 and Figure 4 shown, the method includes:

[0051] Step S1, collect high-resolution satellite data, and preprocess the satellite data; make change detection labels for the preprocessed satellite data; divide the satellite data and the corresponding labels into a training set and a validation set;

[0052] Step S2, apply the training set, and train a change detection model using a joint loss composed of a pixel-level loss and an instance-level loss with the significance of positioning accuracy constraint;

[0053] Step S3, adapt the instance-level evaluation index AP10 to the evaluation of the change detection result; apply the validation set, and evaluate the trained change detection model using the adapted evaluation index AP10;

[0054] Step S4, input the initial change patches and the bi-temporal images obtained by the trained change detection model into the SAM change detection algorithm to obtain change patches.

[0055] In step S1, collect high-resolution satellite data, and preprocess the satellite data; make change detection labels for the preprocessed satellite data; divide the satellite data and the corresponding labels into a training set and a validation set.

[0056] In some embodiments, in step S1, the making of change detection labels for the preprocessed satellite data includes:

[0057] Combine the front and back temporal images, use ArcGIS as a sample marking tool, discriminate and outline the change patches; use the change patches as samples with change detection labels; the marked change patches are for the land cover changes of cultivated land, forest land, orchard land, grassland, water body, road, residential area, greenhouse and construction area outside the built-up area; the sample requires that the marked area of the change patch is greater than 150 square meters.

[0058] Specifically, collecting high-resolution satellite data and preprocessing the satellite data includes: carrying out research on land cover changes in the Guangdong-Hong Kong-Macao Greater Bay Area, selecting domestic high-resolution satellite data from 2023 to 2024 for the image data, including satellites such as GF1, GF2, GF6, and ZY3. The images undergo strict data preprocessing operations, are processed into a 2-meter resolution, and only the RGB channels are retained. The image processing process is as Figure 5 shown.

[0059] In step S2, the training set is applied to train the change detection model using a combined loss composed of a pixel-level loss and an instance-level loss with significance for localization accuracy constraints.

[0060] In some embodiments, in the step S2, the combined loss composed of the pixel-level loss and the instance-level loss with significance for localization accuracy constraints is:

[0061] loss = l change +l inst

[0062]

[0063] where loss represents the combined loss; l change represents the pixel-level loss function calculated based on dice; l inst represents the instance loss function; L inst,i represents the prediction result; P inst,i represents the label of the sample; dice() represents the dice calculation; N represents the total number of instances.

[0064] Specifically, the instance-level loss function and the instance-level accuracy evaluation index are used to constrain the localization accuracy of the model. Both algorithms are decoupled from the change detection network and can be embedded into any change detection model, such as ChangeMamba (Chen et al., 2024), SNUNet-CD (Fang et al., 2021). Therefore, it is planned to optimize the localization accuracy of the model based on the pixel-level change detection network FCCDN (Chen et al., 2022b) as the benchmark structure and embed the instance-level constraints.

[0065] Currently, the mainstream constraint strategies for network learning still remain at the pixel-level constraints, lacking optimization for the localization accuracy of change patches, resulting in serious misdetection and missed detection of small patches, which also limits the practical application ability of the algorithm. Therefore, the project will design an instance-level change patch constraint algorithm to add constraints on the localization accuracy of each patch while retaining a certain contour accuracy, improving the ability of the change detection model to locate change patches.

[0066] The overall schematic diagram of the calculation of the instance-level loss function adopted in this embodiment is as Figure 6 shown. To help the network better optimize, a combined loss composed of a relatively easy-to-optimize pixel-level loss and an instance-level loss with significance for localization accuracy constraints is adopted. The instance-level constraint obtains the patch instances of the result and the label through connected component extraction, calculates the dice for the corresponding label instances and the result instances in sequence and accumulates them to obtain the instance-level loss function of the entire image. The final loss function can be expressed as follows:

[0067] loss = lchange +l inst

[0068]

[0069] Among them, loss represents the combined loss; l change represents the pixel-level loss function calculated based on dice; l inst represents the instance loss function; L inst,i represents the prediction result; P inst,i represents the label of the sample; dice() represents the dice calculation; N represents the total number of instances.

[0070] In order to improve the calculation efficiency of the loss function and avoid the connected component calculation operation during the loss calculation process, it is proposed to calculate the connected components of the label in advance and not calculate the connected components of the real-time output result to improve the efficiency. During the loss calculation process, directly use the bounding rectangle of each changed instance in the label to enclose the loss calculation range, and calculate only the dice loss between the label and the prediction result within this range. In theory, this processing method is equivalent to directly calculating the one-to-one loss between instances, and this processing can also avoid the problem of instance matching.

[0071] In step S3, adapt the instance-level evaluation metric AP10 to the evaluation of the change detection result; apply the validation set and evaluate the trained change detection model using the adapted evaluation metric AP10.

[0072] In some embodiments, in the step S3, the adapting the instance-level evaluation metric AP10 to the evaluation of the change detection result includes:

[0073] AP10 assigns the accuracy evaluation of the entire image to each changed patch instance, calculates the IoU between the predicted patch and the true patch in units of the instance, regards the patch with IoU greater than 0.1 as a correctly predicted patch, that is, TP; regards the patch with IoU less than 0.1 as a false detection patch, that is, FP; regards the true patch not detected as a missed detection patch, that is, FN; if there are multiple predicted patches with IoU greater than 0.1 for the same true value, then the one with the highest confidence is TP, and the others are all FP.

[0074] Specifically, existing change detection algorithms basically use pixel-level metrics (IoU, kappa, etc.) to evaluate the model accuracy. However, this per-pixel evaluation method does not have the ability to evaluate whether the model can accurately locate the change area. In order to better evaluate the evaluation ability of the model for the patch localization accuracy, this embodiment adapts the instance-level evaluation metric AP10 to the evaluation of the change detection result. Generate instances using the change detection result and complete the evaluation based on AP10.

[0075] The detailed schematic diagram is asFigure 7 As shown, simple rectangular boxes are used to represent change instances, with predicted patches P1, P2, P3 and ground truth patches G1, G2. Among them, P3 does not meet the condition that the IoU with the ground truth is greater than 0.1 and is regarded as FP. The IoU of both P1 and P2 is greater than 0.1, but P2 has a higher confidence, so P1 is FP and P2 is TP. Since G2 is not detected, it is regarded as FN. Finally, precision and recall are calculated based on TP, FP, and FN of patch instances:

[0076]

[0077] Samples different precisions according to the recall value, and takes the average value of all sampled precisions as AP10. AP10 evaluates the change detection accuracy with patches as the basic unit, and at the same time takes into account the evaluation of the localization accuracy and contour accuracy of predicted patches. The evaluation results meet the requirements in a large number of change detection engineering applications.

[0078] In step S4, the initial change patches and the bi-temporal images obtained by the trained change detection model are input into the SAM change detection algorithm to obtain change patches.

[0079] In some embodiments, in the step S4, the inputting the initial change patches and the bi-temporal images obtained by the trained change detection model into the SAM change detection algorithm to obtain change patches includes:

[0080] Based on the trained change detection model, initial change patches are obtained, and cue points of the change region are generated according to the initial change patches; the cue points pass through the cue encoder of SAM to obtain cue features with the position information of the change region; the bi-temporal images are respectively encoded by the image encoder of SAM to obtain bi-temporal features; the bi-temporal features are fused by a feature fusion module to obtain change features; the change features and the cue features are jointly input into the decoder of SAM to output change patches.

[0081] The feature fusion module fuses the bi-temporal features based on a cross-layer cross-attention module to obtain change features;

[0082] Such as Figure 9As shown, the cross-layer cross-attention module inputs the features of the previous time phase and the features of the subsequent time phase into two cross-layer self-attention modules respectively. The features of the previous time phase are used as the query vector, and at the same time, the features of the subsequent time phase are used as the key vector and the value vector, and are fed into the first cross-layer self-attention module. The output result vector of the first cross-layer self-attention module and the feature vector of the previous time phase are merged and fed into the first residual connection layer normalization. The output result of the first residual connection layer normalization is fed into the first feed-forward neural network. Then, the output result of the first feed-forward neural network and the output result of the first residual connection layer normalization are combined and input into the second residual link normalization. The output result of the second residual link normalization is used as the query vector, and the features of the subsequent time phase are used as the key vector and the value vector and are combined and input into the second cross-layer self-attention module. The output result of the second residual link normalization and the output result of the second cross-layer self-attention module are combined and input into the third residual connection layer normalization. The output result of the third residual connection layer normalization is fed into the second feed-forward neural network. The output result of the second feed-forward neural network and the output result of the third residual link layer normalization are combined and input into the fourth residual connection layer normalization. The output result of the fourth residual connection layer normalization is the obtained change feature.

[0083] The generation of the hint points of the change region according to the initial change patches includes:

[0084] For the initial change patches, different instance patch regions are obtained through image connected component calculation;

[0085] For each instance patch region, the instance patch region is divided using a grid of a predefined size;

[0086] The center pixel of each grid is selected as the hint point, and the foreground / background category to which the center pixel belongs represents the category of the hint point.

[0087] Specifically, the SAM change detection algorithm framework integrating remote sensing hints: Benefiting from the training of a large amount of image data, SAM shows strong segmentation ability for natural scene images and can complete the contour extraction of ground objects in remote sensing images based on simple point hints. Therefore, in this embodiment, SAM is proposed to cooperate with the output result of the change detection model to obtain accurate change patches and compensate for the lost contour accuracy in the process of optimizing the patch positioning accuracy.

[0088] The adopted SAM change detection framework integrating remote sensing hints is as Figure 8As shown in the figure. Based on the trained change detection model, initial change patches are obtained, and hint points of the change region are generated according to the initial change patches; the hint points pass through the hint encoder of SAM to obtain hint features with the position information of the change region; the dual-temporal images are respectively encoded by the image encoder of SAM to obtain dual-temporal features; the dual-temporal features are fused through a feature fusion module to obtain change features; the change features and the hint features are jointly input into the decoder of SAM to output change patches.

[0089] SAM is designed for single-temporal tasks and cannot handle dual-temporal change detection tasks. Therefore, a change feature extraction module for dual-temporal SAM image features is designed to convert the decoding of SAM into a single-temporal task. This part is the only trainable module in the entire framework. It is both a fusion module for dual-temporal features and can be regarded as an adapter (Adapter) for remote sensing data (Houlsby et al., 2019, Pfeiffer et al., 2020). The feature fusion module fuses dual-temporal features based on a cross-layer cross-attention module to obtain change features.

[0090] SAM supports multiple hint inputs, including points, rectangular boxes, texts, and masks. Among them, the rectangular boxes and texts do not match the output of the change detection model. At the same time, considering that the contour accuracy of the initial change patches output by the model is not high, it is difficult to directly use the mask information of the patches as the hint information of SAM. In this embodiment, a point hint method is proposed to assist SAM in obtaining patches. The method for obtaining SAM hint points is as Figure 10 shown. The specific process includes:

[0091] For the initial change patches, different instance patch regions are obtained through image connected component calculation;

[0092] For each instance patch region, the instance patch region is divided using a grid with a predefined size;

[0093] The center pixel of each grid is selected as a hint point, and the foreground / background category to which the center pixel belongs represents the category of the hint point.

[0094] In summary, the solution proposed by the present invention can meet the dual requirements of high positioning accuracy and high contour accuracy, improve the integrity of large patches, reduce the misdetection and missed detection of small patches, enhance the boundary fit between the detected region and the real change region, and improve the contour accuracy and positioning accuracy of the change detection results in complex scenarios.

[0095] The second aspect of the present invention discloses an instance-level change detection system for collaborative optimization of contour accuracy and positioning accuracy. Figure 11 is a structural diagram of an instance-level change detection system for collaborative optimization of contour accuracy and positioning accuracy according to an embodiment of the present invention; asFigure 11 As shown, the system 100 includes:

[0096] A first processing module 101, configured to collect high-resolution satellite data, preprocess the satellite data; generate change detection labels for the preprocessed satellite data; divide the satellite data and the corresponding labels into a training set and a validation set;

[0097] A second processing module 102, configured to apply the training set and train a change detection model using a combined loss composed of pixel-level loss and instance-level loss with positioning accuracy constraint significance;

[0098] A third processing module 103, configured to adapt the instance-level evaluation metric AP10 to the evaluation of change detection results; apply the adapted evaluation metric AP10 to evaluate the trained change detection model using the validation set;

[0099] A fourth processing module 104, configured to input the initial change patches and bi-temporal images obtained by the trained change detection model into the SAM change detection algorithm to obtain change patches.

[0100] For the system according to the second aspect of the present invention, the first processing module 101 is specifically configured to, the generating change detection labels for the preprocessed satellite data includes:

[0101] Combining pre- and post-temporal images, using ArcGIS as a sample marking tool, discriminating and outlining change patches; using the change patches as samples with change detection labels; the marked change patches are for the land cover changes of cultivated land, forest land, orchard land, grassland, water bodies, roads, residential areas, greenhouse greenhouses, and construction areas outside the built-up area; the sample requires marking change patches with an area greater than 150 square meters.

[0102] Specifically, collecting high-resolution satellite data and preprocessing the satellite data includes: Conducting land cover change research for the Guangdong-Hong Kong-Macao Greater Bay Area, selecting domestic high-resolution satellite data from 2023 to 2024 for image data, including satellites such as GF1, GF2, GF6, and ZY3. The images undergo strict data preprocessing operations, processed into a 2-meter resolution, and only the RGB channels are retained. The image processing process is as Figure 5 shown.

[0103] For the system according to the second aspect of the present invention, the second processing module 102 is specifically configured to, the combined loss composed of pixel-level loss and instance-level loss with positioning accuracy constraint significance is:

[0104] loss = l change + l inst

[0105]

[0106] Among them, loss represents the combined loss; l change represents the pixel-level loss function calculated based on dice; l inst represents the instance loss function; L inst,i represents the prediction result; P inst,i represents the label of the sample; dice() represents the dice calculation; N represents the total number of instances.

[0107] Specifically, an instance-level loss function and an instance-level accuracy evaluation index are used to constrain the positioning accuracy of the model. Both algorithms are decoupled from the change detection network and can be embedded into any change detection model, such as ChangeMamba (Chen et al., 2024), SNUNet-CD (Fang et al., 2021). Therefore, it is planned to optimize the positioning accuracy of the model based on the pixel-level change detection network FCCDN (Chen et al., 2022b) as the benchmark structure and embed instance-level constraints.

[0108] Currently, the mainstream constraint strategies for network learning still remain at the pixel-level constraint, lacking optimization for the positioning accuracy of change patches, resulting in serious misdetection and missed detection of small patches, which also limits the practical application ability of the algorithm. Therefore, the project will design an instance-level change patch constraint algorithm to add constraints on the positioning accuracy of each patch while retaining a certain contour accuracy, improving the ability of the change detection model to locate change patches.

[0109] The overall schematic diagram of the instance-level loss function calculation adopted in this embodiment is as Figure 6 shown. To help the network better optimize, a combined loss composed of a relatively easy-to-optimize pixel-level loss and an instance-level loss with positioning accuracy constraints is adopted. For the instance-level constraint, the patch instances of the result and the label are obtained through connected component extraction, and the dice is calculated and accumulated for the corresponding label instances and result instances in sequence to obtain the instance-level loss function of the entire image. The final loss function can be expressed as follows:

[0110] loss = l change + l inst

[0111]

[0112] Among them, loss represents the combined loss; l change represents the pixel-level loss function calculated based on dice; l inst represents the instance loss function; L inst,i represents the prediction result; P inst,iThe label representing the sample; dice() represents the dice calculation; N represents the total number of instances.

[0113] To improve the calculation efficiency of the loss function and avoid the connected component calculation operation during the loss calculation, it is proposed to calculate the connected components of the label in advance and not calculate the connected components of the real-time output result to improve the efficiency. During the loss calculation process, directly use the bounding rectangle of each changed instance in the label to enclose the loss calculation range, and calculate only the dice loss between the label and the prediction result within this range. In theory, this processing method is equivalent to directly calculating the one-to-one loss between instances, and this processing can also avoid the problem of instance matching.

[0114] According to the system of the second aspect of the present invention, the third processing module 103 is specifically configured that the adaptation of the instance-level evaluation index AP10 to the change detection result evaluation includes:

[0115] AP10 assigns the accuracy evaluation of the entire image to each changed patch instance, calculates the IoU between the predicted patch and the ground truth patch in units of the instance, regards the patch with IoU greater than 0.1 as a correctly predicted patch, that is, TP; regards the patch with IoU less than 0.1 as a false detection patch, that is, FP; regards the undetected ground truth patch as a missed detection patch, that is, FN; if there are multiple predicted patches with IoU greater than 0.1 for the same ground truth, then the one with the highest confidence is TP, and the others are FP.

[0116] Specifically, existing change detection algorithms basically use pixel-level indicators (IoU, kappa, etc.) to evaluate the model accuracy. However, this per-pixel evaluation method does not have the ability to evaluate whether the model can accurately locate the change area. To better evaluate the evaluation ability of the model for the patch localization accuracy, this embodiment adapts the instance-level evaluation index AP10 to the change detection result evaluation. Generate instances using the change detection result and complete the evaluation based on AP10.

[0117] The detailed schematic diagram is as Figure 7 shown. Represent the changed instances with simple rectangular frames. There are predicted patches P1, P2, P3 and ground truth patches G1, G2. Among them, P3 does not meet the condition that the IoU with the ground truth is greater than 0.1 and is regarded as FP. The IoUs of P1 and P2 are both greater than 0.1, but the confidence of P2 is higher. Therefore, P1 is FP and P2 is TP. Since G2 is not detected, it is regarded as FN. Finally, calculate precision and recall based on TP, FP, FN of the patch instances:

[0118]

[0119] Sample different precisions according to the recall value, and take the average of all sampled precisions as AP10. AP10 evaluates the change detection accuracy with patches as the basic unit, and at the same time takes into account the evaluation of the positioning accuracy and contour accuracy of the predicted patches. The evaluation results meet the requirements in a large number of change detection engineering applications.

[0120] According to the system of the second aspect of the present invention, the fourth processing module 104 is specifically configured to input the initial change patches and the bi-temporal images obtained by the trained change detection model into the SAM change detection algorithm, and the obtained change patches include:

[0121] Obtain initial change patches based on the trained change detection model, and generate prompt points for the change area according to the initial change patches; the prompt points pass through the prompt encoder of SAM to obtain prompt features with the position information of the change area; input the bi-temporal images into the image encoder of SAM for encoding respectively to obtain bi-temporal features; fuse the bi-temporal features through the feature fusion module to obtain change features; the change features and the prompt features are jointly input into the decoder of SAM to output change patches.

[0122] The feature fusion module fuses the bi-temporal features based on the cross-layer cross-attention module to obtain change features;

[0123] As Figure 9 shown, the cross-layer cross-attention module inputs the pre-temporal phase features and the post-temporal phase features into two cross-layer self-attention modules respectively. The pre-temporal phase features are used as query vectors, and at the same time the post-temporal phase features are used as key vectors and value vectors and sent into the first cross-layer self-attention module. The output result vector of the first cross-layer self-attention module and the pre-temporal phase feature vector are merged and sent into the first residual connection layer for normalization. The output result of the first residual connection layer normalization is sent into the first feed-forward neural network. Then, the output result of the first feed-forward neural network and the output result of the first residual connection layer normalization are combined and input into the second residual link normalization. The output result of the second residual link normalization is used as the query vector, and the post-temporal phase features are used as the key vector and value vector combination and input into the second cross-layer self-attention module. The output result of the second residual link normalization and the output result of the second cross-layer self-attention module are combined and input into the third residual connection layer normalization. The output result of the third residual connection layer normalization is sent into the second feed-forward neural network. The output result of the second feed-forward neural network and the output result of the third residual link layer normalization are combined and input into the fourth residual connection layer normalization. The output result of the fourth residual connection layer normalization is the obtained change feature.

[0124] The generating of the prompt points for the change area according to the initial change patches includes:

[0125] For the initial changed patch, different instance patch regions are obtained through image connected component calculation;

[0126] For each of the instance patch regions, the instance patch region is divided using a grid of predefined size;

[0127] Each grid center pixel is selected as a hint point, and the foreground / background category to which the center pixel belongs represents the category of the hint point.

[0128] Specifically, the SAM change detection algorithm framework integrating remote sensing hints: Benefiting from the training with a large amount of image data, SAM shows strong segmentation ability for natural scene images and can complete the contour extraction of ground objects in remote sensing images based on simple point hints. Therefore, in this embodiment, SAM is proposed to cooperate with the output result of the change detection model to obtain accurate changed patches and compensate for the lost part of the contour accuracy in the process of optimizing the patch positioning accuracy.

[0129] The adopted SAM change detection framework integrating remote sensing hints is as Figure 8 shown. Based on the trained change detection model, initial changed patches are obtained, and hint points of the changed region are generated according to the initial changed patches; the hint points pass through the hint encoder of SAM to obtain hint features with the position information of the changed region; the dual-temporal images are respectively encoded by the image encoder of SAM to obtain dual-temporal features; the dual-temporal features are fused through a feature fusion module to obtain change features; the change features and the hint features are jointly input into the decoder of SAM to output changed patches.

[0130] SAM is designed for single-temporal tasks and cannot handle dual-temporal change detection tasks. Therefore, a change feature extraction module for dual-temporal SAM image features is designed to convert the decoding of SAM into a single-temporal task. This part is the only trainable module in the whole framework. It is both a fusion module for dual-temporal features and can be regarded as an adapter (Adapter) for remote sensing data (Houlsby et al., 2019, Pfeiffer et al., 2020). The feature fusion module fuses the dual-temporal features based on a cross-layer cross-attention module to obtain change features.

[0131] SAM supports multiple hint inputs, including points, rectangular boxes, texts, and masks, among which the rectangular boxes and texts do not match the output of the change detection model. At the same time, considering that the contour accuracy of the initial changed patches output by the model is not high and it is difficult to directly use the mask information of the patches as the hint information of SAM, this embodiment proposes to use the point hint method to assist SAM in obtaining patches. The method for obtaining SAM hint points is as Figure 10 shown. The specific process includes:

[0132] For the initial changed patches, different instance patch regions are obtained through image connected component calculation.

[0133] For each of the instance patch regions, the instance patch region is divided using a grid of a predefined size.

[0134] Each grid center pixel is selected as a hint point, and the foreground / background category to which the center pixel belongs represents the category of the hint point.

[0135] A third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in an instance-level change detection method for co-optimizing contour accuracy and positioning accuracy according to any one of the first aspects disclosed in the present invention are implemented.

[0136] Figure 12 FIG. is a structural diagram of an electronic device according to an embodiment of the present invention. As Figure 12 shown, the electronic device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a carrier network, near field communication (NFC), or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the outer shell of the electronic device, or an external keyboard, a touchpad, or a mouse, etc.

[0137] Those skilled in the art can understand that Figure 12 the structure shown in is only a structural diagram of a part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0138] A fourth aspect of the present invention discloses a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps in an instance-level change detection method for co-optimizing contour accuracy and positioning accuracy according to any one of the first aspects disclosed in the present invention are implemented.

[0139] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification. The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several variations and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. An instance-level change detection method with collaborative optimization of contour accuracy and positioning accuracy, characterized in that: The method comprises: Step S1: collect high-resolution satellite data and preprocess the satellite data; label the preprocessed satellite data for change detection; divide the satellite data and corresponding labels into a training set and a validation set; Step S2: Using the training set, a change detection model is trained using a joint loss consisting of a pixel-level loss and an instance-level loss with a positioning accuracy constraint. Step S3: adapting the instance-level evaluation index AP10 to the change detection result evaluation; applying the validation set and using the adapted evaluation index AP10 to evaluate the trained change detection model; Step S4: input the initial change spots and the dual-phase image obtained by the trained change detection model into the SAM change detection algorithm to obtain change spots; Based on the trained change detection model, an initial change pattern is obtained, and prompt points of the change area are generated according to the initial change pattern; the prompt points pass through the SAM prompt encoder to obtain prompt features with change area location information; the dual-phase image is input into the SAM image encoder and encoded separately to obtain dual-phase features; the dual-phase features are fused through the feature fusion module to obtain change features; the change features and prompt features are input into the SAM decoder together to output the change pattern.

2. The instance-level change detection method for collaboratively optimizing contour accuracy and positioning accuracy according to claim 1, characterized in that: In step S1, performing change detection tagging on the pre-processed satellite data includes: Combined with before and after time phase images, ArcGIS is used as a sample marking tool to identify and outline change patches; the change patches are used as samples with change detection labels; the marked change patches are for changes in cultivated land, woodland, gardens, grasslands, water bodies, roads, residential areas, greenhouses and construction areas outside the built-up area; the sample requires the marking of change patches with an area greater than 150 square meters.

3. The instance-level change detection method for collaboratively optimizing contour accuracy and positioning accuracy according to claim 1, characterized in that: In step S2, the joint loss composed of the pixel-level loss and the instance-level loss with positioning accuracy constraint is: in, loss Indicates joint loss; Indicates based on dice Computed pixel-level loss function; represents the instance loss function; Indicates the prediction result; Indicates the label of the sample; dice ()express dice calculate; N Indicates the total number of instances.

4. The instance-level change detection method for collaboratively optimizing contour accuracy and positioning accuracy according to claim 1, characterized in that: In step S3, adapting the instance-level evaluation indicator AP10 to the change detection result evaluation includes: AP10 distributes the accuracy evaluation of the entire image to each change patch instance, and calculates the IoU between the predicted patch and the true value patch based on the instance. The patch with an IoU greater than 0.1 is regarded as a correctly predicted patch, namely TP; the patch with an IoU less than 0.1 is regarded as a false positive patch, namely FP; the true value patch that is not detected is regarded as a missed detection patch, namely FN; if there are multiple predicted patches with an IoU greater than 0.1 for the same true value, the one with the highest confidence is TP, and the others are FP.

5. The instance-level change detection method for collaboratively optimizing contour accuracy and positioning accuracy according to claim 1, characterized in that: In step S4, the feature fusion module fuses the bi-temporal features based on the cross-layer cross attention module to obtain the change feature; The cross-layer cross attention module inputs the previous phase features and the subsequent phase features into two cross-layer self-attention modules respectively, takes the previous phase features as the query vector, and the subsequent phase features as the key vector and the value vector, and sends them to the first cross-layer self-attention module. The output result vector of the first cross-layer self-attention module and the previous phase feature vector are combined and sent to the first residual connection layer normalization. The output result of the first residual connection layer normalization is sent to the first feedforward neural network, and then the output result of the first feedforward neural network and the output result of the first residual connection layer normalization are combined and input into the second residual connection normalization. , the output result of the second residual link normalization is used as the query vector, the posterior phase feature is used as the key vector and the value vector combination and input into the second cross-layer self-attention module, the output result of the second residual link normalization and the output result of the second cross-layer self-attention module are combined and input into the third residual connection normalization, the output result of the third residual connection normalization is sent to the second feedforward neural network, the output result of the second feedforward neural network and the output result of the third residual link layer normalization are combined and input into the fourth residual connection layer normalization, and the output result of the fourth residual connection layer normalization is the obtained change feature.

6. The instance-level change detection method for collaboratively optimizing contour accuracy and positioning accuracy according to claim 1, characterized in that: In step S4, generating prompt points of the change area according to the initial change pattern includes: For the initial change patch, different instance patch regions are obtained by image connected domain calculation; For each of the example pattern spots, dividing the example pattern spots into regions using a grid of a predefined size; The central pixel of each grid is selected as a cue point, and the foreground / background category to which the central pixel belongs represents the category of the cue point.

7. An instance-level change detection system for collaborative optimization of contour accuracy and positioning accuracy, characterized in that: The system comprises: The first processing module is configured to collect high-resolution satellite data and preprocess the satellite data; perform change detection labeling on the preprocessed satellite data; and divide the satellite data and corresponding labels into a training set and a validation set; The second processing module is configured to apply the training set to train a change detection model using a joint loss consisting of a pixel-level loss and an instance-level loss with a positioning accuracy constraint. The third processing module is configured to adapt the instance-level evaluation index AP10 to the change detection result evaluation; apply the validation set and use the adapted evaluation index AP10 to evaluate the trained change detection model; A fourth processing module is configured to input the initial change spots obtained by the trained change detection model and the dual-phase image into a SAM change detection algorithm to obtain change spots; Based on the trained change detection model, an initial change pattern is obtained, and prompt points of the change area are generated according to the initial change pattern; the prompt points pass through the SAM prompt encoder to obtain prompt features with change area location information; the dual-phase image is input into the SAM image encoder and encoded separately to obtain dual-phase features; the dual-phase features are fused through the feature fusion module to obtain change features; the change features and prompt features are input into the SAM decoder together to output the change pattern.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps in the instance-level change detection method for collaborative optimization of contour accuracy and positioning accuracy described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the instance-level change detection method for collaboratively optimizing contour accuracy and positioning accuracy according to any one of claims 1 to 6 are implemented.

Images