Semantic scene graph matching-based change detection method and device, equipment and medium

Abstract

The invention relates to a change detection method and device based on semantic scene graph matching, equipment and a medium, and the method comprises the steps: detecting the semantic category of a target description text, generating a candidate region in a dual-temporal image through open vocabulary detection, generating a mask through an instance segmentation model after screening, and carrying out the refined screening, thereby obtaining a high-quality target instance. And constructing a structured semantic scene graph, establishing a cross-time-phase node matching relationship based on a multi-dimensional matching rule, identifying a change target instance, fusing pixel-level evidence to generate and optimize a change graph, identifying spatial correlation dynamic change between instances through relationship reasoning, and outputting a structured result. According to the method, the instance quality is guaranteed through multiple rounds of screening, the false drop rate is reduced, by means of structured expression and association reasoning, the change details of a single instance are presented, spatial relation evolution between instances is clearly presented, and deep practical information is provided for decision making.

Description

Change detection methods, devices, equipment, and media based on semantic scene graph matching Technical Field

[0001] This application relates to the field of image scene detection technology, and in particular to a change detection method, apparatus, device and medium based on semantic scene graph matching. Background Technology

[0002] Change detection is a key technology for comprehensively detecting, confirming, and deeply analyzing changes on the Earth's surface. It is widely used in many important fields such as land and resources monitoring and natural disaster analysis. Its core value lies in providing accurate information on Earth's changes for various high-level decision-making tasks. It is of great significance for ensuring the rational use of resources, improving the efficiency of disaster emergency response, and strengthening target monitoring capabilities.

[0003] Currently, change detection technologies are mainly divided into two categories: one is the mainstream pixel-based deep learning method, which has made some progress in change localization, but only focuses on where the change occurs, lacking in-depth analysis of what the change is and how it occurs; the other is the unsupervised change detection paradigm that has emerged in recent years based on visual basic models (such as SAM) and visual-language models (such as CLIP). With its zero-shot generalization ability and semantic feature representation advantages, it has shown potential in unlabeled scenarios, but it has not yet broken away from the traditional comparison paradigm.

[0004] However, existing technologies suffer from significant drawbacks: First, they suffer from prominent representation mismatch issues. Existing methods rely on low-dimensional, unstructured pixel grids for analysis, making it difficult to adapt to the needs of understanding high-dimensional, structured scenes and failing to achieve deep understanding of changes. Second, they have stringent requirements for pixel alignment. In real-world scenarios, registration errors caused by factors such as perspective and sensor pose are common, easily leading to false feature differences and false detections. Third, they lack semantic understanding and interpretability, only able to locate changes but unable to output crucial information such as the semantic category of the change, the evolution process, and changes in entity relationships. Fourth, they lack structured reasoning capabilities, making it difficult to address the instance-level change detection needs in open scenarios. These problems severely limit the application effectiveness of change detection technology in advanced decision-making tasks, necessitating a novel technical solution. Summary of the Invention

[0005] Therefore, it is necessary to address the aforementioned technical issues by providing a change detection method based on semantic scene graph matching that can achieve instance-level semantic detection without annotation or training, while also possessing high interpretability and avoiding false detections.

[0006] A change detection method based on semantic scene graph matching includes: acquiring dual-temporal images and target description text, wherein the dual-temporal images are scene images of the same monitoring area acquired at two different times, and the target description text includes a set of semantic categories to be monitored specified by the user; generating candidate target regions corresponding to the semantic categories on the dual-temporal images respectively through open vocabulary detection according to each semantic category in the target description text, retaining valid candidate boxes through confidence filtering and adaptive screening, generating instance masks corresponding to the valid candidate boxes using an instance segmentation model, and then obtaining high-quality target instances by screening the instance masks; constructing structured semantic scene graphs according to the target instances corresponding to each of the dual-temporal images respectively, and establishing cross-temporal node matching relationships between the dual-temporal structured semantic scene graphs based on multi-dimensional matching rules of semantic similarity, geometric correlation, and local structural consistency; identifying changed target instances based on the cross-temporal node matching relationships, generating an initial change graph by fusing multi-source pixel-level correlation evidence, and conducting change relationship inference through cross-temporal node matching relationships and neighborhood semantic distribution differences to identify the dynamic change process and change relationships of instances, and obtaining structured change results.

[0007] In one embodiment, the dual-temporal images include at least one of remote sensing images and monitoring images, and there is no need to perform pixel-level alignment on the dual-temporal images in advance.

[0008] In one embodiment, when filtering the instance mask: the semantic alignment confidence obtained by open vocabulary detection is weighted and fused with the segmentation confidence obtained by the instance segmentation model, and then combined with a set fusion confidence threshold to perform an initial screening of the instance mask; the instance mask after the initial screening is normalized and subjected to adaptive threshold binarization to remove instance masks with blurred contours, thus completing the second screening; the instance mask obtained after the second screening is filtered for noise fragments and over-merged regions by setting a minimum area threshold and a maximum area threshold, and then context consistency is checked based on the semantic category distribution in the instance neighborhood to remove semantically misaligned instances, thus obtaining the target instances corresponding to each of the two temporal images.

[0009] In one embodiment, a structured semantic scene graph is constructed based on the target instances corresponding to each of the two temporal images, including: using each target instance corresponding to the two scene images as a node in the basic construction graph, with each node carrying attributes, including the target instance's category label, spatial mask, visual feature vector extracted by DINOv2, geometric center point coordinates, and instance area; for all nodes in the same scene image, calculating the centroid Euclidean distance between any two nodes corresponding to target instances, and establishing an undirected adjacency edge between the two nodes when the centroid Euclidean distance is less than or equal to a preset adjacency distance threshold, the undirected adjacency edge representing the spatial neighborhood relationship between the two target instances; and constructing the structured semantic scene graph corresponding to each of the two temporal images based on the nodes and the edges between the nodes.

[0010] In one embodiment, the multi-dimensional matching rule is expressed as:

[0011] In the above formula, , , , These represent the weight coefficients for semantic distance, geometric distance, structural distance, and uncertainty distance, respectively. This represents the semantic distance defined by the cosine distance between DINOv2 features. Geometric distance representing the crossover ratio and area deviation of the composite mask. Structural distance, used to measure the consistency of the distribution of local neighborhood categories of a node. This represents the uncertainty distance used to penalize matches with low confidence. , They represent The scene image acquired at time 1 Each node and The scene image acquired at time 1 Each node.

[0012] In one embodiment, identifying changed target instances based on the cross-temporal node matching relationship includes: for the node set of the dual-temporal structured semantic scene graph, determining the matching status of each node one by one based on the optimal cross-temporal node matching relationship obtained by the Hungarian algorithm; if a node in the scene graph at time t2 has no corresponding matching node in the scene graph at time t1, it is determined to be a newly added changed target instance; if a node in the scene graph at time t1 has no corresponding matching node in the scene graph at time t2, it is determined to be a deleted changed target instance; if nodes in the scene graphs at two time times have a matching relationship, but the category labels of the corresponding nodes are inconsistent, or the differences in the visual features and geometric features of the nodes exceed a preset threshold, it is determined to be a modified changed target instance; if the category labels of the matching nodes in the scene graphs at two time times are consistent, and the differences in the visual features and geometric features do not exceed a preset threshold, it is determined to be an unchanged instance; then the newly added changed target instance, the deleted changed target instance, and the modified changed target instance are taken as the changed target instances.

[0013] In one embodiment, the structured change result includes: change type, change process, and change relationship.

[0014] This application also provides a change detection device based on semantic scene graph matching, characterized in that the device includes: a data acquisition module, used to acquire dual-temporal images and target description text, wherein the dual-temporal images are scene images of the same monitoring area acquired at two different times, and the target description text includes a set of semantic categories to be monitored specified by the user; a high-quality target instance extraction module, used to generate candidate target regions corresponding to the semantic categories on the dual-temporal images respectively through open vocabulary detection according to each semantic category in the target description text, retain effective candidate boxes after confidence filtering and adaptive screening, generate instance masks corresponding to the effective candidate boxes using an instance segmentation model, and then... The system filters the instance masks to obtain high-quality target instances. A cross-temporal node matching relationship construction module is used to construct structured semantic scene graphs based on the target instances corresponding to each of the two temporal images. Based on multi-dimensional matching rules of semantic similarity, geometric correlation, and local structural consistency, a cross-temporal node matching relationship is established between the two temporal structured semantic scene graphs. A change detection result output module is used to identify changed target instances based on the cross-temporal node matching relationship, fuse multi-source pixel-level correlation evidence to generate an initial change graph, and perform change relationship reasoning through the cross-temporal node matching relationship and neighborhood semantic distribution differences to identify the dynamic change process and change relationship of instances, thereby obtaining structured change results.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above-described change detection method based on semantic scene graph matching.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described change detection method based on semantic scene graph matching.

[0017] The aforementioned change detection method, apparatus, device, and medium based on semantic scene graph matching generate candidate target regions corresponding to semantic categories in dual-temporal images based on the semantic categories in the target description text using open vocabulary detection. Valid candidate boxes are retained through confidence filtering and adaptive selection. Instance masks corresponding to the valid candidate boxes are generated using an instance segmentation model. High-quality target instances are obtained by filtering the instance masks. Structured semantic scene graphs are constructed based on the target instances corresponding to each dual-temporal image. Cross-temporal node matching relationships are established between the dual-temporal structured semantic scene graphs based on multi-dimensional matching rules of semantic similarity, geometric correlation, and local structural consistency. Changed target instances are identified based on these cross-temporal node matching relationships. An initial change graph is generated by fusing multi-source pixel-level association evidence based on the spatial information of the changed target instances. After optimizing the initial change graph, relational reasoning is performed by analyzing the differences in neighborhood semantic distribution of changed target instances in the dual-temporal scene to identify dynamic changes in spatial associations between instances, thus obtaining structured change results. This method can ensure the high quality and purity of target instances through multiple rounds of refined screening, providing reliable data support for cross-time matching and significantly reducing the false detection rate. Furthermore, through structured expression and associative reasoning, the change results not only include the change details of individual instances, but also clearly present the dynamic evolution of spatial relationships between instances, providing more in-depth and practical information support for subsequent decision-making. Attached Figure Description

[0018] Figure 1 is a flowchart illustrating a change detection method based on semantic scene graph matching in one embodiment; Figure 2 is a flowchart illustrating the processing framework of a change detection method based on semantic scene graph matching in one embodiment; Figure 3 is a schematic diagram of the local visualization results of this method and other methods on the EGY-BCD dataset in an experiment; Figure 4 is a schematic diagram of the local visualization results of this method and other methods on the DSIFN dataset in an experiment; Figure 5 is a schematic diagram of the local visualization results of this method and other methods on the LEVIR-CD dataset in an experiment; Figure 6 is a schematic diagram of the detection results under simulated registration error in an experiment; Figure 7 is a schematic diagram of the detection results on unregistered image pairs in the real world in an experiment; Figure 8 is a schematic diagram of unsupervised semantic change detection using this method in an experiment; Figure 9 is a schematic diagram of an example of interpretable change detection using this method in an experiment; Figure 10 is a structural block diagram of a change detection device based on semantic scene graph matching in one embodiment; Figure 11 is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] As shown in Figure 1, this application provides a change detection method based on semantic scene graph matching, which specifically includes the following steps: Step S100, acquiring dual-temporal images and target description text, wherein the dual-temporal images are scene images acquired at two different times in the same monitoring area, and the target description text includes a set of semantic categories to be monitored specified by the user.

[0021] Step S110: Based on the semantic categories in the target description text, candidate target regions corresponding to the semantic categories are generated on the dual-temporal images by open vocabulary detection. Valid candidate boxes are retained after confidence filtering and adaptive screening. Instance masks corresponding to the valid candidate boxes are generated using an instance segmentation model. High-quality target instances are obtained by screening the instance masks.

[0022] Step S120: Construct structured semantic scene graphs based on the target instances corresponding to each of the two temporal images, and establish cross-temporal node matching relationships between the two temporal structured semantic scene graphs based on multi-dimensional matching rules of semantic similarity, geometric correlation and local structural consistency.

[0023] Step S130: Identify changed target instances based on cross-temporal node matching relationships, generate an initial change map by fusing multi-source pixel-level correlation evidence, and conduct change relationship reasoning through cross-temporal node matching relationships and neighborhood semantic distribution differences to identify the dynamic change process and change relationships of instances and obtain structured change results.

[0024] This paper proposes a semantic structure-oriented change detection method for automatically identifying object-level changes and their semantic relationships from dual-temporal remote sensing images or other scene images. This method simulates the process of human visual system perceiving change: first, semantic objects in the scene are extracted; second, cross-temporal object correspondences are established; and finally, change inference is performed based on the differences between objects and their relationships. The overall method, as shown in Figure 2, is designed as a three-stage pipeline framework, including: Change Definition and Instance Extraction (CDIE) step S110, Scene Graph Construction and Matching (SGCM) step S120, and Change Detection and Relational Reasoning (CDRR) step S130. Through these three stages and steps, this method can achieve interpretable instance-level change detection and semantic-level relationship analysis while ensuring high recall.

[0025] In step S100, the dual-temporal images include at least one of remote sensing images and surveillance images, and pixel-level alignment of the dual-temporal images is not required beforehand. The scene image has dimensions H×W×3, where H is the image height, W is the image width, and "3" represents the RGB three channels. The set of user-defined semantic categories to be monitored in the target description text can be represented as follows: .in For the first There are several text category hints (e.g., "buildings", "vegetation", etc.), where K is the total number of categories specified by the user.

[0026] In step S110, the goal of open vocabulary instance localization is to recall as many candidate target regions as possible in the input image, i.e., the scene image, based on the semantic category specified by the user. Instance segmentation and semantic refinement are to obtain high-quality instance masks based on the candidate boxes and further filter unreliable targets through spatial and semantic rules to make them suitable as nodes for subsequent graph construction.

[0027] In this embodiment, the LangSAM detection branch based on GroundingDINO is used for each text prompt. Perform open-lexical object detection and output a set of candidate object regions and their semantic alignment scores. For the first... The text prompt yields:

[0028] In the above formula, This indicates the candidate target area for the target category suggestion. Indicates the first The coordinate vectors of each candidate bounding box (four-dimensional bounding box parameters). Indicates the first Candidate boxes and text prompts The semantic alignment confidence score ranges from [0,1]. Indicates the prompt The number of candidate boxes detected.

[0029] Next, the candidate target regions suggested by all categories are merged to obtain a global candidate set:

[0030] In the above formula, This represents the union of the candidate boxes corresponding to all category suggestions. This represents the total number of candidate boxes, and |·| represents the cardinality of the set.

[0031] Furthermore, to eliminate obviously irrelevant or low-quality candidate boxes while minimizing the loss in recall, a lightweight confidence threshold is set. This refers to the initial screening threshold for candidate boxes. For all candidate boxes... Filter and retain only those that meet the requirements. The sample size is [not specified]. Due to the significant differences in scene complexity across different images, using a fixed number of candidate boxes can easily introduce too many redundant candidates in simple scenes or truncate valid targets in complex scenes. Therefore, a dynamic Top-K mechanism is introduced to adaptively control the number of candidate boxes. The process is as follows:

[0032] In the above formula, This indicates the number of candidate boxes that are adaptively retained. This represents the basic candidate quantity quota. This indicates that the number of reserves controls the total number of candidates. The growth rate coefficient This indicates the maximum number of allowed candidate boxes, used to limit the upper bound. · This indicates the floor function.

[0033] After a light confidence filtering, the remaining candidate boxes are sorted from highest to lowest confidence, and the top K_dyn candidate boxes are retained as valid candidate boxes. This ensures the recall of potential positive samples while avoiding an uncontrolled candidate set size. Finally, the open-vocabulary instance localization stage outputs a set of text-semantically aligned candidate target regions, i.e., valid candidate boxes, providing initial positional constraints for subsequent instance segmentation.

[0034] In this embodiment, a high-quality instance mask is obtained based on the candidate bounding boxes, and unreliable targets are further filtered out using spatial and semantic rules to make them suitable as nodes for subsequent graph construction. This sub-step includes three stages: instance mask prediction, mask binarization and spatial filtering, and semantic consistency refinement.

[0035] Specifically, when screening instance masks: the semantic alignment confidence obtained through open vocabulary detection is weighted and fused with the segmentation confidence obtained by the instance segmentation model. Then, the instance masks are initially screened by combining the set fusion confidence threshold. Next, the instance masks after the initial screening are normalized and subjected to adaptive threshold binarization to remove instance masks with blurred contours, thus completing the second screening. For the instance masks obtained after the second screening, noise fragments and over-merged regions are filtered by setting minimum area thresholds and maximum area thresholds. Finally, context consistency is checked based on the semantic category distribution in the instance neighborhood to remove semantically misaligned instances, thus obtaining the target instances corresponding to each of the two temporal images.

[0036] In this embodiment, the set of valid candidate boxes is represented as:

[0037] In the above formula, Indicates the first One candidate box, This represents the text alignment confidence level obtained from GroundingDINO. This indicates the total number of candidate boxes.

[0038] Furthermore, the Segment Anything Model (SAM) is used for each candidate box. After segmentation, we get:

[0039] In the above formula, Represents pixels Belongs to the The confidence level of each instance.

[0040] Furthermore, to integrate semantic detection confidence and segmentation quality, a fusion confidence representation is introduced as follows:

[0041] In the above formula, Indicates semantic score weights, Indicates the segmentation score weights. Further set the fusion confidence threshold. Mask quality threshold. Only retain those that meet the requirements. The instance mask filters out candidates with significantly insufficient semantics or segmentation quality.

[0042] Furthermore, to obtain clear instance regions that can be used for geometric and topological analysis, a soft mask is used. Normalization and binarization, where normalization is performed on each instance. ,make:

[0043] In the above formula, This represents the maximum pixel value of the mask across the entire image. Next, the Otsu method is used to calculate an adaptive threshold for each instance. And generate a binary mask, the process is represented as follows:

[0044] In the above formula, if ,but ,otherwise ,in, Indicates the first An instance in pixels The binary value at the location (1 for foreground, 0 for background). This represents the threshold adaptively determined by the Otsu method.

[0045] Furthermore, to eliminate noise debris and abnormally large areas, constraints are imposed on the mask area:

[0046] In the above formula, Representation of instances The total number of foreground pixels, This represents the minimum allowed instance area threshold. This represents the maximum allowed instance area threshold. Where the area is less than... The mask is typically noise debris with an area larger than [missing information]. The mask is generally an over-merged region, and all of them are removed.

[0047] Furthermore, considering that in complex remote sensing scenes, there may still be instances where semantics are inconsistent with the context (e.g., a shadow is misidentified as "water"), this method introduces semantic consistency filtering based on local context. This includes filtering for any instance... The system analyzes the category distribution of other instances within its neighboring region. If an instance's category is significantly inconsistent with the main category of its neighborhood, it is marked as untrustworthy and removed. For example, in an area entirely composed of buildings, if a small "water body" instance appears surrounded by buildings, it is considered to have an unreasonable context and is filtered out. Through step S100, this stage outputs a set of target instances with accurate spatial location and reliable semantic labels, providing a foundation for subsequent scene graph construction and change inference.

[0048] In step S120, to explicitly represent target instances and their spatial relationships, this invention represents the image at each moment as a Structured Scene Graph (SSG), and establishes cross-temporal object correspondences through graph matching. Let... , Indicates two points in time. Indicates time The input image, :time Corresponding scene diagram; A collection of nodes, representing instance objects. Let be the set of edges, representing the spatial adjacency relationships between instances.

[0049] In this embodiment, a structured semantic scene graph is constructed based on the target instances corresponding to each of the two temporal images. This includes: using each target instance corresponding to the two scene images as a node in the basic construction graph, with each node carrying attributes including the target instance's category label, spatial mask, visual feature vector extracted by DINOv2, geometric center point coordinates, and instance area; calculating the centroid Euclidean distance between any two nodes corresponding to the target instances for all nodes in the same scene image; establishing an undirected adjacency edge between the two nodes when the centroid Euclidean distance is less than or equal to a preset adjacency distance threshold; and constructing the structured semantic scene graph corresponding to each of the two temporal images based on the nodes and the edges between them.

[0050] Specifically, when constructing nodes, a mask is applied to each instance that has already passed semantic and spatial filtering. Build the corresponding nodes Each node is assigned multiple attributes, including: instance category tags (such as buildings, roads, vegetation, etc.). Binary mask of the instance Instance visual feature vector Its dimension is D, and it is obtained by applying the DINOv2 feature map. The coordinates of the geometric center point of the instance are obtained by average pooling the region. , represented as Instance area That is, mask Number of mid-foreground pixels Therefore, a node can be represented as:

[0051] In the above formula, t represents the time index.

[0052] Specifically, in establishing spatial adjacency relationships between instances, in the node set Internally constructed edge set .set up This is the adjacency distance threshold, used to determine whether two instances are adjacent. For any two nodes... and If the Euclidean distance between their centers satisfies:

[0053] Then establish an undirected edge between the two. To improve construction efficiency, it can be based on the center point of all nodes. Construct a KD-tree and use nearest neighbor retrieval to quickly filter candidate adjacency pairs. The final result is: , .

[0054] In this embodiment, to achieve cross-time object-level change analysis, it is necessary to... and Establish node-level matching relationships between them, that is, at each time step Node search in The corresponding node at time. Let... From the set of nodes To the set of nodes The matching function, = indicates a node At any moment The matching index; :node and The matching cost between them. In this application, the designed matching cost function comprehensively considers multi-dimensional matching rules of four types of information: semantic, geometric, structural, and uncertain information, and is expressed as:

[0055] In the above formula, , , , These represent the weight coefficients for semantic distance, geometric distance, structural distance, and uncertainty distance, respectively. This represents the semantic distance defined by the cosine distance between DINOv2 features. Geometric distance representing the crossover ratio and area deviation of the composite mask. Structural distance, used to measure the consistency of the distribution of local neighborhood categories of a node. This represents the uncertainty distance used to penalize matches with low confidence. , They represent The scene image acquired at time 1 Each node and The scene image acquired at time 1 Each node.

[0056] Specifically, semantic distance The following formula is used to calculate:

[0057] In the above formula, "·" represents the vector dot product, and "||·||2" represents the L2 norm.

[0058] Specifically, geometric distance The following formula is used to calculate:

[0059] In the above formula, This represents the cross-union ratio of the two masks, i.e. , , Represents the area of ​​two instances. , This represents the weighting coefficient between IoU and area difference.

[0060] Specifically, structural distance The following formula is used to calculate:

[0061] In the above formula, Represents a node The semantic histogram vector of the neighborhood, where each dimension represents the number of times different categories appear in the neighborhood.

[0062] Specifically, the distance of uncertainty The following formula is used to calculate:

[0063] In the above formula, , Representing nodes respectively and The overall confidence level can be derived from the fusion of the detection and segmentation stages.

[0064] In step S130, semantic change detection not only needs to determine "which objects have changed," but also needs to establish cross-temporal instance correspondences. Without cross-temporal object matching, it is impossible to semantically interpret changes or achieve higher-level relational reasoning. Based on establishing object matching relationships, this method further detects instance-level changes and infers the evolution of object semantics and relationships between objects over time.

[0065] In this embodiment, identifying changed target instances based on cross-temporal node matching relationships includes: for the node set of the dual-temporal structured semantic scene graph, based on the optimal cross-temporal node matching relationship obtained by the Hungarian algorithm, determining the matching status of each node one by one. If a node in the scene graph at time t2 has no corresponding matching node in the scene graph at time t1, it is determined to be a newly added changed target instance. If a node in the scene graph at time t1 has no corresponding matching node in the scene graph at time t2, it is determined to be a deleted changed target instance. If there is a matching relationship between nodes in the scene graphs at two different times, but the category labels of the corresponding nodes are inconsistent, or the differences in the visual features and geometric features of the nodes exceed a preset threshold, it is determined to be a modified changed target instance. If the category labels of the matching nodes in the scene graphs at two different times are consistent, and the differences in the visual features and geometric features do not exceed a preset threshold, it is determined to be an unchanged instance. Then, the newly added changed target instances, deleted changed target instances, and modified changed target instances are taken as changed target instances.

[0066] Specifically, object matching across time is modeled as a linear assignment problem, and a multimodal cost matrix is ​​used. The solution is as follows: The optimal partial matching is:

[0067] In the above formula, For optimal matching, To effectively match sets, The cost is for multimodal matching. This optimization problem is solved using the Hungarian algorithm.

[0068] Next, by projecting the mask of the changed instance back into the image domain, the region of interest (RoI) can be obtained:

[0069] Within the RoI region, an initial change map is generated using DST-based multi-source fusion, and then the final structural change result M_final is obtained through morphological operations, including the change type of the image target, the change process, and the change relationship.

[0070] Furthermore, in addition to determining whether an object has changed, this method also focuses on changes in the relationships between the changed object and the scene to improve the interpretability of the overall change detection results. For each "modified" node... (or other nodes of interest), examine their neighborhood structure at two time points respectively. Let... :node At any moment The set of neighboring nodes; Indicates matching node At any moment The set of neighboring nodes, h( ) represents the semantic histogram vector obtained by statistically analyzing the categories of neighboring nodes.

[0071] Define the context change vector as follows:

[0072] The vectors described above depict changes in the semantic environment surrounding the changing object. For example, when a "road" node disappears in a certain area and a "building" node is added in the same location, the corresponding context change vector will reflect the functional shift from "transportation facilities" to "building land".

[0073] In this paper, the performance of the proposed method is comprehensively evaluated through experimental results on three challenging public change detection (CD) datasets: LEVIR-CD, DSIFN, and EGY-BCD. These datasets cover different geographical scenes, spatial resolutions, and change types, providing reliable benchmarks for evaluating the generalization ability of change detection algorithms. Key parameter settings are as follows: initial building proposals are filtered using a confidence threshold of 0.3. During the construction of spatial semantic graphs (SSGs), adjacent edges connect nodes within a radius of 1 / 20 of the image's length and width. This matching process is optimized through two iterations. Finally, changes are confirmed through a hierarchical fusion process, and small false alarms with an area less than 150 pixels are removed.

[0074] To evaluate algorithm performance, experiments employed several standard metrics calculated based on components of the confusion matrix: true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN). Key metrics included overall precision (OA), recall (R), accuracy (P), F1 score (F1), and intersection-over-union ratio (IoU), defined as follows. In the building change detection (BCD) task, the average intersection-over-union ratio (mIoU) was also obtained by calculating the average IoU between the "changed" and "unchanged" classes.

[0075]

[0076] The proposed method was evaluated on three widely used remote sensing CD datasets (LEVIR-CD, EGY-BCD, and DSIFN). Evaluation employed five standard metrics: accuracy, precision, recall, F1 score, and mean intersection-over-union ratio (MUI). The invention was compared with various unsupervised methods, including traditional pixel-based methods (e.g., CVA, DCVA, IRMAD), learning-based baseline methods (e.g., SiameseCNN, Autoencoder), and recent methods based on visual fundamental models (VFM) (e.g., MergeSAM, AnyChange, UCD-SCM).

[0077] First, the results on the EGY-BCD dataset, a high-resolution benchmark dataset focusing on building-level changes in urban areas, are presented in Figure 3. As shown, although this method is unsupervised, its results are highly consistent with human perception, focusing on actual building changes while effectively avoiding false alarms. This demonstrates the advantages of user-defined change detection based on DINO features. The quantitative results in Table 1 show that this method achieves optimal performance across all five metrics. Its F1 score and mIoU exceed the AnyChange method by 22.59 and 17.23 points respectively, and its F1 score is more than 20 points higher than most SAM-based unsupervised methods.

[0078] Table 1 shows the performance on the EGY-BCD dataset.

[0079] Next, experimental results on the DSIFN dataset are presented. The dataset's 2-meter spatial resolution makes distinguishing individual buildings challenging. As shown in Figure 4, even with some loss of fine structural details, this method can still locate changed building clusters with reasonable accuracy. The quantitative results in Table 2 show that this method achieves best performance on DSIFN, with an F1 score of 41.68 and an mIoU of 52.62, outperforming other SAM-based methods. However, the performance improvement is relatively small (F1 score improvement +0.48, mIoU improvement +4.93), highlighting the performance degradation that occurs when migrating from high-resolution imagery to coarse-resolution imagery.

[0080] Table 2 Quantitative results on the DSIFN dataset

[0081] Finally, further evaluation was conducted on the LEVIR-CD dataset. As shown in Figure 5, compared with other methods, the change map generated by this method is clearer, with fewer false negatives and false alarms. Table 3 shows the quantitative results, indicating that this method consistently outperforms all baseline methods across five metrics. Compared with recent VFM-based methods (such as AnyChange and UCD-SCM), this invention achieves a significant improvement in F1 score (up to +6.91) and also improves in mIoU (up to +3.37), indicating higher precision and a better precision-recall balance. These results confirm the robustness of this method in typical building change detection scenarios.

[0082] Table 3 Quantitative results on the LEVIR-CD dataset

[0083] The results demonstrate the advantages of graph-driven semantic reasoning over traditional pixel-level differential or embedding matching. Our method consistently achieves superior results across all benchmarks and evaluation metrics. The observed performance gains highlight the beneficial effects of combining semantic instance-level reasoning with multi-scale pixel-level verification. Our graph-driven paradigm effectively bridges the gap between symbolic structures and underlying signals, providing both interpretability and robustness in unconstrained change detection scenarios.

[0084] To evaluate the robustness of this method framework under spatial misregistration, experiments were conducted on synthetic perturbations and real-world misregistered remote sensing image pairs. As shown in Figure 6, registration errors were first simulated by applying geometric transformations (including translation, rotation, and flipping) to the input image pairs in a standard reference, where red circles mark the displaced building areas correctly processed by the method of this invention. Even under significant displacement, this method accurately detects real changes while suppressing false alarms caused by misaligned pixels. This robustness stems from the target-centric paradigm of this invention, where high-rise structures are matched through graph reasoning rather than direct pixel comparison.

[0085] To further evaluate performance under unconstrained conditions, the proposed method was applied to pairs of raw, unregistered satellite images with realistic distortion, as shown in Figure 7. Colored dots represent corresponding geospots between the two views, highlighting strong spatial distortion. These examples exhibit significant spatial inconsistencies. Notably, the proposed method maintains accurate change detection, unaffected by deformation artifacts, demonstrating its strong generalization ability under challenging conditions. The robustness of the pipeline is attributed to graph-constrained verification, a design that ensures that noisy or offset pixel responses are filtered out unless supported by consistent target-level evidence.

[0086] A fundamental challenge in change detection is the lack of a clear semantic definition of "what constitutes a meaningful change." Traditional methods often detect all apparent differences indiscriminately, resulting in outputs containing irrelevant noise and lacking user-specific focus.

[0087] To address this issue, this method introduces a zero-shot, unsupervised semantic change detection framework that supports user-defined variation queries without requiring task-specific labels or training. By leveraging the open-vocabulary representation capabilities of SAM and DINO, the method generates semantically meaningful region-level features. These features enable the model to identify target-level and relation-level changes consistent with higher-level concepts (e.g., "building demolition," "vehicle appearance") even in the absence of explicit supervision. As shown in Figure 8, the method bridges the gap between low-level signal comparison and high-level semantic reasoning, achieving user-defined, controllable open-world change detection.

[0088] A key advantage of the proposed framework lies in its interpretability, both in the inference process and in the final output. Unlike the black-box operation of traditional deep learning models, the process of this method is mathematically grounded, and the inference steps are transparent and traceable. Each component is well-defined, making the decision-making process interpretable and reproducible.

[0089] In terms of graph construction, this method further enhances interpretability by representing each node as a semantically segmented object instance (rather than the low-level superpixels or image patches commonly used in previous works). This instance-level abstraction enables the model to reason about discrete, meaningful entities, thereby inherently making change detection more interpretable. Structural changes such as the appearance, disappearance, or reconnection of objects are directly mapped to graph-level dynamic changes that conform to human cognition.

[0090] Beyond the reasoning process, the output of this method is also highly interpretable. As shown in Figure 9, the model in this method can not only identify which objects appear or disappear, but also capture changes in relationships (such as newly formed or broken spatial connections). These insights are organized into structured graph representations and can be further converted into human-readable text descriptions. This dual representation supports both low-level visual inspection and high-level decision-making, making this system suitable for downstream applications that require interpretable reasoning.

[0091] The aforementioned change detection method based on semantic scene graph matching relies on open vocabulary detection, instance segmentation, and multi-round refined screening. First, GroundingDINO-driven open vocabulary detection responds to user-defined semantic categories. Then, SAM segmentation generates a mask, fusing semantic alignment confidence and segmentation confidence, Otsu adaptive binarization, area threshold filtering, and contextual semantic consistency verification through multiple rounds of screening. This ensures the semantic reliability and spatial accuracy of the target instance, providing high-quality data support for subsequent matching and significantly reducing the false detection rate. Furthermore, by constructing a structured semantic scene graph of instance nodes and spatial adjacency edges, image information is abstracted into a semantically rich structured expression. Combined with multi-dimensional matching rules incorporating semantic distance, geometric overlap, structural distribution, and uncertainty, along with the Hungarian algorithm, optimal cross-temporal node matching is achieved. This completely eliminates the reliance on pixel-level precise alignment found in traditional methods and exhibits strong robustness to registration errors caused by viewpoint shifts and sensor pose differences. By leveraging instance-level change classification and relational reasoning through neighborhood semantic histogram comparison, this system can not only accurately identify changes in added, deleted, and modified instances, but also capture the dynamic evolution of spatial relationships between instances, such as the addition or disappearance of adjacency relationships, and output structured results. This upgrades change detection from simply knowing where it changes to knowing what changes and how it changes. Furthermore, its unsupervised design, which requires no labeled data or model training, combined with the category adaptation capabilities of open-vocabulary detection, can flexibly respond to customized monitoring needs in different scenarios, exhibiting strong generalization capabilities. This provides more in-depth and practical technical support for advanced decision-making scenarios such as land resource monitoring, disaster assessment, and urban governance.

[0092] It should be understood that although the steps in the flowchart of Figure 1 are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in Figure 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0093] In one embodiment, as shown in Figure 10, a change detection device based on semantic scene graph matching is provided, including: a data acquisition module 200, a high-quality target instance extraction module 210, a cross-temporal node matching relationship construction module 220, and a change detection result output module 230. The data acquisition module 200 is used to acquire dual-temporal images and target description text. The dual-temporal images are scene images of the same monitoring area acquired at two different times. The target description text includes a set of semantic categories to be monitored specified by the user. The high-quality target instance extraction module 210 is used to generate candidate target regions corresponding to the semantic categories on the dual-temporal images based on each semantic category in the target description text through open vocabulary detection, and retain valid target regions after confidence filtering and adaptive selection. Candidate boxes are generated using an instance segmentation model to create instance masks corresponding to the valid candidate boxes. High-quality target instances are then obtained by filtering the instance masks. A cross-temporal node matching relationship construction module 220 is used to construct structured semantic scene graphs based on the target instances corresponding to each of the two temporal images. Based on multi-dimensional matching rules of semantic similarity, geometric correlation, and local structural consistency, a cross-temporal node matching relationship is established between the two temporal structured semantic scene graphs. A change detection result output module 230 is used to identify changed target instances based on the cross-temporal node matching relationship, fuse multi-source pixel-level correlation evidence to generate an initial change map, and perform change relationship inference through the cross-temporal node matching relationship and neighborhood semantic distribution differences to identify the dynamic change process and change relationship of instances, thereby obtaining structured change results.

[0094] Specific limitations regarding the change detection device based on semantic scene graph matching can be found in the limitations of the change detection method based on semantic scene graph matching mentioned above, and will not be repeated here. Each module in the aforementioned change detection device based on semantic scene graph matching can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0095] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in Figure 11. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a change detection method based on semantic scene graph matching. The display screen of the computer device may be a liquid crystal display screen or an e-ink display screen. The input device of the computer device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad provided on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0096] Those skilled in the art will understand that the structure shown in Figure 11 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0097] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring dual-temporal images and target description text, wherein the dual-temporal images are scene images of the same monitoring area acquired at two different times, and the target description text includes a set of semantic categories to be monitored specified by a user; generating candidate target regions corresponding to the semantic categories on the dual-temporal images respectively by open vocabulary detection based on each semantic category in the target description text; retaining valid candidate boxes after confidence filtering and adaptive screening; and generating the target regions using an instance segmentation model. The instance mask corresponding to the effective candidate box is then filtered to obtain high-quality target instances. A structured semantic scene graph is constructed based on the target instances corresponding to each of the two temporal images. A cross-temporal node matching relationship is established between the two temporal structured semantic scene graphs based on multi-dimensional matching rules of semantic similarity, geometric correlation, and local structural consistency. Changing target instances are identified based on the cross-temporal node matching relationship. An initial change map is generated by fusing multi-source pixel-level correlation evidence. Change relationship inference is performed through the cross-temporal node matching relationship and differences in neighborhood semantic distribution to identify the dynamic change process and change relationship of instances, thus obtaining structured change results.

[0098] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon. When executed by a processor, the computer program performs the following steps: acquiring dual-temporal images and target description text, wherein the dual-temporal images are scene images of the same monitoring area acquired at two different times, and the target description text includes a set of semantic categories to be monitored specified by a user; generating candidate target regions corresponding to the semantic categories on the dual-temporal images respectively by open vocabulary detection based on each semantic category in the target description text; retaining valid candidate boxes after confidence filtering and adaptive screening; and generating the valid candidate boxes using an instance segmentation model. The corresponding instance mask is then used to filter high-quality target instances. Structured semantic scene graphs are constructed based on the target instances corresponding to each of the two temporal images. Cross-temporal node matching relationships are established between the two temporal structured semantic scene graphs based on multi-dimensional matching rules of semantic similarity, geometric correlation, and local structural consistency. Changing target instances are identified based on the cross-temporal node matching relationships. An initial change graph is generated by fusing multi-source pixel-level correlation evidence. Change relationship reasoning is conducted through cross-temporal node matching relationships and differences in neighborhood semantic distribution to identify the dynamic change process and change relationships of instances, resulting in structured change results.

[0099] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A change detection method based on semantic scene graph matching, characterized in that, The method includes: acquiring dual-temporal images and target description text, wherein the dual-temporal images are scene images of the same monitoring area acquired at two different times, and the target description text includes a set of semantic categories to be monitored specified by the user; generating candidate target regions corresponding to the semantic categories on the dual-temporal images respectively through open vocabulary detection based on each semantic category in the target description text, retaining valid candidate boxes through confidence filtering and adaptive screening, generating instance masks corresponding to the valid candidate boxes using an instance segmentation model, and then obtaining high-quality target instances by screening the instance masks; constructing structured semantic scene graphs based on the target instances corresponding to each of the dual-temporal images respectively, and establishing cross-temporal node matching relationships between the dual-temporal structured semantic scene graphs based on multi-dimensional matching rules of semantic similarity, geometric correlation, and local structural consistency; identifying changed target instances based on the cross-temporal node matching relationships, generating an initial change map by fusing multi-source pixel-level correlation evidence, and conducting change relationship inference through cross-temporal node matching relationships and neighborhood semantic distribution differences to identify the dynamic change process and change relationships of instances, and obtaining structured change results.

2. The change detection method based on semantic scene graph matching according to claim 1, characterized in that, The dual-temporal images include at least one of remote sensing images and monitoring images, and there is no need to perform pixel-level alignment on the dual-temporal images in advance.

3. The change detection method based on semantic scene graph matching according to claim 1, characterized in that, When filtering the instance mask: the semantic alignment confidence obtained by open vocabulary detection is weighted and fused with the segmentation confidence obtained by the instance segmentation model, and then the instance mask is initially filtered in combination with the set fusion confidence threshold. The instance masks that have undergone initial screening are normalized and subjected to adaptive threshold binarization to remove instance masks with blurred outlines, thus completing the second screening. The instance masks obtained after the second screening are filtered for noise fragments and over-merged regions by setting minimum and maximum area thresholds. Then, context consistency is checked based on the semantic category distribution in the neighborhood of the instance to remove semantically misaligned instances, thus obtaining the target instances corresponding to each of the two temporal images.

4. The change detection method based on semantic scene graph matching according to claim 3, characterized in that, Based on the target instances corresponding to each of the two temporal images, a structured semantic scene graph is constructed, including: using each target instance corresponding to the two scene images as a node in the basic construction graph, with each node carrying attributes, including the target instance's category label, spatial mask, visual feature vector extracted by DINOv2, geometric center point coordinates, and instance area; for all nodes in the same scene image, calculating the centroid Euclidean distance between any two nodes corresponding to target instances, and when the centroid Euclidean distance is less than or equal to a preset adjacency distance threshold, establishing an undirected adjacency edge between the two nodes, the undirected adjacency edge representing the spatial neighborhood relationship between the two target instances; and constructing the structured semantic scene graph corresponding to each of the two temporal images based on the nodes and the edges between the nodes.

5. The change detection method based on semantic scene graph matching according to claim 4, characterized in that, The multi-dimensional matching rule is expressed as follows: In the above formula, 、 、 、 These represent the weight coefficients for semantic distance, geometric distance, structural distance, and uncertainty distance, respectively. This represents the semantic distance defined by the cosine distance between DINOv2 features. Geometric distance representing the crossover ratio and area deviation of the composite mask. Structural distance, used to measure the consistency of the distribution of local neighborhood categories of a node. This represents the uncertainty distance used to penalize matches with low confidence. 、 They represent The scene image acquired at time 1 Each node and The scene image acquired at time 1 Each node.

6. The change detection method based on semantic scene graph matching according to claim 5, characterized in that, Identifying changed target instances based on the cross-temporal node matching relationship includes: for the node set of the dual-temporal structured semantic scene graph, judging the matching status of each node one by one based on the optimal cross-temporal node matching relationship obtained by the Hungarian algorithm; if a node in the scene graph at time t2 has no corresponding matching node in the scene graph at time t1, it is determined as a newly added changed target instance; if a node in the scene graph at time t1 has no corresponding matching node in the scene graph at time t2, it is determined as a deleted changed target instance; if there is a matching relationship between nodes in the scene graphs at two time times, but the category labels of the corresponding nodes are inconsistent, or the differences in the visual features and geometric features of the nodes exceed a preset threshold, it is determined as a modified changed target instance; if the category labels of the matching nodes in the scene graphs at two time times are consistent, and the differences in the visual features and geometric features do not exceed a preset threshold, it is determined as an unchanged instance; then the newly added changed target instance, the deleted changed target instance, and the modified changed target instance are taken as the changed target instance.

7. The change detection method based on semantic scene graph matching according to any one of claims 1-6, characterized in that, The results of structured change analysis include: the type of change, the process of change, and the relationships between changes.

8. A change detection device based on semantic scene graph matching, characterized in that, The device includes: a data acquisition module for acquiring dual-temporal images and target description text, wherein the dual-temporal images are scene images of the same monitoring area acquired at two different times, and the target description text includes a set of semantic categories to be monitored specified by the user; and a high-quality target instance extraction module for generating candidate target regions corresponding to the semantic categories in the dual-temporal images based on each semantic category in the target description text through open vocabulary detection, retaining valid candidate boxes through confidence filtering and adaptive screening, generating instance masks corresponding to the valid candidate boxes using an instance segmentation model, and then obtaining high-quality target instance extraction by screening the instance masks. The system includes a target instance quality module; a cross-temporal node matching relationship construction module, which constructs structured semantic scene graphs based on the target instances corresponding to each of the two temporal images, and establishes cross-temporal node matching relationships between the two temporal structured semantic scene graphs based on multi-dimensional matching rules of semantic similarity, geometric correlation, and local structural consistency; and a change detection result output module, which identifies changed target instances based on the cross-temporal node matching relationships, fuses multi-source pixel-level correlation evidence to generate an initial change graph, and performs change relationship inference through cross-temporal node matching relationships and neighborhood semantic distribution differences to identify the dynamic change process and change relationships of instances and obtain structured change results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images