Vision large model and spatiotemporal knowledge graph driven language-vision change detection method

Abstract

This invention relates to the fields of artificial intelligence, computer vision, and remote sensing change detection, and discloses a language-visual change detection method driven by a large visual model and spatiotemporal knowledge graph. The method includes: constructing prior fuzzy semantic information and generating vector geographic entities based on a large visual model and large remote sensing data; constructing a spatiotemporal change geographic knowledge graph and converting it into a full-spatiotemporal context geographic description language, then deconstructing it into abstract high-dimensional knowledge semantic features using a natural language processing model; extracting multi-scale visual features from dual-temporal remote sensing images using an encoder to enhance local-global perception capabilities and decoding the output visual features; aligning the knowledge semantics with the visual features, calculating the contribution of the abstract high-dimensional semantic features to the visual features, suppressing noise, and upsampling to the resolution of the original image; and constructing a joint loss function to optimize and train the model. This invention significantly improves the interpretability, accuracy, and adaptability to complex scenarios in change detection tasks.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, and remote sensing change detection, and in particular to a visual large model and spatiotemporal knowledge graph-driven language-visual change detection method. Background Technology

[0002] Change detection is a visual task in deep learning, which involves artificial intelligence, computer vision and remote sensing change detection. Its purpose is to intelligently identify the changed areas in two periods of remote sensing images by using deep learning algorithms.

[0003] The emergence of large language models such as DeepSeek has significantly improved the coupling ability between natural language instructions and the semantic space of remote sensing images, thereby promoting a paradigm shift in the field of remote sensing from "single-modal visual understanding" to "multi-modal (language-visual) semantic understanding." However, current research still has the following key shortcomings:

[0004] (1) Existing language-visual models are simple in their description of language, but they are difficult to cover complex geographic context information;

[0005] (2) Existing deep learning-based multimodal change detection models lack effective constraints for aligning complex language and visual features;

[0006] Based on the shortcomings of the aforementioned research, the inventors developed a language-visual change detection method that can be jointly driven by a large visual model and a spatiotemporal knowledge graph. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the prior art by providing a visual large model and spatiotemporal knowledge graph-driven language-visual change detection method. This method aims to construct a reasonable professional domain knowledge framework for deep learning visual operations and significantly improve the interpretability, accuracy, and adaptability to complex scenarios of change detection tasks.

[0008] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0009] A visual large-scale model and spatiotemporal knowledge graph-driven language-visual change detection method includes the following steps:

[0010] S1 uses a large visual model to segment dual-temporal remote sensing images into vector geographic entities. A deep learning network is trained based on remote sensing big data to obtain prior fuzzy semantic information. The obtained prior fuzzy semantic information is then mapped to the vector geographic entities according to their spatial location to form attribute information.

[0011] S2, construct a spatiotemporal change geographic knowledge graph based on attribute information, transform the spatiotemporal change geographic knowledge graph into a full spatiotemporal context geographic description language, and deconstruct the full spatiotemporal context geographic description language into abstract high-dimensional knowledge semantic features based on a natural language processing model;

[0012] S3, based on a twin architecture, uses an encoder to extract multi-scale features from dual-temporal remote sensing images, enhances global perception of visual features by compensating for local context, and uses a decoder to aggregate multi-scale features into output visual features.

[0013] S4, Align Knowledge Semantics-Visual Features, Calculate the contribution of abstract high-dimensional knowledge semantic features to visual features, optimize visual features based on contribution and output, and upsample the output to the same resolution as the original image;

[0014] S5. Construct a joint loss function and train and optimize the model by optimizing the knowledge-visual consistency loss and the class imbalance loss.

[0015] Preferably, step S1 includes the following steps:

[0016] S11, employing a large visual model to analyze previous temporal remote sensing images. and post-temporal remote sensing images Adaptive object segmentation is performed and converted into vector data to obtain pre-temporal vector geographic entities. and post-temporal vector geographic entities ;

[0017] ;

[0018] Among them, SAM The large visual model used;

[0019] S12. A Transformer network is used and network weights are trained based on remote sensing big data. Based on the trained network weights, the prior fuzzy semantic information of the preceding time-phase remote sensing images and the prior fuzzy semantic information of the following time-phase remote sensing images are obtained respectively.

[0020] S13. The prior fuzzy semantic information of the preceding time phase and the prior fuzzy semantic information of the following time phase obtained in S12 are mapped one-to-one to the preceding time phase vector geographic entities according to their spatial locations. and post-temporal vector geographic entities The above is used to construct attribute information.

[0021] Preferably, step S2 includes the following steps:

[0022] S21. Based on attribute information and using the ternary relationship of "entity-relationship-attribute", construct a spatiotemporal change geographic knowledge graph, including: spatial topological constraints, entity attributes and temporal changes;

[0023] Spatial topological constraints include adjacency and containment; entity attributes include geometric shape and category transfer; temporal changes include changing entities and changing categories;

[0024] S22 transforms the spatiotemporal change geographic knowledge graph into a full-spatiotemporal context geographic description language;

[0025] S23 uses a natural language processing model to transform the spatiotemporal contextual geographic description language into abstract, high-dimensional knowledge semantic features:

[0026] ;

[0027] Among them, BERT The natural language processing model used. , , These are, respectively, the spatial topological constraint description, the geographic attribute description, and the temporal variation description of the input. , , These are the output semantic features of spatial topological constraints, geographic attributes, and time-varying characteristics, respectively.

[0028] Preferably, step S3 includes the following steps:

[0029] S31 employs a twin network architecture, selects an encoder, and extracts previous temporal remote sensing images using a shared weight approach. Multiscale visual features and post-temporal remote sensing images Multi-scale visual features, the process is as follows:

[0030] ;

[0031] in, For the twin network used, For the encoder used, For previous time-phase remote sensing images Multi-scale visual features. For previous time-phase remote sensing images The visual features at the i-th scale, For post-temporal remote sensing images Multi-scale visual features. For post-temporal remote sensing images The visual feature at the i-th scale, i And i is an integer;

[0032] S32, fused with previous temporal remote sensing imagery Multiscale visual features and post-temporal remote sensing images Multi-scale visual features, the process is as follows:

[0033] ;

[0034] in, For feature concatenation function, For the corresponding fused feature at scale i, i And i is an integer;

[0035] S33 uses a decoder to output visual features, the process of which is as follows:

[0036] ;

[0037] in, The decoder used employs layer-by-layer upsampling, with V representing the output visual features;

[0038] ;

[0039] ;

[0040] ;

[0041] in, For feature concatenation function, For Mamba modules based on local window enhancement, This is the upsampling layer.

[0042] Preferably, the encoder includes one image block layer, multiple downsampling layers, and multiple Mamba modules based on local window enhancement, generating multi-scale visual feature scales as follows: , , , ,in, and These represent the length and width of the original image, as well as the set feature channel dimensions, respectively.

[0043] The image segmentation layer divides the input high-resolution image into N×N small image blocks to represent local details;

[0044] The downsampling layer is used to reduce high-resolution visual features to low-resolution visual features;

[0045] The Mamba module based on local window enhancement includes two branches. The first branch takes X as input features and connects it to a convolutional layer, a SiLU activation function, and a window-shifting-based SSM module. The second branch takes X as input features and connects it to a convolutional layer and a SiLU activation function. The output features of the first and second branches are combined using the Concat function and connected to a normalization layer. The calculation formula is as follows:

[0046] ;

[0047] Where Q is the output feature, Convol(·) is the convolutional layer, SiLU(·) is the activation function, Concat(·) represents the concatenate function, Norm(·) is the normalization layer, and LSSM(·) represents the window-shift-based SSM module. A window-shift-based local context scanning method is designed to form sequence features, which are then converted into discretized parameters through a linear layer.

[0048] ;

[0049] Where A and B are features transformed through a linear layer, and For discretization parameters, Linear(·) represents the linear layer, △ is the specified time scale parameter used for the discrete time term within the corresponding time scale, and I is the identity matrix;

[0050] The discretization parameters are calculated as follows:

[0051] ;

[0052] Where C represents the feature transformed through the linear layer, x t For input, y t For output, h t This represents an intermediate, learnable state, and the result is obtained using global convolution:

[0053] ;

[0054] in, The kernel is a structured convolution kernel, where P is the sequence length, x is the input, and y is the output. It is trained by convolution and performs autoregressive inference using linear recursion.

[0055] Preferably, step S4 includes the following steps:

[0056] S41, employs an attention mechanism to compute semantic features of spatial topological constraint knowledge. Geographic attribute knowledge semantic features semantic features of knowledge about time changes Contribution to visual feature V:

[0057] ;

[0058] Where V represents visual features. Semantic features of spatial topological constraints Geographic attribute knowledge semantic features semantic features of knowledge about time changes Attention is a feature used to evaluate the similarity of visual feature V. The attention mechanism is represented as:

[0059] ;

[0060] Among them, Linear For a linear projection layer, V represents visual features, and F represents abstract, high-dimensional semantic features of the input knowledge graph, including... , , R is the corresponding output G is the channel dimension of the feature. The transpose of the feature;

[0061] S42 treats knowledge semantic features as individual experts, employs a gating network to select effective experts by calculating their contributions, reduces semantic noise, and outputs optimized visual features based on the contributions of the semantic experts. The process is as follows:

[0062] ;

[0063] Among them, GateN For a gated network, K≤3 is used as the specified number of effective experts. And m is an integer;

[0064] S43 upsamples the optimized visual features O to the same resolution as the original image and outputs the probability of the segmentation category:

[0065] ;

[0066] Among them, Softmax Conv is used to output segmentation probabilities. It is a convolutional layer. This is the upsampling layer.

[0067] Preferably, step S5 includes the following steps:

[0068] S51, Construct the joint loss function;

[0069] The loss function is:

[0070] ;

[0071] in, For knowledge-visual consistency loss, For class imbalance loss, and As a regulating weight, it is used for regulation. and The balance;

[0072] S52, constructing a knowledge-visual consistency loss;

[0073] By defining knowledge-visual feature pairs as positive sample pairs and randomly sampled feature pairs as negative sample pairs, a dynamic contrast matrix is ​​constructed using cosine similarity to maximize the mutual information entropy of positive sample pairs while minimizing the association probability of negative sample pairs, as follows:

[0074] ;

[0075] Where V represents visual features, and F represents abstract, high-dimensional semantic features of the input knowledge graph description, including... , , The parameter τ is a temperature coefficient used to adjust the distribution of similarity scores, and N is the total number of pixels;

[0076] S53, construct class imbalance loss;

[0077] Using the binary cross-entropy function, we have:

[0078] ;

[0079] Where N is the total number of pixels, Y is the true label, and Output is the varying segmentation probability.

[0080] The beneficial effects of adopting the above technical solution are as follows:

[0081] 1. This invention constructs a spatiotemporal change knowledge graph by combining a large visual model and remote sensing big data. Compared with existing traditional discrete and single language tags, it forms a structured knowledge system description of spatiotemporal geographic context covering the entire scene. It organically integrates and correlates the three major elements of spatial topological constraints, geographic attributes and continuous temporal changes. This integrated geographic context description essentially builds a reasonable professional domain knowledge framework for deep learning visual operations, significantly improving the interpretability, accuracy and adaptability to complex scenes of change detection tasks.

[0082] 2. This invention enhances the visual representation ability of the Mamba network for local features by adopting a window-drift-based local context scanning method, thereby improving the robustness of change detection.

[0083] 3. This invention achieves dynamic filtering of visual information that enhances high-dimensional semantic consistency by designing three dimensions: spatial topological constraints, geographic attributes, and temporal change knowledge, while suppressing irrelevant visual noise. This enables the model to inject accurate domain prior knowledge into visual semantic understanding based on detailed spatiotemporal geographic context of the entire scene, allowing the model to distinguish complex scenes of "different objects with the same spectrum and different spectra of the same object" and further understand the dynamic change process of geographic entities.

[0084] 4. This invention improves the accuracy, generalization and interpretability of the model in remote sensing change detection scenarios by designing a joint loss function and optimizing the model by constraining knowledge-visual consistency and class imbalance. Attached Figure Description

[0085] Figure 1 This is a flowchart from an embodiment of the present invention.

[0086] Figure 2 This is a flowchart of the overall network architecture in an embodiment of the present invention.

[0087] Figure 3 This is an architecture diagram of the Mamba change detection network based on local context enhancement in an embodiment of the present invention.

[0088] Figure 4 This is a structural diagram of the Mamba module based on local window enhancement in an embodiment of the present invention. Detailed Implementation

[0089] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0090] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have

[0091] The specific orientation, construction, and operation in that specific orientation should not be construed as limiting the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0092] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0093] This application discloses a visual large model and spatiotemporal knowledge graph-driven language-visual change detection method. Its purpose is to build a reasonable professional domain knowledge framework for deep learning visual operations and significantly improve the interpretability, accuracy and adaptability to complex scenarios of change detection tasks.

[0094] Specifically, refer to Figure 1 and Figure 2 The visual big data model and spatiotemporal knowledge graph-driven language-visual change detection method includes the following steps:

[0095] S1. Design a priori fuzzy segmentation driven by a large visual model. Use a large visual model to segment dual-temporal remote sensing images into vector geographic entities. Train a deep learning network based on remote sensing big data to obtain priori fuzzy semantic information. Then, according to the spatial location, map the obtained priori fuzzy semantic information onto the vector geographic entities to form attribute information.

[0096] In this embodiment, the two time phases are time phase 1 and time phase 2, where time phase 1 is the previous time phase remote sensing image and time phase 2 is the subsequent time phase remote sensing image.

[0097] Specifically, S1 includes the following steps:

[0098] S11, employing a large visual model to analyze previous temporal remote sensing images. and post-temporal remote sensing images Adaptive object segmentation was performed separately and converted into vector data to obtain pre-temporal vector geographic entities. and post-temporal vector geographic entities ;

[0099] ;

[0100] Among them, SAM The visual large model used in this embodiment is the segment anything model. For pre-temporal vector geographic entities, For later-phase vector geographic entities;

[0101] S12. A Transformer network is used and network weights are trained based on remote sensing big data. Based on the trained network weights, the prior fuzzy semantic information of the preceding time-phase remote sensing images and the prior fuzzy semantic information of the following time-phase remote sensing images are obtained respectively.

[0102] Among them, remote sensing big data refers to existing open-source remote sensing semantic segmentation datasets.

[0103] S13. The prior fuzzy semantic information of the preceding time phase and the prior fuzzy semantic information of the following time phase obtained in S12 are mapped one-to-one to the preceding time phase vector geographic entities according to their spatial locations. and post-temporal vector geographic entities The above is used to construct attribute information.

[0104] The attribute information includes front-phase attribute information and back-phase attribute information.

[0105] S2, construct a spatiotemporal change geographic knowledge graph based on attribute information, transform the spatiotemporal change geographic knowledge graph into a full spatiotemporal context geographic description language, and deconstruct the full spatiotemporal context geographic description language into abstract high-dimensional knowledge semantic features based on a natural language processing model;

[0106] S21. Based on the previous and subsequent time-phase attribute information and using the "entity-relationship-attribute" ternary relationship, a spatiotemporal change geographic knowledge graph is constructed. The spatiotemporal change geographic knowledge graph includes: spatial topological constraints, entity attributes and temporal changes.

[0107] Spatial topological constraints include adjacency and containment; entity attributes include geometric shape and category transfer; temporal changes include changing entities and changing categories.

[0108] S22 transforms the spatiotemporal change geographic knowledge graph into a full-spatiotemporal context geographic description language;

[0109] Specifically, the conversion rules are as follows:

[0110] The spatial topology constraints are described as follows: the pre-temporal spatiotemporal topology is described as follows: (Entity A) is located in the (direction) of (Entity B), ...; the post-temporal spatiotemporal topology is described as follows: (Entity A) is located in the (direction) of (Entity B), ...

[0111] The geographic attribute description is as follows: The spatiotemporal attribute description of the previous phase is as follows: (Entity A) has a category of (Category), ...; The spatiotemporal attribute description of the subsequent phase is as follows: (Entity A) has a category of (Category), ...

[0112] The temporal change is described as follows: the temporal and spatiotemporal change is described as follows: (Entity A) is spatially aligned with (Entity B), ...; (Entity A) changes temporally with (Entity B), ...

[0113] S23 uses a natural language processing model to transform the spatiotemporal contextual geographic description language into abstract, high-dimensional knowledge semantic features:

[0114] ;

[0115] Among them, BERT The natural language processing model used. , , These are, respectively, the spatial topological constraint description, the geographic attribute description, and the temporal variation description of the input. , , These are the output semantic features of spatial topological constraints, geographic attributes, and time-varying characteristics, respectively.

[0116] S3, based on a twin architecture, uses an encoder to extract multi-scale features from dual-temporal remote sensing images, enhances global perception of visual features by compensating for local context, and uses a decoder to aggregate multi-scale features into output visual features.

[0117] It constructs an encoder and decoder by designing a Mamba change detection network based on local context enhancement, and finally outputs visual features;

[0118] Specifically, refer to Figure 3 It includes the following steps:

[0119] S31 employs a twin network architecture, selects an encoder, and extracts previous temporal remote sensing images using a shared weight approach. Multiscale visual features and post-temporal remote sensing images Multi-scale visual features, the process is as follows:

[0120] ;

[0121] in, For the twin network used, For the encoder used, For previous time-phase remote sensing images Multi-scale visual features. For previous time-phase remote sensing images The visual features at the i-th scale, For post-temporal remote sensing images Multi-scale visual features. For post-temporal remote sensing images The visual feature at the i-th scale, i And i is an integer;

[0122] Among them, reference Figure 4 The encoder includes one image block layer, multiple downsampling layers, and multiple Mamba modules based on local window enhancement. The generated multi-scale visual feature scales are as follows: , , , ,in, and These represent the length and width of the original image, as well as the set feature channel dimensions, respectively.

[0123] The image patch layer divides the input high-resolution image into N×N small image patches to represent local details;

[0124] Downsampling layers are used to reduce high-resolution visual features to low-resolution visual features;

[0125] The Mamba module based on local window enhancement includes two branches. The first branch takes X as the input feature and connects it to a convolutional layer, a SiLU activation function, and a window-shifting SSM module. The second branch takes X as the input feature and connects it to a convolutional layer and a SiLU activation function. Here, X represents the input feature, which is not specific to any particular parameter, but simply represents an input feature. Furthermore, the input feature of the second branch can be different from that of the first branch.

[0126] The Concat function is used to merge the output features of the first and second branches, and then the results are connected to the output of a normalized layer. The calculation formula is as follows:

[0127] ;

[0128] Where Q is the output feature, Convol(·) is the convolutional layer, SiLU(·) is the activation function, Concat(·) represents the concatenate function, Norm(·) is the normalization layer, and LSSM(·) represents the window-shift-based SSM module. A window-shift-based local context scanning method is designed to form sequence features, which are then converted into discretized parameters through a linear layer.

[0129] ;

[0130] Where A and B are features transformed through a linear layer, and For discretization parameters, Linear(·) represents the linear layer, △ is the specified time scale parameter used for the discrete time term within the corresponding time scale, and I is the identity matrix;

[0131] The discretization parameters are calculated as follows:

[0132] ;

[0133] Where C represents the feature transformed through the linear layer, x t For input, y t For output, h t This represents an intermediate learning state, and the result is obtained using global convolution:

[0134] ;

[0135] in, The kernel is a structured convolution kernel, where P is the sequence length, x is the input, and y is the output. It is trained by convolution and performs autoregressive inference using linear recursion.

[0136] S32, fused with previous temporal remote sensing imagery Multiscale visual features and post-temporal remote sensing images Multi-scale visual features, the process is as follows:

[0137] ;

[0138] in, For feature concatenation function, For the corresponding fused feature at scale i, i And i is an integer;

[0139] S33 uses a decoder to output visual features, the process of which is as follows:

[0140] ;

[0141] in, The decoder used employs layer-by-layer upsampling, with V representing the output visual features;

[0142] ;

[0143] ;

[0144] ;

[0145] in, For feature concatenation function, For Mamba modules based on local window enhancement, This is the upsampling layer.

[0146] S4, Align Knowledge Semantics-Visual Features, Calculate the contribution of abstract high-dimensional knowledge semantic features to visual features, optimize visual features based on contribution and output, and upsample the output to the same resolution as the original image;

[0147] S41, employs an attention mechanism to compute semantic features of spatial topological constraint knowledge. Geographic attribute knowledge semantic features semantic features of knowledge about time changes Contribution to visual feature V:

[0148] ;

[0149] Where V represents visual features. Semantic features of spatial topological constraints Geographic attribute knowledge semantic features semantic features of knowledge about time changes Attention is a feature used to evaluate the similarity of visual feature V. The attention mechanism is represented as:

[0150] ;

[0151] Among them, Linear For a linear projection layer, V represents visual features, and F represents abstract, high-dimensional semantic features of the input knowledge graph, including... , , R is the corresponding output G is the channel dimension of the feature. The transpose of the feature;

[0152] S42 treats knowledge semantic features as individual experts, employs a gating network to select effective experts by calculating their contributions, reduces semantic noise, and outputs optimized visual features based on the contributions of the semantic experts. The process is as follows:

[0153] ;

[0154] Among them, GateN For a gated network, K≤3 is used as the specified number of effective experts. And m is an integer;

[0155] S43 upsamples the optimized visual features O to the same resolution as the original image and outputs the probability of the segmentation category:

[0156] ;

[0157] Among them, Softmax Conv is used to output segmentation probabilities. It is a convolutional layer. This is the upsampling layer.

[0158] S5. Construct a joint loss function and train and optimize the model by optimizing the knowledge-visual consistency loss and the class imbalance loss.

[0159] S51, Construct the joint loss function;

[0160] The loss function is:

[0161] ;

[0162] in, For knowledge-visual consistency loss, For class imbalance loss, and As a regulating weight, it is used to regulate and The balance;

[0163] S52, constructing a knowledge-visual consistency loss;

[0164] By defining knowledge-visual feature pairs as positive sample pairs and randomly sampled feature pairs as negative sample pairs, a dynamic contrast matrix is ​​constructed using cosine similarity to maximize the mutual information entropy of positive sample pairs while minimizing the association probability of negative sample pairs, as follows:

[0165] ;

[0166] Where V represents visual features, and F represents abstract, high-dimensional semantic features of the input knowledge graph description, including... , , The parameter τ is a temperature coefficient used to adjust the distribution of similarity scores, and N is the total number of pixels;

[0167] S53, construct class imbalance loss;

[0168] Using the binary cross-entropy function, we have:

[0169] ;

[0170] Where N is the total number of pixels, Y is the true label, and Output is the varying segmentation probability.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A language-visual change detection method driven by a large visual model and spatiotemporal knowledge graph, characterized in that, Includes the following steps: S1 uses a large visual model to segment dual-temporal remote sensing images into vector geographic entities. A deep learning network is trained based on remote sensing big data to obtain prior fuzzy semantic information. The obtained prior fuzzy semantic information is then mapped to the vector geographic entities according to their spatial location to form attribute information. S2, construct a spatiotemporal change geographic knowledge graph based on attribute information, transform the spatiotemporal change geographic knowledge graph into a full spatiotemporal context geographic description language, and deconstruct the full spatiotemporal context geographic description language into abstract high-dimensional knowledge semantic features based on a natural language processing model; S3, based on a twin architecture, uses an encoder to extract multi-scale features from dual-temporal remote sensing images, enhances global perception of visual features by compensating for local context, and uses a decoder to aggregate multi-scale features into output visual features. S4, Align Knowledge Semantics-Visual Features, Calculate the contribution of abstract high-dimensional knowledge semantic features to visual features, optimize visual features based on contribution and output, and upsample the output to the same resolution as the original image; S5. Construct a joint loss function and train and optimize the model by optimizing the knowledge-visual consistency loss and the class imbalance loss.

2. The visual large model and spatiotemporal knowledge graph-driven language-visual change detection method according to claim 1, characterized in that, S1 includes the following steps: S11, employing a large visual model to analyze previous temporal remote sensing images. and post-temporal remote sensing images Adaptive object segmentation is performed and converted into vector data to obtain pre-temporal vector geographic entities. and post-temporal vector geographic entities ; ； Among them, SAM The large visual model used; S12. A Transformer network is used and network weights are trained based on remote sensing big data. Based on the trained network weights, the prior fuzzy semantic information of the preceding time-phase remote sensing images and the prior fuzzy semantic information of the following time-phase remote sensing images are obtained respectively. S13. The prior fuzzy semantic information of the preceding time phase and the prior fuzzy semantic information of the following time phase obtained in S12 are mapped one-to-one to the preceding time phase vector geographic entities according to their spatial locations. and post-temporal vector geographic entities The above is used to construct attribute information.

3. The visual large model and spatiotemporal knowledge graph-driven language-visual change detection method according to claim 2, characterized in that, S2 includes the following steps: S21. Based on attribute information and using the ternary relationship of "entity-relationship-attribute", construct a spatiotemporal change geographic knowledge graph, including: spatial topological constraints, entity attributes and temporal changes; Spatial topological constraints include adjacency and containment; entity attributes include geometric shape and category transfer; temporal changes include changing entities and changing categories; S22 transforms the spatiotemporal change geographic knowledge graph into a full-spatiotemporal context geographic description language; S23 uses a natural language processing model to transform the spatiotemporal contextual geographic description language into abstract, high-dimensional knowledge semantic features: ； Among them, BERT The natural language processing model used. , , These are, respectively, the spatial topological constraint description, the geographic attribute description, and the temporal variation description of the input. , , These are the output semantic features of spatial topological constraints, geographic attributes, and time-varying characteristics, respectively.

4. The visual large model and spatiotemporal knowledge graph-driven language-visual change detection method according to claim 3, characterized in that, S3 includes the following steps: S31 employs a twin network architecture, selects an encoder, and extracts previous temporal remote sensing images using a shared weight approach. Multiscale visual features and post-temporal remote sensing images Multi-scale visual features, the process is as follows: ； in, For the twin network used, For the encoder used, For previous time-phase remote sensing images Multi-scale visual features. For previous time-phase remote sensing images The visual features at the i-th scale, For post-temporal remote sensing images Multi-scale visual features. For post-temporal remote sensing images The visual feature at the i-th scale, i And i is an integer; S32, fused with previous temporal remote sensing imagery Multiscale visual features and post-temporal remote sensing images Multi-scale visual features, the process is as follows: ； in, For feature concatenation function, For the corresponding fused feature at scale i, i And i is an integer; S33 uses a decoder to output visual features, the process of which is as follows: ； in, The decoder used employs layer-by-layer upsampling, with V representing the output visual features; ； ； ； in, For feature concatenation function, For Mamba modules based on local window enhancement, This is the upsampling layer.

5. The visual large model and spatiotemporal knowledge graph-driven language-visual change detection method according to claim 4, characterized in that, The encoder includes one image block layer, multiple downsampling layers, and multiple Mamba modules based on local window enhancement. The generated multi-scale visual feature scales are as follows: , , , ,in, and These represent the length and width of the original image, as well as the set feature channel dimensions, respectively. The image segmentation layer divides the input high-resolution image into N×N small image blocks to represent local details; The downsampling layer is used to reduce high-resolution visual features to low-resolution visual features; The Mamba module based on local window enhancement includes two branches. The first branch takes X as input features and connects it to a convolutional layer, a SiLU activation function, and a window-shifting-based SSM module. The second branch takes X as input features and connects it to a convolutional layer and a SiLU activation function. The output features of the first and second branches are combined using the Concat function and connected to a normalization layer. The calculation formula is as follows: ； Where Q is the output feature, Convol(·) is the convolutional layer, SiLU(·) is the activation function, Concat(·) represents the concatenate function, Norm(·) is the normalization layer, and LSSM(·) represents the window-shift-based SSM module. A window-shift-based local context scanning method is designed to form sequence features, which are then converted into discretized parameters through a linear layer. ； Where A and B are features transformed through a linear layer, and For discretization parameters, Linear(·) represents the linear layer, △ is the specified time scale parameter used for the discrete time term within the corresponding time scale, and I is the identity matrix; The discretization parameters are calculated as follows: ； Where C represents the feature transformed through the linear layer, x t For input, y t For output, h t This represents an intermediate, learnable state, and the result is obtained using global convolution: ； in, The kernel is a structured convolution kernel, where P is the sequence length, x is the input, and y is the output. It is trained by convolution and performs autoregressive inference using linear recursion.

6. The visual large model and spatiotemporal knowledge graph-driven language-visual change detection method according to claim 4, characterized in that, S4 includes the following steps: S41, employs an attention mechanism to compute semantic features of spatial topological constraint knowledge. Geographic attribute knowledge semantic features semantic features of knowledge about time changes Contribution to visual feature V: ； Where V represents visual features. Semantic features of spatial topological constraints Geographic attribute knowledge semantic features semantic features of knowledge about time changes Attention is a feature used to evaluate the similarity of visual feature V. The attention mechanism is represented as: ； Among them, Linear For a linear projection layer, V represents visual features, and F represents abstract, high-dimensional semantic features of the input knowledge graph, including... , , R is the corresponding output G is the channel dimension of the feature. The transpose of the feature; S42 treats knowledge semantic features as individual experts, employs a gating network to select effective experts by calculating their contributions, reduces semantic noise, and outputs optimized visual features based on the contributions of the semantic experts. The process is as follows: ； Among them, GateN For a gated network, K≤3 is used as the specified number of effective experts. And m is an integer; S43 upsamples the optimized visual features O to the same resolution as the original image and outputs the probability of the segmentation category: ； in, To be used for outputting segmentation probabilities, It is a convolutional layer. This is the upsampling layer.

7. The visual large model and spatiotemporal knowledge graph-driven language-visual change detection method according to claim 1, characterized in that, The S5 Includes the following steps: S51, Construct the joint loss function; The loss function is: ； in, For knowledge-visual consistency loss, For class imbalance loss, and As a regulating weight, it is used for regulation. and The balance; S52, constructing a knowledge-visual consistency loss; By defining knowledge-visual feature pairs as positive sample pairs and randomly sampled feature pairs as negative sample pairs, a dynamic contrast matrix is ​​constructed using cosine similarity to maximize the mutual information entropy of positive sample pairs while minimizing the association probability of negative sample pairs, as follows: ； Where V represents visual features, and F represents abstract, high-dimensional semantic features of the input knowledge graph description, including... , , The parameter τ is a temperature coefficient used to adjust the distribution of similarity scores, and N is the total number of pixels; S53, construct class imbalance loss; Using the binary cross-entropy function, we have: ； Where N is the total number of pixels, Y is the true label, and Output is the varying segmentation probability.

Images