Remote sensing image-text retrieval method and system based on nine-intersection model and laplace alignment

Abstract

The application discloses a remote sensing image-text retrieval method and system based on a nine-intersection model and Laplace alignment, and aims at the problem of difficult image-text semantic alignment under the conditions of few samples and zero samples. The method extracts global and local text features from the text description, wherein the local features are extracted by a large language model to extract entities and spatial relationships, are divided into three categories of topology, direction and distance according to the nine-intersection model, a dynamic weighted adjacency matrix is constructed and is extracted through graph convolution; global and local visual features are extracted from the remote sensing image, wherein the local features are enhanced in multiple scales by using an asymmetric dynamic receptive field, more branches are used for low-frequency features and fewer branches are used for high-frequency features, and a visual relationship graph is constructed; the local and global features are locally guided and fused; a loss function containing a Laplace operator difference loss is constructed to constrain the consistency of the image-text spatial structure. The application improves the accuracy and robustness of remote sensing image-text retrieval under the conditions of few samples and zero samples.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing data processing and artificial intelligence, specifically to a remote sensing image and text retrieval method and system based on the nine-intersection model and Laplace alignment, which is used to improve the semantic alignment capability and retrieval performance between remote sensing images and text descriptions. Background Technology

[0002] With the rapid development of remote sensing imaging technology, the amount of remote sensing image data is growing exponentially, making the efficient management and utilization of massive amounts of remote sensing data an urgent need. Cross-modal retrieval technology, as an important bridge connecting images and text, can enable the retrieval of relevant remote sensing images through natural language queries, and has broad application prospects. Large-scale vision-language pre-trained models (such as CLIP and ALIGN) have made significant progress in cross-modal retrieval in the natural image domain, demonstrating strong performance in zero-shot and few-shot tasks by learning the alignment relationship between images and text. However, when directly transferred to the remote sensing domain, the effect is limited due to the unique characteristics of remote sensing data (such as sparse targets, complex spatial structure, and high background redundancy). How to design an adaptation mechanism for the characteristics of remote sensing scenes, especially under conditions of few-shot and zero-shot data lacking a large amount of labeled data, to achieve stable and accurate cross-modal retrieval has become a key problem that current research urgently needs to overcome.

[0003] Existing technologies mainly include the following two aspects: 1. Feature Representation Methods for Cross-Modal Retrieval Remote sensing cross-modal retrieval aims to match corresponding remote sensing images through natural language text queries, or conversely, to achieve image-to-text retrieval, and is widely used in fields such as geographic information extraction, urban planning, and disaster monitoring. Due to the unique characteristics of remote sensing data, such as imaging methods, scale variations, target sparsity, and complex layouts, existing retrieval methods have mainly evolved from coarse-grained alignment based on global features to refined methods incorporating local region perception and structured modeling. Traditional methods often rely on convolutional neural networks (CNNs) to extract overall image features, combined with language models to encode text descriptions, and optimize the cross-modal mapping space through contrastive learning. Subsequently, to address issues such as dense targets, small target distribution, and complex spatial relationships in remote sensing scenes, researchers have introduced techniques such as target detection, region attention, and graph convolutional networks (GCNs) to model fine-grained entities and their relationships in both visual and linguistic modalities, improving the ability to perceive structure and model local details. Although the aforementioned advancements have improved cross-modal retrieval performance in remote sensing scenarios to some extent, existing methods still face significant challenges in terms of domain adaptability, structural consistency modeling, and robustness when dealing with zero-sample or few-sample conditions.

[0004] 2. Application of visual language models in cross-modal retrieval In recent years, Visual-Language Models (VLMs), such as CLIP and ALIGN, have made groundbreaking progress in cross-modal retrieval tasks in the field of natural images. These models learn the alignment relationships between images and text by pre-training on large-scale image-text pairing datasets, achieving high-precision retrieval under zero-shot and few-shot conditions without requiring extensive annotation. Methods represented by CLIP employ independent encoders to extract image and text features, and optimize through contrastive loss to maintain semantic consistency of features from different modalities in a unified space, thereby achieving efficient retrieval from natural language descriptions to image content. Numerous studies have validated the superior performance of VLMs on datasets such as COCO and Flickr30k in tasks including zero-shot retrieval, image-text matching, and image-text generation.

[0005] Furthermore, VLMs have demonstrated strong cross-modal transfer capabilities and broad adaptability in fields such as medical imaging, art retrieval, and video understanding, driving the development of multimodal learning technologies. Especially in application scenarios lacking extensive manual annotation, VLMs, relying on their large-scale pre-training characteristics, have exhibited superior data efficiency and generalization ability, becoming an important technical approach in the current cross-modal retrieval field. However, despite their excellent performance in natural image domains, direct transfer to remote sensing data still faces significant challenges, including large differences in domain distribution, significant variations in target scale, and complex spatial structures, affecting cross-modal alignment effectiveness and retrieval accuracy.

[0006] Disadvantages of existing technology: (1) Insufficient cross-modal structural alignment capability limits the improvement of retrieval accuracy. Most traditional cross-modal retrieval methods adopt simple feature alignment strategies, ignoring the structural correspondence between images and text at the level of local entities and relationships. Although existing visual language models (such as CLIP) have good alignment results in natural scenes, in the field of remote sensing, due to the large differences in domain distribution and complex structural features, there is a lack of effective local structural consistency modeling, resulting in coarse inter-modal alignment and reducing the accuracy and robustness of the retrieval system.

[0007] (2) Insufficient support for adaptation under conditions of few or zero samples. Remote sensing data annotation is costly and samples are scarce, and practical applications often face the need for retrieval with few or zero samples. Existing methods generally rely on large-scale labeled data for optimization, and lack structure enhancement and alignment mechanisms for conditions of small data volume. This leads to a significant drop in model performance when labeled data is limited, and insufficient cross-domain transferability and generalization, making it difficult to meet the requirements of efficiency and stability of retrieval systems in practical remote sensing applications. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention discloses a remote sensing image and text retrieval method and system based on the nine-intersection model and Laplace alignment. By using the nine-intersection model to perform structured modeling of text spatial relationships, enhancing multi-scale target perception of remote sensing images with asymmetric dynamic receptive fields, and constraining cross-modal spatial structure consistency with Laplace operator difference loss, the accuracy and robustness of remote sensing image and text retrieval under conditions of few samples and zero samples are significantly improved.

[0009] The remote sensing image retrieval method based on the nine-intersection model and Laplace alignment designed in this invention includes: Global text features are extracted from the text descriptions of remote sensing images using a pre-trained language model, and local text features are obtained through the following steps: entities and spatial relationships are extracted using a large language model, and classified into three categories according to the nine-intersection model: topology, orientation, and distance. A relation-weighted adjacency matrix is ​​constructed and learnable dynamic weights are assigned to different types. The features are then extracted through a graph convolutional network. Global and local visual features are extracted from remote sensing images. The global visual features are obtained by extracting the overall scene of the image using a visual Transformer. The local features are obtained by the following steps: multi-scale enhancement is performed using an asymmetric dynamic receptive field, with more receptive field branches for low-frequency features and fewer branches for high-frequency features, and a visual relationship graph is constructed based on the enhanced features. Local text features are fused with global text features, and local features are fused with global visual features, respectively, to obtain fused text features and fused visual features; during fusion, attention weights are generated from local features to optimize global features. Construct a loss function with Laplacian operator difference loss to constrain the spatial structural consistency between fused visual features and fused text features; Cross-modal retrieval based on the fusion of visual features and fusion of text features.

[0010] Furthermore, the asymmetric dynamic receptive field strategy specifically includes: Residual networks are used to extract feature maps of different depths to obtain low-frequency features. and high frequency characteristics ; Regarding the low-frequency features Multi-scale convolution is performed using at least three receptive field branches of different sizes to process the high-frequency features. Multi-scale convolution is performed using two receptive field branches of different sizes; The local features are obtained by fusing the low-frequency features enhanced by multi-scale with the high-frequency features.

[0011] Furthermore, the extraction of local text features also includes: Using the entities extracted from the large language model as nodes, node embedding representations are obtained through a word embedding model; Based on the relation types classified by the nine-intersection model, different learnable weights are assigned to the edges to construct a relation-weighted adjacency matrix A, where topological relations, directional relations, and distance relations correspond to the weights, respectively. , , ; The text structural features are obtained by extracting structured features from the text semantic graph using a three-layer graph convolutional network.

[0012] Furthermore, the Laplacian operator difference loss is calculated in the following manner: Calculate the spatial relationship feature matrix corresponding to the fused visual features respectively. The spatial relationship feature matrix corresponding to the fused text features Discrete Laplace operator:

[0013] Calculate the Laplacian operator difference loss:

[0014] in, For the sample size, For the first i One image, For the first i A text description.

[0015] Furthermore, the loss function is a weighted combination of a global alignment term and a Laplacian operator difference loss term; The global alignment term uses triplet loss. This is used to optimize the overall distance between the fused visual features and the fused text features in the common feature space; The loss function is expressed as:

[0016] in, For the Laplacian operator difference loss, , Weighting parameters to balance the impact of different loss terms.

[0017] Furthermore, the local guidance mechanism in the fusion step specifically includes: For visual modalities, through learnable parameter matrices , Global visual features and local features Project onto the hidden space and compute the attention matrix. :

[0018] in, To hide spatial dimensions; Using the attention matrix For the global visual features Optimization is performed to obtain the global features of local optimization. ; A multi-level fusion strategy is used to obtain fused visual features:

[0019] in, , , For learnable weights, This indicates element-wise multiplication.

[0020] Furthermore, a local guidance mechanism symmetrical to that of the visual modality is adopted for the text modality to obtain fused text features; wherein, the fusion method of global text features and local text features is the same as that of the visual modality, and the local guidance mechanism of the visual modality and the text modality adopts independently learned parameter matrices and does not share parameters.

[0021] Furthermore, the method is applied to remote sensing cross-modal retrieval tasks under conditions of few or zero samples; the large language model adopts the DeepSeek model.

[0022] Furthermore, the pre-trained language model includes RoBERTa or BERT, the visual Transformer includes ViT, the convolutional neural network includes ResNet, and the graph convolutional network is a three-layer graph convolutional network structure.

[0023] Based on the same inventive concept, this invention also discloses a system for implementing a remote sensing image and text retrieval method based on the nine-intersection model and Laplace alignment, comprising: The text feature extraction module is used to perform the text feature extraction steps of the remote sensing image retrieval method based on the nine-intersection model and Laplacian alignment. The pre-trained language model extracts global text features from the text description of the remote sensing image, and uses a large language model to extract entities and spatial relationships, classifies them according to the nine-intersection model, constructs a dynamic weighted adjacency matrix, and extracts local text features through a graph convolutional network. The visual feature extraction module is used to perform the visual feature extraction steps of the remote sensing image retrieval method based on the nine-intersection model and Laplacian alignment. It extracts global visual features by visual Transformer or convolutional neural network, and extracts local features and constructs a visual relationship graph by using an asymmetric dynamic receptive field strategy. The feature fusion module is used to perform the fusion steps of the remote sensing image retrieval method based on the nine-intersection model and Laplacian alignment, and to fuse local text features with global text features and local features with global visual features to obtain fused text features and fused visual features. The cross-modal alignment module is used to perform the cross-modal alignment steps of the remote sensing image retrieval method based on the nine-intersection model and Laplacian alignment, construct a loss function including the Laplacian operator difference loss, and constrain the spatial structural consistency of the fused visual features and fused text features. The retrieval module performs cross-modal retrieval based on the fusion of visual features and fusion of text features.

[0024] Compared with existing remote sensing cross-modal retrieval technologies, this invention makes systematic innovations in feature modeling, cross-modal alignment, and adaptive enhancement, and has the following advantages and beneficial technical effects: (1) Enhanced fine-grained perception capability of complex spatial structures By introducing object detection and graph convolutional network modeling of spatial relationships between objects in the visual modality, and extracting entities and their semantic relationships based on a large language model in the text modality, this invention constructs bimodal local structural features. This effectively captures fine-grained structural information in remote sensing images and text descriptions. Compared to traditional methods based on global feature alignment, this invention can accurately model and understand local detailed relationships in remote sensing scenes with densely distributed multiple objects, significant scale variations, and complex spatial layouts, significantly improving the retrieval system's ability to represent complex scenes and its retrieval accuracy.

[0025] (2) Improved cross-modal structural consistency and retrieval robustness This invention designs a local guided feature fusion mechanism and a hierarchical graph alignment cross-modal loss function to simultaneously optimize the alignment between visual and textual elements at both the global semantic level and the local node relationship level. By dynamically fusing local and global features and achieving structural alignment at the local entity level, this invention effectively mitigates the impact of intermodal heterogeneity differences on retrieval performance and improves the consistency and discriminability of the cross-modal feature space. Especially under conditions of few or no samples, this invention maintains strong retrieval performance and cross-domain generalization ability, overcoming the problem of significant performance degradation in traditional methods under data scarcity, and possesses higher practical value and promotion potential.

[0026] (3) It improves the ability to express global semantics and local spatial relationships in a coordinated manner. This invention does not rely solely on single-modality or single-scale features for image-text matching. Instead, it extracts global semantic features and local structural features from both text descriptions and remote sensing images, and adaptively optimizes global features through a local guidance mechanism. On the text side, a large language model is used to extract entities and spatial relationships, and a nine-intersection model is used to structurally model topological, directional, and distance relationships, enabling explicit expression of spatial semantics in the text description. On the visual side, an asymmetric dynamic receptive field enhances the perception of targets at different scales, and a visual relationship graph is further constructed to depict the spatial layout relationships between targets. Therefore, this invention establishes a closer connection between global scene semantics, local target information, and spatial structural relationships, avoiding the problems of overly coarse global features, insufficient utilization of local information, and inconsistencies in image-text spatial relationships found in traditional methods. This further improves the accuracy, stability, and interpretability of remote sensing image-text retrieval results. Attached Figure Description

[0027] Figure 1 These are remote sensing images at different scales.

[0028] Figure 2 This is a schematic diagram of the dynamic receptive field structure in this invention.

[0029] Figure 3 This is a schematic diagram of the local text representation structure.

[0030] Figure 4 This is a schematic diagram of the localized fusion mechanism structure in this invention.

[0031] Figure 5 A schematic diagram of the remote sensing image retrieval method based on the nine-intersection model and Laplace alignment of the present invention. Detailed Implementation

[0032] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0033] Example 1 This embodiment discloses a remote sensing image and text retrieval method based on the nine-intersection model and Laplacian alignment. The core of this method includes visual local structure modeling, text local structure modeling, local guided feature fusion, and hierarchical cross-modal alignment. The specific process includes: 1. Experimental Environment and Training Parameters This example experiment uses an Intel(R) Xeon(R) W-2123 3.60GHz processor, 64GB of memory, Ubuntu 16.04 64-bit operating system, and an NVIDIA A100 Tensor Core GPU. Other environment features include CUDA version 11.7, Python version 3.10, and modeling based on the PyTorch deep learning framework version 1.13.0.

[0034] During training, a batch size of 100 was used, employing the Adam optimizer with an initial learning rate of 0.0002, linear decay at a rate of 0.5, and a decay period of 25 epochs, for a total of 70 training epochs. During training, the remote sensing images were uniformly scaled to 224x224. To improve the model's robustness, a series of data augmentations were performed, including image color perturbation, image scale transformation, and image flipping. Additionally, the text truncation was set to 64, insufficient for empty characters (…). <unk>The embedding space for both images and text is 512, and the boundary value for the triplet loss is set to 0.2.

[0035] 2. This embodiment discloses an overview of a remote sensing image and text retrieval method based on the nine-intersection model and Laplace alignment. This invention addresses the significant structural differences between remote sensing images and text descriptions by proposing a method for collaborative modeling of global and local features. First, in the visual modality, global features of the image are extracted using a Visual Transformer (ViT) or Convolutional Neural Network (CNN). In the text modality, pre-trained language models (such as RoBERTa) are used to extract overall semantic features. To capture key targets and their spatial layout information in remote sensing images, a target detection module based on dynamic receptive fields is further introduced to detect local target regions and construct a visual relation graph. Local structural features are then extracted using a Graph Convolutional Network (GCN). Simultaneously, for the text descriptions, a Large Language Model (LLM) is employed to extract entities and relationships, constructing a text semantic graph and performing structured modeling. This achieves multi-granularity feature representation across both modalities, laying the foundation for fine-grained cross-modal alignment.

[0036] The Visual Relational Representation Module (V-RS) design addresses the limitations of traditional visual representation methods, which typically rely on convolutional neural networks (CNNs) for feature extraction or coarse relation modeling. While these methods can capture global features of remote sensing images, the multi-scale nature of these images presents challenges. Figure 1 Fine-grained target information can easily be lost in the noise, leading to insufficient attention paid to important objects in images by the model. Furthermore, visual representation methods typically do not consider spatial relationships between targets, or only perform coarse relationship modeling. This prevents the model from accurately understanding and utilizing the spatial layout and interrelationships between targets in an image, thus limiting the expressive power of features and the accuracy of retrieval. To address these issues, this invention designs a Visual Relationship Module (V-RS). On one hand, the V-enc. branch is designed to achieve fine extraction of visual features through multi-scale feature extraction and dynamic receptive field feature enhancement; on the other hand, the V-spa branch is designed to perform efficient spatial relationship modeling of targets in remote sensing images, thereby enhancing the spatial representation of visual representations.

[0037] Locally-Guided Cross-Modal Feature Fusion Mechanism: To address the scale inconsistency and information conflict issues inherent in traditional cross-modal feature fusion, this invention designs a Local-Guided Feature Fusion (LGFF) mechanism. Through dynamic interaction between local target features and global scene features, it optimizes the representation distribution of global features using fine-grained local information. Specifically, within the visual and text modalities, attention weights are generated based on local features, and global feature representations are reweighted. A multi-level feature fusion strategy is employed to ensure the guiding role of local information in global representation. This mechanism effectively alleviates the problems of local feature weakening or redundant interference, improving the consistency of intra-modal representation and the stability of cross-modal matching.

[0038] Because the global visual features of remote sensing images typically cover a large geographical area, they are often limited by low resolution, resulting in insufficient fine-grained representation of key targets. Conversely, local remote sensing targets provide rich, fine-grained information, including target categories, spatial distribution, and structural relationships. Similarly, text descriptions at the global level mainly summarize the semantic information of the scene, while local text entities precisely characterize target objects and their spatial relationships. Directly concatenating or simply weighting and fusing global and local features may lead to information redundancy, feature conflicts, or target ambiguity, thus affecting the stability of cross-modal alignment. To address this issue, we propose a Locally Guided Fusion Mechanism (LGFF), such as... Figure 4 As shown, this mechanism employs multi-level local attention, enabling local information to not only serve as supplementary information but also actively optimize the representation structure of global features, thereby enhancing cross-modal alignment capabilities.

[0039] 3. Specific Implementation of Visual Feature Extraction Global Visual Feature Extraction: This invention uses a publicly available remote sensing image-text pairing dataset for training and testing. Each remote sensing image is accompanied by multiple text descriptions to provide local semantic information about the image. The image data is uniformly adjusted to a fixed size and normalized. The text data is first processed to remove stop words, and then entities and spatial relationships are extracted from the text descriptions using the DeepSeek large language model. The extracted entities are used as nodes, and the spatial relationships between entities are used as edges. According to the definition of the nine-intersection model, spatial relationships are classified into three categories: topological relationships, directional relationships, and distance relationships, thus constructing a spatial relationship vocabulary. Table 1 shows the specific contents of the spatial relationship vocabulary constructed in this invention.

[0040] Table 1 Spatial Relationship Terminology

[0041] For the input remote sensing image, global features are first extracted based on a visual rancer (ViT) or convolutional neural network (CNN). In this embodiment, ResNet18 is used as the backbone network to extract image features. The feature map extracted for each residual block is defined as follows: ,in The input is the original image. Low-frequency and high-frequency features are obtained using the following formula:

[0042]

[0043] in, Represents residual block After upsampling and The low-frequency characteristics after addition are obtained similarly. High-frequency features. This results in feature maps of different depths.

[0044] Asymmetric Dynamic Receptive Field Enhancement: A method for dynamic receptive field enhancement is proposed to optimize multi-size remote sensing images, such as... Figure 2 As shown. The main idea is to use multiple convolutional kernels of different sizes on the input feature maps at different scales, and automatically select appropriate convolutional kernels to combine the extracted features to generate channel attention information, thereby achieving feature extraction and enhancement at different scales. The specific steps are as follows: First, input the feature map. , dimension , respectively , Convolutional kernels with different receptive field sizes, for Perform convolution operations to obtain feature maps with multiple receptive fields. and The corresponding elements of these feature maps are added together and fused to obtain a fused feature map.

[0045] (1) Secondly, the fused feature map Perform global average pooling to obtain dimension . Channel feature map The process is defined as follows: and through Fully connected extraction yields channel attention. Then Convolution is performed with convolution kernels of different receptive field sizes to obtain channel attention information corresponding to multiple receptive fields.

[0046] (2) (3) Finally, the multi-receptive-field convolution features , ...and channel attention information Multiply the results and output the attention features of each receptive field separately. , ...and these feature maps are fused to obtain the final multi-receptive-field channel attention feature map. .

[0047] Based on the above process, Θ is defined here, where Θ(·) represents the dynamic receptive field feature enhancement module, used to perform multi-scale convolution, channel attention weighting, and feature fusion on the input feature map, thereby outputting the enhanced feature representation. Since low-frequency features contain richer details, more receptive field branches of different sizes are designed, while high-frequency features contain fewer image domain features, so only two receptive field branches are designed. The processing of this multi-level feature enhancement module can be represented as follows: (4) (5) Finally, the enhanced high-frequency features are fused with the low-frequency features to form a non-spatial feature. ,in, This indicates the low-frequency characteristics after enhancement by multiple receptive field branches. This indicates the high-frequency features enhanced by fewer receptive field branches. These are not the final global visual features used for retrieval, but rather enhanced visual representations prior to modeling local visual structures. They provide more comprehensive multi-scale visual information for subsequent object detection, local region feature extraction, and visual relationship graph construction. Specifically, global visual features are primarily used to represent the overall scene semantics of remote sensing images, such as overall category information like airports, ports, and residential areas; while... It focuses more on preserving and enhancing local targets, edge textures, and multi-scale region information in images, providing a foundation for local target detection and spatial relationship modeling. Therefore, It can be regarded as an intermediate enhancement feature between the original image features and the local visual structure features.

[0048] Visual Relationship Graph Construction: Based on Enhanced Visual Features This paper uses a target detection module to detect local target regions in remote sensing images and extracts the corresponding local visual features for each target region. The V-SPA branch mentioned in this paper refers to the visual spatial relationship modeling branch, which models and structurally represents the spatial relationships of the detected remote sensing targets. Specifically, a visual relationship graph is constructed using the detected target regions as graph nodes and the spatial positional relationships between targets as graph edges, such as adjacent, contained, located on the left, located on the right, near, far, etc. Subsequently, a graph convolutional network is used to propagate and aggregate features from the visual relationship graph to obtain local visual features containing spatial structural information. .

[0049] Thus, two types of complementary features are formed in the visual modality: one type is the global visual features extracted by the visual Transformer or convolutional neural network. One type is used to describe the overall scene semantics of an image; the other type is composed of... Local visual features obtained after object detection, visual relationship graph construction, and graph convolutional network processing This is used to describe target entities and their spatial layout relationships in an image. Subsequently, in the local guided feature fusion mechanism, This will serve as local guidance information for global visual features. Attention optimization and multi-level fusion are performed to obtain fused visual features for cross-modal retrieval: .

[0050] Visual Relationship Graph Construction: Based on the enhanced features, local target regions are detected by the target detection module, and a visual relationship graph is constructed. A V-spa branch is designed to efficiently model the spatial relationships of targets in remote sensing images, enhancing the spatial representation of visual information. The V-spa branch, or Visual Spatial RelationBranch, takes as input the local target regions obtained by the target detection module and their corresponding region features. Its main function is to explicitly model the spatial positional and structural relationships between different targets in the remote sensing image. Specifically, detected local targets are used as nodes in the visual relationship graph, with node features represented by the visual features of the corresponding target region. Spatial relationships between targets are used as edges, which can include adjacent, contained, overlapping, located on the left, located on the right, near, far, etc. Through this method, the V-spa branch organizes the originally scattered local target features into a visual relationship graph with spatial structural constraints. Subsequently, a Graph Convolutional Network (GCN) is used to propagate node features and aggregate relationships on the visual relationship graph, enabling each target node to fuse contextual information from its neighboring targets. This results in a visual local feature V_l containing target entity information and spatial layout relationships, which is used for subsequent cross-modal fusion and alignment. Efficient spatial relationship modeling of targets in remote sensing images is performed to enhance the spatial representation of visual representation. Using detected targets as nodes and spatial relationships between targets as edges, a visual relationship graph structure is constructed. Local structural features are extracted using a Graph Convolutional Network (GCN) to obtain the visual local features. This is used for subsequent cross-modal fusion and alignment.

[0051] 4. The specific process of feature extraction in this paper Global text feature extraction: For the input text description, the overall semantic features are extracted using a pre-trained language model (such as RoBERTa) and used as the global text feature Tg.

[0052] Using the DeepSeek large language model, combined with a specific Prompt design, entities and relationships are extracted from the text descriptions of remote sensing images. For example... Figure 5 As shown in the blue box, the text description of a remote sensing image contains multiple entities with clear meanings (such as "building", "airplanes", "epuipment", "tarmac") and their spatial relationships (such as "are present on", "in the center", "on either side of"). This structured data can clearly depict the entities and their spatial relationships in the text, providing a foundation for subsequent text graph construction.

[0053] Text semantic graph construction and GCN feature extraction: Based on entities and their relationships extracted by the LLM (deepseek) model, a text graph was constructed. Here, V represents the extracted entity node, i.e., the geographic entity or target object mentioned in the text, and E represents the spatial relationship between entities. Specifically, the entity node set is... , where n is the number of entities extracted from the text. It is the embedded representation of the i-th entity, encoded by the word embedding model (Word2Vec), that is... Where Entity_i represents the i-th entity object extracted from the text description by the large language model, such as buildings, roads, airplanes, water bodies, parking lots, and other targets with clear visual or geographical meaning; Embedding This represents the entity embedding mapping function, used to convert discrete entity words or entity phrases into continuous vector representations; v_i represents the initial node feature vector corresponding to the i-th entity node. This vector serves as the feature representation of entity nodes in the text semantic graph, used to subsequently construct the node feature matrix, and input into the graph convolutional network for entity relationship propagation and local text structure feature extraction. Spatial relationships between entities Indicates connection and The relationship type.

[0054] Using extracted entities as nodes, node embedding representations are obtained through a word embedding model. Spatial relationships between entities. As edges, based on the relation types obtained from the nine-intersection model, different learnable dynamic weights are assigned to the edges, constructing a relation-weighted adjacency matrix A, where topological relations, directional relations, and distance relations correspond to the weights, respectively. , , These learnable dynamic weight parameters are automatically updated during model training to optimize the importance allocation of different types of spatial relationships. The constructed text graph can be fully represented as follows:

[0055] Relation-weighted adjacency matrix A:

[0056] in , , The relation-weighted adjacency matrix A, consisting of learnable dynamic weight parameters, represents the connection relationships and their strengths between entity nodes in the text semantic graph. For each element in matrix A... If the spatial relationship between the i-th entity node and the j-th entity node If it belongs to a topological relationship, then... Assign the learnable weight w corresponding to the topological relationship; if If it is a directional relationship, then... Assign the learnable weight w corresponding to the directional relationship; if If it belongs to a distance relationship, then... Assign the learnable weight w corresponding to the distance relationship; if there is no spatial relationship between the two that can be classified into the above three categories, then... The value is assigned to 0. The resulting adjacency matrix A not only indicates whether connections exist between entity nodes, but also dynamically characterizes the importance of different spatial relationships based on their type, providing a foundation for structured feature extraction in subsequent graph convolutional networks. It is automatically updated during model training to optimize the importance allocation of different types of spatial relationships. Simultaneously, the node matrix is ​​obtained:

[0057] in d The embedding dimension is represented. Structured features are extracted from the text semantic graph using a three-layer Graph Convolutional Network (GCN). In each GCN layer, the feature update formula for each node is as follows:

[0058] in, It is the node feature matrix of the k-th layer, initially... , It is a learnable adjacency matrix, where H^{(k)} is the node feature matrix of the k-th layer, and initially H^{(0)}=X; This is a normalized adjacency matrix obtained by further processing the aforementioned relation-weighted adjacency matrix A, used to control the information propagation process between different entity nodes in the text semantic graph. Specifically, A represents the original spatial relational connections between entity nodes and their relational weights, where topological relations, directional relations, and distance relations are represented by different learnable weight parameters. To enable the graph convolutional network to retain node information while stably aggregating neighboring node information during node feature propagation, self-connections are added to A and normalized, resulting in... .therefore, A and W are not two independent matrices, but rather graph convolution propagation matrices obtained by self-connecting and normalizing A. W^{(k)} is the trainable weight matrix of the k-th layer, and δ(·) is the non-linear activation function ReLU, used to enhance feature representation. It is the trainable weight matrix of the k-th layer. It is the non-linear activation function ReLU, used to enhance feature representation. From this, the final local text representation can be obtained:

[0059] in It provides the final feature representation for all text entities, captures fine-grained spatial relationships in the text, and provides more accurate text semantic representation for cross-modal learning, which can be used for subsequent visual-text cross-modal matching.

[0060] Specific implementation of local guided feature fusion To address the scale inconsistency and information conflict issues inherent in traditional cross-modal feature fusion, this invention employs a Local Guided Feature Fusion (LGFF) mechanism for feature fusion. This mechanism optimizes the representation distribution of global features by leveraging local fine-grained information through dynamic interaction between local target features and global scene features.

[0061] For visual modalities, the importance distribution of global features is first calculated using local information:

[0062] in, , It is a learnable parameter matrix. It is the attention matrix, used to calculate global features. and local features The degree of matching between them. Next, using... global features Optimize:

[0063] It is a global feature optimized locally, meaning that the region of interest for the global feature is adjusted based on local target information. Finally, a multi-level fusion strategy is adopted:

[0064] in, This represents the visual characteristics after fusion. This represents the global visual features after local attention optimization. Representing local visual features; , , These are learnable weight parameters used to adaptively adjust the contribution of different feature components to the fusion result. The symbol ⊙ represents element-wise multiplication, i.e., the Hadamard product, which requires... and They have the same feature dimensions and are multiplied item by item along the corresponding dimensions. By introducing ⊙ This approach allows for explicit modeling of the interaction between global scene semantics and local target structure, enabling fused features to retain overall scene information while highlighting discriminative information related to local targets and spatial relationships. The point set theorem representing the matrix can also utilize the same local guidance mechanism to obtain fused text features.

[0065] in, , , These are learnable weights, which are introduced additionally. This is used to model the intersection of local and global features, ensuring the complementarity of information at different scales. In this way, scene-level global information is preserved while highlighting local target details, improving cross-modal matching performance.

[0066] 5. Specific implementation of the cross-modal alignment loss function In cross-modal retrieval tasks, existing methods mainly employ the triplet loss function.

[37] Triplet loss is used to optimize the alignment of image and text features. However, traditional triplet loss functions only consider the overall feature similarity between images and text, ignoring the fine-grained spatial relationships between them, which is particularly important in remote sensing image retrieval tasks. To address these issues, an improved cross-modal alignment loss function, called Spatially-Aware Cross-Modal Alignment Loss (SACMA Loss), is proposed. This loss function incorporates spatial features and optimizes cross-modal retrieval results by minimizing the alignment error between image and text features in a common feature space.

[0067] The triplet loss function is used to ensure that relevant image-text pairs are closer in the feature space than unrelated image-text pairs. ( )and ( Let and represent image and text features respectively. The triplet loss can be expressed as:

[0068] in, Indicates the number of samples. and These are related image and text features. and These are irrelevant text features. and These are irrelevant text features. Represents a margin constant. This represents the distance metric between features (Euclidean distance is used in this paper).

[0069] To enhance the effectiveness of cross-modal spatial relation alignment, we introduce the Laplacian operator difference into the spatial feature alignment loss.

[38] Loss. The Laplacian operator can highlight edges and rapidly changing regions in the feature matrix, capturing higher-order spatial variations. By calculating the Laplacian differences between image and text features to measure their similarity, it ensures that images and text maintain higher consistency in the common feature space, thereby enhancing the model's perception and understanding of spatial features. and The spatial relationship features extracted through the V-RS and T-RS modules are shown in the following details: First, in order to capture higher-order spatial transformations of the feature matrix, for the feature matrix... and Calculate its discrete form of the Laplace operator:

[0070] Then, by calculating the Laplacian operator difference between the image modality and the text modality, their consistency in spatial features is ensured, as follows:

[0071] Finally, to optimize the overall model performance, the improved triple spatialization loss function and the cross-modal feature alignment loss function are weighted and combined:

[0072] Among them, among them, and These are weighting parameters used to balance the effects of different loss terms.

[0073] 6. Training and Retrieval Process The remote sensing image dataset and its corresponding text dataset are input into the network model, and the total loss function is continuously optimized by the Adam optimizer. In order to adjust the network parameters, the optimal remote sensing image text retrieval model can be obtained.

[0074] During the retrieval phase, for the input query image or query text, the trained model extracts fused visual features and fused text features, calculates the similarity between the fused visual features and fused text features, and returns the retrieval results in descending order of similarity.

[0075] 7. Alternative Implementation Methods 7.1 Alternative methods for feature extraction modules In the visual modality feature extraction stage, in addition to using the Visual Transformer (ViT) as the basic encoder, other types of visual feature extraction networks, such as the Swing Transformer, EfficientNet, and ConvNeXt, can also be selected to adapt to different computing resources and performance requirements. Text modality feature extraction can also be implemented based on different language models, such as DeBERTaV3, T5, and OPT, to replace RoBERTa and enhance semantic representation capabilities under few-shot conditions.

[0076] 7.2 Alternative Design to Local Structural Modeling Methods In the visual local relation modeling stage, in addition to directly extracting targets based on object detection, unsupervised or weakly supervised clustering methods (such as region generators and clusterers) can be used to determine local regions of interest, thereby establishing a local node graph. In the text modality, in addition to extracting based on explicit relations, syntactic analysis and dependency parsing trees can also be used to construct the connection relationships between entities, forming a semantic graph, thereby further enhancing the flexibility of text local structure modeling.

[0077] 7.3 Variant Implementation of Cross-Modal Fusion and Alignment Mechanisms For local guided feature fusion, the weighted cross-attention mechanism can be replaced with a multi-level gated fusion mechanism, or feature fusion can be achieved based on a Transformer interactive encoder (such as a Co-Attention Encoder) to further improve fusion accuracy. In the process of hierarchical graph alignment, in addition to using the Hungarian matching strategy for local node alignment, the Optimal Transport method can be introduced to optimize the matching relationship between visual and text nodes in a more granular way.

[0078] Scalable application scenarios of this invention Besides remote sensing image and text retrieval, the local modeling and hierarchical alignment mechanism proposed in this invention can also be applied to other cross-modal retrieval tasks that require handling complex spatial structures and fine-grained semantic relationships, including but not limited to: Urban surveillance image and descriptive text retrieval system; Cross-modal retrieval system for medical images and clinical reports; A system for retrieving and comparing satellite multi-temporal imagery and change description texts; Semantic matching application of drone aerial images and navigation command text.

[0079] Example 2 Based on the same inventive concept, this invention also discloses a system for implementing a remote sensing image and text retrieval method based on the nine-intersection model and Laplace alignment, comprising: The text feature extraction module is used to perform the text feature extraction steps of the remote sensing image retrieval method based on the nine-intersection model and Laplacian alignment. The pre-trained language model extracts global text features from the text description of the remote sensing image, and uses a large language model to extract entities and spatial relationships, classifies them according to the nine-intersection model, constructs a dynamic weighted adjacency matrix, and extracts local text features through a graph convolutional network. The visual feature extraction module is used to perform the visual feature extraction steps of the remote sensing image retrieval method based on the nine-intersection model and Laplacian alignment. It extracts global visual features by visual Transformer or convolutional neural network, and extracts local features and constructs a visual relationship graph by using an asymmetric dynamic receptive field strategy. The feature fusion module is used to perform the fusion steps of the remote sensing image retrieval method based on the nine-intersection model and Laplacian alignment, and to fuse local text features with global text features and local features with global visual features to obtain fused text features and fused visual features. The cross-modal alignment module is used to perform the cross-modal alignment steps of the remote sensing image retrieval method based on the nine-intersection model and Laplacian alignment, construct a loss function including the Laplacian operator difference loss, and constrain the spatial structural consistency of the fused visual features and fused text features. The retrieval module performs cross-modal retrieval based on the fusion of visual features and fusion of text features.

[0080] Since the system described in Embodiment 2 of this invention is the same system used to implement the remote sensing image retrieval method based on the nine-intersection model and Laplace alignment in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this system based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All systems used in any method of this invention's embodiments fall within the scope of protection of this invention.

[0081] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / unk>

Claims

1. A remote sensing image and text retrieval method based on the nine-intersection model and Laplace alignment, characterized in that, include: Global text features are extracted from the text descriptions of remote sensing images using a pre-trained language model, and local text features are obtained through the following steps: entities and spatial relationships are extracted using a large language model, and classified into three categories according to the nine-intersection model: topology, orientation, and distance. A relation-weighted adjacency matrix is ​​constructed and learnable dynamic weights are assigned to different types. The features are then extracted through a graph convolutional network. Global and local visual features are extracted from remote sensing images. The global visual features are obtained by extracting the overall scene of the image using a visual Transformer. The local features are obtained by the following steps: multi-scale enhancement is performed using an asymmetric dynamic receptive field, with more receptive field branches for low-frequency features and fewer branches for high-frequency features, and a visual relationship graph is constructed based on the enhanced features. Local text features are fused with global text features, and local features are fused with global visual features, respectively, to obtain fused text features and fused visual features; during fusion, attention weights are generated from local features to optimize global features. Construct a loss function with Laplacian operator difference loss to constrain the spatial structural consistency between fused visual features and fused text features; Cross-modal retrieval based on the fusion of visual features and fusion of text features.

2. The method according to claim 1, characterized in that, The asymmetric dynamic receptive field strategy specifically includes: Residual networks are used to extract feature maps of different depths to obtain low-frequency features. and high frequency characteristics ; Regarding the low-frequency features Multi-scale convolution is performed using at least three receptive field branches of different sizes to process the high-frequency features. Multi-scale convolution is performed using two receptive field branches of different sizes; The local features are obtained by fusing the low-frequency features enhanced by multi-scale with the high-frequency features.

3. The method according to claim 1, characterized in that, The extraction of local text features also includes: Using the entities extracted from the large language model as nodes, node embedding representations are obtained through a word embedding model; Based on the relation types classified by the nine-intersection model, different learnable weights are assigned to the edges to construct a relation-weighted adjacency matrix A, where topological relations, directional relations, and distance relations correspond to the weights, respectively. , , ; The text structural features are obtained by extracting structured features from the text semantic graph using a three-layer graph convolutional network.

4. The method according to claim 1, characterized in that, The Laplace operator difference loss is calculated in the following manner: Calculate the spatial relationship feature matrix corresponding to the fused visual features respectively. The spatial relationship feature matrix corresponding to the fused text features Discrete Laplace operator: Calculate the Laplacian operator difference loss: in, For the sample size, For the first i One image, For the first i A text description.

5. The method according to claim 1, characterized in that, The loss function is a weighted combination of a global alignment term and a Laplacian operator difference loss term. The global alignment term uses triplet loss. This is used to optimize the overall distance between the fused visual features and the fused text features in the common feature space; The loss function is expressed as: in, For the Laplacian operator difference loss, , Weighting parameters to balance the impact of different loss terms.

6. The method according to claim 1, characterized in that, The local guidance mechanism in the fusion step specifically includes: For visual modalities, through learnable parameter matrices , Global visual features and local features Project onto the latent space and compute the attention matrix. : in, To hide spatial dimensions; Using the attention matrix For the global visual features Optimization is performed to obtain the global features of local optimization. ; A multi-level fusion strategy is used to obtain fused visual features: in, , , For learnable weights, This indicates element-wise multiplication.

7. The method according to claim 6, characterized in that, For the text modality, a local guidance mechanism symmetrical to that of the visual modality is adopted to obtain fused text features. The fusion method of global text features and local text features is the same as that of the visual modality. Furthermore, the local guidance mechanisms of the visual modality and the text modality adopt independently learned parameter matrices and do not share parameters.

8. The method according to claim 1, characterized in that: The method is applied to remote sensing cross-modal retrieval tasks under conditions of few or zero samples; the large language model adopts the DeepSeek model.

9. The method according to claim 1, characterized in that: The pre-trained language model includes RoBERTa or BERT, the visual Transformer includes ViT, and the convolutional neural network includes ResNet; the graph convolutional network is a three-layer graph convolutional network structure.

10. A system for implementing the remote sensing image retrieval method based on the nine-intersection model and Laplace alignment as described in any one of claims 1-9, characterized in that, include: The text feature extraction module is used to perform the text feature extraction steps of claim 1, which extract global text features from the text description of the remote sensing image by a pre-trained language model, extract entities and spatial relationships by a large language model, classify them by a nine-intersection model, construct a dynamic weighted adjacency matrix and extract local text features through a graph convolutional network. The visual feature extraction module is used to perform the visual feature extraction steps of claim 1, extract global visual features by a visual Transformer or a convolutional neural network, and extract local features and construct a visual relationship graph by using an asymmetric dynamic receptive field strategy. The feature fusion module is used to perform the fusion step described in claim 1, and to fuse local text features with global text features and local features with global visual features, respectively, to obtain fused text features and fused visual features; A cross-modal alignment module is used to perform the cross-modal alignment steps of claim 1, construct a loss function including Laplacian operator difference loss, and constrain the spatial structural consistency of fused visual features and fused text features; The retrieval module performs cross-modal retrieval based on the fusion of visual features and fusion of text features.

Images