Remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure

Abstract

The present application relates to a remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure, belonging to the technical field of remote sensing image-text cross-modal retrieval. It comprises: inputting the remote sensing image and text to be retrieved into a trained knowledge enhancement and asymmetric structure visual-language basic model, and realizing cross-modal retrieval through model processing; obtaining the retrieval result to obtain the matching retrieval output of the remote sensing image and the text, and completing the cross-modal retrieval task. The present application solves the problem that there is significant information asymmetry between the current remote sensing image and the text modal, which limits the retrieval performance. The present application realizes efficient modal fine-grained shared feature learning by designing a cross-modal asymmetric Kolmogorov-Arnold adapter fine-tuning method; at the same time, knowledge is extracted from ConceptNet and a remote sensing knowledge graph to generate knowledge-enhanced sentences to enrich the text semantics, thereby bridging the semantic gap between the modal and improving the remote sensing image-text retrieval performance.

Description

TECHNICAL FIELD

[0001] The present application relates to a remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure, belonging to the technical field of remote sensing image-text cross-modal retrieval. BACKGROUND

[0002] Remote sensing image-text cross-modal retrieval aims to establish the correlation between remote sensing image visual information and text semantic information, realizes remote sensing image retrieval based on text description or text matching based on remote sensing image, and plays an important role in land resource investigation, environmental monitoring, urban planning and other fields, and has become a research hotspot in the cross field of remote sensing technology and cross-modal learning. Traditional cross-modal retrieval methods mostly rely on hand-designed feature matching strategies, which are difficult to capture the deep correlation between the fine-grained features of complex ground objects in remote sensing images and text semantics, and the retrieval accuracy is limited.

[0003] In recent years, visual-linguistic models based on deep learning have promoted the development of cross-modal retrieval technology with their powerful feature extraction and semantic modeling capabilities. Such models learn the correlation rules of general images and texts through pre-training, and then adapt to specific tasks through fine-tuning. However, when they are applied to remote sensing image-text cross-modal retrieval, the following problems exist:

[0004] (1) At the data level, there are significant differences in scene complexity and semantic hierarchy between remote sensing images and natural images. Remote sensing images contain a large amount of ground object details (such as airport runways, farmland boundaries, vegetation types, etc.), and have unique spatial distribution rules, while text descriptions are often concise and abstract, and difficult to completely cover the semantic information in the image, resulting in a serious information asymmetry between the two.

[0005] (2) At the model level, existing models mostly use symmetric feature fusion methods, without considering the differences in characteristics between remote sensing images and text modalities. Remote sensing images require fine-grained visual feature mining, while text requires strengthening of key entity and semantic correlation, and simple symmetric fusion cannot achieve accurate alignment of the two. At the same time, general pre-trained models lack knowledge in the field of remote sensing (such as the difference between "cultivated land" and "abandoned land", and the spatial feature difference between "runway" and "highway"), and have insufficient understanding of professional semantics, further exacerbating the cross-modal semantic gap.

[0006] Therefore, there is an urgent need for a cross-modal retrieval method that can adapt to the characteristics of the remote sensing field, enhance text semantic expression, and optimize the interaction of modal features, to improve the accuracy and robustness of remote sensing image-text retrieval. SUMMARY

[0007] The technical problem solved by the present application is that the present application provides a remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure to solve the problem that the remote sensing image and text modal in the prior art have significant information asymmetry, which limits the retrieval performance, and the problem that the model is difficult to realize fine-grained feature sharing and semantic precise alignment; and the present application improves the accuracy and robustness of remote sensing image-text retrieval.

[0008] The technical solution of the present application is a remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure, which comprises:

[0009] S1, inputting the remote sensing image and text to be retrieved into a trained knowledge enhancement and asymmetric structure visual-linguistic basic model to realize cross-modal retrieval through model processing;

[0010] S2, obtaining the retrieval result to obtain the matching retrieval output of the remote sensing image and the text, and completing the cross-modal retrieval task.

[0011] Further, the retrieval process performed by the trained knowledge enhancement and asymmetric structure visual-linguistic basic model comprises the following steps:

[0012] S11, cross-modal asymmetric Kolmogorov-Arnold adapter feature interaction is performed;

[0013] The group Kolmogorov-Arnold network is introduced into the adapter structure, combined with the modal-specific attention mechanism, which includes differential attention and hierarchical attention; the visual features extracted from the input remote sensing image and the semantic features extracted from the text are subjected to fine-grained modal shared feature learning to obtain the preliminary fused cross-modal feature representation;

[0014] S12, semantic supplement is performed through a knowledge enhancement module: extracting common sense knowledge from ConceptNet and extracting domain-specific knowledge from a remote sensing knowledge graph; performing entity recognition on the input text and constructing a knowledge graph, generating a knowledge enhanced sentence based on the knowledge graph, fusing the knowledge enhanced sentence with the original text semantics, enriching the text semantic features, and further interacting the enhanced text semantic features with the preliminary fused cross-modal feature representation in S12 to strengthen the cross-modal semantic association.

[0015] Further, in S11, the cross-modal asymmetric Kolmogorov-Arnold adapter feature interaction comprises:

[0016] Assume is the index of the input channel, and the total number of channels is divided into groups, each group containing channels, where denotes the corresponding group index; the group-wise Kolmogorov-Arnold network operates on the input vector is expressed as:

[0017] (1)

[0018] where, is a group-wise Kolmogorov-Arnold network; is a mapping function; denotes the th group corresponding transformation function; is a linear weight; denotes the composition of two functions;

[0019] After rewriting, we have:

[0020] (2)

[0021] The GR-KAN layer first performs a group-wise rational function transformation, and then a linear layer is used to realize the overall mapping; the process is as follows:

[0022] (3)

[0023] where, is a linear layer; is a group-wise rational function;

[0024] The differential attention mechanism maps the query, key and value vectors to the output, and calculates the attention score using the query and key vectors, and then weights and sums the value vectors; a pair of softmax functions is introduced to eliminate the noise in the attention score; the calculation formula is as follows:

[0025] (4)

[0026] where, are the query, key and value vectors respectively, and and are both split into two parts and for subsequent differential calculation; d is the dimension of the query, key and value vectors, are the parameters of respectively; is a learnable scalar, and softmax(.) is the softmax function;

[0027] Re-parameterize as:

[0028] (5)

[0029] wherein, is a learnable vector; is a constant for initializing ;

[0030] The visual features extracted by the differential attention mechanism of the cross-modal asymmetric Kolmogorov-Arnold adapter are input to the shared layer and interact with the semantic features extracted from the text to obtain a preliminary fusion of cross-modal feature representations.

[0031] Before being elevated to the original input dimension, a gating mechanism is introduced to dynamically adjust the contribution of each modality feature.

[0032] Further, the S11 comprises:

[0033] The cross-modal asymmetric Kolmogorov-Arnold adapter is constructed to address the difference in visual feature dimension and semantic feature dimension of remote sensing images. A modality-specific attention mechanism is used to assign dynamic weights to different modality features, achieving feature alignment and fine-grained fusion.

[0034] Further, in the S12, the knowledge-enhanced sentence generation process comprises:

[0035] The original text is subjected to entity recognition using natural language processing tools, and the identified entities are used to retrieve associated knowledge from ConceptNet and RSKG to construct a text knowledge graph in the form of knowledge triples. The knowledge-enhanced sentence that supplements the text semantics is generated through text knowledge graph reasoning. The text relationship in the form of knowledge triples is as follows:

[0036] Common relationships are selected from all relationship types supported by ConceptNet;

[0037] By combining RSKG and ConceptNet, the triple list obtained from ConceptNet is filtered to remove triple lists whose head entity and tail entity do not belong to the RSKG entity set ;

[0038] Subsequently, the filtered ConceptNet triple list is merged with the RSKG triple list to construct a new triple list , wherein , represents the number of triples retrieved from ConceptNet; represents the number of triples obtained from the RSKG graph; represents the number of triples obtained from the combined ConceptNet and RSKG, represents the triples obtained from the combined ConceptNet and RSKG.

[0039] Further, in the S1, the knowledge-enhanced and asymmetric structure visual-language base model training process comprises:

[0040] Collect remote sensing image-text pair datasets, and divide the datasets into training sets, validation sets and test sets;

[0041] The constructed knowledge-enhanced and asymmetric structure visual-language base model is trained using the training set, the model parameters are optimized by a cross-modal retrieval loss function, the training strategy is adjusted using the validation set, and the model retrieval performance is verified by the test set, wherein the loss function specifically comprises:

[0042] The knowledge-enhanced and asymmetric structure visual-language base model uses the following loss function to measure the semantic similarity of the two modalities:

[0043] (6)

[0044] wherein, represents a matched sample pair; is a text that does not match the remote sensing image is a remote sensing image that does not match the text is a remote sensing image that does not match the text is a remote sensing image that does not match the text is a cross-modal constraint boundary value; ;

[0045] The calculation formula of the cosine similarity is as follows:

[0046] (7)

[0047] wherein n is the number of remote sensing image-text pairs; is the mth remote sensing image; is the mth text.

[0048] Further, in the S11, the group K-A network of the cross-modal asymmetric K-A adapter dynamically adapts the mapping relationship between the different feature categories of the remote sensing image and the semantic description of the text through a pre-defined combination of multiple loss function groups, and the cross-modal asymmetric K-A adapter further introduces a hierarchical attention mechanism to extract key information words and aggregate their representations to construct a sentence vector, and the specific formula is as follows:

[0049] (8)

[0050] (9)

[0051] (10)

[0052] The specific steps include:

[0053] First, the words are encoded by a bidirectional GRU to obtain word-level annotations and passed into a single-layer multi-layer perceptron to obtain the hidden representation of the word-level annotations ; ;

[0054] Next, the importance of the word is measured by the similarity with the word-level context vector and the normalized importance weight is obtained by the softmax function ; ;

[0055] Finally, the word-level annotations are weighted and summed according to the weights to obtain the sentence vector ; wherein the word-level context vector is a high-level representation of a fixed query;

[0056] The sentence vector obtained above is then encoded by a bidirectional GRU to obtain sentence-level annotations and the sentence-level context vector is introduced to measure the importance of the obtained sentence, and the specific formula is as follows:

[0057] (11)

[0058] (12)

[0059] (13)

[0060] Similar to the above word-level processing method, the same strategy is adopted to obtain the hidden representation of the sentence vector, the sentence-level importance weight , and the text vector for integrating all sentence information.

[0061] Further, the present application also includes a post-processing step of the trained knowledge-enhanced and asymmetric structure visual-language base model retrieval result: based on the prior knowledge in the field of remote sensing, the similarity of the preliminary retrieval result is reordered and the error matching is corrected. The present application also provides a knowledge-enhanced and asymmetric structure remote sensing image-text retrieval system, which comprises a module for executing the knowledge-enhanced and asymmetric structure remote sensing image-text retrieval method. ​

[0062] The application also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the knowledge-enhanced and asymmetric structure-based remote sensing image-text retrieval method when executing the program.

[0063] The application has the following beneficial effects:

[0064] The application realizes efficient modal fine-grained shared feature learning, extracts knowledge from ConceptNet and a remote sensing knowledge graph, generates knowledge-enhanced sentences to enrich text semantics, thereby bridging the semantic gap between modalities and improving the performance of remote sensing image-text retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0065] Fig. 1 FIG. 1 is a schematic diagram of the knowledge-enhanced and asymmetric structure-based remote sensing image-text retrieval method of the application;

[0066] Fig. 2 FIG. 4 is a schematic diagram of the knowledge graph construction process framework provided in the embodiment of the application;

[0067] Fig. 3 FIG. 5 is a schematic diagram of the process framework of the cross-modal asymmetric Kolmogorov-Arnold adapter module provided in the embodiment of the application. DETAILED DESCRIPTION

[0068] Embodiment 1: As shown in the knowledge-enhanced and asymmetric structure-based remote sensing image-text retrieval method, the method comprises the following steps: Figs. 1-3

[0069] S1, input the remote sensing image and text to be retrieved into the trained knowledge-enhanced and asymmetric structure-based visual-linguistic basic model, and realize cross-modal retrieval by model processing;

[0070] S2, obtain the retrieval result to obtain the matching retrieval output of the remote sensing image and the text, and complete the cross-modal retrieval task.

[0071] Further, the retrieval process performed by the trained knowledge-enhanced and asymmetric structure-based visual-linguistic basic model comprises the following steps:

[0072] S11, cross-modal asymmetric Kolmogorov-Arnold adapter feature interaction is performed;

[0073] ​The group Kolmogorov-Arnold network is introduced into the adapter structure, combined with the modal specific attention mechanism, including differential attention and hierarchical attention; the visual features extracted from the input remote sensing image and the semantic features extracted from the text are subjected to fine-grained modal shared feature learning, and the preliminary fused cross-modal feature representation is obtained;

[0074] S12, semantic supplement by a knowledge enhancement module: extracting common sense knowledge from ConceptNet and extracting domain-specific knowledge from a remote sensing knowledge graph; performing entity recognition on the input text and constructing a knowledge graph, generating a knowledge-enhanced sentence based on the knowledge graph, fusing the knowledge-enhanced sentence with the original text semantics, enriching the text semantic features, and further interacting the enhanced text semantic features with the preliminary fused cross-modal feature representation in S12 to strengthen the cross-modal semantic association.

[0075] Further, in the S11, the cross-modal asymmetric Kolmogorov-Arnold adapter feature interaction includes:

[0076] Assume is the index of the input channel, and the total number of channels is divided into groups, each group containing channels, where represents the corresponding group index; the group Kolmogorov-Arnold network operates on the input vector is expressed as:

[0077] (1)

[0078] where, is the group Kolmogorov-Arnold network; is a mapping function; represents the transformation function corresponding to the group; is a linear weight; represents the composition of two functions;

[0079] After rewriting, we get:

[0080] (2)

[0081] The GR-KAN layer first performs a group rational function transformation, and then a linear layer is connected to realize the overall mapping; the process is as follows:

[0082] (3)

[0083] where, is a linear layer; Rational functions for grouping;

[0084] The differential attention mechanism maps the query, key and value vectors to the output, and calculates the attention score using the query and key vectors, and then weights and sums the value vectors; a pair of softmax functions are introduced to eliminate the noise in the attention score; the calculation formula is as follows:

[0085] (4)

[0086] wherein, are the query (query), key (key) and value (value) vectors respectively, and and are both split into two parts and for subsequent differential calculation; d is the dimension of the query, key and value vectors, are parameters of respectively; is a learnable scalar, and softmax(.) is a softmax function;

[0087] In order to coordinate the dynamic changes of the learning process, the is re-parameterized as:

[0088] (5)

[0089] wherein, is a learnable vector; is a constant used to initialize ;

[0090] In order to realize the complementary fusion between modalities, the visual features extracted by the differential attention mechanism of the cross-modal asymmetric Kolmogorov-Arnold adapter are input to the shared layer, and the semantic features extracted from the text are interactively shared to obtain the preliminary fusion of cross-modal feature representation;

[0091] When the semantic features extracted from the text have information missing or expression fuzzy, the semantic clues provided by the visual features can effectively enhance the understanding and reasoning ability of the model to the overall semantics, in order to avoid the imbalance of information interaction between modalities, before being upgraded to the original input dimension, a gating mechanism is introduced to dynamically adjust the contribution of each modal feature, thereby improving the stability and balance of cross-modal fusion.

[0092] Further, the S11 comprises:

[0093] The cross-modal asymmetric Kolmogorov-Arnold adapter is constructed, and in view of the difference between the visual feature dimension and the semantic feature dimension of the remote sensing image, a modal-specific attention mechanism is used to assign dynamic weights to different modal features, so as to realize feature alignment and fine-grained fusion.

[0094] Further, in the S12, the knowledge-enhanced sentence generation process comprises:

[0095] Perform entity recognition on the original text using natural language processing tools, retrieve associated knowledge in ConceptNet and RSKG based on the recognized entities, construct a text knowledge graph in the form of knowledge triples, and generate knowledge-enhanced sentences that supplement the text semantics through text knowledge graph reasoning; the text relationship in the form of knowledge triples is as follows:

[0096] From all the relationship types supported by ConceptNet, the commonly used relationships are selected, which are: UsedFor, ReceivesAction, HasA, Causes, HasProperty, CreatedBy, DefinedAs, AtLocation, HasSubEvent, MadeUpOf, HasPrerequisite, Desires, NotDesires, IsA and CapableOf, to improve the relevance and accuracy of semantic expression; By combining RSKG and ConceptNet, the triple list obtained from ConceptNet is filtered to remove triple lists whose head entity and tail entity do not belong to the RSKG entity set to ensure that external knowledge focuses on the field of remote sensing;

[0098] Then, the filtered ConceptNet triple list is merged with the RSKG triple list to construct a new triple list wherein , represents the number of triples retrieved from ConceptNet; represents the number of triples obtained from the RSKG graph; represents the number of triples obtained from the combined ConceptNet and RSKG, represents the triples obtained from the combined ConceptNet and RSKG.

[0099] Further, in the S1, the knowledge-enhanced and non-symmetric structure visual-language base model training process comprises:

[0100] Collect remote sensing image-text paired datasets, and divide the datasets into training sets, validation sets and test sets;

[0101] The constructed knowledge-enhanced and asymmetric structure visual-language base model is trained using the training set to optimize the model parameters with a cross-modal retrieval loss function, the training strategy is adjusted using the validation set, and the model retrieval performance is verified through the test set;

[0102] At the beginning of training, the parameters of the ViT image encoder and the BERT text encoder are kept frozen and do not participate in updating during the training process; the GR-KAN layer, the modal-specific attention mechanism and the shared layer in the cross-modal asymmetric Kolmogorov-Arnold adapter are randomly initialized; the related mapping parameters of knowledge retrieval and sentence generation in the knowledge enhancement module are randomly initialized.

[0103] During the training process, a random flip data augmentation strategy is used (for remote sensing images); all images are uniformly adjusted to 224x224 size, and the features are finally linearly projected to a 512-dimensional public space; the dropout value is set to 0.2, and the boundary value in the cross-modal loss function is set to 0.2.

[0104] The loss function specifically includes:

[0105] During the training process, in order to realize the cross-modal alignment of remote sensing images and text, the knowledge-enhanced and asymmetric structure visual-language base model uses the following loss function to measure the semantic similarity of the two modalities:

[0106] (6)

[0107] wherein, represents a matched sample pair; is a remote sensing image that does not match the text; is a remote sensing image that does not match the text; is the boundary value of the cross-modal constraint; ;

[0108] The calculation formula of the cosine similarity is as follows:

[0109] (7)

[0110] wherein, n is the number of remote sensing image-text pairs; is the mth remote sensing image; is the mth text.

[0111] The smaller the value of the loss function, the closer the predicted value is to the true value.

[0112] Furthermore, in S11, the swarm-based Kolmogorov-Arnold network of the cross-modal asymmetric Kolmogorov-Arnold adapter dynamically adapts the mapping relationship between different land cover features (such as vegetation, buildings, etc.) in remote sensing images and text semantic descriptions through a combination of predefined loss functions, thereby improving the domain adaptability of feature interactions. The cross-modal asymmetric Kolmogorov-Arnold adapter also introduces a hierarchical attention mechanism to extract key information words and aggregate their representations to construct sentence vectors, as shown in the following formula:

[0113] (8)

[0114] (9)

[0115] (10)

[0116] The specific steps include:

[0117] First, words are analyzed using a bidirectional GRU. Encode to obtain word-level annotations And it is fed into a single-layer multilayer perceptron to obtain Hidden representation ;

[0118] Next, through With word-level context vectors The importance of a word is measured by its similarity to the word's meaning, and a normalized importance weight is obtained using the softmax function. ;

[0119] Finally, the word-level annotations are weighted and summed based on these weights to obtain the sentence vector. Among them, word-level context vectors It is a high-level representation of a fixed query "which word is the most informative word", similar to the query mechanism used in memory networks;

[0120] The resulting sentence vectors are then processed using a bidirectional GRU. Encode to obtain sentence-level comments And introduce sentence-level context vectors The formula for measuring the importance of the obtained sentences is shown below:

[0121] (11)

[0122] (12)

[0123] (13)

[0124] Similar to the word-level processing method described above, the same strategy is used to obtain the hidden representation of the sentence vector , the sentence-level importance weight , and the text vector for integrating all sentence information .

[0125] Further, the present application also includes a post-processing step for the trained knowledge-enhanced and asymmetric structure visual-language base model retrieval result: based on prior knowledge in the field of remote sensing (such as the distribution logic of surface features, typical scene text association rules), the preliminary retrieval result is re-ordered in similarity and the error matching is corrected, to improve the accuracy and reliability of the final retrieval output.

[0126] Corresponding content: expand the method flow, add a retrieval result post-processing link, further optimize the result using remote sensing field knowledge, embody the improvement of the application in practical application. The present application also provides a remote sensing image-text retrieval system based on knowledge enhancement and asymmetric structure, which comprises a module for executing the remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure.

[0127] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure when executing the program.

[0128] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure when executing the program.

[0129] (14)

[0130] As shown in Table 1 below, the performance comparison of the remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure of the present application and other retrieval methods on RSITMD and RSICD datasets is given.

[0131] Table 1 is a comparison of retrieval performance on RSICD and RSITMD test sets

[0132]

[0133] The remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure provided by the present application is superior in both RSICD and RSITMD data sets, and compared with the mainstream parameter efficient fine-tuning method, the average recall rate is improved by 7.81%-12.89%, and compared with the full parameter fine-tuning GeoRSCLIP model, the average recall rate is improved by 2.14%-2.71%. The above experimental results show that the remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure provided by the present application has better performance than other retrieval methods, and the effectiveness and superiority of the retrieval method provided by the present application are verified.

[0134] In order to further verify the superiority of the method of the present application, qualitative analysis is carried out, and the visualization result shows that other methods are prone to ignore fine-grained semantic information during retrieval, and there are inaccurate matching or missed detection phenomena, while the remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure of the present application can accurately identify key entities and their spatial relationships in the image with the help of external knowledge, and can accurately match the target in Top-1 retrieval, and has better performance in fine-grained semantic understanding and cross-modal alignment, and can effectively reduce false matching even in complex scenes.

[0135] The knowledge graph structure of the present application integrates the common sense knowledge of ConceptNet and the specific knowledge of the remote sensing field of RSKG, identifies the key entities in the text and generates knowledge enhanced sentences, effectively makes up for the insufficient coverage of the text description to the complex semantics of the remote sensing image, narrows the semantic gap between the modalities, and enhances the integrity and accuracy of the text semantics.

[0136] The cross-modal asymmetric Kolmogorov-Arnold adapter of the present application introduces a group Kolmogorov-Arnold network to improve the nonlinear expression ability of the model, and combines a modal-specific attention mechanism to realize efficient sharing and complementation of fine-grained features between remote sensing images and text modalities, alleviate the information asymmetry problem between the two, and improve the balance and robustness of cross-modal feature alignment.

[0137] The experimental results on multiple benchmark data sets (RSICD, RSITMD) show that the remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure of the present application is significantly superior to the mainstream parameter efficient fine-tuning method and the full parameter fine-tuning model in terms of average recall rate and other indicators. In addition, the asymmetric structure design and knowledge enhancement mechanism can be extended to other cross-modal tasks in the remote sensing field (such as remote sensing image text description, text-based remote sensing image generation, etc.), and has good generalization ability.

[0138] Although the present application has been described in detail by the preferred embodiments mentioned above, it should be appreciated that the above description should not be considered as limiting the present application. Various modifications and alternatives to the present application will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present application should be defined by the appended claims.

[0139] Embodiment 2: a remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure, the method comprising:

[0140] Step 1: constructing a training sample of a trained knowledge-enhanced and asymmetric structure visual-linguistic base model, containing a training set, a validation set and a test set; before training starts, uniformly adjusting the remote sensing images to a fixed size of 224x224 to ensure the consistency of input data;

[0141] Step 2: selecting GeoRSCLIP as the basic framework, wherein ViT is used as the feature extractor of remote sensing images, and BERT is used as the feature extractor of text;

[0142] Step 3: constructing a trained knowledge-enhanced and asymmetric structure visual-linguistic base model includes a cross-modal asymmetric Kolmogorov-Arnold adapter and a knowledge enhancement module. The corresponding processing processes of the cross-modal asymmetric Kolmogorov-Arnold adapter and the knowledge enhancement module are as follows:

[0143] Selecting RSICD and RSITMD datasets, providing corresponding text descriptions for each remote sensing image. The remote sensing image-text retrieval dataset is divided into three parts: training set, validation set and test set, with a ratio of 80%, 10% and 10%;

[0144] The training set is used for model training, the validation set is used for adjusting hyperparameters and preventing overfitting. The test set is used to evaluate the performance of the final model; the remote sensing images in the training set, the validation set and the test set are processed by size adjustment, and their pixel values are standardized to the range that the model can process, so as to improve the stability and convergence speed of training;

[0145] Constructing a cross-modal asymmetric Kolmogorov-Arnold adapter; a general visual-linguistic model has good performance after being pre-trained on large-scale natural image-text data, but the modality difference between remote sensing images and text limits its performance in remote sensing image-text retrieval tasks; in order to realize modality fine-grained shared feature learning, the present application introduces a group Kolmogorov-Arnold network into the adapter, and combines a modality-specific attention mechanism to construct a cross-modal asymmetric Kolmogorov-Arnold adapter;

[0146] A knowledge enhancement module is constructed; since the text description is difficult to completely cover the semantic information of the remote sensing image, a semantic gap exists between the modalities; in order to enrich the text semantic expression, the application extracts common sense and domain-specific knowledge from ConceptNet and a remote sensing knowledge graph, identifies entities in the text and constructs a knowledge graph, and generates knowledge enhancement sentences, so as to narrow the gap between the modalities.

[0147] The application introduces external knowledge modeling in image-text retrieval in deep learning: the text description corresponding to the remote sensing image is segmented and tagged, all nouns are extracted as keywords, and the plural form is restored to the singular to standardize the vocabulary.

[0148] Retrieving from the remote sensing knowledge graph: taking the keyword as the initial node, retrieving the one-hop neighbor node and edge constituted triple (in the form of, retaining all semantic relationships:

[0149] Retrieving from the common sense knowledge base: through the official API, the triple related to the keyword is obtained, the triple in which at least one of the head entity and the tail entity belongs to the RSKG entity set is screened out, and 15 common relationships are retained.

[0150] For a given keyword, the knowledge triple related to the keyword can be obtained through the official API provided by ConceptNet. In this study, the keyword list is The triple set retrieved from ConceptNet is constructed, denoted as , wherein represents the number of triples retrieved from ConceptNet. In order to ensure language consistency, triples containing non-English words are excluded in the retrieval process. In terms of relationship selection, 15 common relationships are selected from all relationship types supported by ConceptNet, which are: UsedFor, ReceivesAction, HasA, Causes, HasProperty, CreatedBy, DefinedAs, AtLocation, HasSubEvent, MadeUpOf, HasPrerequisite, Desires, NotDesires, IsA and CapableOf, in order to improve the relevance and accuracy of semantic expression. By combining RSKG and ConceptNet, the triple list obtained from ConceptNet is screened, and the triple in which the head entity and the tail entity do not belong to the RSKG entity set is excluded, so as to ensure that the external knowledge focuses on the remote sensing field. Then, the screened ConceptNet triple and the RSKG triple list are merged to construct a new triple list where , represents the number of triples retrieved from ConceptNet; represents the number of triples obtained from RSKG; represents the number of triples obtained from the combined ConceptNet and RSKG. However, mining all related entities and relationships comprehensively may introduce redundant information.

[0151] Before encoding the selected knowledge triples , they need to be converted into simple structured knowledge sentences. Specifically, for each triple 〈head entity, relation, tail entity〉, the head entity and tail entity are kept unchanged, and the relation is rewritten semantically according to the preset conversion template, thereby generating a grammatically correct and semantically clear sentence. After conversion, a list of knowledge sentences is obtained. Finally, the knowledge sentences in the list are merged to form a comprehensive knowledge sentence corresponding to the description text , denoted as .

[0152] The optimization strategy of the network is as follows:

[0153] In order to realize the cross-modal alignment of RS images and texts, the knowledge-enhanced and asymmetric structure-based visual-language base model adopts the following loss function to measure the semantic similarity of the two modalities:

[0154]

[0155] where represents the matched sample pair; is the text that does not match the RS image ; is the RS image that does not match the text ; is the boundary value of the cross-modal constraint; ; the calculation formula of the cosine similarity is as follows:

[0156]

[0157] The specific process during training is as follows:

[0158] The final feature is linearly projected to a 512-dimensional common space, same as the GeoRSCLIP model. For RS images, all images are adjusted to a fixed size of 224×224 for training. In terms of parameter settings, the dropout value and The initial learning rate of the network is set to 1e-5, and weight decay of 0.7 is performed after every 5 training cycles. During the training process, the Adam optimizer is used to optimize the network parameters, the batch size is set to 256, and the total training period is set to 20. To ensure the stability and reliability of the experimental results, the experiment is evaluated by k-fold cross-validation, and the average value of 5-fold cross-validation (k = 5) is finally taken for performance evaluation and presentation.

[0159] The specific embodiments of the application are described in detail above with reference to the accompanying drawings, but the application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the application.

Claims

1. A remote sensing image-text retrieval method based on knowledge enhancement and asymmetric structure, characterized in that: The method comprises: S1, inputting the to-be-retrieved remote sensing image and text into a trained knowledge-enhanced and asymmetric-structure visual-language base model, and realizing cross-modal retrieval through model processing; S2, obtaining a retrieval result to obtain a matching retrieval output of the remote sensing image and the text, and completing the cross-modal retrieval task; The retrieval process of the trained knowledge-enhanced and asymmetric-structure visual-language base model comprises the following steps: S11, cross-modal asymmetric Kolmogorov-Arnold adapter feature interaction is performed; A group Kolmogorov-Arnold network is introduced into the adapter structure, and a modal-specific attention mechanism is combined, which includes differential attention and hierarchical attention; the visual features extracted from the input remote sensing image and the semantic features extracted from the text are subjected to fine-grained modal shared feature learning to obtain a preliminary fused cross-modal feature representation; S12, semantic supplement is performed through a knowledge enhancement module: extracting common sense knowledge from ConceptNet and extracting domain-specific knowledge from a remote sensing knowledge graph; performing entity recognition on the input text and constructing a knowledge graph, generating a knowledge-enhanced sentence based on the knowledge graph, fusing the knowledge-enhanced sentence with the original text semantics, enriching the text semantic features, and further interacting the enhanced text semantic features with the preliminary fused cross-modal feature representation in S12 to strengthen the cross-modal semantic association; In S11, the cross-modal asymmetric Kolmogorov-Arnold adapter feature interaction comprises: Assume is the index of input channel, in the case of dividing the total number of channels into groups, each group contains channels, where denotes the corresponding group index; the operation of group-wise Kolmogorov-Arnold network on the input vector is expressed as: （1） wherein, is a group Kolmogorov-Arnold network; is a mapping function; denotes the group corresponding transformation function; is a linear weight; denotes the composition of two functions; After rewriting: （2） The GR-KAN layer first performs grouping of the rational function The transformation, followed by a linear layer, implements the overall mapping; the process is as follows: （3） wherein is a linear layer; is a grouped rational function; The differential attention mechanism maps the query, key and value vectors to the output, calculates the attention score using the query and key vectors, and then weights and sums the value vectors; a pair of softmax functions are introduced to eliminate the noise in the attention score; the calculation formula is as follows: （4） where, are the query, key and value vectors, respectively, and and are split into two parts and for the subsequent difference computation; d is the dimension of the query, key and value vectors, are parameters of ; is a learnable scalar and softmax(.) is the softmax function; Will Reparameterized as follows: （5） wherein, is a learnable vector; is a constant used to initialize is a constant used to initialize The visual feature input extracted by the cross-modal asymmetric Kolmogorov-Arnold adapter is input into the shared layer, interacts with the semantic feature extracted from the text, and obtains a preliminary fused cross-modal feature representation; Before being upgraded to the original input dimension, a gating mechanism is introduced to dynamically adjust the contribution degree of each modal feature. 2.The knowledge enhancement and asymmetric structure based remote sensing image-text retrieval method according to claim 1, characterized in that: S11 comprises: A cross-modal asymmetric Kolmogorov-Arnold adapter is constructed, and in view of the difference between the visual feature dimension of the remote sensing image and the semantic feature dimension, a modal-specific attention mechanism is used to dynamically assign weights to different modal features, so as to realize feature alignment and fine-grained fusion. 3.The knowledge enhancement and asymmetric structure based remote sensing image-text retrieval method according to claim 1, characterized in that, In S12, the knowledge-enhanced sentence generation process comprises: An entity recognition is performed on the original text by using a natural language processing tool, associated knowledge is retrieved in ConceptNet and RSKG according to the recognized entity, a text knowledge graph is constructed in the form of a knowledge triple, and a knowledge-enhanced sentence that supplements the text semantics is generated through reasoning of the text knowledge graph; the text relationship in the form of a knowledge triple is as follows: Commonly used relationships are selected from all relationship types supported by ConceptNet; By combining RSKG with ConceptNet, the list of triples obtained from ConceptNet is filtered out the list of triples whose head entity and tail entity are not both in the set of RSKG entities ; Subsequently, the filtered list of ConceptNet triples is merged with the list of RSKG triples to build a new list of triples where represents the number of triples retrieved from ConceptNet; represents the number of triples obtained from the RSKG graph; represents the number of triples obtained from the combined ConceptNet and RSKG, represents the triples obtained from the combined ConceptNet and RSKG.​​ 4.The knowledge enhancement and asymmetric structure based remote sensing image-text retrieval method according to claim 1, characterized in that, In S1, the knowledge-enhanced and asymmetric-structure visual-language base model training process comprises: A remote sensing image-text paired dataset is collected, and the dataset is divided into a training set, a validation set and a test set; The constructed knowledge-enhanced and asymmetric structure visual-language base model is trained using the training set to optimize the model parameters with a cross-modal retrieval loss function, the training strategy is adjusted using the validation set, and the model retrieval performance is verified through the test set, wherein the loss function specifically includes: The knowledge-enhanced and asymmetric structure visual-language base model uses the following loss function to measure the semantic similarity of the two modalities: （6） wherein, represent pairs of samples that match; are text that does not match the remote sensing image are remote sensing images that do not match the text; are text that does not match the remote sensing image are remote sensing images that do not match the text; are boundary values of the cross-modal constraint; ; The calculation formula of the cosine similarity is as follows: （7） Wherein, n is the number of remote sensing image-text pairs; is the mth remote sensing image; is the mth text. 5.The knowledge enhancement and asymmetric structure based remote sensing image-text retrieval method according to claim 1, characterized in that, In S11, the group Kohonen-Arnold network of the cross-modal asymmetric Kohonen-Arnold adapter dynamically adapts the mapping relationship between different feature categories of the remote sensing image and the text semantic description through a pre-defined combination of multiple loss functions, and the cross-modal asymmetric Kohonen-Arnold adapter also introduces a hierarchical attention mechanism to extract key information words and aggregate their representations to construct a sentence vector, and the specific formula is as follows: （8） （9） （10） The specific steps include: First, words are analyzed using a bidirectional GRU. Encode to obtain word-level annotations And it is fed into a single-layer multilayer perceptron to obtain Hidden representation ; Next, the importance of the word is measured by the similarity with the word-level context vector and the normalized importance weight is obtained by the softmax function ;​ Finally, the sentence vector is obtained by weighted sum of the word-level annotations according to the weights ; where the word-level context vector is a high-level representation of the fixed query; The sentence vector obtained above is then encoded by a bidirectional GRU to obtain a sentence-level annotation and introduce a sentence-level context vector to measure the importance of the obtained sentence, and the specific formula is as follows: （11） （12） （13） Similar to the word-level processing method described above, the same strategy is used to obtain the hidden representation of the sentence vector , the sentence-level importance weight , and the text vector for integrating all the sentence information . 6.The knowledge enhancement and asymmetric structure based remote sensing image-text retrieval method according to claim 1, characterized in that, It also includes a post-processing step for the retrieval results of the trained knowledge-enhanced and asymmetric structure visual-language base model: based on prior knowledge in the remote sensing field, the similarity of the preliminary retrieval results is reordered and the error matching is corrected.

7. A remote sensing image-text retrieval system based on knowledge enhancement and asymmetric structure, characterized in that, The system includes a module for performing the knowledge-enhanced and asymmetric structure-based remote sensing image-text retrieval method according to any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the knowledge-enhanced and asymmetric structure-based remote sensing image-text retrieval method according to any one of claims 1 to 6.

Images