Small sample remote sensing image increment identification method and system based on cross-modal attribute feature fusion

Abstract

The invention discloses a small sample remote sensing image increment recognition method based on cross-modal attribute feature fusion for the application requirement of open type remote sensing scene image recognition with limited available sample size. The method comprises the following steps: 1, acquiring a remote sensing image sample and text description of a corresponding category; 2, constructing a priori semantic learnable attribute library; 3, performing projection transformation on the multi-modal information to obtain uniform-dimension feature representation; 4, constructing a cross-modal attention field fusion module, and realizing cross-modal bidirectional interactive fusion of vision-attribute and text-attribute; 5, performing joint training based on cross-modal comparison alignment loss; and 6, realizing new category remote sensing image recognition without incremental training through the designed prototype classifier. When facing a small sample class incremental learning scene with large intra-class difference and high inter-class similarity of remote sensing images, the method can show better small sample class incremental remote sensing image recognition performance than an existing advanced method for newly added classes without incremental training.

Description

A method and system for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion Technical Field

[0001] This invention relates to the fields of computer science and technology and surveying and mapping geographic information service technology. In particular, it addresses the problem of incremental recognition of small-sample remote sensing images in open scenarios and designs a method and system for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion. Background Technology

[0002] Image recognition is one of the core tasks in the field of remote sensing image interpretation, and it is a key technical support for Earth observation applications such as environmental monitoring, disaster assessment, and land planning. Faced with the continuous emergence of new ground feature image categories in the open world, how to enable models to continuously and accurately adapt to new categories under limited labeled samples has become a significant challenge for current intelligent remote sensing interpretation. In recent years, deep learning has significantly improved the accuracy of remote sensing image recognition, but its performance heavily relies on large-scale, high-quality labeled samples, and fixed models struggle to incrementally recognize new categories while maintaining the ability to recognize existing categories. Especially in applications in the field of remote sensing imagery, on the one hand, remote sensing data labeling is highly specialized and costly, making it difficult to obtain sufficient samples, particularly for images containing special ground features or rare targets; on the other hand, practical applications often require dynamic recognition of newly emerging image categories, demanding that models possess continuous evolution capabilities. To address these challenges, Few-Shot Class Incremental Learning (FSCIL) for remote sensing scenarios has emerged, aiming to enable models to acquire progressive image recognition capabilities using a very small number of labeled samples. However, the large intra-class differences, high inter-class similarities, and complex backgrounds of remote sensing images make it difficult for existing FSCIL methods to fully capture effective information related to fine-grained class attributes due to sample scarcity, severely limiting their performance. Furthermore, the incremental training process struggles to balance learning new classes with retaining knowledge of old classes, resulting in a significant catastrophic forgetting problem. Meanwhile, with the increasing real-time acquisition of remote sensing data, existing methods requiring additional incremental training cannot be effectively deployed and used plug-and-play on edge platforms such as satellites and drones. Improving image recognition performance for new classes under the limitations of incremental training-free applications has become a new challenge. Summary of the Invention

[0003] To address the problems existing in the prior art, the present invention aims to provide a method and system for incremental recognition of few-sample remote sensing images based on cross-modal attribute feature fusion. This method utilizes attribute features to guide the model to fully mine transferable fine-grained category attribute features of remote sensing images during base class training, and achieves cross-modal feature fusion through a bidirectional attention field. The guided model absorbs rich, fine-grained visual-text attribute information from multiple modalities, thereby improving the discriminative power of the old and new classes in the mapping space distribution, and thus achieving excellent incremental recognition performance of few-sample remote sensing images even without incremental training.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a small-sample remote sensing image incremental recognition method based on cross-modal attribute feature fusion, the method comprising: acquiring remote sensing image samples and their corresponding text descriptions; constructing an attribute feature library containing prior semantic attributes and learnable attributes; performing projection transformations on the original image information, category text information, and attribute information respectively to obtain a feature representation of a unified dimension; constructing a cross-modal bidirectional attention field to realize cross-modal feature interaction fusion of visual-attribute and text-attribute; performing joint training based on multimodal contrast alignment loss constraints; and after basic training, designing a prototype classifier to achieve new class image recognition without incremental training.

[0005] Optionally, the acquisition of remote sensing image samples and their corresponding text descriptions includes: acquiring remote sensing image sample data; generating corresponding text descriptions for each remote sensing image sample category by providing prompts to a large language model, describing the main ground features, scene characteristics, and attribute information that the image category may contain; or acquiring category text that is friendly to multimodal text encoders by combining templates and class names.

[0006] Optionally, the construction of the attribute feature library containing prior semantic attributes and learnable attributes includes: using a large language model to generate multiple sets of prior semantic attributes based on public knowledge in the field of remote sensing image interpretation, such as major land features, scene layout, color texture, etc.; using a multimodal large model text encoder to convert the aforementioned prior semantic attribute text into semantic feature vectors to form a prior semantic attribute library; randomly initializing multiple sets of learnable attribute vectors to form a learnable attribute library; and concatenating the semantic feature vectors and learnable attribute vectors to form a complete attribute feature library. : in Represents a priori semantic attribute library; This indicates a learnable attribute library; It is a concatenation function along the feature dimension; The total number of attributes. The number of prior semantic attributes. The number of learnable attribute vectors; For attribute feature dimensions.

[0007] Projection transformations are performed on the original image information, category text information, and attribute information to obtain a feature representation of a unified dimension. This includes: using a visual projection layer to project the original image information into a low-dimensional space to obtain the image feature matrix. : The attribute projection layer is used to project attribute information onto the same low-dimensional space to obtain the attribute feature matrix. : A text projection layer is used to project the categorical text information into the same low-dimensional space, resulting in the categorical feature matrix. : in , These are the original image information and the category text information, respectively. The original image information refers to the feature output of the original image after it has been segmented into multiple image tokens and input into a visual encoder such as a visual attention model. The category text information only contains the classification feature output of the original category text in a text encoder such as a contrastive language-visual model. , , These are dedicated projection matrices for three categories of features: image, attribute, and text. For the number of visual tokens, To unify the projection dimensions.

[0008] Optionally, the construction of a cross-modal bidirectional attention field to achieve cross-modal feature interaction fusion of vision and attributes includes: calculating attribute-enhanced visual vectors. And the comprehensive attention weight matrix from visual to attribute The specific process is as follows: For the input image feature matrix and attribute feature matrix Through multi-head attention mechanism, As keys and values, As a query, calculate the visual-to-attribute attention weights. and attribute-enhanced visual vectors : in, The multi-head attention mechanism is described in the following calculation process: For the h-th attention head, the attention enhancement vector is calculated through linear projection and scaled dot product attention. : in, These are the query, key, and value matrices for a single attention head. The projected weight matrix is ​​a learnable matrix. and For the projection dimension, the visual-to-attribute attention weight matrix for: Attention weights of each attention head and attention enhancement vector After the calculations are complete, the multi-head outputs are spliced ​​together and finally projected: in, The output projection matrix is ​​used; finally, the comprehensive attention weights from visual to attribute are calculated. Defined as the average of the weight matrices of all attention heads: Attribute vectors for computational vision enhancement and attribute-to-visual attention weights : in, for Calculate the attention weight matrix of the h-th attention head; construct a visual-attribute bidirectional attention field. : Among them, subscript This means that the variable is the first element of the original matrix. Line number The elements of the column; visual fusion features are calculated through visual-attribute bidirectional attention field contraction: Optionally, a cross-modal bidirectional attention field is constructed to achieve cross-modal feature interaction fusion of text and attributes, including: computing attribute-enhanced text vectors. and text-to-attribute attention weights : in, for The attention weight matrix of the h-th attention head is calculated.

[0009] Calculate the attribute vector for text enhancement and attribute-to-text attention weights : in, for The attention weight matrix of the h-th attention head is calculated.

[0010] Constructing a text-attribute bidirectional attention field : Calculate text fusion features by shrinking the attention field: in This represents element-wise multiplication.

[0011] Optionally, optimizing model parameters based on multimodal contrastive alignment loss to ensure cross-modal feature consistency includes: defining a contrastive loss function. : in Represents cosine similarity. and It can be replaced with any feature matrix. Specifically The matrix contains the first 1 eigenvector This refers to the temperature parameter.

[0012] Define multimodal contrast alignment loss : in, To compare loss functions, For visual fusion features, This is a text fusion feature.

[0013] The training is based on modal contrast alignment loss constraints, that is, using As a loss function, an optimizer is defined that simultaneously optimizes the parameters of the encoder and the fusion module using the gradient backpropagation algorithm.

[0014] Optionally, after the basic training, a prototype classifier is designed to achieve new category recognition without incremental training, including: for each category Calculate its visual fusion prototype vector representation : For images of new categories, classification is based on the nearest neighbor prototype: in The visual fusion feature of the i-th sample in this class. The prediction results of the prototype classifier. Indicates category The number of training samples.

[0015] In another aspect, the present invention provides a system for deploying the small-sample remote sensing image incremental recognition method based on cross-modal attribute feature fusion, comprising: an image acquisition module for acquiring remote sensing image data and corresponding text descriptions; an attribute library construction module for constructing an attribute feature library containing prior semantic attributes and learnable attributes; a feature projection module for performing projection transformations on the original image information, category text information, and attribute information to obtain a unified-dimensional feature representation; a visual-attribute fusion module for achieving cross-modal feature interaction fusion of text and attributes through a cross-modal bidirectional attention field mechanism; a text-attribute fusion module for achieving cross-modal feature interaction fusion of text and attributes through a cross-modal bidirectional attention field mechanism; a contrast alignment training module for optimizing model parameters through multimodal contrast alignment loss; and a classifier module for designing a prototype classifier to achieve new class image recognition without incremental training.

[0016] Compared with existing technologies, this invention has the following advantages: It introduces an attribute-guided cross-modal feature fusion mechanism, significantly improving the recognition accuracy for small samples. Attributes, acting as semantic bridges, can guide the model to extract highly discriminative fine-grained features even under conditions of scarce samples, effectively alleviating the problem of low recognition accuracy caused by large intra-class differences and high inter-class similarity in remote sensing images.

[0017] This invention enables remote sensing image recognition of new categories without incremental training, significantly improving model usability and deployment efficiency. The model parameters are fixed after only one basic training session, incurring no training overhead when facing the recognition needs of new categories. This not only avoids the catastrophic forgetting problem that may occur during incremental model training but also improves the usability and deployment efficiency of the model on edge devices such as satellites and drones. Attached Figure Description

[0018] Figure 1 is a flowchart of the method of the present invention.

[0019] Figure 2 is a visualization of the visual-attribute bidirectional attention field weights for a specific sample.

[0020] Figure 3 shows the average attention visualization heatmap of the visual-attribute bidirectional attention field of a specific sample across the visual modality. Detailed Implementation

[0021] To describe the implementation process of the present invention in detail, the embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. It should be noted that the embodiments described herein are only used to better describe the present invention, and the present invention can be implemented in many forms and is not limited to the examples described herein.

[0022] The specific implementation steps of this invention are shown in Figure 1, and include the following steps: Step S01: Obtain remote sensing image samples and their corresponding text descriptions. It should be noted that remote sensing images may contain visible light, infrared, multispectral, or hyperspectral data. The text description includes the main ground features, scene features, and attribute information that the image may contain. The generation method is to provide prompts to a large language model, such as: "Please describe in detail the typical visual information of the remote sensing image about 'category name', including main ground features, scene layout, color texture, and other characteristics." A corresponding text description is generated for each remote sensing image sample category, describing the main ground features, scene features, and attribute information that the image may contain; or a combination of templates and category names is used, such as: "This is a remote sensing image about 'category name'." Category text that is friendly to a multimodal text encoder is obtained.

[0023] Step S02: Construct an attribute feature library containing prior semantic attributes and learnable attributes. Using a large language model, prior semantic attribute text is generated based on publicly available knowledge in the field of remote sensing image interpretation, including major features, scene layout, color, and texture. This prior semantic attribute text is then transformed into prior attribute vectors using a multimodal large-scale text encoder, such as a contrastive language-visual model. Simultaneously, multiple sets of learnable attribute vectors are randomly initialized, and the two are concatenated to form a complete attribute feature library. in Represents a priori semantic attribute library; This indicates a learnable attribute library; It is a concatenation function along the feature dimension; The total number of attributes. The number of prior semantic attributes. The number of learnable attribute vectors; For attribute feature dimensions.

[0024] Step S03: Perform projection transformations on the original image information, category text information, and attribute information respectively to obtain a feature representation with a unified dimension.

[0025] The original image information is projected into a low-dimensional space using a visual projection layer to obtain the image feature matrix. : A text projection layer is used to project the categorical text information into the same low-dimensional space, resulting in the categorical feature matrix. : The attribute projection layer is used to project attribute information onto the same low-dimensional space to obtain the attribute feature matrix. : in , These are the original image information and the category text information, respectively. The original image information is the feature output after the original image is segmented into multiple image tokens and input into a visual encoder such as a visual attention model. The category text information only contains the classification feature output of the original category text in a text encoder such as a contrastive language-visual model. , , These are dedicated projection matrices for three categories of features: image, attribute, and text. For the number of visual tokens, To unify the projection dimensions.

[0026] Step S04: Construct a cross-modal bidirectional attention field to achieve cross-modal feature interaction fusion of visual-attribute and text-attribute.

[0027] 1) Visual-attribute attention field fusion: computationally enhancing visual vectors with attributes And the comprehensive attention weight matrix from visual to attribute The specific process is as follows: For the input image feature matrix and attribute feature matrix Through multi-head attention mechanism, As keys and values, As a query, calculate the visual-to-attribute attention weights. and attribute-enhanced visual vectors : in, The multi-head attention mechanism is described in the following calculation process: For the h-th attention head, the attention enhancement vector is calculated through linear projection and scaled dot product attention. : in, These are the query, key, and value matrices for a single attention head. The projected weight matrix is ​​a learnable matrix. and For the projection dimension, the visual-to-attribute attention weight matrix for: Attention weights of each attention head and attention enhancement vector After the calculations are complete, the multi-head outputs are spliced ​​together and finally projected: in, The output projection matrix is ​​used; finally, the comprehensive attention weights from visual to attribute are calculated. Defined as the average of the weight matrices of all attention heads: Attribute vectors for computational vision enhancement and attribute-to-visual attention weights : in, for The attention weight matrix of the h-th attention head is calculated.

[0028] Constructing a visual-attribute bidirectional attention field : Among them, subscript This means that the variable is the first element of the original matrix. Line number The elements of the column.

[0029] Calculate visual fusion features by contracting the visual-attribute bidirectional attention field: 2) Text-Attribute Attention Field Fusion: Computational Attribute-Enhanced Text Vectors and text-to-attribute attention weights : in, for The attention weight matrix of the h-th attention head is calculated.

[0030] Calculate the attribute vector for text enhancement and attribute-to-text attention weights : in, for The attention weight matrix of the h-th attention head is calculated.

[0031] Constructing a text-attribute bidirectional attention field : Calculate text fusion features by shrinking the attention field: in This represents element-wise multiplication, and MHA represents multi-head attention mechanism.

[0032] Step S05: Perform joint training based on the multimodal contrastive alignment loss bundle.

[0033] Define the contrastive loss function : in Represents cosine similarity. and For any characteristic matrix, Specifically The matrix contains the first 1 eigenvector For temperature parameters. (Please complete the meaning of the first letter that appears); Define the multimodal contrast alignment loss. : in, To compare loss functions, For visual fusion features, This is a text fusion feature.

[0034] The training is based on modal contrast alignment loss constraints, that is, using As a loss function, an optimizer is defined that simultaneously optimizes the parameters of the encoder and the fusion module using the gradient backpropagation algorithm.

[0035] Step S06: After basic training, design a prototype classifier to achieve new class image recognition without incremental training. For each class... Calculate its visual fusion prototype vector representation : For images of new categories, classification is based on the nearest neighbor prototype: in The visual fusion feature of the i-th sample in this class. The prediction results of the prototype classifier. Indicates category The number of training samples.

[0036] The embodiments of the present invention can effectively handle incremental learning tasks with few samples in remote sensing scenarios, and significantly improve the stability and accuracy of the model when continuously learning new categories.

[0037] Compared with the incremental learning methods for few samples in the prior art, the recognition accuracy of the present invention is higher, as shown in Table 1.

[0038] Table 1 shows a comparison of the recognition accuracy of the embodiments of the present invention with other advanced few-shot incremental learning methods requiring incremental training on the NWPU-RESISC45 remote sensing dataset. The NWPU-RESISC45 dataset contains 45 remote sensing scene categories, totaling 31,500 images, with 700 images per category and an image size of 256×256×3. In the experiments, the 45 categories were divided into 25 base classes and 20 incremental classes, with the incremental classes further divided into 4 incremental stages. The training and test sets for the base class portion were split in a 3:1 ratio. For the incremental stages, the training set for each stage was generated by randomly selecting 5 training samples from each category, while the test set maintained its original ratio.

[0039] The results show that, in different testing phases, the recognition performance of the embodiments of the present invention is superior to existing methods, with an average accuracy of 84.86%, representing a 6.80% improvement over the best method, FACT+G-MFCN. Compared with existing few-shot incremental learning methods that rely on additional incremental training techniques, the present invention exhibits better stability in continuous incremental learning without the need for incremental training. The results also show that, across multiple incremental sessions, the embodiments of the present invention demonstrate more stable performance and superior resistance to forgetting.

[0040] Table 1. Comparison of recognition results of the proposed method and advanced few-shot incremental learning methods on the NWPU-RESISC45 dataset (bold for best, underline for second best) | Method | Base Class Stage | Incremental Stage 1 | Incremental Stage 2 | Incremental Stage 3 | Incremental Stage 5 | Average Accuracy | |---|---|---|---|---|---| | Linear | 92.49 | 14.39 | 14.05 | 13.00 | 12.78 | 29.34 | | SPPR | 94.18 | 83.02 | 73.45 | 66.57 | 59.37 | 75.32 | | CEC | 91.81 | 75.19 | 74.15 | 71.56 | 67.23 | 75.99 | | FACT | 92.99 | 77.58 | 73.40 | 69.77 | 66.00 | 75.95 | | C-FSCIL | 94.12 | 81.18 | 72 .7665.5559.2074.56PCOFA94.5283.6875.4467.8861.3076.56CEC+auto93.1477.9874.6772.6569.2877.54SSFE92.7079.4074.7672.8970.2678.00CEC+G-MFCN92.4080.2674.5572.6567.5377.48FACT+G-MFCN93.0080.2975.6972.8568.6078.06Method of this invention99.4589.6882.0079.0574.1484.86 Tables 2 and 3 illustrate the cross-modal visual-attribute attention field in this invention. Figure 2 shows the numerical distribution of the attention field for two samples labeled "airplane" and "airport," with peak values ​​indicating a high degree of matching between a specific attribute and a certain region of the image sample. This demonstrates that the attention field after multimodal alignment can distinguish attributes based on their matching degree with the sample, and simultaneously map important attributes to specific spatial locations within the sample image; its distribution is as expected. Figure 3 selects samples with four different labels, including the aforementioned samples, and shows the attention field strength at specific locations within the image samples. Red-green indicates that the region receives significant attention from key attributes. This reveals that the injection of attribute modality can guide the attention field to regions rich in remote sensing sample classification information, thereby guiding the model to extract key information from the visual modality and achieving incremental classification of remote sensing samples.

[0041] This invention addresses incremental learning scenarios with small sample classes in remote sensing images characterized by significant intra-class differences and high inter-class similarities. By leveraging an attribute-guided bidirectional attention mechanism, it significantly captures and extracts fine-grained information, improving model recognition accuracy and enhancing anti-forgetting capabilities. It requires only a single-stage training phase in the base class stage to accumulate robust foundational knowledge. For newly added categories, it demonstrates superior incremental remote sensing image recognition performance compared to existing advanced methods without the need for incremental training. This approach facilitates deployment on edge computing devices such as drones or satellites, enabling its application in open scenarios such as disaster emergency response and dynamic monitoring of land and resources.

[0042] On the other hand, this invention also provides a system for deploying a small-sample remote sensing image incremental recognition method based on cross-modal attribute feature fusion, comprising: an image acquisition module for acquiring remote sensing image data and corresponding text descriptions; an attribute library construction module for constructing an attribute feature library containing prior semantic attributes and learnable attributes; a feature projection module for performing projection transformations on the original image information, category text information, and attribute information to obtain a unified-dimensional feature representation; a visual-attribute fusion module for achieving cross-modal feature interaction fusion of text and attributes through a cross-modal bidirectional attention field mechanism; a text-attribute fusion module for achieving cross-modal feature interaction fusion of text and attributes through a cross-modal bidirectional attention field mechanism; a contrast alignment training module for optimizing model parameters through multimodal contrast alignment loss; and a classifier module for designing a prototype classifier to achieve new category image recognition without incremental training.

[0043] It should be noted that although the technical solutions and implementation details of the present invention have been described in detail through specific embodiments, their purpose is only to help those skilled in the art understand the core concepts and technical points of the present invention, and is not intended to limit the scope of protection of the present invention. Any equivalent modifications or substitutions made under the guidance of the concept and spirit of the present invention are included within the scope defined by the claims of the present invention.

Claims

1. A method for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion, characterized in that, include: Acquire remote sensing image samples and their corresponding text descriptions; construct an attribute feature library containing prior semantic attributes and learnable attributes; perform projection transformations on the original image information, category text information, and attribute information respectively to obtain a unified dimension feature representation; construct a cross-modal bidirectional attention field to achieve cross-modal feature interaction and fusion of visual-attribute and text-attribute; Joint training is performed based on multimodal contrast alignment loss constraints; after basic training, a prototype classifier is designed to achieve new class image recognition without incremental training.

2. The method for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion according to claim 1, characterized in that, The process of obtaining remote sensing image samples and their corresponding text descriptions includes: obtaining raw remote sensing image data; generating corresponding text descriptions for each remote sensing image sample category by providing prompts to a large language model, describing the main ground features, scene characteristics, and attribute information that the image category may contain; or obtaining category text that is friendly to multimodal text encoders by combining templates and class names.

3. The method for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion according to claim 1, characterized in that, The construction of the attribute feature library, which includes prior semantic attributes and learnable attributes, includes: using a large language model to generate multiple sets of prior semantic attributes based on knowledge from the remote sensing image interpretation domain, such as major land features, scene layout, color, and texture; using a multimodal large model text encoder to convert the aforementioned prior semantic attribute text into semantic feature vectors, forming a prior semantic attribute library; randomly initializing multiple sets of learnable attribute vectors to form a learnable attribute library; and concatenating the prior semantic attributes and learnable attributes to form a complete attribute feature library. ： in Represents a priori semantic attribute library; This indicates a learnable attribute library; It is a concatenation function along the feature dimension; The total number of attributes. The number of prior semantic attributes. The number of learnable attribute vectors; For attribute feature dimensions.

4. The method for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion according to claim 1, characterized in that, The step of performing projection transformations on the original image information, category text information, and attribute information to obtain a feature representation of a unified dimension includes: using a visual projection layer to project the original image information into a low-dimensional space to obtain an image feature matrix. ： The attribute projection layer is used to project attribute information onto the same low-dimensional space to obtain the attribute feature matrix. ： A text projection layer is used to project the categorical text information into the same low-dimensional space, resulting in the categorical feature matrix. ： in 、 These are the original image information and the category text information, respectively. The original image information is the feature output after the original image is segmented into multiple image tokens and input into the multimodal model visual encoder. The category text information only contains the classification feature output of the original category text in the text encoder of the contrast language-visual model and other large multimodal models. ， ， These are dedicated projection matrices for three categories of features: image, attribute, and text. For the number of visual tokens, To unify the projection dimensions.

5. The method for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion according to claim 1, characterized in that, Cross-modal bidirectional attention fields are constructed through cross-modal attention field fusion to achieve cross-modal feature interaction fusion of visual-attribute and text-attribute modes, including: 1) Visual-attribute attention field fusion: calculating attribute-enhanced visual vectors. And the comprehensive attention weight matrix from visual to attribute The specific process is as follows: For the input image feature matrix and attribute feature matrix Through multi-head attention mechanism, As keys and values, As a query, calculate the visual-to-attribute attention weights. and attribute-enhanced visual vectors ： in, The multi-head attention mechanism is described in the following calculation process: For the h-th attention head, the attention enhancement vector is calculated through linear projection and scaled dot product attention. ： in, These are the query, key, and value matrices for a single attention head; The projective weight matrix is ​​a learnable matrix. and For projection dimensions; This represents the matrix transpose operation; from which the visual-to-attribute attention weight matrix is ​​calculated. ： Attention weight matrix of each attention head and attention enhancement vector After the calculations are complete, the multi-head outputs are spliced ​​together and then projected in the final output: in, The output projection matrix is ​​used; finally, the comprehensive attention weights from visual to attribute are calculated. Defined as the average of the weight matrices of all attention heads: Attribute vectors for computational vision enhancement and attribute-to-visual attention weights ： in, for Calculate the attention weight matrix of the h-th attention head; construct a visual-attribute bidirectional attention field. ： Among them, subscript This means that the variable is the first element of the original matrix. Line number The elements of the column; visual fusion features are calculated through visual-attribute bidirectional attention field contraction: in, 1) Element-wise multiplication of two matrices; 2) Text-attribute attention field fusion: computing attribute-enhanced text vectors. and text-to-attribute attention weights ： in, for Calculate the attention weight matrix for the h-th attention head. Calculate the attribute vector for text enhancement. and attribute-to-text attention weights ： in, for Calculate the attention weight matrix of the h-th attention head; construct a text-attribute bidirectional attention field. ： Calculate text fusion features by shrinking the attention field: in This represents element-wise multiplication of two matrices.

6. The method for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion according to claim 1, characterized in that, The joint training based on multimodal contrastive alignment loss constraints includes: defining a contrastive loss function. ： in Represents cosine similarity. and It can be replaced with any feature matrix. Specifically The matrix contains the first 1 eigenvector Define the temperature parameter; define the multimodal contrast alignment loss. ： in, To compare loss functions, For visual fusion features, For text fusion features; the joint training based on multimodal contrastive alignment loss constraints, that is, utilizing As a loss function, an optimizer is defined to simultaneously optimize the parameters of the encoder and the fusion module using the gradient backpropagation algorithm.

7. The method for incremental recognition of small-sample remote sensing images based on cross-modal attribute feature fusion according to claim 1, characterized in that, After the basic training, the following prototype classifier is designed to achieve new class image recognition without incremental training, including: for each class Calculate its visual fusion prototype vector representation ： For images of new categories, classification is based on the nearest neighbor prototype: in The visual fusion feature of the i-th sample in this class. The prediction results of the prototype classifier. Indicates category The number of training samples.

8. A system for implementing the small-sample remote sensing image incremental recognition method based on cross-modal attribute feature fusion as described in any one of claims 1-7, characterized in that, include: Image acquisition module: acquires remote sensing image data and corresponding text descriptions; Attribute library construction module: constructs an attribute feature library containing prior semantic attributes and learnable attributes; Feature projection module: performs projection transformations on the original image information, category text information, and attribute information to obtain a unified dimension feature representation; Cross-modal attention field fusion module: Constructs a cross-modal bidirectional attention field to achieve cross-modal feature interaction fusion of visual-attribute and text-attribute; Joint training module: Performs joint training based on multimodal contrastive alignment loss constraints; Classifier module: Design a prototype classifier to achieve image recognition without incremental training.

Images