Large model driven small sample remote sensing target detection method

Abstract

The invention discloses a large-model-driven small sample remote sensing target detection method, which belongs to the technical field of artificial intelligence, and comprises the following steps: carrying out multi-scale feature extraction on a to-be-detected remote sensing image to obtain a multi-scale feature map and generate a scene-level feature vector; inputting the scene-level feature vector into a pre-trained large model, generating a scene text describing the current remote sensing image and a category text generated based on scene text extension, and encoding the scene text and the category text into a scene context instruction vector; calculating a weight through a scene context instruction vector, and carrying out weighted fusion on the multi-scale feature map to obtain a feature enhancement vector; inputting the feature enhancement vector into a pre-trained small sample remote sensing target detection model; and outputting a final detection box list as a detection result, wherein each final detection box comprises a final category and bounding box coordinates. According to the invention, the problems of strong sample dependence, insufficient scene generalization ability and unstable detection precision in the existing remote sensing target detection technology can be solved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a large-model-driven method for detecting remote sensing targets in small samples. Background Technology

[0002] With the continuous improvement of remote sensing image resolution and the enhancement of satellite data acquisition capabilities, remote sensing target detection technology has been widely applied in fields such as land resource surveys, ecological environment monitoring, agricultural assessment, urban planning, and disaster early warning. The core task of this technology is to identify and locate targets of interest, such as buildings, vehicles, ships, roads, or crops, from complex remote sensing images. However, existing deep learning target detection methods typically rely on large-scale labeled samples for model training. When the number of samples is limited or the scene differences are significant, model performance often declines noticeably. Especially under small sample conditions, the model struggles to effectively learn discriminative feature representations, resulting in low detection accuracy and insufficient generalization ability.

[0003] Traditional few-sample object detection research mostly employs strategies such as metric learning, transfer learning, or meta-learning, achieving classification or localization by calculating the feature similarity between support samples and query samples. These methods perform well in natural scenes, but in remote sensing image tasks, they are often affected by the complexity of ground features, background noise, and scale variations. For example, natural disturbances such as clouds, shadows, and water bodies can blur or distort texture features in remote sensing images, leading to confusion in feature matching and target localization. Furthermore, remote sensing images exhibit significant cross-regional and cross-temporal differences; different satellites, imaging conditions, and geographical environments can cause substantial variations in the spectral characteristics of similar targets, making it difficult for traditional few-sample detection models to maintain robustness under such multi-source conditions.

[0004] In recent years, large-scale models such as GPT, BLIP, and LLaVA have demonstrated significant advantages in multimodal understanding and instruction learning, capable of extracting high-level semantic information from text descriptions and guiding downstream tasks in natural language. These models possess powerful context awareness and semantic reasoning capabilities, providing new insights for remote sensing image analysis. However, current research on remote sensing target detection largely focuses on visual feature enhancement or network structure optimization, failing to fully utilize the language understanding capabilities of large-scale models to provide semantic instruction support for small-sample detection. Existing methods generally lack a mechanism to dynamically adjust feature selection and inference strategies based on scene context semantics, resulting in insufficient generalization performance in complex and diverse remote sensing scenes, and detection results are easily biased due to geographical and temporal differences.

[0005] Therefore, in order to address the problems of weak generalization ability, unstable detection accuracy, and lack of scene semantic guidance in remote sensing target detection models under small sample conditions, there is an urgent need for a remote sensing target detection method that combines the scene context instruction mechanism of large models, so as to achieve high-precision target recognition and adaptive detection across scenes and tasks. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a large model-driven method for detecting remote sensing targets with small samples, thereby solving the problems of strong sample dependence, insufficient scene generalization ability, and unstable detection accuracy in existing remote sensing target detection technologies.

[0007] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0008] This invention provides a large-model-driven method for detecting remote sensing targets in small samples, comprising:

[0009] Multi-scale feature extraction is performed on the remote sensing image to be detected to obtain a multi-scale feature map, and scene-level feature vectors are generated based on the multi-scale feature map;

[0010] The scene-level feature vector is input into a pre-trained large model to generate scene text describing the current remote sensing image and category text generated based on the scene text. The scene text and category text are then encoded into scene context instruction vectors. The pre-trained large model is used to generate text describing the scene based on the feature vector or scene text.

[0011] Weights are calculated using scene context instruction vectors, and multi-scale feature maps are weighted and fused to obtain feature enhancement vectors.

[0012] The feature enhancement vector is input into the detection head of the pre-trained few-sample remote sensing target detection model to generate multiple candidate boxes, and the score of each category in each candidate box is calculated separately.

[0013] The category with the highest score in each candidate box that exceeds the threshold is retained as the final category. Non-maximum suppression is applied to all retained candidate boxes belonging to the same category to remove overlapping redundant detection boxes.

[0014] Output the final list of detection boxes as the detection result. Each final detection box includes the final category and bounding box coordinates.

[0015] The aforementioned large-model-driven small-sample remote sensing target detection method involves extracting multi-scale features from the remote sensing image to be detected, obtaining a multi-scale feature map, and generating a scene-level feature vector based on the multi-scale feature map, including:

[0016] A multi-scale convolutional feature extraction network is used to extract multi-scale features from the remote sensing image to be detected, resulting in a multi-scale feature map composed of feature maps of different resolutions. The multi-scale feature maps are then concatenated to obtain a fused feature tensor. Global pooling is then performed on the fused feature tensor to obtain a scene-level feature vector.

[0017] The aforementioned large-model-driven small-sample remote sensing target detection method, wherein scene-level feature vectors are input into a pre-trained large model to generate scene text describing the current remote sensing image and category text generated based on scene text expansion, and the scene text and category text are encoded into scene context instruction vectors, including:

[0018] A large-scale LLM model is pre-trained using scene-level feature vectors s as conditional input to generate scene text T describing the overall semantics of the current remote sensing image. ctx ; Based on scene text T ctx The pre-trained large model LLM is guided to expand each category in the pre-defined category vocabulary C to generate the corresponding category text T. cls ; the scene text T ctx and category text T cls After being structurally spliced ​​according to the preset prompt template, the complete text is formed. The complete text is then input into the pre-trained text embedding module, and after text encoding and mapping operations, it is projected onto the shared semantic space where the multi-scale feature map is located to obtain the context instruction vector g.

[0019] The aforementioned large-model-driven small-sample remote sensing target detection method, which calculates weights through scene context command vectors and performs weighted fusion of multi-scale feature maps to obtain a feature enhancement vector, includes:

[0020] The scene context instruction vector g is used to calculate the channel weight w through the projection matrix W.

[0021] ,

[0022] In the formula, The preset projection matrix; Use the Sigmoid activation function;

[0023] The channel weights w include the channel weights of features at each scale. The channel weights of features at each scale are weighted and fused onto the multi-scale feature maps to obtain the feature enhancement vector F. g .

[0024] The aforementioned large-model-driven small-sample remote sensing target detection method involves training the small-sample remote sensing target detection model, which includes:

[0025] Repeat the following steps until the overall loss function converges or the maximum number of training epochs is reached:

[0026] A small sample detection task is performed, which includes a query image with full annotations and K support samples independently sampled for each category in a predefined category vocabulary C; the full annotation of the query image includes the true categories of all targets to be detected and their corresponding bounding box coordinates;

[0027] The query image in the current training round is subjected to multi-scale feature extraction, scene text and category text generation and encoding, and weighted fusion of multi-scale feature maps in sequence, resulting in the multi-scale feature map and scene context command vector g of the query image in the current training round. x and feature enhancement vector F gx ;

[0028] Based on the scene context instruction vector g x Calculate the category prototypes of all categories in the pre-defined category vocabulary C for the current training round. ;

[0029] Calculate the detection loss for the current training round based on the model prediction results. Prototype alignment loss Loss of consistency with instructions ;

[0030] The feature enhancement vector is input into the detection head of the small sample remote sensing target detection model, and the output class probability distribution, bounding box position offset and target confidence are used as the model prediction results. The detection loss is calculated based on the difference between the model prediction results and the ground truth labels of the query image; the ground truth labels include class labels, bounding box positions and target labels.

[0031] Loss detection in the current training round The calculation formula is:

[0032] ,

[0033] In the formula, The classification loss is calculated based on the predicted class probability distribution and the true class label. The bounding box regression loss is calculated based on the predicted bounding box position offset and the true bounding box position. The target confidence loss is calculated based on the predicted target confidence and the true target label, where 1 indicates the presence of a target and 0 indicates background.

[0034] Prototype alignment loss in the current training round The calculation formula is:

[0035] ,

[0036] In the formula, This represents the number of positive samples in the query image during the current training round. Positive samples are candidate boxes whose Intersection over Union (IoU) with the ground truth bounding boxes is greater than a threshold. It is a d-dimensional feature vector extracted from the i-th positive sample candidate box in the current training round; For the current training round category Category prototype, It is the true class of the i-th positive candidate box in the current training round; The square of the Euclidean distance;

[0037] Current training round instruction consistency loss The calculation formula is:

[0038] ,

[0039] In the formula, Calculate the KL divergence between the two distributions; softmax() is the normalization function that converts the vector into a probability distribution; H is a linear mapping matrix used to linearly map the feature enhancement vector Fgx of the current training round to the d'-dimensional semantic space; G is a linear projection matrix used to linearly map the scene context instruction vector gx of the current training round to the d'-dimensional semantic space.

[0040] Loss is detected based on the current training round. Prototype alignment loss Loss of consistency with instructions The weighted calculation of the overall loss function L is given by the following formula:

[0041] ,

[0042] In the formula, and These are the preset tradeoff coefficients between prototype alignment loss and instruction consistency loss.

[0043] The aforementioned large-model-driven small-sample remote sensing target detection method, wherein the scene context instruction vector g x Calculate the category prototypes of all categories in the pre-defined category vocabulary C for the current training round. ,include:

[0044] The supporting sample subset S of category l in the preset category vocabulary C C,l It includes K samples: k is the supporting sample subset S C,l The index of the sample, ranging from 1 to K; This represents the feature vector of the k-th supporting sample;

[0045] Repeatedly execute the following category prototype calculation process for a single category in the current training round to obtain the category prototypes for all categories in the preset category vocabulary C for the current training round. :

[0046] Calculate the class prototype of class l in the current training round. This includes: inputting all K supporting samples of category l into the feature extraction network. The feature representation of the supporting sample of category l is obtained. The calculation formula is: In the formula, T represents the transpose operation; F SUP It is a K×d matrix, where each row is a d-dimensional feature vector of a sample; It is a matrix The k-th row; based on the scene context instruction vector g of the current training round. x and the feature representation of supporting samples Calculate the importance of each supporting sample in category l, and the importance weight of the supporting samples in category l. The calculation formula is: In the formula, The importance weight of category l supporting samples Includes K elements; For the scene context instruction vector g x The vector obtained by linear projection onto the d-dimensional feature space. for The transpose of the function; softmax is the normalization function, which transforms the sample to obtain the importance weights of class l supporting the sample. ,and Weight the importance of the samples supported by category l. After weighted summation of the feature vectors of the corresponding supporting samples and normalization, the class prototype of class l in the current training epoch is obtained. The calculation formula is: In the formula, This represents the L2 norm operation.

[0047] The aforementioned large-model-driven small-sample remote sensing target detection method, wherein calculating the score for each category in each candidate bounding box includes:

[0048] Adjust the nonmaximum suppression threshold based on the scene context instruction vector g. Category confidence threshold Matching temperature parameters :

[0049] Non-maximum suppression threshold The calculation formula is:

[0050] ,

[0051] Confidence threshold The calculation formula is:

[0052] ,

[0053] Matching temperature parameters The calculation formula is:

[0054] ,

[0055] In the formula, Use the Sigmoid activation function; (·) is a smooth positive value mapping function; , and These are the preset linear mapping matrices. , and transpose; , and The preset linear transformation bias is used; the score calculation formula for candidate box m category l is:

[0056] ,

[0057] In the formula, m is the candidate box index, and the total number of candidate boxes output by the detection head is M; l is the category index, and the total number of categories in the preset category vocabulary C is C'; and These are the preset weighting coefficients for the fusion of classification score and prototype similarity. ; Let m be the base class probability of class l of candidate box; For similarity function, In the formula, yes transpose; It is a d-dimensional feature vector extracted from the candidate box m; The final category prototype for the pre-computed category l; express and The dot product; after the training of the small-sample remote sensing target detection model is completed, the class prototypes of the same category in all training rounds are averaged to obtain the final class prototypes p for each category. c *, It is p c One of the elements in *.

[0058] The aforementioned large-model-driven small-sample remote sensing target detection method retains the class with the highest score exceeding a threshold in each candidate box as the final class. Non-maximum suppression is applied to all retained candidate boxes belonging to the same class to remove overlapping redundant detection boxes. The final detection box list is output as the detection result. Each final detection box includes the final class and bounding box coordinates, including: retaining categories with scores greater than the confidence threshold. The candidate boxes are selected; for each candidate box, the category with the highest score is taken as its final category; the highest score and corresponding category of each candidate box are obtained; the highest score greater than the confidence threshold is retained. The candidate boxes are selected, and the category corresponding to the highest score among the retained candidate boxes is taken as the final category. Non-maximum suppression (NMS) is applied to all retained boxes belonging to the same category to remove overlapping redundant detection boxes: For all retained candidate boxes belonging to the same category, NMS is applied based on their bounding box coordinates and confidence scores. Candidate boxes of the same category are sorted in descending order of confidence score, and the highest-scoring box is selected sequentially. Boxes with an intersection-union (IoU) exceeding the NMS threshold are removed. It processes all other bounding boxes until all boxes have been processed; the final list of detected bounding boxes is output as the detection result, and each final detected bounding box contains the final category, bounding box coordinates and confidence score.

[0059] The aforementioned large-model-driven few-sample remote sensing target detection method, in its training of the few-sample remote sensing target detection model, also includes: calculating the category prototypes of all categories in the pre-set category vocabulary C for the current training round. Previously, the supporting samples of each category were augmented using the scene text of the current training round.

[0060] In the aforementioned large model-driven small sample remote sensing target detection method, the scene context instruction vector is cached and updated according to scene region or time window; when continuously input remote sensing images belong to the same geographical region or the same time window, the cached scene context instruction vector g is reused; when the region changes or the time window switches, the scene context instruction vector is recalculated and updated.

[0061] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0062] The large-model-driven small-sample remote sensing target detection method of the present invention introduces a pre-trained large model with natural language understanding and semantic reasoning capabilities to generate scene context instruction vectors related to the remote sensing image scene. These vectors are then used as semantic guidance signals to dynamically adjust the feature extraction and reasoning strategies of the detection model. This enables cross-scene, highly robust target detection under limited sample conditions, solving the problems of strong sample dependence, insufficient scene generalization ability, and unstable detection accuracy in existing remote sensing target detection technologies.

[0063] This invention, based on the conventional object detection process, introduces a pre-trained large model to calculate and generate scene context instruction vectors, forming a semantically driven dynamic adaptive detection framework, specifically embodied in:

[0064] (1) A scene context instruction vector is generated using a large model to transform visual features into high-level semantic descriptions and introduce scene knowledge to assist in detection decisions. By extracting scene visual features to generate scene-level feature vectors, and then using LLM to generate semantic descriptions, which are encoded as scene context instruction vectors g, the common sense and reasoning capabilities of the large language model are enabled to provide real-time and adaptive semantic priors for visual detection tasks, thereby significantly improving detection performance and robustness in complex and open scenes.

[0065] (2) The scene context instruction vector guides the construction of feature enhancement vector as input to the small sample remote sensing target detection model; and during the training process, the scene context instruction vector is calculated based on the query image of the current training round, and the category prototype is calculated based on the scene context instruction vector. The category prototype of the same category changes dynamically in different training batches, that is, the prototype alignment mechanism is adopted. The scene context instruction vector participates in the entire feature modeling process, dynamically adjusts the feature channel weight and category prototype construction, so that the small sample target has a more robust feature expression ability.

[0066] (3) The inference strategy adaptive mechanism driven by scene context instruction vectors adjusts the strategy in real time according to the scene context instruction vectors during the inference stage, and adaptively optimizes parameters such as NMS threshold and confidence threshold according to the scene context instruction vectors to enhance detection stability in complex scenarios.

[0067] To improve the efficiency and stability of the inference phase, the scene context instruction vector g is managed using a caching mechanism. The scene context instruction vector is cached and updated according to scene region or time window. When consecutively input remote sensing images belong to the same geographic region or the same time window, the cached scene context instruction vector g is directly reused; the scene context instruction vector is recalculated and updated only when the region changes or the time window switches.

[0068] To further improve the scenario adaptability of category prototype construction, the category prototype in the current training round is calculated. Previously, semantically consistent enhancement processing was performed on the supporting samples of all categories in category vocabulary C based on the scene text generated in the current training round, so that the category prototypes built on the enhanced supporting samples have higher stability and generalization ability under different scene conditions.

[0069] Before using the model, the category prototypes for all categories to be detected have been pre-calculated and stored. In other words, the category prototypes for all categories within the pre-defined category vocabulary C have been obtained. When a new image to be detected is input into the pre-trained small-sample remote sensing object detection model, there is no need to recalculate the prototypes; the stored prototypes can be directly used for matching, which greatly improves inference speed.

[0070] Therefore, this invention can effectively overcome the problem of decreased detection accuracy in multi-source, multi-temporal, and multi-scale remote sensing scenarios using traditional methods, improve the model's adaptability, generalization performance, and computational efficiency in complex environments, and solve the problems of strong sample dependence, insufficient scene generalization ability, and unstable detection accuracy in existing small-sample remote sensing target detection technologies.

[0071] The target detection method proposed in this invention can fully utilize the semantic understanding capability of large models and the feature learning mechanism of small samples to achieve accurate detection and cross-domain adaptation in complex remote sensing scenes, and significantly improve the detection accuracy, generalization performance and computational efficiency of the model. Attached Figure Description

[0072] Figure 1 This is a schematic diagram illustrating the construction process of a large-model-driven small-sample remote sensing target detection method according to the present invention. Detailed Implementation

[0073] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0074] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0076] Example 1:

[0077] This embodiment introduces a large model-driven method for small-sample remote sensing target detection, including:

[0078] S1: Perform multi-scale feature extraction on the remote sensing image to be detected to obtain a multi-scale feature map, and generate a scene-level feature vector based on the multi-scale feature map;

[0079] S2: Input the scene-level feature vector into the pre-trained large model to generate scene text describing the current remote sensing image and category text generated based on the scene text, and encode the scene text and category text into scene context instruction vector; the pre-trained large model is used to generate text describing the scene based on the feature vector or scene text;

[0080] S3: Calculate weights using scene context instruction vectors, and then perform weighted fusion of multi-scale feature maps to obtain feature enhancement vectors;

[0081] S4: Input the feature enhancement vector into the detection head of the pre-trained few-sample remote sensing target detection model to generate multiple candidate boxes, and calculate the score for each category in each candidate box.

[0082] S5: Retain the category with the highest score in each candidate box that exceeds the threshold as the final category, perform non-maximum suppression on all retained candidate boxes belonging to the same category, and remove overlapping redundant detection boxes;

[0083] S6: Output the final list of detection boxes as the detection results. Each final detection box includes the final category and its coordinates in the remote sensing image.

[0084] This embodiment mainly introduces, for example... Figure 1 The process of generating scene context command vectors and the process of generating feature enhancement vectors guided by scene context command vectors are shown below:

[0085] Step S1 includes:

[0086] The preprocessed remote sensing image I is used to extract multi-scale feature maps through a multi-scale convolutional feature extraction network, and then the multi-scale feature maps are fused. After the operation, a scene-level feature vector s is obtained through global pooling; specifically, it includes:

[0087] The original remote sensing image is subjected to size normalization, pixel normalization, denoising, and data enhancement such as random flipping, rotation, and color perturbation to obtain a preprocessed image. The preprocessed image is then input into a multi-scale convolutional feature extraction network to obtain a multi-scale feature map composed of feature maps of different resolutions. The multi-scale convolutional feature extraction network can be any network capable of extracting multi-scale features, such as FPN, BiFPN, ResNet+FPN, HRNet backbone, or Swin Transformer. The number of multi-scale feature maps is determined based on the network architecture design and task requirements. In this embodiment, the multi-scale feature map set includes feature maps of four scales. The multi-scale feature maps are concatenated along either the channel or scale dimension to obtain the fused feature tensor F. And obtain the scene-level feature vector s through global pooling operation. ,in It can be either global average pooling or global adaptive pooling.

[0088] In this embodiment, the above process expression is:

[0089] Multi-scale feature map set In the formula, I represents the preprocessed remote sensing image, and E(·) is a multi-scale convolutional feature extraction network; the fused feature tensor In the formula, concat(·) is the feature concatenation operation, where the scene-level feature vector is concatenated. In the formula, Pool(·) represents global average pooling.

[0090] Step S2 includes:

[0091] A large-scale LLM model is pre-trained using scene-level feature vectors s as conditional input to generate scene text T describing the overall semantics of the current remote sensing image. ctx ; Based on scene text T ctx The pre-trained large model LLM is guided to expand each category in the pre-defined category vocabulary C to generate the corresponding category text T. cls This process aims to obtain category description information that reflects different categories under the semantic conditions of the current scene. This step does not rely on prior judgments about the actual categories existing in the current image.

[0092] In this embodiment, the above process expression is:

[0093] In the formula, T ctx For the pre-trained large model LLM, scene text describing the current remote sensing image is generated based on scene-level feature vectors; T cls The pre-trained large model LLM generates category text based on scene text and a pre-set category vocabulary C.

[0094] In one specific embodiment, the scene text T ctx The description is as follows: The town has dense buildings and contains rivers and roads; the predefined category vocabulary C has a total of 3 categories C', including three categories: C={C1,C2,C3}, where C1 represents buildings, C2 represents rivers, and C3 represents roads, with category text T. cls To generate extended versions of the three categories: buildings, rivers, and roads.

[0095] The scene text T ctx and category text T cls After being structurally spliced ​​according to the preset prompt template, a complete text is formed. The complete text is then input into a pre-trained text embedding module, and after text encoding and mapping operations, it is projected onto the shared semantic space where the multi-scale feature map is located to obtain the context instruction vector g. The scene context instruction vector g is used to represent the semantic prior information of the current remote sensing image.

[0096] After text encoding and mapping operations, projecting onto the shared semantic space where the multi-scale feature map resides means: after text encoding and through a learnable linear mapping layer, the text semantic vector is projected onto a shared semantic space with the same channel dimension as the multi-scale feature map. The text embedding module is implemented based on existing mature text encoders, such as BERT, RoBERTa, and CLIP text encoders. Its basic structure includes: a text segmenter, a word embedding layer, and a multi-layer Transformer semantic encoder. The input text is first converted into a sequence of subwords by the segmenter, then mapped into a vector representation by the word embedding layer, and finally encoded by the Transformer network to obtain a fixed-dimensional text semantic vector.

[0097] In this embodiment, the above process expression is:

[0098] In the formula This represents the text encoding and mapping operations of the text embedding module. This indicates that the scene text T is concatenated according to a structured splicing function based on a preset template. ctx and category text T cls Concatenate the context instruction vector. Semantic priors used to characterize the current remote sensing image serve as guiding signals for subsequent feature enhancement and inference control.

[0099] Step S3 includes:

[0100] The scene context command vector g is used to calculate the channel weights through the projection matrix W. In the formula, This is a preset projection matrix used to map vector g into a preceding vector of weight coefficients; The activation function is Sigmoid. The channel weights w include the channel weights of features at each scale. In this embodiment, w includes w1, w2, w3, and w4. w1, w2, w3, and w4 are used to weight and fuse the multi-scale feature maps F1~F4 respectively to obtain the feature enhancement vector F. g The calculation formula is: .

[0101] The projection matrix W is generated through random initialization using Xavier initialization. The projection matrix W is generated during the training of the small-sample remote sensing target detection model to minimize the feature enhancement vector F. g The loss function used for subsequent task calculations serves as the optimization objective. A learnable parameter matrix is ​​obtained by iteratively updating and eventually converging from randomly initialized initial values ​​using a gradient descent algorithm. Semantic priors are utilized to select the importance of features at different scales. This weighting process enables the small-sample remote sensing target detection model to automatically focus on feature regions more relevant to the target category in different scenarios.

[0102] In some specific embodiments, to improve the efficiency and stability of the inference phase, the scene context instruction vector g is managed using a caching mechanism. The scene context instruction vector is cached and updated according to scene region or time window. Specifically, when consecutively input remote sensing images belong to the same geographic region or the same time window, their scene semantics have strong consistency. Therefore, it is not necessary to regenerate scene text and scene context instruction vector g for each image. The system detects the geographic region number or timestamp of the input image, and when it belongs to the same region or the same time window as the previous image, it directly reuses the cached scene context instruction vector g; the scene context instruction vector is only recalculated and updated when the region changes or the time window switches.

[0103] Example 2

[0104] Based on the same inventive concept as Embodiment 1, this embodiment describes a specific implementation of a large-model-driven small-sample remote sensing target detection method. This embodiment mainly introduces, as follows: Figure 1 The following describes the process of constructing category prototypes guided by scene context instruction vectors and training the few-sample remote sensing target detection model:

[0105] In step S4, the training of the small-sample remote sensing target detection model includes:

[0106] During training, the category prototypes are dynamically adjusted based on query samples. A joint optimization approach using support samples and query samples is employed, and the detection loss between the predicted results and the ground truth labels is calculated. Prototype alignment loss Loss of consistency with instructions The model parameters are dynamically updated by weighted summation. After each iteration, the parameters with the minimum loss are recorded to prevent overfitting, and training terminates when a preset convergence condition is met. Specifically, this includes:

[0107] Repeat steps X1 through X3 until the loss converges or the maximum number of training epochs is reached:

[0108] X1: Sample a small-sample detection task, including a query image with full annotations and K support samples independently sampled for each category in a predefined category vocabulary C:

[0109] A query sample and a set of support samples are sampled from the dataset. The query sample includes a query image and its complete annotation. The complete annotation of the query image includes the true categories of all objects to be detected and the corresponding bounding box coordinates, which are used to calculate the detection loss. For each category of the predefined category vocabulary C, K images of the corresponding category are randomly sampled from the dataset, K∈{1,5}, which constitute the support sample subsets of the corresponding categories. The set of all support sample subsets is the support sample set.

[0110] X2: Perform steps S1 to S3 on the query image of the current training round, which means sequentially performing multi-scale feature extraction, scene text and category text generation and encoding, and weighted fusion of multi-scale feature maps to obtain the multi-scale feature map and scene context command vector g of the query image of the current training round. x and feature enhancement vector F gx ;

[0111] Based on the scene context instruction vector g x Calculate the category prototypes of all categories in the pre-defined category vocabulary C for the current training round. The specific calculation process is as follows:

[0112] The supporting sample subset S of category l in the preset category vocabulary C C,l It includes K samples: k is the supporting sample subset S C,l The index of the sample, ranging from 1 to K; This represents the feature vector of the k-th supporting sample;

[0113] Calculate the class prototype of class l in the current training round. :

[0114] Input all K support samples of category l into the feature extraction network. The feature representation of the supporting sample of category l is obtained. The calculation formula is:

[0115] ,

[0116] In the formula, These are the K supporting samples for category l; T represents the transpose operation; F SUP It is a K×d matrix, where each row is a d-dimensional feature vector of a sample; It is a matrix The k-th row; in this embodiment, the feature extraction network You can choose a 1×1 convolutional layer, a fully connected layer, or a multilayer perceptron (MLP).

[0117] Based on the scene context instruction vector g of the current training round x and the feature representation of supporting samples Calculate the importance of each supporting sample in category l, and the importance weight of the supporting samples in category l. The calculation formula is:

[0118] ,

[0119] In the formula, The importance weight of category l supporting samples Includes K elements; For the scene context instruction vector g x The vector obtained by linear projection onto the d-dimensional feature space. for The transpose of the function; softmax is the normalization function, which transforms the sample to obtain the importance weights of class l supporting the sample. ,and ;

[0120] Weight the importance of the sample supporting category l. After weighted summation of the feature vectors of the corresponding supporting samples and normalization, the class prototype of class l in the current training epoch is obtained. The calculation formula is:

[0121] ,

[0122] In the formula, Represents the L2 norm operation; A d-dimensional vector; the class prototype of class l in the current training epoch. It is a d-dimensional vector.

[0123] Repeat the above process of calculating the category prototype for a single category in the current training round to obtain the category prototypes for all categories in the preset category vocabulary C for the current training round. ;

[0124] X3: Calculate the detection loss for the current training round based on the model prediction results. Prototype alignment loss Loss of consistency with instructions Specifically, this includes:

[0125] The feature enhancement vector is input into the detection head of the small sample remote sensing target detection model, and the output class probability distribution, bounding box position offset and target confidence are used as the model prediction results. The detection loss is calculated based on the difference between the model prediction results and the ground truth labels of the query image; the ground truth labels include class labels, bounding box positions and target labels.

[0126] Loss detection in the current training round The calculation formula is:

[0127] ,

[0128] In the formula, The classification loss is calculated using FocalLoss based on the predicted class probability distribution and the true class label. The bounding box regression loss is calculated using GIoU Loss based on the predicted bounding box position offset and the true bounding box position. The target confidence loss is calculated using a weighted binary cross-entropy loss based on the predicted target confidence and the true target label. In the target label, 1 indicates the presence of a target and 0 indicates background.

[0129] Prototype alignment loss in the current training round The calculation formula is:

[0130] ,

[0131] In the formula, This represents the number of positive samples in the query image during the current training round. Positive samples are candidate boxes whose Intersection over Union (IoU) with the ground truth bounding boxes is greater than a threshold. In a specific embodiment, candidate boxes with IoU > 0.5 are considered positive samples. It is a d-dimensional feature vector extracted from the i-th positive sample candidate box in the current training round; For the current training round category Category prototype, It is the true class of the i-th positive candidate box in the current training round; The square of the Euclidean distance is used; the prototype alignment loss is used to constrain the distance between positive sample features in the query image and the corresponding class prototype.

[0132] Current training round instruction consistency loss The calculation formula is:

[0133] ,

[0134] In the formula, The KL divergence between two distributions is calculated to measure their consistency. `softmax()` is a normalization function that transforms the vector into a probability distribution. `H` is a linear mapping matrix used to linearly map the feature augmentation vector `Fgx` from the current training round to a d'-dimensional semantic space. `G` is a linear projection matrix used to linearly map the scene context instruction vector `gx` from the current training round to a d'-dimensional semantic space. Instruction consistency loss is used to minimize the KL divergence between the feature augmentation vector and the scene context instruction vector in the prediction space, achieving cross-modal alignment.

[0135] The three losses mentioned above participate in backpropagation during the training phase to simultaneously optimize the performance of target detection and recognition, the semantic consistency of category prototypes, and the ability of scene context instruction vectors to constrain the model's inference strategy.

[0136] Loss is detected based on the current training round. Prototype alignment loss Loss of consistency with instructions The weighted calculation of the overall loss function L is given by the following formula:

[0137] ,

[0138] In the formula, and These are the preset tradeoff coefficients between prototype alignment loss and instruction consistency loss.

[0139] After training, the class prototypes of the same category from all training rounds are averaged to obtain the final class prototypes p for each category. c *; The final category prototype of category l is , It is p c One of the elements in *.

[0140] Before using the model, the category prototypes for all categories to be detected have been pre-calculated and stored. In other words, the category prototypes for all categories in the pre-defined category vocabulary C have been obtained. When a new image to be detected is input into the pre-trained small sample remote sensing object detection model, there is no need to recalculate the prototypes; the stored prototypes can be directly called for matching, which can greatly improve the inference speed.

[0141] In one specific embodiment, to further improve the scenario adaptability of category prototype construction, the category prototype of the current training round is calculated. Previously, a text-driven augmentation function was used to semantically augment the support samples of all categories in the category vocabulary C based on the scene text generated in the current training epoch. Combining the support samples of each category with the scene text of the current training epoch allows the support samples to exhibit appearance features consistent with the semantics of the current scene in terms of brightness, texture, color, occlusion pattern, or geometric appearance. The augmented support samples are closer to the true distribution of the current scene, resulting in higher stability and generalization ability of the category prototypes subsequently built based on the augmented support samples under different scene conditions.

[0142] Example 3

[0143] Based on the same inventive concept as Embodiments 1 and 2, this embodiment describes a specific implementation of a large-model-driven small-sample remote sensing target detection method. This embodiment mainly introduces... Figure 1 The process of generating the test results is shown below:

[0144] In step S4, the score for each category in each candidate box is calculated, including:

[0145] Adjust the nonmaximum suppression threshold based on the scene context instruction vector g. Category confidence threshold Matching temperature parameters This enables adaptive detection strategies for different remote sensing scenarios.

[0146] Non-maximum suppression threshold The calculation formula is:

[0147] ,

[0148] Confidence threshold The calculation formula is:

[0149] ,

[0150] Matching temperature parameters The calculation formula is:

[0151] ,

[0152] In the formula, Use the Sigmoid activation function; (·) is a smooth positive value mapping function; , and These are the preset linear mapping matrices. , and transpose; , and The preset linear transformation bias; linear mapping matrix , and and linear transformation bias , and Initialize randomly at the start of training, optimize through backpropagation during training, and save as fixed parameters after training, using them as a fixed mapping function during the inference phase; match temperature parameters. Used to adjust matching sensitivity;

[0153] The formula for calculating the score of candidate box m for category l is:

[0154] ,

[0155] In the formula, m is the candidate box index, and the total number of candidate boxes output by the detection head is M; l is the category index, and the total number of categories in the preset category vocabulary C is C'; and These are the preset weighting coefficients for the fusion of classification score and prototype similarity. ; Given the base class probability of candidate box m for category l, the detector head outputs the original category scores of all C categories of candidate box m. After transformation by the activation function, the base probability distribution of candidate box m for each category is obtained. The base class probability of candidate box m for category l is the element value of the corresponding category l position in the base probability distribution. This is a similarity function used to measure the degree of matching between the candidate bounding box features and the category prototype.

[0156] ,

[0157] In the formula, It is a d-dimensional feature vector extracted from the candidate box m; The final category prototype for the pre-computed category l; express and The dot product.

[0158] The d-dimensional vector is obtained by cropping the region of the candidate box m from the feature enhancement vector Fg using ROI Align and then passing it through a feature extraction network. In this embodiment, the candidate box feature vector ϕ(q) m The extraction method is as follows: First, the candidate bounding box m generated by the detector head is mapped to the spatial coordinates of the feature enhancement vector Fg; then, the local feature region corresponding to the candidate bounding box is extracted from Fg using ROIAlign; finally, the extracted local features are input into the feature extraction network. This is then transformed into a fixed-dimensional d-dimensional feature vector. Feature extraction network You can choose a 1×1 convolutional layer, a fully connected layer, or a multilayer perceptron (MLP).

[0159] Steps S5 and S6 include:

[0160] The category score is greater than the confidence threshold. The candidate boxes are selected; among the remaining candidate boxes, the category with the highest score is taken as its final category.

[0161] Obtain the highest score and corresponding category for each candidate box; retain the highest score that is greater than the confidence threshold. The candidate boxes are selected, and the category corresponding to the highest score in the selected candidate box is the final category.

[0162] Non-maximum suppression is applied to all retained bounding boxes belonging to the same category to remove overlapping redundant detection boxes:

[0163] For all retained candidate boxes belonging to the same category, non-maximum suppression (NMS) is performed based on their bounding box coordinates and confidence scores. Candidate boxes of the same category are sorted in descending order of confidence scores, and the highest-scoring box is selected in turn. Boxes with an intersection-union ratio (IoU) exceeding the NMS threshold are removed. Repeat this process for all other bounding boxes until all boxes have been processed. Non-maximum suppression threshold. This is used to determine which bounding boxes are considered duplicate detections, thus determining the density of the final detection result. The expression is: In the formula, The set of candidate boxes to be retained.

[0164] The final list of detection boxes is output as the detection result. Each final detection box includes the final category, bounding box coordinates, and confidence score.

[0165] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0167] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0168] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0169] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A large-model-driven method for detecting remote sensing targets in small samples, characterized in that, include: Multi-scale feature extraction is performed on the remote sensing image to be detected to obtain a multi-scale feature map, and scene-level feature vectors are generated based on the multi-scale feature map; The scene-level feature vector is input into a pre-trained large model to generate scene text describing the current remote sensing image and category text generated based on the scene text. The scene text and category text are then encoded into scene context instruction vectors. The pre-trained large model is used to generate text describing the scene based on the feature vector or scene text. Weights are calculated using scene context instruction vectors, and multi-scale feature maps are weighted and fused to obtain feature enhancement vectors. The feature enhancement vector is input into the detection head of the pre-trained few-sample remote sensing target detection model to generate multiple candidate boxes, and the score of each category in each candidate box is calculated separately. The category with the highest score in each candidate box that exceeds the threshold is retained as the final category. Non-maximum suppression is applied to all retained candidate boxes belonging to the same category to remove overlapping redundant detection boxes. Output the final list of detection boxes as the detection result. Each final detection box includes the final category and bounding box coordinates.

2. The large-model-driven small-sample remote sensing target detection method according to claim 1, characterized in that, The process involves multi-scale feature extraction of the remote sensing image to be detected to obtain a multi-scale feature map, and generating a scene-level feature vector based on the multi-scale feature map, including: A multi-scale convolutional feature extraction network is used to extract multi-scale features from the remote sensing image to be detected, resulting in a multi-scale feature map composed of feature maps of different resolutions. The multi-scale feature maps are concatenated to obtain the fused feature tensor. Global pooling is performed on the fused feature tensor to obtain scene-level feature vectors.

3. The large-model-driven small-sample remote sensing target detection method according to claim 1, characterized in that, The process involves inputting scene-level feature vectors into a pre-trained large model to generate scene text describing the current remote sensing image and category text expanded from the scene text. The scene text and category text are then encoded into scene context instruction vectors, including: A large-scale LLM model is pre-trained using scene-level feature vectors s as conditional input to generate scene text T describing the overall semantics of the current remote sensing image. ctx ; Based on scene text T ctx The pre-trained large model LLM is guided to expand each category in the pre-defined category vocabulary C to generate the corresponding category text T. cls ; The scene text T ctx and category text T cls After being structurally spliced ​​according to the preset prompt template, the complete text is formed. The complete text is then input into the pre-trained text embedding module, and after text encoding and mapping operations, it is projected onto the shared semantic space where the multi-scale feature map is located to obtain the context instruction vector g.

4. The large-model-driven small-sample remote sensing target detection method according to claim 3, characterized in that, The step of calculating weights through scene context instruction vectors and then weighting and fusing multi-scale feature maps to obtain a feature enhancement vector includes: The scene context instruction vector g is used to calculate the channel weight w through the projection matrix W. ， In the formula, The preset projection matrix; Use the Sigmoid activation function; The channel weights w include the channel weights of features at each scale. The channel weights of features at each scale are weighted and fused onto the multi-scale feature maps to obtain the feature enhancement vector F. g .

5. The large-model-driven small-sample remote sensing target detection method according to claim 1, characterized in that, Training a small-sample remote sensing target detection model includes: Repeat the following steps until the overall loss function converges or the maximum number of training epochs is reached: A small sample detection task is performed, which includes a query image with full annotations and K support samples independently sampled for each category in a predefined category vocabulary C; the full annotation of the query image includes the true categories of all targets to be detected and their corresponding bounding box coordinates; The query image in the current training round is subjected to multi-scale feature extraction, scene text and category text generation and encoding, and weighted fusion of multi-scale feature maps in sequence, resulting in the multi-scale feature map and scene context command vector g of the query image in the current training round. x and feature enhancement vector F gx ; Based on the scene context instruction vector g x Calculate the category prototypes of all categories in the pre-defined category vocabulary C for the current training round. ; Calculate the detection loss for the current training round based on the model prediction results. Prototype alignment loss Loss of consistency with instructions ; The feature enhancement vector is input into the detection head of the small sample remote sensing target detection model, and the output class probability distribution, bounding box position offset and target confidence are used as the model prediction results. The detection loss is calculated based on the difference between the model prediction results and the ground truth labels of the query image; the ground truth labels include class labels, bounding box positions and target labels. Loss detection in the current training round The calculation formula is: ， In the formula, The classification loss is calculated based on the predicted class probability distribution and the true class label. The bounding box regression loss is calculated based on the predicted bounding box position offset and the true bounding box position. The target confidence loss is calculated based on the predicted target confidence and the true target label, where 1 indicates the presence of a target and 0 indicates background. Prototype alignment loss in the current training round The calculation formula is: ， In the formula, This represents the number of positive samples in the query image during the current training round. Positive samples are candidate boxes whose Intersection over Union (IoU) with the ground truth bounding boxes is greater than a threshold. It is a d-dimensional feature vector extracted from the i-th positive sample candidate box in the current training round; For the current training round category The category prototype, It is the true class of the i-th positive candidate box in the current training round; The square of the Euclidean distance; Current training round instruction consistency loss The calculation formula is: ， In the formula, Calculate the KL divergence between the two distributions; softmax() is the normalization function that converts the vector into a probability distribution; H is a linear mapping matrix used to linearly map the feature enhancement vector Fgx of the current training round to the d'-dimensional semantic space; G is a linear projection matrix used to linearly map the scene context instruction vector gx of the current training round to the d'-dimensional semantic space. Loss is detected based on the current training round. Prototype alignment loss Loss of consistency with instructions The weighted calculation of the overall loss function L is given by the following formula: ， In the formula, and These are the preset tradeoff coefficients between prototype alignment loss and instruction consistency loss.

6. The large-model-driven small-sample remote sensing target detection method according to claim 5, characterized in that, The instruction vector based on the scene context g x Calculate the category prototypes of all categories in the pre-defined category vocabulary C for the current training round. ,include: The supporting sample subset S of category l in the preset category vocabulary C C,l It includes K samples: k is the supporting sample subset S C,l The index of the sample, ranging from 1 to K; This represents the feature vector of the k-th supporting sample; Repeatedly execute the following category prototype calculation process for a single category in the current training round to obtain the category prototypes for all categories in the preset category vocabulary C for the current training round. : Calculate the class prototype of class l in the current training round. ,include: Input all K support samples of category l into the feature extraction network. The feature representation of the supporting sample of category l is obtained. The calculation formula is: ， In the formula, T represents the transpose operation; F SUP It is a K×d matrix, where each row is a d-dimensional feature vector of a sample; It is a matrix The k-th row; Based on the scene context instruction vector g of the current training round x and the feature representation of supporting samples Calculate the importance of each supporting sample in category l, and the importance weight of the supporting samples in category l. The calculation formula is: ， In the formula, The importance weight of category l supporting samples Includes K elements; For the scene context instruction vector g x The vector obtained by linear projection onto the d-dimensional feature space. for The transpose of the function; softmax is the normalization function, which transforms the sample to obtain the importance weights of class l supporting the sample. ,and ; Weight the importance of the sample supporting category l. After weighted summation of the feature vectors of the corresponding supporting samples and normalization, the class prototype of class l in the current training epoch is obtained. The calculation formula is: ， In the formula, This represents the L2 norm operation.

7. The large-model-driven small-sample remote sensing target detection method according to claim 1, characterized in that, The calculation of the score for each category in each candidate box includes: Adjust the nonmaximum suppression threshold based on the scene context instruction vector g. Category confidence threshold Matching temperature parameters : Non-maximum suppression threshold The calculation formula is: ， Confidence threshold The calculation formula is: ， Matching temperature parameters The calculation formula is: ， In the formula, Use the Sigmoid activation function; (·) is a smooth positive value mapping function; , and These are the preset linear mapping matrices. , and Transpose of; , and The preset linear transformation bias; The formula for calculating the score of candidate box m for category l is: ， In the formula, m is the candidate box index, and the total number of candidate boxes output by the detection head is M; l is the category index, and the total number of categories in the preset category vocabulary C is C'; and These are the preset weighting coefficients for the fusion of classification score and prototype similarity. ; Let m be the base class probability of class l of candidate box; For similarity function, ， In the formula, yes Transpose of; It is a d-dimensional feature vector extracted from the candidate box m; The final category prototype for the pre-computed category l; express and The dot product; after the training of the small-sample remote sensing target detection model is completed, the class prototypes of the same category in all training rounds are averaged to obtain the final class prototypes p for each category. c *, It is p c One of the elements in *.

8. The large-model-driven small-sample remote sensing target detection method according to claim 7, characterized in that, The category with the highest score in each candidate box that exceeds the threshold is retained as the final category. Non-maximum suppression is applied to all retained candidate boxes belonging to the same category to remove overlapping redundant detection boxes. Output the final list of detection boxes as the detection results. Each final detection box includes the final category and bounding box coordinates, including: The category score is greater than the confidence threshold. The candidate boxes are selected; among the remaining candidate boxes, the category with the highest score is taken as its final category. Obtain the highest score and corresponding category for each candidate box; retain the highest score that is greater than the confidence threshold. The candidate boxes are selected, and the category corresponding to the highest score in the selected candidate box is the final category. Non-maximum suppression is applied to all retained bounding boxes belonging to the same category to remove overlapping redundant detection boxes: For all retained candidate boxes belonging to the same category, non-maximum suppression (NMS) is performed based on their bounding box coordinates and confidence scores. Candidate boxes of the same category are sorted in descending order of confidence scores, and the highest-scoring box is selected in turn. Boxes with an intersection-union ratio (IoU) exceeding the NMS threshold are removed. Continue processing all other boxes until all boxes have been processed; The final list of detection boxes is output as the detection result. Each final detection box includes the final category, bounding box coordinates, and confidence score.

9. The large-model-driven small-sample remote sensing target detection method according to claim 5 or 6, characterized in that, Training a small-sample remote sensing target detection model also includes: Calculate the category prototypes of all categories in the pre-defined category vocabulary C for the current training round. Previously, the supporting samples of each category were augmented using the scene text of the current training round.

10. The large-model-driven small-sample remote sensing target detection method according to claim 1, characterized in that, The scene context instruction vector is cached and updated according to scene region or time window; when consecutively input remote sensing images belong to the same geographical region or the same time window, the cached scene context instruction vector g is reused. When the region changes or the time window switches, the scene context instruction vector is recalculated and updated.

Images