Small sample target detection method and system based on cross-modal knowledge fusion

Abstract

The invention discloses a small sample target detection method and system based on cross-modal knowledge fusion, and mainly solves the problems of scarce visual features and poor detection effect in small sample target detection in the prior art. The implementation scheme comprises the following steps: constructing a small sample target detection data set by using a remote sensing image data set, and dividing a training sample and a test sample; a more efficient baseline unit, a semantic feature extraction unit and a cross-modal semantic fusion unit are respectively established, the more efficient baseline unit and the semantic feature extraction unit are connected in parallel and then connected in series with a fusion unit, and a remote sensing image small sample target detection model based on cross-modal knowledge fusion is formed; training a remote sensing image small sample target detection model by using the training sample; and inputting a test sample into the trained small sample target detection model to obtain target positioning and classification. According to the method, the semantic features are introduced, the problem of scarcity of visual features in a small sample scene is effectively solved, the small sample target detection effect is improved, and the method can be used for environment monitoring, urban planning and disaster early warning.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and remote sensing image processing, and particularly relates to a small sample target detection method and system, which can be used in disaster monitoring and emergency response, environmental and ecological monitoring, military and security detection and the like. BACKGROUND

[0002] Remote sensing image target detection is one of the important research contents in the field of remote sensing, and is widely used in land planning, disaster monitoring, military reconnaissance and the like. The purpose of remote sensing image target detection is to determine whether a target of interest exists in a remote sensing image and to determine the position of the target. The labeling process of remote sensing images is intensive, time-consuming and costly, and especially in large-size and complex background remote sensing images, the difficulty of manual labeling is further increased, which brings serious challenges to remote sensing image target detection, resulting in a decline in detection performance. In this regard, small sample target detection FSOD has gradually become an important research direction in the field of remote sensing image processing.

[0003] Existing small sample target detection methods mainly rely on visual features, but under the condition of sample scarcity, the model is difficult to learn sufficient feature representation, leading to classification confusion and performance decline. Although traditional methods such as TFA perform well in natural images, their performance is significantly reduced in remote sensing images due to problems such as variable target direction and large scale difference.

[0004] The patent document with the application number CN202211642380.X discloses a neural network target detection method guided by cross-modal knowledge, which includes the following implementation schemes: constructing a target detection model; obtaining a set of labeled sample images; inputting the set of sample images into the target detection model for training to obtain a trained target detection model. This method introduces text modal knowledge on the basis of existing neural network small sample target detection methods, solves the problem of insufficient class generalization information in small sample image data, and improves the performance of neural network small sample target detection. However, since it ignores the problem of text modal overfitting, the model is prone to fall into a local optimum after multiple rounds of training.

[0005] The patent document with the application number CN202311448952.5 discloses a MultiFNet architecture method based on a multi-modal target detection architecture. It includes: first, using the ResNet50-FPN algorithm to extract the features of the candidate targets from the input image, and using the BERT-BiGRU algorithm to extract the text features. Then, through the collaborative double attention module, the image embedding and semantic embedding are spliced to generate an image and text feature sequence of a complete sample, which is then sent to the Transformer module for further processing. This method uses advanced target detection and semantic extraction technology based on ResNet50-FPN, BERT-BiGRU, collaborative double attention module and Transformer to improve the accuracy of target detection. However, due to the use of only the collaborative double attention module for image embedding and semantic embedding direct splicing fusion method, the domain difference between the text modal and the visual modal is ignored, which easily introduces a large amount of noise and affects the final detection effect.

[0006] At the same time, the above two inventions are small sample target detection algorithms designed for natural image characteristics, and they are not improved for remote sensing image characteristics, so they are difficult to be directly applied to remote sensing image small sample target detection. SUMMARY

[0007] The present application aims to overcome the shortcomings of the prior art and provides a small sample target detection method and system based on cross-modal knowledge fusion to reduce the noise caused by the domain difference between the text modal and the visual modal, reduce the overfitting of text knowledge, and improve the relevance to remote sensing images.

[0008] To achieve the above purpose, the technical scheme of the present application includes:

[0009] 1. A small sample target detection method based on cross-modal knowledge fusion, characterized in that it comprises:

[0010] (1) Constructing a small sample target detection dataset using a remote sensing image dataset, and dividing to obtain training samples and test samples;

[0011] (2) Constructing a remote sensing image small sample target detection model based on cross-modal knowledge fusion:

[0012] Establish a more efficient baseline unit composed of a backbone network, a feature pyramid network FPN, a region proposal network RPN, a region of interest extraction network ROI and a prediction head cascade, which is used to extract visual features of remote sensing images;

[0013] Establish a semantic feature extraction unit including a gated recurrent unit GRU and a bidirectional encoder network BERT, which is used to extract discriminative semantic feature representations and effectively suppress model overfitting phenomenon;

[0014] The cross-modal semantic fusion unit including a feedforward neural network and a cross-attention mechanism is established, which is used for key-value mapping matching of visual features and features of each category, and weighted fusion;

[0015] The more efficient baseline unit is connected in parallel with the semantic feature extraction unit, and then connected in series with the fusion unit, to form a remote sensing image small sample target detection model based on cross-modal knowledge fusion.

[0016] (3) The remote sensing image small sample target detection model is trained by using training samples;

[0017] (4) The test sample is input into the trained small sample target detection model, and target positioning and classification are obtained

[0018] Further, the cross-modal semantic fusion unit established in the (2) has the following structures and functions of each part:

[0019] The feedforward neural network includes a multi-layer dimension reduction full connection structure, which realizes cross-modal alignment of visual and text features through nonlinear transformation, and maps the aligned cross-modal to a shared feature space;

[0020] The cross-attention mechanism first maps the semantic features and the visual features into key-value pairs, then calculates the similarity matrix of the two, and finally selects the top 3 semantic features with the highest similarity for weighted fusion.

[0021] 2. A small sample target detection system based on cross-modal knowledge fusion, comprising:

[0022] A more efficient baseline module is used for extracting visual features of remote sensing images.

[0023] A semantic feature extraction module is used for extracting discriminative semantic feature representation while effectively suppressing model overfitting phenomenon.

[0024] A cross-modal semantic fusion module is used for key-value mapping matching of visual features and features of each category, and weighted fusion.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] Firstly, the present application uses a more efficient baseline unit, only freezes the backbone network, and fine-tunes the feature pyramid network FPN, the region proposal network RPN, the region of interest extraction network ROI and the prediction head, so that the spatial features of the remote sensing image can be captured more efficiently.

[0027] Secondly, the semantic feature extraction unit is used to effectively capture the long-term dependence in the text semantic information, and the random mask mechanism is used to reduce the overfitting phenomenon in the training process, so that the category text semantic features are more efficiently extracted.

[0028] Thirdly, the cross-modal semantic fusion unit is used to map the semantic features and the visual features into key-value pairs respectively, calculate the similarity matrix of the two and weighted fusion, so that the cross-modal alignment of the visual and text features is realized, the domain difference is reduced, the introduction of noise is reduced, and the semantic features and the visual features are more efficiently fused. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flowchart of the small sample target detection method based on cross-modal knowledge fusion of the present application;

[0030] Figure 2 is a remote sensing image small sample target detection model diagram constructed in the method of the present application:

[0031] Figure 3 is a more efficient baseline unit structure diagram of the present application;

[0032] Figure 4 is a semantic feature extraction unit structure diagram of the present application;

[0033] Figure 5 is a cross-modal semantic fusion unit structure diagram of the present application;

[0034] Figure 6 is a block diagram of the small sample target detection system based on cross-modal knowledge fusion of the present application;

[0035] Figure 7 is a visual comparison diagram of target detection on the DIOR dataset using the two-stage fine-tuning method TFA of the present application and the existing two-stage fine-tuning method TFA respectively;

[0036] Figure 8 is a visual comparison diagram of target detection on the NWPU-VHR-10.v2 dataset using the two-stage fine-tuning method TFA of the present application and the existing two-stage fine-tuning method TFA respectively. DETAILED DESCRIPTION

[0037] In order to enable the personnel in the technical field to better understand the present application scheme, the technical solutions in the present application implementation will be described clearly and completely below in combination with the drawings in the present application embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, other embodiments obtained by the personnel in the field without making creative labor should belong to the protection scope of the present application.

[0038] It should be noted that the step numbers in the specification and claims of the present application are only for the purpose of clearly describing the embodiments of the present application, facilitating understanding, and the order of the serial numbers is not limited.

[0039] Embodiment one, small sample target detection method based on cross-modal knowledge fusion

[0040] The present example is to detect targets in remote sensing images by constructing an end-to-end target detection network.

[0041] Referring to Figure 1 The implementation steps of the present example include the following:

[0042] Step 1, obtain training samples and test samples.

[0043] 1.1) Obtain DIOR dataset and NWPUVHR-10.v2 dataset from a public website;

[0044] 1.2) Randomly divide the DIOR dataset and the NWPU VHR-10.v2 dataset into new classes and base classes according to a preset ratio;

[0045] 1.3) Divide the base class into a base class training set and a base class test set according to a set ratio, and select K labeled samples for each new class as a new class training set, while reserving a number of samples for each new class as a new class test set.

[0046] Step 2, establish a more efficient baseline model unit.

[0047] Referring to Figure 3 This step includes:

[0048] 2.1) Construct a more efficient baseline unit composed of a backbone network, a feature pyramid network FPN, a region proposal network RPN, a region of interest extraction network ROI, and a prediction head:

[0049] The backbone network includes a plurality of residual convolution operations, which is used to extract useful information from the input picture, analyze the local and global features of the image step by step, and finally output feature maps of different levels;

[0050] The feature pyramid network FPN utilizes a top-down path and a lateral connection mechanism to combine high-order semantic information of deep features with spatial details of shallow features, realizing multi-scale feature aggregation;

[0051] The region proposal network RPN includes an anchor generator and a non-maximum suppression, which first generates candidate anchor boxes covering multiple scales and aspect ratios through the anchor generator, and then through the non-maximum suppression, finally retains the most representative high-quality proposal region;

[0052] The interest region extraction network ROI includes pooling and alignment operations, which are used to unify the features of all candidate regions into the same size, facilitating subsequent classification and positioning.

[0053] The prediction head includes a classification branch and a regression branch, which are used to predict the category probability distribution of the target and the fine adjustment of the bounding box, respectively.

[0054] 2.2) Only freeze the backbone network, and fine-tune the feature pyramid network FPN, the region proposal network RPN, the interest region extraction network ROI, and the prediction head, to obtain the convolution feature map C3 from the backbone network.

[0055] Step 3, establish a semantic feature extraction unit.

[0056] Referring to Figure 4 , this step includes:

[0057] 3.1) Use the CLIP network to extract the word vector f text_word of each category in each category description text text.

[0058] f text_word = clip(text);

[0059] 3.2) Construct a text modal knowledge extraction unit composed of GRU and BERT network, which is used to further extract the word vector f text_word extracted by CLIP, that is, first apply word-level mask M1 to f text_word , then input the masked semantic features into the first GRU layer to capture the long-term dependency relationship in the text semantic information; then integrate and average the word vectors through the BERT m network to obtain the vector-level semantic features; then apply the vector-level mask M2 to the vector-level semantic features for random masking, and finally input the second GRU layer to obtain the semantic features f text .

[0060] f text = GRU(M2(BERT m (GRU(M1(f text_word )))))。

[0061] Step 4, establish a cross-modal semantic fusion unit.

[0062] Referring to Figure 5 , this step includes:

[0063] In view of the domain difference between visual features and semantic features, a cross-modal semantic fusion unit including feature mapping, feature matching, and feature fusion is constructed.

[0064] The feature map is firstly generated by using the feedforward neural network FFN to map the visual features f vis and semantic features f text Perform key mapping to obtain visual feature key mapping f vis_1 and semantic feature key mapping f text_1 :

[0065] f text_1 =FFN(f text )

[0066] f vis_1 =FFN(f vis )

[0067] The feature matching is based on the visual feature key mapping f vis_1 and semantic feature key mapping f text_1 , first calculate the similarity matrix between visual features and semantic features: and similarity matrix Sorting to get the three most similar categories: top3 cls =top3(similarity matrix ), and then based on the top 3 most similar categories cls Get the similarity of the top 3 most similar semantic features from the similarity matrix: top3 value =similarity matrix [top3 cls ];

[0068] The feature fusion is based on the top 3 most similar categories of semantic feature similarity value and the semantic features of the top three most similar categories f text_1 [top3 cls ], perform weighted feature fusion to obtain weighted text features: f text_add =softmax(top3 value )*f text_1 [top3 cls ], and then weighted text feature f text_add With visual features f vis_1 Add to get the fusion feature: f fusion =f vis_1 +f text_add .

[0069] Step 5: Build a small sample target detection model for remote sensing images based on cross-modal knowledge fusion.

[0070] The higher-efficiency baseline model unit established in step 2 is connected in parallel with the semantic feature extraction unit established in step 3, and then connected in cascade with the cross-modal semantic fusion unit established in step 4, to form a remote sensing image small sample target detection model based on cross-modal knowledge fusion, as shown in Figure 2

[0071] Step 6, training the remote sensing image small sample target detection model based on cross-modal knowledge fusion.

[0072] In this embodiment, it is divided into two stages of base class training and new class fine-tuning, and the implementation includes:

[0073] 6.1) Base class training:

[0074] 6.1.1) Set the training parameters:

[0075] Set the maximum number of iterations to 36000, the initial learning rate to 0.01, the learning rate to 0.001 after 24000 iterations, the learning rate to 0.0001 after 32000 iterations; select the optimizer as the SGD optimizer, the weight decay as 0.0001, and the momentum value as 0.9;

[0076] 6.1.2) Select the loss function as the cross-entropy loss function;

[0077] 6.1.3) Take minimizing the loss function as the goal, calculate the loss value between the predicted result and the true label under the supervision of a large amount of data through forward propagation, calculate the gradient of the loss function to each layer parameter of the model by using the back propagation algorithm, and update the model parameters based on the gradient descent method until the maximum number of iterations is reached, to obtain an initial trained base class detection model with strong feature extraction capability.

[0078] 6.2) New class fine-tuning:

[0079] 6.2.1) Set the training parameters:

[0080] Set the maximum number of iterations to 8000, the initial learning rate to 0.005, the learning rate to 0.0005 after 5000 iterations, the learning rate to 0.00005 after 7000 iterations, the loss function as the cross-entropy loss function, the optimizer as the SGD optimizer, the weight decay as 0.0001, and the momentum value as 0.9.

[0081] 6.1.2) Select the loss function as the cross-entropy loss function;

[0082] ​6.2.2) Targeting at minimizing the loss function, the initial trained base class detection model parameters are migrated and loaded, and the backbone network is frozen. Under the supervision of a small amount of data, the loss value between the predicted result and the real label is calculated through forward propagation, the parameter gradient of the model part layer is calculated by using the back propagation algorithm, and the model part parameters are fine-tuned based on the gradient descent algorithm until the maximum iteration number is reached, and a fine-tuned model with new class detection capability is obtained.

[0083] Step 7: Input the test set obtained in step 1 into the fine-tuned model with new class detection capability after training to obtain the position of the target bounding box and the target class result.

[0084] Example Two: Small Sample Target Detection System Based on Cross-Modal Knowledge Fusion

[0085] Reference Figure 6 The present example includes: a more efficient baseline module 1, a semantic feature extraction module 2, and a cross-modal semantic fusion module 3, wherein the semantic feature extraction module 2 includes a gated recurrent unit (GRU) sub-module 21 and a bidirectional encoder network (BERT) sub-module 22; the cross-modal semantic fusion module 3 includes a feedforward neural network (FNN) sub-module 31 and a cross-attention sub-module 32. The working principle of the entire system is as follows:

[0086] The baseline module 1 extracts preliminary visual features through the depth residual structure of the backbone network, and then extracts multi-scale remote sensing image visual features through the scale mapping of the feature pyramid network, and outputs the remote sensing image visual features to the cross-modal semantic fusion module 3.

[0087] The semantic feature extraction module 2 is used to extract discriminative semantic feature representations in the set category text corpus, wherein the gated recurrent unit (GRU) sub-module 21 is used to capture the long-term dependency relationship in the semantic information of the given category text corpus to obtain preliminary extracted text features; the bidirectional encoder network (BERT) sub-module 22 is used to further process the text features preliminarily extracted by the GRU sub-module 21, and through the self-attention mechanism, the final semantic features are obtained by realizing the bidirectional encoding of the context information, and the semantic features are output to the cross-modal fusion module 3.

[0088] The cross-modal semantic fusion module 3 is used to perform key-value mapping matching and weighted fusion of the visual features and the semantic features, wherein the feedforward neural network (FNN) sub-module 31 is used to realize the cross-modal alignment of the visual and text features; the cross-attention sub-module 32 is used to perform weighted fusion of the aligned semantic features and visual features to obtain the final fusion features.

[0089] It should be noted that: the above function modules can be realized by software, hardware, firmware or any combination thereof, in whole or in part. When implemented by software, it can be realized in whole or in part in the form of program instruction product. Program instruction product includes one or a group of program instructions. When loading and executing program instructions on a computer, the flow or function is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Program instructions can be stored in a computer readable and writable storage medium, or transferred from one computer readable and writable storage medium to another computer readable and writable storage medium.

[0090] The direct coupling or communication connection between the modules shown or discussed in the embodiments can be realized by the indirect coupling or communication connection of some interfaces, devices or modules. The function modules and sub-modules in the embodiments can be dynamically in one processing component, or each module can be physically present alone, or two or more modules can be dynamically in one processing component. When the above dynamic components are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer readable and writable storage medium. The storage medium can be a memory, a magnetic disk or an optical disk, etc.

[0091] The effects of the present application can be further illustrated by the following simulation results:

[0092] I. Simulation conditions

[0093] 1. The model training environment and algorithm test environment of the simulation experiment of the present application are unified, the computing device used is NVIDIA GeForce GTX 3080 GPU, the operating system used is Ubuntu 20.04.7LST, the CUDA version is 11.1, the Python version is 3.8.12, and the Torch version is 1.8.0.

[0094] 2. The remote sensing image target detection data sets used are DIOR data set and NWPU VHR-10.v2 data set, respectively. The new class and base class of the two data sets are divided, and the new class training set, new class test set, base class training set and base class test set are divided. The base class training set and the base class test set are used for the base class training stage, and the new class training set and the new class test set are used for the new class fine-tuning stage.

[0095] 3. Three different new class and base class division methods are adopted for the DIOR data set, which are division 1, division 2 and division 3, in order to fully verify the effectiveness of the application. The new class and base class of the three different divisions are selected as shown in Table 1:

[0096] Table 1 DIOR data set base class and new class division

[0097]

[0098] Each division in Table 1 contains 15 base classes and 5 new classes, where the 15 base classes keep the original test set and training set division of the DIOR dataset; and the 5 new classes, each of which randomly selects K = 3, 5, 10, 20 target instances as a new class training set, and the rest as a new class test set.

[0099] 4. Two different new class and base class division methods are adopted for the NWPU VHR-10.v2 dataset, namely division 1 and division 2, to fully verify the effectiveness of the application. The new class and base class selection of the two different divisions are as follows:

[0100] Table 2 Division of base classes and new classes of NWPU VHR-10.v2 dataset

[0101]

[0102] Each division in Table 2 contains 7 base classes and 3 new classes, where the 7 base classes keep the original test set and training set division of the NWPU VHR-10.v2 dataset; and each of the 3 new classes randomly selects K = 3, 5, 10, 20 target instances as a new class training set, and the rest as a new class test set.

[0103] II. Simulation experiment content results

[0104] Simulation 1, under the above conditions, the application and the existing 7 small sample target detection methods FRW, MetaRCNN, FsDetView, P-CNN, TFA, DeRFCN, G-FSDet are used respectively to detect different sample quantities 3-shot, 5-shot, 10-shot, 20-shot on the dataset in Table 1, and the class average accuracy (mAP%) of the detection is compared, and the results are shown in Table 3.

[0105] Table 3 Comparison of class average accuracy (mAP%) of the application and other methods on the DIOR dataset

[0106]

[0107] The class average accuracy mAP is calculated from the precision P and the recall R.

[0108] The precision P measures the proportion of correctly predicted samples predicted as positive classes, and the recall R is defined as the proportion of all correctly predicted positive and negative samples in the total number of samples, and the calculation formulas of the two are as follows:

[0109]

[0110]

[0111] Wherein, TP represents correctly predicting positive class as positive class, FP represents wrongly predicting negative class as positive class, and FN represents wrongly predicting positive class as negative class;

[0112] The average accuracy AP is the integral value of the PR curve, and the calculation formula is as follows:

[0113]

[0114] The class average accuracy mAP is the average value of the average accuracy AP of each class, and the formula is as follows

[0115]

[0116] As can be seen from Table 4, the application achieves the best effect on three different divisions, for example, in the sample quantity 20-shot setting of division 1, the class average precision mAP of the application reaches 39.87, which is more than 10% higher than 35.37 of the most advanced small sample target detection algorithm DeRFCN of natural images, and more than 1 point higher than 38.83 of the most advanced small sample target detection algorithm G-FSDet in the field of remote sensing.

[0117] The application performs target detection simulation on the DIOR data set, and the visualization result is as shown in Figure 7 7(a) is a TFA simulation detection result graph, and 7(b) is a simulation detection result graph of the application. Figure 7 It can be seen that the missing detection rate of the application method is greatly reduced, and the detection accuracy is greatly improved.

[0118] Simulation 2, under the above conditions, the application and the existing seven small sample target detection methods FRW, MetaRCNN, FsDetView, P-CNN, TFA, DeRFCN, G-FSDet are used respectively to detect different sample quantities 3-shot, 5-shot, 10-shot, 20-shot on the data set divided in Table 2, and the class average accuracy (mAP%) of the detection is compared, and the result is shown in Table 5.

[0119] Table 5 Comparison of the application method and other methods on NWPUVHR-10.v2 data set (mAP%)

[0120]

[0121] As can be seen from Table 5, the application achieves the best effect in two different divisions, for example, in the sample quantity 10-shot setting of division 1, the class average precision mAP of the application reaches 84.4, which is more than 30% higher than the 62.95 of the natural image most advanced small sample target detection algorithm DeRFCN, and more than 17 points higher than the 71.82 of the remote sensing field most advanced small sample target detection algorithm G-FSDet.

[0122] The application performs target detection simulation on the NWPUVHR-10.v2 data set, and the visual results are as shown in Figure 8 8(a) is a TFA simulation detection visualization result graph, and 8(b) is an application simulation detection visualization result graph. Figure 8 As can be seen from the above, the missing detection rate of the application method is greatly reduced, and the detection accuracy is greatly improved.

[0123] The above simulation results show that the application successfully alleviates the problem of insufficient visual features in the small sample scene by introducing class semantic information, and effectively improves the effect of remote sensing image small sample target detection.

[0124] Appendix: The sources of the above existing methods are as follows:

[0125] FRW, Few-shot object detection via Feature Reweighting, published in International Conference on Computer Vision Workshop in 2019.

[0126] Meta RCNN, Meta R-CNN: Towards General Solver for Instance-level Low-shot Learning, published in International Conference on Computer Vision in 2019.

[0127] FsDetView, Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild, published in European Conference on Computer Vision in 2020.

[0128] P-CNN, Prototype-CNN for Few-Shot Object Detection in Remote Sensing Images, published in IEEE Transactions on Geoscience and Remote Sensing in 2022.

[0129] TFA, Frustratingly Simple Few-Shot Object Detection, published in International Conference on Machine Learning in 2020.

[0130] DeRFCN, DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection, published in International Conference on Computer Vision in 2021.

[0131] G-FSDet, Generalized few-shot object detection in remote sensing images, published in ISPRS Journal of Photogrammetry and Remote Sensing in 2023.

Claims

1. A small sample target detection method based on cross-modal knowledge fusion, characterized in that: include: (1) Use remote sensing image datasets to construct a small sample target detection dataset and divide it into training samples and test samples; (2) Constructing a small sample target detection model for remote sensing images based on cross-modal knowledge fusion: Establish a more efficient baseline unit consisting of a backbone network, a feature pyramid network FPN, a region proposal network RPN, a region of interest extraction network ROI and a prediction head cascade for extracting visual features of remote sensing images; A semantic feature extraction unit consisting of a gated neural network (GRU) and a bidirectional encoder network (BERT) is established to extract discriminative semantic feature representations while effectively suppressing model overfitting. A cross-modal semantic fusion unit consisting of a feedforward neural network and a cross-attention mechanism is established to perform key-value mapping matching between visual features and features of each category and perform weighted fusion. The unit of the more efficient baseline is connected in parallel with the semantic feature extraction unit, and then connected in series with the fusion unit to form a remote sensing image small sample target detection model based on cross-modal knowledge fusion; (3) Using training samples to train the small sample target detection model of remote sensing images; (4) Input the test sample into the trained small sample target detection model to obtain target positioning and classification.

2. The method according to claim 1, characterized in that In (1), a small sample target detection dataset is constructed using a remote sensing image dataset, and training samples and test samples are obtained by dividing the dataset into two groups. The implementation includes: 2a) Based on the DIOR dataset and the NWPU VHR-10.v2 dataset, the new classes and base classes are randomly divided according to the preset ratio; 2b) Based on the division of new classes and base classes, the base class is divided into a base class training set and a base class test set according to a preset ratio. K labeled samples are selected for each new class as the new class training set, and a number of samples are reserved for each new class as the new class test set.

3. The method according to claim 1, characterized in that The structure and function of each part of the more efficient baseline unit established in (2) are as follows" The backbone network includes multiple layers of residual convolution operations, which are used to extract useful information from the input image, gradually analyze the local and global features of the image, and finally output feature maps at different levels; The feature pyramid network (FPN) uses a top-down path and a lateral connection mechanism to combine the high-order semantic information of deep features with the spatial details of shallow features to achieve multi-scale feature aggregation; The RPN includes an anchor generator and non-maximum suppression. It first generates candidate anchor boxes covering multiple scales and aspect ratios through the anchor generator, and then uses non-maximum suppression to finally retain the most representative high-quality proposal regions. The region of interest extraction network ROI, including pooling and alignment operations, is used to unify the features of all candidate regions into the same size to facilitate subsequent classification and positioning; The prediction head includes a classification branch and a regression branch, which are used to predict the category probability distribution of the target and refine the bounding box respectively; These parts are connected in series.

4. The method according to claim 1, wherein The structure and function of each part of the semantic feature extraction unit established in (2) are as follows: The gated neural network GRU includes two core components, an update gate and a reset gate, which are cascaded together. It is used to selectively retain or forget historical information through a gating mechanism, effectively capturing long-term dependencies in text semantic information. The bidirectional encoder network BERT includes a multi-layer self-attention encoder stack structure, which implements bidirectional encoding of contextual information through a self-attention mechanism to obtain vector-level semantic features; The two networks GRU and BERT mentioned above are connected in series.

5. The method according to claim 1, wherein The structure and function of each part of the cross-modal semantic fusion unit established in (2) are as follows: The feedforward neural network includes a multi-layer dimensionality reduction fully connected structure, which realizes cross-modal alignment of visual and text features through nonlinear transformation and maps the aligned cross-modal features to a shared feature space; The cross-attention mechanism first maps semantic features and visual features into key-value pairs respectively, then calculates the similarity matrix between the two, and finally selects the three types of semantic features with the highest similarity for weighted fusion.

6. The method according to claim 1, characterized in that In (3), the training samples are used to train the small sample target detection model of remote sensing images, which is divided into two stages: base class training and new class fine-tuning, wherein: In the base class training phase, the model is initialized and trained using a base class dataset containing a large number of samples to obtain a pre-trained model with basic detection capabilities; In the new class fine-tuning stage, based on the pre-trained model, a new class dataset containing a small number of samples is used to perform transfer learning and parameter fine-tuning to obtain a final optimized model adapted to new class detection.

7. The method according to claim 6, characterized in that In the base class training phase, the model is initially trained using a base class dataset containing a large number of samples, which includes: Set the maximum number of iterations to 36,000, the initial learning rate to 0.01, the learning rate to 0.001 after 24,000 iterations, and the learning rate to 0.0001 after 32,000 iterations; The loss function is selected as the cross entropy loss function, the optimizer is selected as the SGD optimizer, the weight decay is 0.0001, and the momentum value is 0.9; With the goal of minimizing the loss function, under the supervision of a large amount of data, the loss value between the predicted result and the true label is calculated through forward propagation, and the gradient of the loss function with respect to the parameters of each layer of the model is calculated using the backpropagation algorithm. The model parameters are iteratively updated based on the gradient descent method until the maximum number of iterations is reached, thereby obtaining an initial trained base class detection model with strong feature extraction capabilities.

8. The method according to claim 6, characterized in that In the new class fine-tuning stage, based on the pre-trained model, a new class dataset containing a small number of samples is used to perform transfer learning and parameter fine-tuning, which includes: Set the maximum number of iterations to 8000, the initial learning rate to 0.005, the learning rate to 0.0005 after 5000 iterations, and the learning rate to 0.00005 after 7000 iterations. The loss function is the cross entropy loss function, the optimizer is the SGD optimizer, the weight decay is 0.0001, and the momentum value is 0.

9. With the goal of minimizing the loss function, the parameters of the base class detection model after initial training are migrated and loaded, and the backbone network is frozen. Under the supervision of a small amount of data, the loss value between the predicted result and the true label is calculated through forward propagation. The parameter gradient of some layers of the model is calculated using the backpropagation algorithm, and some model parameters are fine-tuned based on the gradient descent algorithm until the maximum number of iterations is reached, thereby obtaining a fine-tuned model with the ability to detect new classes.

9. A small sample target detection system based on cross-modal knowledge fusion, characterized in that: include: A more efficient baseline module for extracting visual features from remote sensing images; Semantic feature extraction module, used to extract discriminative semantic feature representations while effectively suppressing model overfitting; The cross-modal semantic fusion module is used to perform key-value mapping matching between visual features and features of each category and perform weighted fusion.

10. The system according to claim 9, characterized in that: The semantic feature extraction module includes a gated neural network GRU submodule for effectively capturing long-term dependencies in text semantic information and a bidirectional encoder network BERT submodule for bidirectional encoding of contextual information to obtain vector-level semantic features; The cross-modal semantic fusion module includes a feedforward neural network sub-module for achieving cross-modal alignment of visual and text features and mapping the aligned cross-modal features to a shared feature space, and a cross-attention sub-module for measuring the similarity between text semantics and visual features and performing weighted fusion.

Images