Method for detecting remote sensing image target based on meta-knowledge adaptive migration network

Abstract

A few-shot remote sensing image target detection method based on meta-knowledge adaptive migration network belongs to the field of target detection of deep learning. The present application aims at the problems that the existing few-shot target detection method is limited in the ability to obtain effective information from a small amount of samples, domain transfer, and high similarity target is easy to cause misclassification, and designs a new few-shot target detection method. It includes: a spatial frequency joint enhancement model is designed, the dual enhancement of the query image features is realized by fusing the information of the spatial and frequency domains; an adaptive re-weighted loss function is designed, the sensitivity of new class detection in the limited sample scene is enhanced by adaptively reshaping the standard cross-entropy loss of the new class; a similarity meta-loss function is designed, the normalized class similarity of the support image set is included into the meta-loss function, and the feature discriminability is enhanced by re-weighting the meta-loss. The present application is used for few-shot remote sensing image target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to a few-shot remote sensing image target detection method based on meta-knowledge adaptive migration network, and belongs to the field of target detection of deep learning. BACKGROUND

[0002] Target detection plays a crucial role in object recognition and scene interpretation in very high resolution (VHR) remote sensing images. Due to the progress of deep learning techniques, target detection methods have made significant progress in recent years. The effectiveness of these methods largely depends on the quality and quantity of data used during the training process. However, obtaining labeled data is often expensive, and even unattainable, which is particularly evident in remote sensing applications.

[0003] Few-shot learning (FSL) can alleviate this problem, which aims to solve problems in new tasks or new classes by learning knowledge from existing classes or tasks. This usually involves using a small number of samples as training data, and then using the information provided by these samples to make inferences and generalizations. In the past few years, few-shot learning has made great progress, especially in the field of few-shot classification. Compared with classification tasks, few-shot target detection is a more complex task because it requires accurate classification and localization of each region of interest (ROI). In addition, most existing few-shot target detection methods are developed based on natural images. However, the development of few-shot target detection algorithms for VHR remote sensing images still lags behind. Since there is a fundamental difference in the perspective they capture, models trained on natural images cannot be directly migrated to the few-shot target detection task of remote sensing images. High-resolution remote sensing images mainly provide an overhead view of geospatial objects, while natural scene images capture the outline of objects. Given the higher complexity and diversity presented in high-resolution remote sensing images, few-shot target detection on such images faces challenges such as large scale, complex structure, different perspectives, and class imbalance. Therefore, applying a target detector trained on natural scene images to the high-resolution remote sensing image target detection task often encounters challenges.

[0004] Currently, most few-shot object detectors still rely on traditional deep learning-based object detection frameworks such as Faster R-CNN and YOLO, which are known for their powerful feature extraction capabilities. However, in the few-shot learning scenario, these frameworks face inherent challenges, including class imbalance, high intra-class variance, and low inter-class distance. In addition, the scarcity of supervision signals can exacerbate these challenges, especially when dealing with low-density sampling data, which has issues such as large intra-class variance, small inter-class distance, and data bias. Specifically, training an object detector based solely on limited training data can lead to overfitting and poor generalization. However, when combined with a large amount of base class data, the highly imbalanced dataset can cause the detector to favor the base classes, making it difficult to accurately detect instances of new classes. Therefore, training a reliable few-shot detector that can extract high-quality candidate boxes from non-robust features to better detect new class objects is a challenging task.

[0005] Existing methods in few-shot object detection have problems such as domain shift between base classes and new classes, misclassification caused by high similarity of class features, and limited ability to extract effective information from a small number of samples.

[0006] To address the challenges of large scale, complex structure, diverse perspective, and class imbalance in few-shot remote sensing image object detection, researchers have proposed many improved methods, such as multi-scale feature learning and improved region candidate box generation. However, all these methods require retraining from scratch. SUMMARY

[0007] To address the problem of limited ability to extract effective information from a small number of samples and the problem of misclassification caused by high similarity of objects in existing few-shot object detection methods, the present application provides a few-shot remote sensing image object detection method based on meta-knowledge adaptive transfer network.

[0008] The few-shot remote sensing image object detection method based on meta-knowledge adaptive transfer network of the present application is realized based on an adaptive meta-knowledge transfer network detector composed of a general object detector and a spatial frequency joint enhancement module, and includes two training stages. After completing the training of the two training stages, a few-shot remote sensing object detection model is obtained;

[0009] The first stage: training a general object detector using a meta-learning framework, the general object detector including a backbone network and an output module, and using a base class dataset to complete the training; the base class dataset is composed of sample images;

[0010] The first stage training of the general object detector includes: obtaining a plurality of sufficient learning task data sets from the basic category data set, each learning task data set including a training set and an evaluation set; each training set including a respective instance object image of all basic category data, each instance object image being obtained by cropping from a sample image according to a label; and each evaluation set including one or more sample images;

[0011] The general object detector is iteratively trained and evaluated using the learning task data set, so that the backbone network and the output module learn meta-level knowledge based on the basic category data set and are capable of quickly learning and generalizing on new detection tasks; the meta-knowledge includes general learning strategies and feature representations;

[0012] In the second stage, a spatial frequency joint enhancement module is added between the backbone network and the output module to form an adaptive meta-knowledge transfer network detector using a meta-learning framework; a self-adaptive reweighted loss function is used as the classification loss function in the training process, and a similarity meta-loss function is used as the meta-loss;

[0013] The second stage training of the adaptive meta-knowledge transfer network detector includes: the image data categories in the second stage training data set for training include basic categories and novel categories; a plurality of few-shot learning task data sets are obtained from the second stage training data set; each few-shot learning task data set includes a few-shot support image set and a few-shot query image set; few-shot second stage training images are configured in the few-shot support image set and the few-shot query image set, respectively; the data types in the few-shot support image set include all basic categories and novel categories; and the few-shot second stage training data of the novel categories is included in the few-shot query image set;

[0014] The adaptive meta-knowledge transfer network detector is trained and evaluated using the few-shot learning task data set, so that the meta-level knowledge learned in the first stage training is quickly transferred to the novel category target detection task;

[0015] Each iteration training is implemented by using one few-shot learning task data set;

[0016] The process of each iteration training includes: extracting features of each support image in the few-shot support image set using the backbone network to obtain a plurality of channel support image feature maps, and then using the spatial frequency joint enhancement module to first compress the channel support image feature maps to obtain a support image single-channel feature map, and then using a power enhancement function to enhance the feature values of the support image single-channel feature map to obtain a support image single-channel feature enhancement map;

[0017] The feature of one query image in the few-sample query image set is extracted by using a backbone network to obtain a multi-channel query image feature map, and a spatial frequency joint enhancement module is used to first compress the channels of the multi-channel query image feature map to obtain a single-channel query image feature map, and then a power enhancement function is used to enhance the feature values of the single-channel query image feature map to obtain a single-channel query image feature enhancement map;

[0018] Each single-channel support image feature enhancement map is multiplied with the single-channel query image feature enhancement map in sequence to obtain a feature correlation matrix of each class, the feature correlation matrix is normalized by using a softmax function to obtain an attention weight of each class, the attention weights of all classes are added to obtain a fusion weight, and the fusion weight is activated by using a sigmoid function to obtain an activated fusion weight;

[0019] The activated fusion weight is multiplied with the multi-channel query image feature map to obtain a query image spatial domain enhancement feature map;

[0020] The single-channel query image feature map is subjected to a discrete Fourier transform to obtain a query image frequency domain map, and the query image frequency domain map is subjected to a high-pass filtering to obtain a query image frequency domain enhancement map; the query image frequency domain enhancement map is subjected to an inverse discrete Fourier transform to obtain a query image frequency domain enhancement feature map;

[0021] The multi-channel query image feature map, the query image spatial domain enhancement feature map and the query image frequency domain enhancement feature map are weighted and added to obtain a query image spatial and frequency dual enhancement feature map;

[0022] A region generation network is further used to determine a candidate frame on the query image spatial and frequency dual enhancement feature map to obtain a region of interest, and then the classification result and the regression result of the current query image are obtained;

[0023] The above process is repeated to obtain the classification result and the regression result of each query image in the few-sample query image set;

[0024] After a preset number of iterations or a preset termination condition is met in the second stage, a few-sample remote sensing target detection model is obtained; during the training process, the adaptive meta-knowledge transfer network detector adjusts the network parameters according to the adaptive reweighted loss function, the regression loss function and the similarity meta-loss function;

[0025] For a real-time detection task containing a new class target, the network parameters obtained in the second stage are used as the initial parameters of the few-sample remote sensing target detection model, and rapid learning is performed on the support image set of the new class target detection task to realize the detection of the basic class target and the new class target.

[0026] According to the few-sample remote sensing image target detection method based on the meta-knowledge adaptive transfer network, the activated fusion weight is represented as w(q′j s′ i ):

[0027]

[0028] q′ j is the single-channel feature enhanced map of the jth query image, s′ i is the single-channel feature enhanced map of the ith class of support image, and N is the total number of support image classes of the support image set;

[0029] s i ′=MAP 4 (s i ), (2)

[0030] q′ j =MAP 4 (q j ), (3)

[0031] where s i is the single-channel feature map of the ith class of support image, and q j is the single-channel feature map of the jth query image; MAP is global average pooling;

[0032] The spatial enhanced feature map of the jth query image is represented as S j :

[0033] S j =Q j ·w(q j ′,s i ′), (4)

[0034] where Q j is the jth multi-channel query image.

[0035] According to the few-shot remote sensing image target detection method based on meta-knowledge adaptive migration network of the application, the spatial and frequency enhanced feature map of the jth query image is represented as Q″ j :

[0036] Q″ j =α*S j +β*F′(x,y)+Q j , (5)

[0037] where F′(x,y) is the frequency enhanced feature map of the query image, and (x,y) represents the horizontal and vertical coordinate pair on the frequency enhanced feature map of the query image; where alpha is the weight of the spatial enhanced feature map of the query image, and beta is the weight of the frequency enhanced feature map of the query image.

[0038] According to the few-shot remote sensing image target detection method based on the meta-knowledge adaptive migration network, alpha=0.1, beta=0.01.

[0039] According to the few-shot remote sensing image target detection method based on the meta-knowledge adaptive migration network, the adaptive reweighted loss function is represented as

[0040]

[0041] In the formula, is the predicted value of the jth query image belonging to the ith category, y ij is the true value of the jth query image belonging to the ith category; n is the total number of query images, is the true value adaptive weight, is the predicted value adaptive weight.

[0042]

[0043]

[0044] In the formula, L cls is the cross-entropy loss function:

[0045]

[0046] According to the few-shot remote sensing image target detection method based on the meta-knowledge adaptive migration network, in the model training process, the network parameters are also adjusted according to the similarity meta-loss function

[0047]

[0048] In the formula, W i is the weight of the feature vector in the similarity measurement, v ij is the true probability of the jth query image belonging to the ith category in the similarity measurement, is the predicted probability of the jth query image belonging to the ith category in the similarity measurement.

[0049] According to the few-shot remote sensing image target detection method based on the meta-knowledge adaptive migration network, the general target detector adopts Meta R-CNN as a basic model.

[0050] The method provided by the application reduces the training cost, improves the migration efficiency, and improves the detection performance of the few-shot target detection model.

[0051] ​The method has higher precision classification results for detection tasks with domain offset between base classes and new classes and high similarity of class characteristics, and has a strong ability to obtain effective information from a small number of samples. The effective and rapid adaptive migration of meta-knowledge from the source domain to the target domain is realized, and the detection capability of the network for new class targets is improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a schematic diagram of a two-stage training data set of a network model in the meta-knowledge adaptive migration network-based few-shot remote sensing image target detection method of the present application;

[0053] Figure 2 is a model training and testing schematic diagram of the method of the present application;

[0054] Figure 3 is a second stage schematic diagram of model training in the method of the present application;

[0055] Figure 4 is a visualization schematic diagram of the adaptive reweighted loss function;

[0056] Figure 5 is a detection result diagram of target detection by a few-shot remote sensing target detection model obtained by using the method of the present application;

[0057] Figure 6 is a detection accuracy box plot comparison diagram of target detection by different methods; ours in the diagram represents the method of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0060] The present application will be further described below in combination with the drawings and specific embodiments, but is not limited by the present application.

[0061] Specific embodiment one, in combination Figures 1 to 3 As shown in the combination, the present application provides a meta-knowledge adaptive migration network-based few-shot remote sensing image target detection method, which is realized by an adaptive meta-knowledge migration network detector composed of a general target detector and a spatial frequency joint enhancement module, and includes two training stages. After the training of the two training stages is completed, a few-shot remote sensing target detection model is obtained.

[0062] The first stage: a general object detector is trained by using a meta-learning framework, the general object detector includes a backbone network and an output module, and the training is completed by using a basic class data set; the basic class data set is composed of sample images;

[0063] The first stage training of the general object detector includes: a plurality of sufficient learning task data sets are obtained from the basic class data set, each learning task data set includes a training set and an evaluation set; each training set includes one instance object image of all basic class data, and each instance object image is obtained by cropping from a sample image according to a label; each evaluation set includes one or more sample images; the general object detector is first learned as a meta-learner under the meta-learning framework. The training set of each learning task is displayed to the learner (the general object detector as a learner under the learning framework) one training set at a time. The work of the learner is to try to correctly classify and locate the evaluation set picture target in each learning task.

[0064] The general object detector is iteratively trained and evaluated by using the learning task data set, so that the backbone network and the output module learn the meta-level knowledge based on the basic class data set, and can quickly learn and generalize on a new detection task; the meta-knowledge includes a general learning strategy and a feature representation;

[0065] The second stage: a spatial frequency joint enhancement module is added between the backbone network and the output module by using the meta-learning framework to form an adaptive meta-knowledge transfer network detector; the classification loss function in the training process adopts a self-adaptive reweighted loss function, and the meta-loss adopts a similarity meta-loss function;

[0066] The second stage training of the adaptive meta-knowledge transfer network detector includes: the image data categories in the second stage training data set for training include basic classes and novel classes; a plurality of few-shot learning task data sets are obtained from the second stage training data set; each few-shot learning task data set includes a few-shot support image set and a few-shot query image set; a few-shot second stage training image is configured in the few-shot support image set and the few-shot query image set, respectively; the data types in the few-shot support image set include all basic classes and novel classes; the few-shot second stage training data of the novel class is included in the few-shot query image set;

[0067] The adaptive meta-knowledge transfer network detector is trained and evaluated by using the few-shot learning task data set, so that the meta-level knowledge learned in the first stage training is quickly transferred to the novel class target detection task;

[0068] Each iteration training is realized by using one few-shot learning task data set;

[0069] The process of each iteration training comprises: extracting the feature of each support image in the small sample support image set by using the backbone network to obtain a multi-channel support image feature map, and then using the spatial frequency joint enhancement module to first compress the multi-channel support image feature map to obtain a single-channel support image feature map, and then using a power enhancement function to enhance the feature value of the single-channel support image feature map to obtain a single-channel support image feature enhancement map;

[0070] extracting the feature of a query image in the small sample query image set by using the backbone network to obtain a multi-channel query image feature map, and then using the spatial frequency joint enhancement module to first compress the multi-channel query image feature map to obtain a single-channel query image feature map, and then using a power enhancement function to enhance the feature value of the single-channel query image feature map to obtain a single-channel query image feature enhancement map;

[0071] multiplying each single-channel support image feature enhancement map with the single-channel query image feature enhancement map in turn to obtain a feature correlation matrix of each class, normalizing the feature correlation matrix by using a softmax function to obtain an attention weight of each class, adding the attention weights of all classes to obtain a fusion weight, and then activating the fusion weight by using a sigmoid function to obtain an activated fusion weight;

[0072] multiplying the activated fusion weight with the multi-channel query image feature map to obtain a query image spatial domain enhancement feature map;

[0073] performing discrete Fourier transform on the single-channel query image feature map to obtain a query image frequency domain map, performing high-pass filtering on the query image frequency domain map to obtain a query image frequency domain enhancement map, and then performing inverse discrete Fourier transform on the query image frequency domain enhancement map to obtain a query image frequency domain enhancement feature map;

[0074] weighting and adding the multi-channel query image feature map, the query image spatial domain enhancement feature map and the query image frequency domain enhancement feature map to obtain a query image spatial-frequency dual enhancement feature map;

[0075] then using a region generation network to determine a bounding box on the query image spatial-frequency dual enhancement feature map to obtain a region of interest, and further obtaining a classification result and a regression result of the current query image;

[0076] repeating the above process to obtain a classification result and a regression result of each query image in the small sample query image set;

[0077] after a preset number of iterations or a preset termination condition is met in the second stage, obtaining a small sample remote sensing target detection model; during the training process, the adaptive meta-knowledge transfer network detector adjusts network parameters according to the adaptive reweighted loss function, the regression loss function and the similarity meta-loss function;

[0078] For real-time detection tasks containing new category targets, the network parameters obtained in the second stage are used as the initial parameters of the few-shot remote sensing target detection model, and rapid learning is performed on the support image set of the new category target detection task to realize the detection of the basic category targets and the new category targets in the detection task. The effectiveness of the adaptive meta-knowledge transfer network model is measured by the detection effect of the category targets, especially the new category targets.

[0079] The embodiment takes how to better apply the few-shot target detection task to ultra-high resolution remote sensing images as the starting point, and considers how to adaptively transfer the meta-knowledge learned in the first stage of training the basic detector to the second stage of learning for new categories, while alleviating the challenges of domain shift, category confusion and limited ability to obtain effective information from a small number of samples in the few-shot target detection task. Specifically, the embodiment designs a transfer network for the second stage. First, a spatial frequency joint enhancement module is designed. The module first enhances the spatial information of the query image feature map by fusing the feature information in the support image set to obtain a spatial domain enhanced feature map, and suppresses the low-frequency information in the query image feature map by performing frequency domain enhancement on the query image feature map to obtain a frequency domain enhanced feature map. Then, the frequency domain enhanced feature map and the spatial domain enhanced feature map are fused into the feature map of the query image to obtain the final spatial frequency joint enhanced feature map, which helps the region proposal network (RPN) to extract high-quality candidate boxes.

[0080] The embodiment adopts a conventional few-shot target detection framework, and the training strategy is divided into two stages: the first stage is the training stage of the basic detector. In this stage, a general target detection model is trained using an existing large-scale labeled data set. The basic detector learns the basic feature representation of target detection and the target classifier; the second stage is the transfer learning stage for new categories. In this stage, the model is trained using a small amount of labeled data (only a few samples per category) to transfer the learned knowledge from the basic categories to the new categories. The goal of this stage is to achieve target detection capability for novel categories through few-shot learning. Through the training of the two stages, the model can perform target detection in the few-shot case and effectively detect the targets of the basic categories and the new categories. This two-stage training strategy can solve the problems of sample scarcity and data imbalance in the few-shot target detection task and improve the performance of the model in the few-shot case. The training process is shown in Figure 1 .

[0081] Specifically, it is assumed that the data set contains two non-overlapping categories, referred to as "basic categories (C base ) and "novel categories (C novel )", which correspond to subsets "D base"and" "D novel ". In the base classes, each class has enough labeled samples, while in the new classes, each class has only a small number of labeled samples. By adopting a two-stage training strategy, a model is trained to be able to detect objects in both base classes and new classes in the test phase. In the first stage, a base detector is trained using D base data. The second stage aims to quickly transfer the knowledge learned from the base classes to the new classes. In this way, a few-shot object detection model can be created during testing, which can detect objects in all classes in the base classes and new classes.

[0082] The embodiment adopts a meta-learning strategy and selects Meta R-CNN as the baseline model, aiming to obtain meta-knowledge. As shown in Figure 2 , a K-shot task with N classes can be represented as T = {(s1,..., s N ), q j}, where s i (i = 1, 2,..., N) represents a support image set with K instances for each class, and q j (j = 1, 2,..., n) is a query image from the query set. The model f(q | s) is trained to recognize objects belonging to the class in s i . The embodiment does not use a general training strategy (such as small batch gradient descent for object detection), but uses sub-tasks during training, where each sub-task contains the above training process.

[0083] First, the embodiment designs a spatial frequency joint enhancement module to assist the feature extraction network to extract high-quality proposal boxes, as shown in Figure 3 . This module integrates information from the support image set into the query image set and suppresses low-frequency information in the query image. Therefore, the limited sample information can be fully utilized to achieve double enhancement of the query image features.

[0084] Specifically, first, the data information from the support image dataset is introduced into the query image to achieve spatial domain enhancement of the query image. To simplify the calculation, the channel dimensions of the support image feature map and the query image feature map are mean average pooled to obtain a support image single-channel feature map and a query image single-channel feature map. To further enhance the features of the main information and weaken the information of the insignificant features, a power enhancement function is applied to the obtained single-channel feature map to enhance the feature values, thereby improving the overall quality of the image. The formulas are shown in (2) and (3), where MAP(·) represents mean average pooling. Next, the correlation between the feature map of the query image and the feature map of each class in the support image set is calculated. In this step, the enhanced query image feature map q'j and support image set features s′ i are multiplied to obtain the feature correlation weight of each class. Then the generated feature weight is normalized using the softmax function to ensure that the attention weight of each class is 1. After obtaining the attention weight of each class, the feature weights of all classes are added, and the resulting fusion weight is activated by the sigmoid function. This step enhances the nonlinear expression ability of the module and ensures that only the most relevant information is retained. In this way, the activated fusion weight w(q′ j ,s′ i ) is obtained, as shown in formula 1. Finally, the activated fusion weight is multiplied back into the query image feature map to obtain the spatial domain enhanced feature map S j , thereby obtaining the enhanced information related to the support image set in the query image. This process highlights the pixels with high similarity and enhances the features of the query image.

[0085] Further, the activated fusion weight is represented as w(q′ j ,s′ i ):

[0086]

[0087] where q′ j is the single-channel feature enhancement map of the jth query image, s′ i is the single-channel feature enhancement map of the ith class support image, and N is the total number of support image categories in the support image set.

[0088] s i ′=MAP 4 (s i ), (2)

[0089] q′ j =MAP 4 (q j ), (3)

[0090] where s i is the single-channel feature map of the ith class support image, q j is the single-channel feature map of the jth query image, and MAP is the global average pooling.

[0091] The spatial domain enhanced feature map of the jth query image is represented as S j :

[0092] S j =Q j ·w(q j ′,s i ′), (4)

[0093] where Qj The jth multi-channel query image.

[0094] Secondly, to further enhance the feature map of the query image by exploring improvement from the perspective of frequency domain. First, the feature map is transformed to the frequency domain by using the discrete Fourier transform (DFT), obtaining the frequency domain feature map F(u, v). Then, the frequency domain is moved, and the direct current component (representing the low frequency part) is repositioned to the center of the frequency domain, obtaining the feature map f(u, v). After that, a threshold is set to suppress low frequency information and enhance high frequency information, obtaining the frequency domain enhanced map f'(u, v). The motivation of this method is that high frequency information usually contains more detailed and rich information of the feature map, while low frequency information tends to be more uniform and less informative. After the filtering process, the direct current component needs to be moved back to the original position before converting back to the spatial domain, obtaining the frequency domain feature map f"(u, v) before inverse Fourier transform. Then the frequency domain feature map is converted back to the spatial domain using the inverse Fourier transform, obtaining the frequency domain enhanced feature map F'(x, y). The enhanced feature map shows enhanced details, which is convenient for analysis and interpretation.

[0095] Finally, the enhanced feature maps of the two methods are weighted and superimposed to generate the final space-frequency joint enhanced feature. In order to better understand the process, the formula of the steps involved in the space-frequency double enhancement module is as follows:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] In this embodiment, the jth query image space-frequency double enhanced feature map is represented as Q" j :

[0102] Q" j = α * S j + β * F'(x, y) + Q j , (5)

[0103] In the formula, F'(x, y) is the query image frequency domain enhanced feature map, and (x, y) represents the horizontal and vertical coordinate pair on the query image frequency domain enhanced feature map; in the formula, α is the weight of the query image space domain enhanced feature map, and β is the weight of the query image frequency domain enhanced feature map.

[0104] In the above equation, the value of the transformed signal in the frequency domain is represented by F(u,v), where (x,y) represents its corresponding position on the feature map. Furthermore, W and H represent the width and height of the input feature map, respectively, while u and v represent the horizontal and vertical positions in the frequency domain, respectively. The equation... The term is a complex exponential function of the rotation factor, which captures the phase difference between adjacent sampling points, has an amplitude of 1, and its phase varies depending on the values ​​of u and v. Note that the symbol j represents the imaginary unit, defined as satisfying the equation j 2 =1. To shift the DC component (low-frequency part) to the center of the frequency domain, the image needs to be shifted horizontally and vertically by W / 2 and H / 2 pixels, respectively. The coordinates (crow,col) represent the center point, f′(u,v) represent the value corresponding to position (u,v) in the frequency domain after high-pass filtering. In this embodiment, the filtering radius is set to 3 pixels. After filtering, the frequency domain shift operation is performed through the function f″(u,v), and F′(x,y) represents the final frequency enhancement feature map.

[0105] As an example, experimental analysis showed that setting α and β to 0.1 and 0.01 respectively yielded the best detection results.

[0106] This implementation then introduces a novel loss function, the adaptive reweighted loss, as a more effective alternative to address overfitting and poor generalization in scenarios with few samples. This loss function adaptively focuses on new class domains and dynamically adjusts the weights based on the difference between the predicted and true values, thereby improving the model's adaptability and sensitivity in detecting new classes when samples are limited.

[0107] This implementation devises a function to reweight the ground truth and predicted values ​​of new target categories. The weights are adaptively adjusted based on the target category and detection performance; that is, only new target categories are reweighted, with greater weight given to new target categories with poor detection performance. The reweighted values ​​are then fed into the standard cross-entropy loss function to optimize the cross-entropy loss function in the case of few samples, thereby enhancing the adaptability and sensitivity to new target category detection tasks with limited samples. Finally, the pairwise similarity between category feature vectors is calculated, and the similarity calculation results are summed according to category and normalized to obtain the weight for each category. This weight is then multiplied by the corresponding feature vector to modify the category feature vector value, thereby increasing the inter-class distance and improving the ability to distinguish between highly similar features of different categories.

[0108] In this embodiment, the adaptive reweighting loss function is expressed as follows:

[0109]

[0110] In the formula yi,j is the predicted value of the jth query image belonging to the ith class, ij yi,j is the ground truth value of the jth query image belonging to the ith class; n is the total number of query images, wi,j is the ground truth adaptive weight, wi,j is the predicted adaptive weight;

[0111]

[0112]

[0113] where L is the cross-entropy loss function: cls

[0114]

[0115] Here, the cross-entropy loss function is optimized. The cross-entropy loss function penalizes the items that make incorrect predictions, and the amount of penalty is proportional to the difference between the predicted probability and the true probability. Specifically, the greater the difference between these probabilities, the higher the cross-entropy loss. Conversely, when the predicted probability is close to the true probability, the cross-entropy loss is small.

[0116] The cross-entropy loss function quantifies the difference between the true label and the predicted label, which helps to optimize the classification problem.

[0117] The present embodiment designs a re-weighting function to process the predicted value and the ground truth value of the new class target, as shown in equations 7 and 8. The traditional cross-entropy loss is modified with this re-weighting function to obtain the adaptive re-weighted loss of the present embodiment, as shown in equation 6. By replacing the original value with the re-weighted value, the performance of the model in detecting new class objects is improved.

[0118] Equation representing the final adaptive re-weighted loss function.

[0119] It is worth noting that during the training process in the second stage, there is a significant imbalance in the number of samples between the base classes and the new classes. This class imbalance can cause the cross-entropy loss function to prioritize classes with more samples, leading to an imbalance in the training process. To solve this problem, the adaptive re-weighted loss function is only applied to the new class target objects. By doing so, selective attention to new class targets is increased during the training process, resulting in a more balanced and fair learning process. This re-weighting operation helps to alleviate the effects of class imbalance and ensures that the model pays sufficient attention to the detection of new class targets, despite the limited number of samples available for these classes.

[0120] Figure 4 Equation (9) and equation (6) are three-dimensionally visualized to intuitively understand the effect of the modified loss function. Figure 4 ​The middle layer plot represents the loss function value of the standard cross-entropy loss, and the upper layer plot represents the value of the adaptive re-weighted loss function proposed in this application. By assigning higher weights to targets with larger prediction errors and lower weights to samples with smaller prediction errors, the sensitivity of the loss function to new class targets is enhanced. This means that the loss function becomes more sensitive to incorrect predictions, assigning higher penalties to larger prediction errors, resulting in larger loss values. Conversely, smaller prediction errors are associated with lower loss values.

[0121] Finally, class similarity is introduced in the meta-loss function to address the class confusion problem in the small sample scenario, thereby enhancing the model's ability to distinguish target classes and ultimately improving its detection accuracy.

[0122] Specifically, one of the challenges of meta-learning is to ensure that the model can effectively generalize to new tasks or datasets, especially when the available samples for each specific task or dataset are limited. Meta-Loss is designed to address this challenge. The basic idea behind Meta-Loss is to encourage the model to learn a set of different object attention vectors that can be used to identify different objects or classes in the input. However, in some cases, the similarity between class features can pose a challenge to accurate identification of target classes, especially when the samples for each class are limited. For example, when two classes exhibit similar features, it is difficult for the model to distinguish them well based on the available samples. Therefore, to address this problem, the present embodiment incorporates feature similarity into the meta-loss framework. By integrating feature similarity, the model's ability to distinguish target classes is enhanced, thereby improving its overall detection accuracy.

[0123] To achieve this goal, several steps need to be taken. First, the feature vectors of different classes are calculated after the backbone feature extraction process. These feature vectors encapsulate the unique features of each class and serve as the basis for class identification in the input data. Subsequently, the similarity between the feature vectors of different classes is calculated, resulting in a similarity matrix representing the degree of similarity between each class and every other class. To incorporate feature similarity into the meta-loss, the similarity values are summarized and converted into weights based on class statistics. Specifically, the similarity values of each class are summarized, and this information is used to assign higher weights to classes with larger similarity values. The basic principle behind this approach is that classes with high similarity values are more likely to be confused with each other. Therefore, our goal is to ensure that the model places more emphasis on distinguishing these classes with high feature similarity. The formula can be written as:

[0124]

[0125]

[0126] Here, sim xyto compute v x and v y between them, which falls in the range of 0 to 1. Specifically, v x refers to the feature vector of the x-th class in the support set, while v y represents the feature vector of the y-th class in the same group, where x ranges from 0 to N, and y ranges from 0 to x. N represents the total number of classes. In addition, a is an adjustment factor that fine-tunes the strength of the similarity value. Empirical results show that a value of 0.1 produces the best results. Finally, W x is the weight assigned to each feature vector, which is a key element in the calculation.

[0127] During the model training process, the network parameters are also adjusted according to the similarity meta-loss function

[0128]

[0129] where W i is the weight of the feature vector in the similarity measure, v ij is the true probability that the j-th query image belongs to the i-th class in the similarity measure, is the predicted probability that the j-th query image belongs to the i-th class in the similarity measure.

[0130] Both types of feature vectors come from the support images.

[0131] As an example, the general object detector in this embodiment uses Meta R-CNN as the base model implementation.

[0132] The beneficial effects of the present application are verified by experiments as follows:

[0133] The mean average precision (mAP) is used as the target detection standard to evaluate the detection performance. The superiority of the present method in the large-scale DIOR dataset is verified by experiments, considering different target structures and sizes in the dataset. The detection performance of the adaptive meta-knowledge transfer network of the present application is compared with several existing Few-Shot Object Detection (FSOD) methods, including Faster R-CNN, RepMet, FeatReweighting, TFA, Meta R-CNN, and Prototype-CNN. The comparison results are shown in Table 2. Table 1 lists three methods used for dataset splitting.

[0134] Table 1 Three different new class selection settings, with the remaining classes being base classes.

[0135]

[0136] Table 2​

[0137] The FSOD experiment was performed on the first benchmark / new class split, and mAP was used as the evaluation metric. The best results are shown in bold.

[0138]

[0139] The method was evaluated on the DIOR dataset with split setting using 3- / 5- / 10- / 20- / 30-shot (i.e., only 3 / 5 / 10 / 20 / 30 instances per class) settings. Table 2 shows the experimental results for the first data set split. It can be seen that although the accuracy on the base classes is not as good as the original method, the accuracy on the new classes has been substantially improved. The specific details of the performance improvement are as follows: the 3-shot detection accuracy is improved by 4 percentage points, and the 5-shot and 10-shot detection effects are improved by 1.9 percentage points. Similarly, the 20-shot result shows an increase of 2.6 percentage points, and the 30-shot result shows an increase of 0.9%. Similarly, the experimental results of the other two settings (split setting 2 and split setting 3) also show obvious enhancement. The results are shown in Tables 3 and 4.

[0140] Table 3:

[0141] The FSOD experiment was performed on the second benchmark / new class split, and MAP was used as the evaluation metric. The best results are shown in bold.

[0142]

[0143] Table 4

[0144] The FSOD experiment results of the third benchmark / new class split, using MAP as the evaluation metric, and the best results are shown in bold.

[0145]

[0146] The observation results of Tables 2, 3, and 4 show that the two-stage training strategy of fine-tuning the knowledge from base classes to new classes is essential for better detection performance on new classes. RepNet and FeatReweighting improve the detection accuracy of base classes and new classes, but the detection performance of new classes is still insufficient. RFA significantly enhances the detection performance of all classes, but the results are very unstable and vary greatly with different split methods. Meta R-CNN integrates meta-learning into the two-stage object detection / segmentation method and uses the RoI feature part of Faster / Mask R-CNN to solve the problem of identifying images with complex background and multiple instance objects. Prototype-CNN and the adaptive meta-knowledge transfer network of the present application are both built on the achievements of Meta R-CNN. Prototype-CNN proposes a P-G RPN module to obtain better prospect suggestion frames, while the adaptive meta-knowledge transfer network of the present application first introduces a spatial frequency joint enhancement module to better utilize the features of a small number of sample images, and secondly improves both the classification loss and the meta-loss, achieving better detection performance on new classes.

[0147] Ablation experiments were conducted to demonstrate the effectiveness of the three improvements in the method of the present application, namely the spatial frequency joint enhancement module, the adaptive re-weighting loss function, and the similarity meta-loss function. Various experiments were conducted using different instance number settings and different base class / new class division methods, and the results are listed in Table 5.

[0148] Table 5

[0149] Ablation study of three base / new splits. Use MAP as the evaluation measure and the best result, and display the best result in bold.

[0150]

[0151]

[0152] Analysis of the spatial frequency joint enhancement module: it can integrate support image set information into the query image in a direct, efficient and effective way. Secondly, it extracts the target and background information in the image by using a low-pass filter in the frequency domain of the query image. Experimental results show that the impact of deleting this module on target detection accuracy is the greatest in about 40% of the cases. The results show that the method of the present application can enhance the image in both the spatial domain and the frequency domain. In addition, the effectiveness of integrating fusion information into the spatial attention module to improve the detection accuracy of the query image in the few-shot target detection task is verified.

[0153] Adaptive re-weighted loss function analysis: The cross-entropy loss function is optimized to pay more attention to newly introduced classes that are not well recognized. Experimental results show that removing the improvement to the cross-entropy loss function has the greatest impact on the target detection accuracy in about 60% of the cases. Experiments confirm the effectiveness of the function design. In addition, it is also verified that the network architecture can enhance the attention to new classes through re-weighting, thereby improving the final detection accuracy.

[0154] Similarity meta-loss function analysis: Removing the improvement to the meta-loss function has a smaller impact than removing the improvement to the cross-entropy loss function and removing the spatial-temporal joint enhancement module, but its effectiveness also verifies the importance of incorporating the similarity between different class features into the meta-loss function calculation. This method improves the network's ability to distinguish different classes of targets in the case of few samples in a simple and practical solution, especially when the targets have high feature similarity. This can effectively alleviate the problem of class error classification. Specific embodiments:

[0156] The following multi-round experiments are conducted on a large-scale remote sensing image dataset to evaluate the effectiveness of the method of the present application.

[0157] The dataset includes 23,463 images and 190,288 instances of 20 target classes, with 5,862, 5,863, and 11,738 images in the training, validation, and test sets, respectively. The images in the DIOR dataset are 800x800 pixels in size, with a spatial resolution ranging from 0.5 to 30 meters. Five classes are randomly selected as novel classes. The dataset split uses three methods, as shown in Table 1.

[0158] The baseline method is Meta R-CNN, which uses a two-stage training strategy to address the overfitting problem. In the first stage, only objects in the base classes are considered. In the second stage, K-shot object instances from the novel classes and 3K object instances from the base classes are selected. Experiments are conducted with K values from 3 to 30 to test the detection performance of the method of the present application on the test set containing base and novel classes (C test = C base ∪C novel ).

[0159] The adaptation meta-knowledge transfer network of the present application is developed using Pytorch deep learning framework, and the experiment is performed using a single Nvidia GeForce GTX 1080Ti GPU. The baseline model uses Meta R-CNN with ResNet101 backbone network. The training is performed using SGD optimizer with learning rate of 0.01, momentum coefficient of 0.9 and weight decay of 1x10-4. A two-stage training process is adopted, the first stage only focuses on objects from base classes, and the second stage selects K-shot object instances from novel classes and 3K object instances from base classes for training. Experiments are performed with K values from 3 to 30. During the testing process, the features of each class averaged after the output features of the backbone network are directly loaded. In this way, the support image information can be integrated into the test picture, so as to achieve better detection results. The test experiment is performed on the DIOR dataset, and each experiment contains 20 average feature maps, including 15 base classes. The evaluation includes 11,738 images in the test set. The box plot of the detection performance of the new class is shown as Figure 6 shown. The solid line in the figure represents the median, and the dashed line represents the average. It can be easily found from the figure that the method of the present application is better than the prior art method, and has higher detection accuracy and relatively stable detection performance.

[0160] Finally, the results of the third base class / new class division method are shown. Figure 5 The model is visualized for 30-shot. The first row shows the detection performance of detecting new classes (from C1 to C5) in turn. The second row randomly shows the detection performance of base class objects. The detection effect proves the effectiveness of the method of the present application.

[0161] Although the present application is described herein with reference to particular embodiments, it is to be understood that these examples are merely illustrative of principles and applications of the present application. It should therefore be understood that numerous modifications can be made to the illustrative embodiments and that other arrangements can be devised without departing from the spirit and scope of the present application as defined by the appended claims. It should be understood that the features described in connection with one embodiment can be used in conjunction with other embodiments described herein. It should also be understood that features described in connection with separate embodiments can be used in other described embodiments.

Claims

1. A few-shot remote sensing image target detection method based on meta-knowledge adaptive migration network, characterized in that The adaptive meta-knowledge transfer network detector is composed of a general target detector and a spatial frequency joint enhancement module, and includes two training stages. The first stage is to train the general target detector using a meta-learning framework, and the general target detector includes a backbone network and an output module, and the training is completed using a basic class data set. The basic class data set is composed of sample images. The first stage training of the general target detector includes: obtaining a plurality of sufficient learning task data sets from the basic class data set, each learning task data set including a training set and an evaluation set; each training set includes one instance object image of all basic class data, and each instance object image is obtained by cropping from a sample image according to a label; and each evaluation set includes one or more sample images. The general target detector is iteratively trained and evaluated using the learning task data set, so that the backbone network and the output module learn meta-level knowledge based on the basic class data set and can quickly learn and generalize on new detection tasks; the meta-knowledge includes a general learning strategy and a feature representation; The second stage is to add a spatial frequency joint enhancement module between the backbone network and the output module to form an adaptive meta-knowledge transfer network detector using a meta-learning framework; and the classification loss function in the training process uses an adaptive reweighted loss function, and the meta-loss uses a similarity meta-loss function. The second stage training of the adaptive meta-knowledge transfer network detector includes: the image data categories in the second stage training data set for training include basic categories and novel categories; a plurality of few-shot learning task data sets are obtained from the second stage training data set; each few-shot learning task data set includes a few-shot support image set and a few-shot query image set; a few-shot second stage training image is configured in the few-shot support image set and the few-shot query image set, respectively; the data types in the few-shot support image set include all basic categories and novel categories; and the few-shot query image set includes few-shot second stage training data of novel categories; The adaptive meta-knowledge transfer network detector is trained and evaluated using the few-shot learning task data set, so that the meta-level knowledge learned in the first stage training is quickly transferred to the novel category target detection task; Each iteration training is implemented by a few-shot learning task data set. Each iteration training process includes: using the backbone network to extract features of each support image in the few-shot support image set to obtain a multi-channel support image feature map, and then using the spatial frequency joint enhancement module to first compress the multi-channel support image feature map to obtain a support image single-channel feature map, and then using a power enhancement function to enhance the feature values of the support image single-channel feature map to obtain a support image single-channel feature enhancement map. The feature of one query image in the few-sample query image set is extracted by using a backbone network to obtain a multi-channel query image feature map, and a spatial frequency joint enhancement module is used to first compress the channels of the multi-channel query image feature map to obtain a single-channel query image feature map, and then use a power enhancement function to enhance the feature values of the single-channel query image feature map to obtain a single-channel query image feature enhancement map. Each single-channel support image feature enhancement map is multiplied with the single-channel query image feature enhancement map in turn to obtain a feature correlation matrix of each class, and a softmax function is used to normalize the feature correlation matrix to obtain the attention weight of each class; then the attention weights of all classes are added to obtain a fusion weight, which is activated by a sigmoid function to obtain an activated fusion weight. The activated fusion weight is multiplied with the multi-channel query image feature map to obtain a query image spatial domain enhancement feature map. The single-channel query image feature map is subjected to a discrete Fourier transform to obtain a query image frequency domain map, and then a high-pass filter is used to obtain a query image frequency domain enhancement map; the query image frequency domain enhancement map is subjected to an inverse discrete Fourier transform to obtain a query image frequency domain enhancement feature map. The multi-channel query image feature map, the query image spatial domain enhancement feature map and the query image frequency domain enhancement feature map are weighted and added to obtain a query image spatial and frequency dual enhancement feature map. A region generation network is then used to determine a candidate box on the query image spatial and frequency dual enhancement feature map to obtain a region of interest, and then the classification result and the regression result of the current query image are obtained. The above process is repeated to obtain the classification result and the regression result of each query image in the few-sample query image set. After a preset number of iterations or a preset termination condition is met in the second stage, a few-sample remote sensing target detection model is obtained; during the training process, the adaptive meta-knowledge transfer network detector adjusts the network parameters according to the adaptive reweighted loss function, the regression loss function and the similarity meta-loss function. For real-time detection tasks containing new class targets, the network parameters obtained in the second stage are used as the initial parameters of the few-sample remote sensing target detection model, and rapid learning is performed on the support image set of the new class target detection task to realize the detection of the basic class targets and the new class targets.

2. The few-sample remote sensing image target detection method based on the meta-knowledge adaptive transfer network according to claim 1, characterized in that, The activation function is represented as f(q) j ,s′ i ) In the formula, q′ j is the single-channel feature enhancement map of the jth query image, s′ i is the single-channel feature enhancement map of the ith support image of the ith class, and N is the total number of support image classes of the support image set. s i = MAP 4 (s i ), (2) q' j = MAP 4 (q j ), (3) where s i is the single-channel feature map of the i-th class of support image, q j is the single-channel feature map of the j-th query image; MAP is the global average pooling. The jth query image spatial domain enhanced feature map is represented as S j : S j = Q j · w(q j ′,s i ′),(4) wherein Q j is the jth multi-channel query image.

3. The method of claim 2, wherein the meta-knowledge based adaptive migration network is a meta-knowledge based adaptive migration network for few-shot remote sensing image target detection. The jth query image spatial-frequency dual enhanced feature map is represented as Q" j : Q" j = a * S j + b * F'(x, y) + Q j , (5) where F'(x, y) is the query image frequency domain enhancement feature map, and (x, y) represents the horizontal and vertical coordinate pair on the query image frequency domain enhancement feature map; where α is the weight of the query image spatial domain enhancement feature map, and β is the weight of the query image frequency domain enhancement feature map.

4. The method of claim 3, wherein the meta-knowledge based adaptive migration network is a meta-knowledge based adaptive migration network for few-shot remote sensing image target detection. α＝0.1，β＝0.01。 5. The method of claim 1, wherein the meta-knowledge based adaptive migration network is a meta-knowledge based adaptive migration network for few-shot remote sensing image target detection. The adaptive reweighted loss function is represented as In the formula is the predicted value of the jth query image belonging to the ith category, y ij is the true value of the jth query image belonging to the ith category; n is the total number of query images, is the true value adaptive weight, is the predicted value adaptive weight; where L cls is the cross-entropy loss function:

6. The method of claim 5, wherein the meta-knowledge based adaptive migration network is a meta-knowledge based adaptive migration network for few-shot remote sensing image target detection. In the model training process, the similarity meta-loss function is also used Adjust the network parameters: where W i is the weight of the feature vector in the similarity measure, v ij is the true probability that the jth query image belongs to the ith class in the similarity measure, is the predicted probability that the jth query image belongs to the ith class in the similarity measure.

7. The method of claim 6, wherein the meta-knowledge based adaptive migration network is a meta-knowledge based adaptive migration network for few-shot remote sensing image target detection. The general target detector uses Meta R-CNN as the basic model.

Images