An order-aware incremental instance detection method for remote sensing images

Abstract

The present invention discloses an order-aware remote sensing image instance incremental detection method, which obtains remote sensing image instances; designs an order score calculation function to evaluate the uncertainty and inaccuracy of the inference results of the target detection model according to whether there are real labels, and measures the incremental learning value in the newly added instances and the incremental learning value in the historical instances; uses a weighted training loss based on the order score while fine-tuning the target detection model to balance the contribution of training data with different values. The present invention can continuously learn streaming data with different values, pay full attention to data differences; adaptively sort newly added instances and historical instances, and provide a unified sample weighting direction for model training.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing images, and in particular relates to an order-aware remote sensing image instance increment detection method. Background Art

[0002] As one of the important tasks of remote sensing image interpretation, the target detection task of remote sensing images requires detecting various targets describing surface objects on remote sensing images to achieve the purpose of classifying and accurately locating the targets. The categories of interest in the target detection task of remote sensing images are mainly aircraft, buildings, vehicles, ships, etc., which plays a very critical role in application fields such as object tracking and monitoring, activity detection, scene analysis, urban planning and military research.

[0003] The object detection algorithm based on deep learning automatically learns low-level features such as color, edge, texture, contour and high-level features of macroscopic description semantics through convolutional neural network. Its performance in remote sensing image object detection tasks has long surpassed the traditional detection algorithm of manually designed features. However, it is worth noting that an important setting of deep learning method is that the training data is complete and contains rich enough features.

[0004] In fact, the real world is changing dynamically, and new data will continue to emerge over time. New data comes from two aspects: 1) new instance data of existing categories, and 2) new category data. The categories of interest in remote sensing image target detection tasks are relatively fixed, while the samples of similar targets provided by remote sensing satellites are constantly increasing, constantly enriching the diversity of targets. The diversity of remote sensing image targets, specifically, refers to the diversity in color, size, resolution, background complexity, etc., which is caused by many factors, including imaging factors such as imaging conditions and imaging quality, environmental factors such as weather, season, and geographical location, and semantic factors such as shape and visual appearance.

[0005] In the target detection algorithm of remote sensing images, the deep learning method based on convolutional neural network is increasingly widely used due to its high performance. The good performance of this method relies on a complete dataset and obtains good detection capabilities by learning the rich features of remote sensing images. However, in reality, new remote sensing image data will appear continuously when the categories are fixed. In such an incremental instance scenario, the newly appeared instances contain new features, which require the model to continue learning. At the same time, there are also historical instances that have been learned, which require the model to review to avoid catastrophic forgetting. Since there is already a learning foundation, the instance data has a difference in incremental learning value, which needs to be paid attention to both the data itself and the model learning strategy, but the existing methods either do not pay attention to the difference in data or have a single perspective.

[0006] The methods of remote sensing image target detection are mainly divided into traditional machine learning methods and deep learning-based methods according to the feature extraction method. Traditional machine learning methods need to rely on the designer's prior knowledge to manually design corresponding features for different categories of detection targets, and use HOG features for ship, building and vehicle detection. In addition, there are works that use binary gradient amplitude (BING) features for aircraft detection and local binary patterns (LBP) for ship detection. For visual images, more valuable information is hierarchical, and the features manually extracted by traditional methods have poor robustness. Since 2012, deep neural networks have gained an overwhelming advantage in natural image classification tasks by relying on their powerful feature extraction capabilities. They have been widely and successfully applied in tasks such as semantic segmentation, target detection, speech recognition, and medical image analysis. Applying them to target detection in remote sensing images can well solve the shortcomings of traditional methods.

[0007] There are two development directions for deep learning-based object detection methods depending on whether candidate boxes are generated. The region proposal-based method first generates candidate boxes through a dedicated module, finds foregrounds and adjusts bounding boxes, and then further classifies and adjusts bounding boxes based on previously generated candidate boxes. In R-CNN, a selective search method is used to generate candidate regions, which are uniformly transformed to a fixed size and input into the CNN model to obtain feature vectors for object classification and bounding box positioning. However, many candidate regions in the R-CNN method overlap with each other, resulting in very slow training and prediction speeds. Compared with R-CNN, SPPnet and Fast R-CNN perform feature extraction on the entire image, but candidate regions need to be obtained in advance, which relies on external candidate region methods and is slow. To solve this problem, Faster R-CNN directly generates foreground candidate regions through a fully convolutional candidate region generation network (RPN), and then adds a Fast R-CNN object detector for candidate region classification and regression. RPN and Fast R-CNN detectors share the same convolutional layer, generating higher quality candidate regions than traditional candidate region generation methods, achieving true end-to-end. The features used by previous networks are often limited to the deep layer. Although the deep features have richer semantic information and are more conducive to object classification, the lack of spatial information is not conducive to object positioning. Therefore, FPN proposes a top-down network architecture with horizontal connections, which integrates multiple layers of features to make positioning more accurate.

[0008] The regression-based method uses only one network to complete target category classification and bounding box positioning at the same time. Compared with the method based on region proposal, it can better meet the real-time requirements in terms of computing speed. YOLO simplifies the entire target detection process by dividing the grid and improves the speed. SSD improves detection accuracy by introducing multi-scale features. YOLOv2 and YOLOv3 have improved detection accuracy while maintaining excellent speed through improvements in many aspects such as backbone networks and training strategies. RetinaNet uses feature pyramid networks and loss functions that focus on the imbalance problem of difficult samples to improve detection accuracy.

[0009] Many studies have conducted in-depth explorations on some problems existing in remote sensing images, including category imbalance, complex background, scale change of targets, directed box detection, small targets, etc., and improved the performance of remote sensing image target detection. However, current remote sensing image target detection methods often focus on static data in a closed world, only need to complete learning once, and cannot have the ability to incrementally learn when data continues to expand.

[0010] Some current incremental learning strategies generally solve the problem of catastrophic forgetting when the model encounters new categories, and do not pay attention to the instance increment scenarios in remote sensing image target detection tasks. Some methods in other fields, including active learning methods, evaluate the learning value of unlabeled samples through query algorithms, and can give priority to learning important new instances; methods that focus on difficult samples believe that difficult samples need to be paid attention to during the training process. Most of these existing methods focus on a single level and lack an effective integrated framework method.

[0011] Incremental learning is an effective progressive learning model for streaming data. It originated from cognitive neuroscience research on memory and forgetting. Incremental learning specifically refers to a learning model in which a learning system can continue to learn new knowledge after learning old knowledge. Since incremental learning itself is a relatively broad concept, it needs to be analyzed and studied based on specific scenarios. It can be mainly divided into three scenarios: instance incremental scenario, class incremental scenario, and instance and class incremental scenario. In the instance incremental scenario, the number of categories is fixed, and the data in each category will expand at each learning stage. In the class incremental scenario, the number of categories will change. The last one is a scenario in which the situations in the first two scenarios will occur.

[0012] Incremental learning needs to balance the relationship between new knowledge and old knowledge. It requires both plasticity to acquire new knowledge and stability to consolidate existing knowledge. This is called the dilemma of plasticity and stability. Current incremental learning mainly focuses on the catastrophic forgetting problem in terms of model stability in the category increment scenario. First, relevant research was carried out on the basic visual analysis task of image classification, which can be divided into regularization-based incremental learning and replay-based incremental learning. Regularization-based incremental learning learns new tasks without using old data, and protects the model's memory of old knowledge by constraining the loss function. Representative methods include the forgetfulness-free learning LwF algorithm and the plastic weight consolidation EWC algorithm. The former focuses on keeping the decision boundary as unchanged as possible, while the latter aims to update the internal representation. Replay-based incremental learning allows a portion of representative old data to be added to the new data to train new tasks together. The model reviews and consolidates old knowledge by contacting the old data again. Representative methods include the iCaRL method and the GEM method. The third idea can achieve a better balance between cost consumption and learning results. Therefore, the present invention also adopts this paradigm to overcome forgetting.

[0013] The target detection model is more complex than the image classification model. It not only requires semantic classification capabilities for targets, but also requires bounding box prediction. In order to solve the problem of catastrophic forgetting in target detection tasks, a target detection method for new category increments based on knowledge distillation was proposed. This method has dependencies on old categories and uses an additional selective search method to calculate candidate boxes. Later, some work made improvements on Faster R-CNN. In addition, there is also work to overcome forgetting in the category increment scenario through meta-learning. At present, target detection methods for incremental learning tasks lack the focus on the scenario of increasing instances in the field of remote sensing image target detection, and need more in-depth research on the model's ability to learn new instances and remember historical instances. Summary of the invention

[0014] In view of this, the present invention proposes an order-aware instance incremental detection framework, which includes instance rank and rank-aware incremental learning. In instance rank, a rank score calculation function is designed to evaluate the uncertainty and inaccuracy of the inference results of the original model according to whether there are true labels, and measure the incremental learning value in the newly added instances and the incremental learning value in the historical instances. In rank-aware incremental learning, the proposed rank-aware loss is used while fine-tuning, and the weighted training loss is based on the rank score to balance the contribution of training data with value differences. These two parts are closely connected to form a complete framework method. The present invention has carried out a number of experiments such as comparative experiments, ablation experiments, parameter sensitivity analysis, etc. on two large-scale remote sensing image target detection public datasets with rich categories, and verified the superiority and effectiveness of the method proposed in the present invention.

[0015] The present invention proposes an order-aware remote sensing image instance increment detection method, comprising the following steps:

[0016] Get remote sensing image instances;

[0017] According to whether there are true labels, the order score calculation function is designed to evaluate the uncertainty and inaccuracy of the inference results of the target detection model, and measure the incremental learning value in the newly added instances and the incremental learning value in the historical instances.

[0018] We use a rank-order score-weighted training loss while fine-tuning the object detection model to balance the contribution of training data with different values.

[0019] Furthermore, the uncertainty of a single sample is calculated as follows:

[0020]

[0021] Among them, img i is the i-th image, is the "top-1" confidence of the jth predicted target in image i, Transform(img i ) is to convert img i Perform image transformation operations without destroying the semantic information of the image. i} is to calculate a single image img i The uncertainty of is calculated as follows:

[0022]

[0023] Furthermore, the inaccuracy is calculated based on the difference between the predicted box and the true box. The specific calculation formula is as follows:

[0024] IA(img i )=1-mAP50(TL_img i ,PR_img i )

[0025] Among them, img i is the i-th image, TL(img i ) is img i The real label in PR(img i ) is img i The prediction results in , mAP50 is the average precision calculation method of the coco dataset, which is used here to calculate a single image img i Average precision at IoU threshold 0.5.

[0026] Furthermore, the rank-score is calculated comprehensively based on the uncertainty and inaccuracy of the prediction results. In order to weigh the weight of the two, the specific calculation formula of rank-score is as follows:

[0027] Rs(img i )=α*UC(img i )+β*IA(img i )

[0028] Where a and b are weight hyperparameters.

[0029] Furthermore, the calculation formula of Rank-aware loss is as follows:

[0030] Rank-aware loss(img i )=W(img i ,iter)*loss(img i )

[0031] Among them, loss(img i ) is img i The original training loss, W(img i ,iter) is img i The loss weight at iteration number iter.

[0032] Furthermore, the loss weight calculation formula is as follows:

[0033]

[0034]

[0035] Among them, max_iter is the maximum number of iterations, is the average of the initial scores of the incremental training set.

[0036] Furthermore, the target detection model is Faster R-CNN.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. For the remote sensing image object detection task, an order-aware instance incremental detection framework is proposed for the instance incremental scenario, which can continuously learn streaming data of varying values ​​and pay comprehensive attention to data differences.

[0039] 2. A rank score calculation method based on the uncertainty and inaccuracy of the prediction results is designed to adaptively sort new instances and historical instances, while providing a unified sample weighting direction for model training.

[0040] 3. Experiments were conducted on two widely used remote sensing image datasets. Compared with existing methods, the superiority of the proposed method was verified. The effectiveness of the method itself was verified through ablation experiments and hyperparameter analysis experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The framework diagram of the order-aware instance incremental detection method of the present invention;

[0042] Figure 2 An example diagram of the scoring results of the new instance in the DIOR dataset of the present invention;

[0043] Figure 3 Statistical graph of the number of instances on the DIOR dataset of the present invention;

[0044] Figure 4 Statistical graph of the number of instances on the DOTA dataset of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0046] The present invention is aimed at the target detection task of remote sensing images, and studies the instance increment scenario of adding data of the same category when the category is fixed. The data that has been learned in the initial scenario is called historical instance (old instance). The detection model that has currently learned the historical instance is the original model. The newly added instance (new instance) is the data that appears in the current instance increment scenario and has no label. It is hoped that the model can be updated and optimized according to the changes in the training data.

[0047] The classic learning paradigm of learning through static data is no longer applicable to the incremental instance scenario where the data changes. Currently, incremental learning has attracted much attention among the methods for dealing with streaming data. Specifically, incremental learning can continue to learn new knowledge based on existing learning, making learning continuous. Incremental scenarios have very specific scenarios in different fields and applications, with their own tasks and requirements, and appropriate incremental learning methods need to be designed accordingly.

[0048] In the remote sensing image target detection task, the data that has appeared in the instance increment scenario includes new instances and historical instances. Among the new instances, there are data containing new features and data with similar features to historical instances. There are differences in learning value between different new instances. If the original model can only access new instances, catastrophic forgetting will occur. The continued participation of historical instances in incremental learning can effectively slow down the forgetting phenomenon, but the value of different historical instances in consolidating previous knowledge is different, and the playback volume will be limited by storage resources and training resources, so joint training cannot be achieved. Under the incremental learning paradigm, the importance of instance data is ranked differently. It is necessary to examine the value of the data and give priority to learning. At the same time, based on the existing training, the training method of the model is adjusted according to the differences in the data, so that it can adapt to more diverse data sets after optimization.

[0049] The present invention designs an order-aware instance incremental detection framework, including instance rank and order-aware incremental learning. In instance rank, for unlabeled new instances and learned historical instances, the rank score is designed to automatically evaluate the value of the instance, and the incremental training set is constructed according to the score ranking. Rank-aware incremental learning is to design a rank-aware loss according to the rank score of the sample during the learning stage of the model to balance the contribution of samples of different values ​​to learning. The key to the success of the method lies in scoring and ranking the value of the data. For the new instance, since it is not manually labeled, it is necessary to automatically perceive the importance of the data according to the uncertainty of the model's prediction results of the data without manual visual interpretation. For the real labeled historical instances, according to the uncertainty of the prediction results and the inaccuracy of the real labels, the two properties are combined to find the instances that can maintain the original memory of the original model. The new instances and historical instances that have been sampled and labeled maintain the same dimensional information volume, and can be uniformly incrementally learned after being sorted according to the same rule.

[0050] The rank-aware instance-incremental detection framework (RAIIDF) proposed in this invention has the following structure: Figure 1 As shown in the figure, it includes instance rank at the data level and rank-aware incremental learning at the model level. In instance rank, a rank score calculation function is designed based on the data itself and the response to the original model. Sorting and sampling are performed through value scoring, so that new instances with more unknown features and historical instances that can simulate the previous data distribution can be learned first. Then, before model training, the samples in the incremental training set have a rank-score under a unified learning value assessment rule. The training process gives a reasonable weight to the training loss of the sample according to the rank-score.

[0051] Figure 1 In the RAIIDF structure diagram, the top two sub-figures respectively show the two parts of the present invention: instancerank and rank-aware incremental learning. In instance rank, for unlabeled new instances, the rank-score based on uncertainty is calculated, and the order is in the direction of high values. After annotation, the rank-score based on uncertainty and inaccuracy is calculated together with the historical instances, and the historical instances are scored in the direction of various Rs values. The incremental training data set is finally constructed using the two types of data. In rank-aware incremental learning, the loss function uses rank-aware loss when fine-tuning the model. The two bottom sub-figures respectively show the calculation methods of uncertainty and inaccuracy. Uncertainty comprehensively calculates the top1 confidence of the model's prediction results for the original image and the transformed image. The specific calculation formula is shown in Formula 1. Inaccuracy calculates the degree of conformity of the model to the predicted results and true labels of the image. The specific calculation formula is shown in Formula 3.

[0052] Instance rank

[0053] First, the rank-score is calculated to measure the incremental learning value of the sample, which serves as the basis for the sample rank. Since the two types of data, new instances and historical instances, in the instance increment scenario have different labeling states and incremental values: new instances are unlabeled, and historical instances have real labels; learning the former is to learn new knowledge, and learning the latter is to review old knowledge. There are certain differences in the ranks of new instances and historical instances. Therefore, the instance rank designed by the present invention includes two parts: new instances (New instance rank, NIR) and old instances (Old instance rank, OIR). The specific content will be described below.

[0054] The following describes in detail the instance rank method for both new instance data and historical instance data.

[0055] (1) New instance order

[0056] When the new instance is unlabeled, the value of the new instance is measured by the original model with learning experience. The more uncertain the original model's prediction result for the new instance, the more likely the new instance contains new knowledge. Therefore, the rank-score is calculated based on the uncertainty (UC) of the original model's prediction result for the new instance. The higher the rank-score, the higher the ranking in the value ranking.

[0057] The prediction results in the entire image are actually at the prediction box level. The "top-1" confidence of the prediction box is used to evaluate the uncertainty of a single prediction result in the image. The lower the "top-1" confidence, the less confident the model is about the result. In order to integrate the uncertainty at the image level, the average of the "top-1" confidences of all prediction boxes in the image is taken. In addition, since the information of the original image prediction result is relatively simple, we transform the original image and use the prediction results of the transformed image to enrich the information. The uncertainty of the images before and after the transformation is calculated respectively, and the difference between the prediction results before and after the transformation is calculated and compared. The greater the difference, the less confident the original model is about the image.

[0058] The uncertainty of a single sample is calculated as follows:

[0059]

[0060] Among them, img i is the i-th image, is the "top-1" confidence of the jth predicted target in image i, Transform(img i ) is to convert img iPerform image transformation operations without destroying the semantic information of the image. i} is to calculate a single image img i uncertainty.

[0061] The uncertainty calculation formula for a single image is as follows:

[0062]

[0063] (2) Old instance rank

[0064] When the original model learns new instances, simply leaning towards new knowledge will destroy the existing learning experience, so it is necessary to learn historical instances that can consolidate the existing learning experience. Since the original model fits the distribution of historical instances through learning, the historical instances with more diverse quality of prediction results restore the learning experience of the model and are more valuable to learn. Therefore, the rank-score of the historical instances is calculated based on the quality of the prediction results, and the ranking is in the direction of numerical diversification.

[0065] When the data is labeled, in addition to uncertainty, the accuracy of the prediction results also reflects the quality of the prediction results. Inaccuracy (IA) is calculated based on the difference between the predicted box and the true box. The specific calculation formula is as follows:

[0066] IA(img i )=1-mAP50(TL_img i ,PR_img i ) (3)

[0067] Among them, img i is the i-th image, TL(img i ) is img i True labels (real labels), PR (img i ) is img i The predicted results in mAP50 are the average precision calculation method of the coco dataset, which is used here to calculate the average precision of a single image img i Average precision at IoU threshold 0.5.

[0068] The rank-score is calculated based on the uncertainty and inaccuracy of the prediction results. In order to balance the weight of the two, the weight hyperparameters a and b are introduced. The specific calculation formula of rank-score is as follows:

[0069] Rs(img i )=α*UC(img i)+β*IA(img i ) (4)

[0070] While ranking old instances based on UC and IA rank-scores, unlabeled new instances are manually labeled and have real labels after being ranked based on UC rank-scores. The amount of information at this time is the same as that of old instances and is more complete. We add the IA indicator and recalculate the rank-score of the new instances.

[0071] The following describes the process of rank-aware incremental learning for the entire incremental training set.

[0072] When the original model is fine-tuned, facing the incremental training set of new instances and historical instances with the same rank-score, the learning value of the samples has been evaluated. The present invention uses rank-score to design a ranking-aware incremental learning strategy.

[0073] Specifically, the present invention improves the sample loss function and proposes a Rank-aware loss, which weights the sample loss according to the rank-score, giving a greater weight to samples with higher scores. Since the rank-score is obtained based on the fixed experience of the original model before incremental learning, the learning ability of the model changes during the learning process, and the initial rank-score needs to be weakened. As the number of iterations increases, the sample score is gradually regressed to the initial rank-score average of the data set. The calculation formula of Rank-aware loss is as follows:

[0074] Rank-aware loss(img i )=W(img i ,iter)*loss(img i ) (5)

[0075] Among them, loss(img i ) is img i The original training loss, W(img i ,iter) is img i The loss weight at iteration number iter.

[0076] The loss weight calculation formula is as follows:

[0077]

[0078]

[0079] Among them, max_iter is the maximum number of iterations, is the average of the initial scores of the incremental training set.

[0080] Experiments and Datasets

[0081] The present invention conducts experiments on two commonly used public remote sensing image datasets, DIOR and DOTA, which have sufficient categories for target detection. DIOR has 23,463 images with a size of 800*800, containing 192,472 instance targets, covering 20 common categories. DIOR is divided into training set, validation set and test set, with 5,862, 5,863 and 11,725 ​​images respectively. DOTA has 2,806 images with sizes ranging from 800*800 to 4,000*4,000, containing 188,282 targets, covering 15 categories. DOTA is divided into training set, validation set and test set, with 1,411, 758 and 937 images respectively. Since the test set of DOTA is not available, the training set of DOTA is used for training, and the accuracy is tested on the validation set. In addition, since a uniform size is required, after cropping the original DOTA images, 9123 and 2796 images in the range of 800*800 are generated as available training sets and validation sets respectively.

[0082] Baseline

[0083] (1) Finetuning (FT): It is assumed that there is no difference between the newly added instances, and they are added to the training set in no order. At the same time, the original model is used to fine-tune the parameters.

[0084] (2) ALDOD: Actively select samples for learning based on query strategies. We selected two average-based active query strategies: Avg (based on average 1v2 confidence) and Avg+w (based on average 1v2 confidence while balancing the number of objects of different categories). This method is used on newly added instances.

[0085] (3) Memory Buffer (MB): Add new instances to the training set in an unordered manner while acknowledging the existence of forgetting and retaining historical instances in an unordered manner.

[0086] (4) FOCAL LOSS (FL): It is believed that the difficulty of new samples varies, and the loss of difficult samples should be balanced during model training. This method is used on newly added instances.

[0087] Experimental setup

[0088] To simulate the instance increment scenario, the entire dataset is divided into three datasets, namely, old instances, new instances, and test instances. The original model is trained using the old instances. Later, new instances appear, at which point the model needs incremental learning. The test instance is always fixed before and after the increment to evaluate the accuracy of the trained model. For DIOR, the training set is used as the old instances, the validation set is used as the new instances, and the test set is used as the test instances. For DOTA, half of the training set is randomly selected as the old instances and the other half as the new instances. The original validation set is used as the test instance. The statistics of the divided datasets are shown in Table 1. The old instances and new instances of each dataset have approximately the same number of images and objects.

[0089] Table 1. Dataset after instance incremental scene division

[0090]

[0091] (1)Proportion setting to verify the rank of instance

[0092] In order to explore and compare the performance of different methods (including baselines and the method proposed in this paper), we set different sampling ratios of new and old instances for incremental training. The ratio of new instances ranges from 10% to 80%, with an interval of 10%. When saving old instances, considering the actual storage cost and the mitigation effect of forgetting, the default ratio is set to 25%, and other smaller ratios are set to 5%-20%, with an interval of 5%.

[0093] (2) Training configuration

[0094] The target detection model used in the present invention is Faster R-CNN, which has excellent accuracy in DIOR and DOTA. The backbone network adopts Resnet-50-FPN. The specific implementation is based on Detectron2. The GPU configuration is NVIDIA GeForce GTX2080. The pre-trained model of COCO is used to migrate it to the remote sensing image target detection task. If the method has a random selection operation, it needs to be repeated 5 times. In terms of hyperparameter setting, during training, the initial learning rate is set to 0.00025, the batch size is set to 4, and the epoch is set to 12. The hyperparameters α and β of RAIIDF are set to 1 by default, and the transformation operation of the image is set to rotate 90 degrees clockwise. For the hyperparameters of FL, we refer to the original file and set the γ value to 2. Since FasterR-CNN is a two-stage detection model, the classification loss in the RPN and Fast R-CNN structures becomes a focal loss. In addition, in order to avoid the imbalance between the classification loss and the regression loss, the focal loss value in the classification is converted to the same value as the original classification loss value.

[0095] This paper uses average precision as the evaluation index of the algorithm to comprehensively evaluate the accuracy of the classification and coordinate regression results on the test set. Specifically, three accuracy indicators of COCO are used:

[0096] (1) mAP (mean average precision): The average of all thresholds (0.50: 0.05: 0.95) of IoU (Intersection over Union) and the average precision of all categories. It integrates different positioning accuracy requirements and reflects the average performance.

[0097] (2) AP50: The average precision of all categories under an IoU threshold of 0.5. The positioning accuracy requires that the IoU between the predicted box and the true value is greater than 0.5.

[0098] (3) AP75: Average precision of all categories under an IoU threshold of 0.75. The positioning accuracy requires that the IoU between the predicted box and the true value is greater than 0.75.

[0099] Experimental Results

[0100] Figure 3 and Figure 4The accuracy results of different methods are shown under different proportions of new instances for training on DIOR and DOTA, respectively. By default, the proportion of old instances in MB and RAIDF is 25%. In the experimental results, FT in the baseline has the lowest accuracy in most settings. FT learns new instances on the existing basis without ranking them by value and without manual adjustment, and its performance is the most basic. Under the current parameters and some proportions, the performance of FL, which requires manual adjustment of parameters, is improved compared with FT, which to some extent reflects that FL's focus on difficult samples is not much good for incremental learning. The results of MB on these two datasets are consistent, and its accuracy is much higher than FT. The reason for the good results is that a certain amount of storage space is opened for old instances, which effectively alleviates catastrophic forgetting. Under multiple proportions of new instances on DIOR and DOTA, both methods of ALDOD outperform FT, indicating that new instances with greater value can be actively selected through the query strategy of ALDOD. In addition to the new method proposed in this paper, the methods with the highest accuracy are concentrated in the two most competitive methods in the baseline, MB and ALOD. Compared with all existing methods collected in this paper, our proposed RAIIDF shows significant superiority in all three accuracy metrics, both at small scales and at large scales close to the full dataset, demonstrating the very powerful advantage of RAIIDF.

[0101] (1) Visualization of scoring results

[0102] Figure 2 The figure shows the ranking basis (rank-score) and rankresults of some new instances in DIOR. Each sample shows the prediction results of the original data, the prediction results of the transformed image, and the true label of the image. For the rank-score, the uncertainty body is calculated based on the prediction results of the original model for the new instance. The samples with lower average confidence and greater difference before and after transformation have higher rank-score, higher ranking, and more priority learning.

[0103] (2)Labeling cost for new instance

[0104] New instances are sampled without labels, and then annotated and learned. Therefore, in most studies, the number of images is used to control the sampling amount. Our experiments also follow this approach, but when the number of images is the same, the sampling strategies in different methods will label different numbers of objects. We adopted four different sampling strategies for new instances on two experimental datasets and counted the number of manually labeled objects generated by different methods. The statistical results are shown in Figure 2. Figure 3 and Figure 4As shown in Figure 2, we can see that FT using the random sampling strategy has the largest number of labeled objects at various scales. The two query sampling strategies of RAIDF significantly reduce the number of annotations, and RAIDF selects images containing the least number of objects when achieving the highest performance in most cases, reflecting the advantage of this method in terms of labeling cost.

[0105] (3) Ablation experiment

[0106] Ablation experiments were conducted on the three parts of RAIIDF (NIR, OIR, RAIL) on the DIOR dataset, with four new instance selection ratios and 25% historical instance ratio. The results are shown in Table 2. The addition of each part can improve the performance, and the combination of the three can achieve the highest performance, which shows the internal and overall effectiveness of the method.

[0107] Table 2 Ablation experiments on the DIOR dataset

[0108]

[0109] The two hyperparameters α and β that control weights in the RAIIDF method represent the weights of the uncertainty-based score and the inaccuracy-based score in the comprehensive score. We conducted parameter sensitivity experiments on the DIOR dataset. The results are shown in Table 3. It can be seen that the results are the worst when α=0 and β=1. In the comparative experiment, the RAIID method uses α=1 and β=1 by default. From the results, it is not the best parameter configuration, but the results between different parameter combinations are small. Since the RAIID method has no hyperparameters when selecting new instances, it is selected once and then labeled. Hyperparameter adjustment only exists in the subsequent training stage, and it is also a directional balance between the two.

[0110] Table 3 Parameter sensitivity experimental results on the DIOR dataset, 20% new instances, 25% old instances

[0111]

[0112]

[0113] Table 4 Exploring the impact of data arrival order on the algorithm on the DIOR dataset

[0114]

[0115] For streaming data, there are differences in the distribution of the two data sets that arrive before and after. The final incremental learning results will also be different if the data arrives in different orders. In order to explore the impact of different data arrival orders on the incremental learning method we proposed, the order of data appearance was changed on the DIOR data set. In order to avoid confusion, the definition in Table 1 is continued, but the order of data arrival is: new instances appear first, then old instances appear. In order to ensure the reliability of the results, the MB method and RAIIDF method, which performed better in the baseline, were selected for experimentation, and the experimental results are shown in Table 4. First, under the setting of the reverse order, the method of the present invention itself performs well-surpassing the MB method. Then, the incremental results generated by the two methods under the setting of the opposite order of data arrival are compared, and the average value is taken under different incremental learning data ratios as the comprehensive result of incremental learning. We found that the incremental learning results of different methods are different in different data learning orders. Compared with the MB method, RAIIDF can shorten this difference.

[0116] The data and model levels of the present invention provide a reliable solution for the instance increment scenario, and obtain a detection model that can cope with changes in data distribution and fit the data distribution, while also saving learning costs in terms of annotation, storage, time, etc. In addition, when the current static data set is too large, this method can also be used to divide the data set into learning orders, and through progressive learning, the current learning experience can be used to dynamically adjust subsequent learning tasks to improve learning efficiency.

[0117] The present invention studies the scenario of instance increment (in the target detection task of remote sensing images with fixed categories, data containing new instances are constantly arriving), which is closer to reality than the scenario with closed and fixed data. The present invention analyzes the difference in learning value of data in the scenario of instance increment, and can learn the existing ranked data in a balanced manner in a rank-aware manner based on the sorting of data. On two large datasets, DIOR and DOTA, sufficient experiments are set up from multiple aspects such as performance comparison, scoring result visualization, data annotation, ablation experiments, parameter sensitivity, and data order, and the excellent results demonstrate the effectiveness of the method.

[0118] The beneficial effects of the present invention are as follows:

[0119] 1. For the remote sensing image object detection task, an order-aware instance incremental detection framework is proposed for the instance incremental scenario, which can continuously learn streaming data of varying values ​​and pay comprehensive attention to data differences.

[0120] 2. A rank score calculation method based on the uncertainty and inaccuracy of the prediction results is designed to adaptively sort new instances and historical instances, while providing a unified sample weighting direction for model training.

[0121] 3. Experiments were conducted on two widely used remote sensing image datasets. Compared with existing methods, the superiority of the proposed method was verified. The effectiveness of the method itself was verified through ablation experiments and hyperparameter analysis experiments.

[0122] The word "preferred" as used in the present invention is intended to be used as an example, instance or illustration. Any aspect or design of the present invention described as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included arrangements. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the aforementioned examples.

[0123] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown in the present invention. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0124] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0125] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment, and any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An order-aware remote sensing image instance incremental detection method, characterized in that: The following steps are involved: Get remote sensing image instances; According to whether there are true labels, the order score calculation function is designed to evaluate the uncertainty and inaccuracy of the inference results of the target detection model, and measure the incremental learning value in the newly added instances and the incremental learning value in the historical instances. Use a weighted training loss based on the order score while fine-tuning the object detection model to balance the contribution of training data with different values; The uncertainty of a single sample is calculated as follows: Among them, img i is the i-th image, is the "top-1" confidence of the j-th predicted target in image i, Transform(img i ) is to convert img i Perform image transformation operations without destroying the semantic information of the image. i } is to calculate a single image img i The uncertainty of is calculated as follows: The inaccuracy is calculated based on the difference between the predicted box and the true box, and the specific calculation formula is as follows: IA(img i )=1-mAP50(TL_img i ,PR_img i ) Among them, img i is the i-th image, TL(img i ) is img i The real label in PR(img i ) is img i The prediction results in , mAP50 is the average precision calculation method of the coco dataset, which is used to calculate the single image img i Average precision at IoU threshold 0.5; The order score is calculated comprehensively according to the uncertainty and inaccuracy of the prediction results. In order to balance the weight of the two, the specific calculation formula of the order score is as follows: Rsimg i )=α*UC(img i )+β*IA(img i ) Where a and b are weight hyperparameters.

2. The order-aware remote sensing image instance increment detection method according to claim 1, characterized in that: The calculation formula of order-aware loss is as follows: Rank-aware loss(img i )=W(img i ,iter)*loss(img i ) Among them, loss(img i ) is img i The original training loss, W(img i ,iter) is img i The loss weight at iteration number iter.

3. The order-aware remote sensing image instance increment detection method according to claim 2, characterized in that: The loss weight calculation formula is as follows: Among them, max_iter is the maximum number of iterations, is the average of the initial scores of the incremental training set.

4. The order-aware remote sensing image instance increment detection method according to claim 1, characterized in that: The target detection model is Faster R-CNN.

Images