A remote sensing image open world target detection method and system
By employing the Old-Known model freezing and pseudo-label training strategy within the Old-New-known framework, the problem of forgetting old categories in open-world object detection of remote sensing images is addressed, enabling memory-based learning of old categories and improving detection performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-08
- Publication Date
- 2026-04-10
AI Technical Summary
In open-world object detection of remote sensing images, there is a problem of forgetting old categories. Existing technologies have failed to effectively maintain the memory of old categories, resulting in a decline in detection performance.
By adopting the Old-New Known framework, the classification-regression model parameters of the Old-Known model are frozen, and training is performed using pseudo-labels and auxiliary datasets. Combined with a loss function, this enables memory-based learning of Old-Known categories, thereby improving detection performance.
In open-world target detection of remote sensing images, effectively maintaining the memory of old categories improves the performance of the detection task, reduces the forgetting of old categories, and improves detection accuracy.
Smart Images
Figure CN121482375B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of target detection, and particularly relates to a remote sensing image open world target detection method and system. BACKGROUND
[0002] Remote sensing image target detection is to identify and locate a specific class of target from a remote sensing image. Remote sensing images are taken from different heights, and have the characteristics of scale diversity, special angle of view, high background complexity, etc. Remote sensing image open world target detection (RSOWOD) is a field involving the intersection of remote sensing technology and target detection technology. Compared with remote sensing image target detection, RSOWOD is closer to the real world and has potential application value in the fields of environmental monitoring, urban planning, disaster monitoring, resource management, etc.
[0003] As one of the branches of remote sensing image target detection (RSOD), RSOWOD needs to automatically identify and locate unknown or new classes of targets from remote sensing images and label them as unknown in addition to identifying and locating objects in remote sensing images in an open world environment, and gradually learn unknown or new classes while retaining memory of known classes.
[0004] For natural image target detection, detection frameworks based on Transformer models have the characteristics of simple detection and high performance, and have made great achievements in target detection tasks. Considering the influence of potential unknown classes on target detection tasks, existing technologies have taken the following measures to eliminate the influence of unknown classes: (1) designing an end-to-end detection framework based on a Transformer model to generate pseudo labels for unknown classes; (2) estimating the target probability of different proposals to improve the performance of detection frameworks based on Transformer models.
[0005] However, compared with natural image target detection, the above measures for natural image processing face the challenges of small target pixel ratio and coexistence of multiple classes due to the unique angle of view and large scale of remote sensing images. On the one hand, the small proportion of remote sensing image annotation information leads to slow convergence of the Transformer model on different target detection tasks; on the other hand, due to the coexistence of multiple classes in remote sensing images, old and new classes often appear in the same task. However, the above existing technologies focus on the labeling of unknown classes and do not pay attention to the memory of old classes, resulting in forgetting the labeling of old classes.
[0006] In open-world object detection tasks using remote sensing images, a large number of old and new categories exist. However, current research has not focused on learning the old categories, resulting in many old categories being forgotten. Therefore, it is crucial to improve the performance of object detection tasks by enabling the Transformer model to learn the old categories. Summary of the Invention
[0007] The purpose of this invention is to address the shortcomings of existing technologies in open-world target detection tasks for remote sensing images, which suffer from the loss of numerous old category labels. This invention provides a method for open-world target detection in remote sensing images, employing an Old-New known framework (i.e., an old-new target recognition framework). This framework learns Newknown categories (i.e., new known target categories) while maintaining the memory of Old known categories (i.e., previously known target categories), thereby improving the detection performance of the target detection task.
[0008] To achieve the above objectives, the technical solution provided by this invention is:
[0009] On one hand, the present invention provides an open-world target detection method for remote sensing images, comprising:
[0010] Step 1: Remote Sensing Image Dataset Processing ,in, , express Zhang remote sensing image, Indicates the first Zhang remote sensing image, Indicates the first Zhang remote sensing image, Labels representing the corresponding remote sensing images. Indicates the corresponding number Tags for remote sensing images Indicates correspondence Labels for remote sensing images;
[0011] Step 2: Remote sensing image dataset detection: [This step involves] detecting the remote sensing image dataset. Input a New-Known model that has been constructed and trained for open-world object detection; the output of the New-Known model is a remote sensing image dataset. The category information and location information; specifically, the remote sensing image dataset detection process includes: step (21) dividing the remote sensing image dataset into categories and locations; The backbone network of the New-Known model is input to extract image features. The backbone network convolves the image features to obtain a feature map, reduces the dimensionality of the feature map, and performs flattening and concatenation operations to obtain the feature map. ; step (22) inputting the query embedding into the classification-regression model of the New-Known model to obtain the class information and positioning information of the open-world target of the remote sensing image. ; step (23) inputting the query embedding into the classification-regression model of the New-Known model to obtain the class information and positioning information of the open-world target of the remote sensing image.
[0012] According to one aspect, as a further limitation of the present application, the step one in processing the remote sensing image dataset :
[0013] (1) the remote sensing image dataset has a set of target classes ;
[0014] (2) the label corresponding to the first remote sensing image is represented as , where represents the first vector containing boundary coordinates and object class label, represents the second vector containing boundary coordinates and object class label, represents the vector containing boundary coordinates and object class label.
[0015] (3) based on the remote sensing image dataset , the unknown class set encountered in the detection process of the target detection by the New-Known model is represented as , where represents the unknown class.
[0016] According to one aspect, as a further limitation of the present application, the model constructed in step two is a target detection model for open-world target detection of remote sensing images, which includes not only the New-Known model but also the Old-Known model, wherein:
[0017] The New-Known model is used for target detection of remote sensing images, and the New-Known model and the Old-Known model are trained together; specifically:
[0018] (1) the New-Known model and the Old-Known model have the same network structure, respectively comprising a backbone network, an encoder-decoder module and a classification-regression model; when training the target detection model, the Old-Known model is initialized using the weights learned from the previous task, the New-Known model and the Old-Known model are trained simultaneously, and the parameters of the classification-regression model in the Old-Known model are frozen;
[0019] (2) the data output by the Old-Known model is used to update the parameters of the classification-regression model in the New-Known model through back propagation.
[0020] Based on one aspect, as a further limitation of the application, the backbone network adopts a ResNet-50 network;
[0021] The encoder-decoder module adopts a Transformer model architecture with multi-head self-attention mechanism;
[0022] The classification-regression model adopts a 3-layer convolution with a convolution kernel of 3 and a linear function.
[0023] Based on one aspect, as a further limitation of the application, in step two, the trained model is the New-Known model and the Old-Known model;
[0024] In step two, the New-Known model and the Old-Known model are trained simultaneously, and the model training includes pseudo-label training, auxiliary dataset training and loss function training; wherein:
[0025] Pseudo-label training includes:
[0026] Freezing the parameters of the classification-regression model in the Old-Known model, and updating the branch parameters of the New-Known model through back propagation on the remote sensing image dataset The Old-Known class is hidden, and the branch parameters of the New-Known model are updated through back propagation on the New-Known class and the Old-Known class for pseudo-label training;
[0027] Auxiliary dataset training includes:
[0028] Randomly sampling remote sensing images from the remote sensing image dataset of the previous task to construct an auxiliary dataset , freezing the parameters of the classification-regression model in the Old-Known model, and updating the branch parameters of the New-Known model through back propagation on the auxiliary dataset The Old-Known model is used to assist in learning the Old-Known category, and the auxiliary data set is labeled The New-Known model is trained based on the pseudo-labels of the Old-Known category.
[0029] The loss function training includes:
[0030] The New-Known model is trained on the New-Known category and the Old-Known category, and the remote sensing image data set The total loss is the weighted sum of the New-Known category loss and the Old-Known category loss; the Old-Known model freezes the parameters of the Old-Known branch, the New-Known category is trained based on the pseudo-labels of the Old-Known category, and the auxiliary training is performed through the auxiliary data set The loss function of the auxiliary data set is used to calculate the known category loss.
[0031] According to one aspect, as a further limitation of the present application, the pseudo-label training specifically includes:
[0032] The Old-Known model with frozen parameters is used to generate predicted categories and image positioning information in the classification-regression model of the Old-Known branch; specifically:
[0033] (1) The training process uses a threshold Non-maximum suppression is performed to eliminate the overlap of predicted boxes when the targets in the remote sensing image are closely connected;
[0034] (2) According to the confidence threshold The category information is filtered to obtain the category prediction result of the high-score foreground by filtering the category score;
[0035] (3) The corresponding positioning information is found through the category prediction result of the high-score foreground, and the positioning information is selected as the pseudo-label of the Old-Known category.
[0036] According to one aspect, as a further limitation of the present application, the auxiliary data set training specifically includes:
[0037] For the remote sensing image data set , considering that not every remote sensing image contains the Old-Known category, an auxiliary data set is constructed The Old-Known category is learned, so the data set for training contains the remote sensing image data set , the remote sensing image data set , and the remote sensing image data set . auxiliary data set of task 1 ; wherein:
[0038] , , , representing the task remote sensing pictures, representing the labels of the corresponding remote sensing pictures, representing the task pictures, the task there is a set of target categories ;
[0039] , representing the auxiliary data set of task 1, representing the auxiliary data set of task 2, representing the auxiliary data set of task ;
[0040] Specifically:
[0041] When the task , the remote sensing image data set , the remote sensing image data set only contains new known categories, and the New-Known model learns the new known categories from the labeled true value of the loss function ;
[0042] When the task , the remote sensing image data set , the remote sensing image data set contains old known categories and new known categories, the remote sensing image data set learns the new known categories by using the labeled true value, and learns the old known categories by using the pseudo-label training strategy, and the loss function of the old known categories is .
[0043] On the one hand, as a further limitation of the present application, the loss function training specifically includes:
[0044] The total loss calculation expression of the remote sensing image data set is:
[0045]
[0046] In the formula, represents the weighted sum of the new known category loss and the old known category loss, represents the new known category loss, represents the old known category loss used to balance the remote sensing image data set The weights of the importance of known categories in the loss. The loss function representing the previously known categories. , Indicates batch size. This represents the regression loss of the classification-regression model. This represents the regression branch of a classification-regression model. Represents a remote sensing image dataset The Middle The query output obtained from the remote sensing image through the encoder-decoder module Indicates task The Middle Pseudo-labels for the image This represents the classification loss of a classification-regression model. This represents the classification branch of a classification-regression model;
[0047] Auxiliary dataset The loss function is calculated as follows:
[0048]
[0049] In the formula, Representing auxiliary datasets Batch size in , Indicates batch size. Represents a constant. This represents the regression loss of the classification-regression model. This represents the regression branch of a classification-regression model. Representing auxiliary datasets The Middle The query output of Zhang's image. Indicates task pseudo-tags in This represents the classification loss of a classification-regression model. This represents the regression branch of a classification-regression model.
[0050] Based on one aspect, as a further limitation of the present invention, during the training phase of the New-Known model, the New-Known model learns from the corresponding task... Remote sensing image dataset The process involves identifying unseen targets, classifying them into an unknown category, and detecting previously encountered known categories; specifically:
[0051] The New-Known model is for the task. During detection, first detect the new set of known categories. The categories in the set are then checked against the previously known set of categories. old known classes in the set of old known classes; wherein, the New-Known model is trained on the task unknown classes are detected classes in the set of old known classes;
[0052] the New-Known model is trained on the task unknown classes are detected on a remote sensing image dataset the New-Known model is trained on the task the New-Known model is trained on the task unknown classes are detected classes in the set of old known classes;
[0053] the New-Known model is trained on the task
[0054] In another aspect, the present application also provides a remote sensing image open world object detection system, based on the remote sensing image open world object detection method of the above-mentioned aspect, the remote sensing image open world object detection system of the present application comprises an input unit, a processing unit and an output unit; wherein:
[0055] The remote sensing image dataset is input into the input unit;
[0056] The New-Known model for open world object detection is configured in the processing unit, and the remote sensing image dataset is input into the New-Known model, and the New-Known model processes the remote sensing image dataset ;
[0057] The output unit outputs the class information and the positioning information.
[0058] The present application has the following advantages:
[0059] 1. The object detection model of the present application adopts the Old-New known framework. In the training process, not only is the New known class learned through the training dataset while the memory of the Old known class is maintained, but also the Old known class is assisted in training through the auxiliary dataset. Therefore, after the model training is completed, the performance of the detection task is improved in the process of remote sensing image open world object detection of the image data by using the object detection model.
[0060] 2. Since not every remote sensing image in the remote sensing image dataset contains both Old known and New known categories, this invention, in the open-world target detection task of remote sensing images, adopts a target detection model that only needs to maintain the memory of Old known categories, without having to pay attention to New known categories, thus reducing the forgetting of Old known categories in the target detection process.
[0061] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0062] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:
[0063] Figure 1 The present invention provides a flowchart of an open-world target detection method for remote sensing images;
[0064] Figure 2 The present invention provides a block diagram of the target detection model principle during the training phase of an open-world target detection method for remote sensing images;
[0065] Figure 3 The present invention provides a remote sensing image dataset partitioning table during the experimental effect verification of an open-world target detection method for remote sensing images;
[0066] Figure 4 This invention provides a parameter table for constructing a target detection model in an open-world target detection method for remote sensing images.
[0067] Figure 5 This invention provides a comparative table of experimental results for an open-world target detection method for remote sensing images. Detailed Implementation
[0068] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0069] Please see Figure 1 This invention provides an open-world target detection method for remote sensing images, comprising:
[0070] Step 1: Remote Sensing Image Dataset Processing ,in, , express Zhang remote sensing image, Indicates the first Zhang remote sensing image, Indicates the first Zhang remote sensing image, Labels representing the corresponding remote sensing images. Indicates the corresponding number Tags for remote sensing images Indicates correspondence Labels for remote sensing images.
[0071] Specifically, step one of the embodiments of the present invention involves processing the remote sensing image dataset. During the process: (1) Remote sensing image dataset There is a set of target categories The target detection task is to locate the target position and identify the target category. Therefore, the target category in the embodiments of the present invention is... These are important known conditions that guide the New-Known model to complete training; (2) Corresponding Tags for remote sensing images Represented as ,in, This represents the first vector containing boundary coordinates and object category labels. This represents the second vector containing boundary coordinates and object category labels. Indicates the first (3) Based on remote sensing image datasets During the target detection process using the New-Known model: the set of unknown categories encountered by the target detection. Represented as , express Unknown category.
[0072] It should be noted that the remote sensing image dataset processing in this embodiment of the invention includes not only the processing of the remote sensing image dataset during the target detection model stage, but also the processing of the remote sensing image dataset during the training stage. Specifically, before training the target detection model, this embodiment of the invention also processes the remote sensing image training dataset. The remote sensing image training dataset is processed. Represented as , express One picture, , This refers to the first image. This refers to the second image. Indicates the first One picture, Indicates correspondence The image's tag, , Indicates the corresponding number a label of the i-th picture, a label of the i-th picture, a label of the i-th picture, a label of the i-th picture, a label of the i-th picture, a label of the i-th picture consists of annotation objects, , , a i-th vector containing boundary coordinates and object class label, a i-th vector containing boundary coordinates and object class label, a i-th vector containing boundary coordinates and object class label, a i-th vector containing boundary coordinates and object class label. More specifically, in the process of using a target detection model to perform an open world target detection task of a remote sensing image, the target detection model not only needs to identify an unobserved target in the open world of the remote sensing image and classify the unobserved target into an unknown category, but also needs to detect known categories previously detected by the target detection model. In addition, in the process of training the target detection model, not only the categories in a new known category set
[0073] of a task , but also the old known categories in the new known category set from a task to a task , wherein , the target detection model can also detect categories in an unknown category set . At the end of the task , the training of the task begins. In the process of training the task , the target detection model is incrementally trained on the basis of the target detection results output by the target detection model from the task to the task , and is not retrained from the remote sensing image dataset of the task . Therefore, the target detection model trained for the task inherits the weights of the target detection model trained for the task . The training is performed on the remote sensing image dataset of the task , which is represented as , Remote sensing image dataset, Indicates task All remote sensing images, Indicates the corresponding task Labels for all remote sensing images.
[0074] It should be noted that step one of the embodiments of the present invention not only processes the remote sensing image dataset of task t. Also handle tasks Auxiliary dataset .in, , , Indicates task of One picture, task There is a set of target categories .
[0075] Step 2: Remote sensing image dataset detection: [This step involves] detecting the remote sensing image dataset. Input is a New-Known model that has been built and trained for open-world object detection; output is a remote sensing image dataset. The category information and location information; specifically, the remote sensing image dataset detection process includes: step (21) dividing the remote sensing image dataset into categories and locations; Input the backbone network of the New-Known model to extract image features The backbone network will use image features After convolution, feature maps are obtained. The feature map dimension of each image is represented as follows: The length and width of the feature maps corresponding to each image are varied according to the size of the different input images. For example, the remote sensing image dataset of this embodiment of the invention... Feature map dimensions of each image's corresponding feature map Image features: 512, 1024, 2048 After a 1×1 convolution, the dimension of the feature map is uniformly reduced to 256. The length and width of the feature map are then flattened and concatenated. The dimensionality-reduced feature map, after being flattened and concatenated, yields the feature map. Step (22) will feature Input the encoder-decoder module of the New-Known model to obtain the query embedding of the remote sensing image. Step (23) embeds the query Input the classification-regression model of the New-Known model to obtain the category and location information of open-world targets in remote sensing images.
[0076] Specifically, the model constructed in step two of the embodiment of the present application is a target detection model for target detection in an open world of remote sensing images, and the target detection model comprises a New-Known model and an Old-Known model, wherein: the New-Known model is used for target detection of the remote sensing images, and the New-Known model and the Old-Known model jointly participate in training; specifically: (1) the New-Known model and the Old-Known model have the same network structure and respectively comprise a backbone network, an encoder-decoder module and a classification-regression model; when training the target detection model, the Old-Known model is initialized using the weights learned from a previous task, the New-Known model and the Old-Known model are simultaneously trained, and the parameters of the classification-regression model in the Old-Known model are frozen; (2) the data output by the Old-Known model is updated by the New-Known model through back propagation, and the parameters of the classification-regression model in the New-Known model are updated.
[0077] Preferably, the New-Known model and the Old-Known model of the embodiment of the present application both adopt an enhanced deformable architecture, comprising a backbone network, an encoder-decoder module and a classification-regression model. The backbone network adopts a ResNet-50 network, the encoder-decoder module adopts a Transformer model with a multi-head self-attention mechanism, and the classification-regression model comprises a classification model and a regression model , the classification model is composed of a 3-layer convolution with a convolution kernel of 3, and the regression model is composed of a linear function.
[0078] Specifically, referring to Figure 2 , in step two of the embodiment of the present application, the model trained is the New-Known model and the Old-Known model; more specifically, the New-Known model and the Old-Known model are simultaneously trained, and the model training comprises: (1) pseudo-label training, (2) auxiliary dataset training and (3) loss function training; wherein: (1) since each remote sensing image in the remote sensing image dataset has the characteristics of large size and small target, the remote sensing image dataset In addition to the known categories included in the current task, the dataset also includes multiple hidden Old Known categories. Furthermore, because existing technologies do not consider learning Old Known categories during remote sensing image target detection, the process of detecting Old Known categories in existing technologies suffers from forgetting these categories. To maintain the memory of Old Known categories, this embodiment of the invention employs a pseudo-label training method to train the target detection model. Therefore, the pseudo-label training in this embodiment includes: freezing the parameters of the classification-regression model in the Old-Known model, and training the target detection model on the remote sensing image dataset. The hidden Old-Known category is labeled, and the branch parameters of the New-Known model are updated through backpropagation. Pseudo-label training is performed on the New-Known category and the Old-Known category; (2) Auxiliary dataset training includes: constructing an auxiliary dataset by randomly sampling remote sensing images from the remote sensing image dataset of the previous task. Freeze the parameters of the classification-regression model in the Old-Known model, and use the auxiliary dataset. Input the Old-Known model to help learn Old-Known categories, and label the auxiliary dataset. The classification-regression model of the New-Known model is trained on pseudo-labels of the Old-Known category; (3) Loss function training includes: the new known branch of the New-Known model is trained on the New-Known category and the Old-Known category, remote sensing image dataset The total loss is a weighted sum of the losses for the new known classes and the old known classes; the parameters of the old known branches are frozen in the Old-Known model, and the New-Known classes are trained based on the pseudo-labels of the Old-Known classes. Auxiliary training is performed using an auxiliary dataset. The loss function is used to calculate the loss for a known class.
[0079] For more details, please continue reading. Figure 2 The pseudo-label training of this embodiment includes: generating predicted categories and image localization information using the old known branches in the classification-regression model of the Old-Known model with frozen parameters; specifically: (1) during the training process, since the distance between targets in the remote sensing image is relatively close, the predicted bounding boxes may overlap. In order to eliminate overlap, this embodiment uses a threshold. (2) Non-maximum suppression operation is performed to eliminate the overlap of prediction boxes when targets in remote sensing images are closely connected; (3) The embodiments of the present invention also use confidence thresholds. (3) Filter category information and filter category scores to obtain category prediction results for high-scoring prospects; (4) Find corresponding location information through the category prediction results for high-scoring prospects and select the location information as pseudo-labels for Old-Known categories.
[0080] For more details, please continue reading. Figure 2 The auxiliary dataset training in this embodiment of the invention specifically includes: for remote sensing image datasets Considering that not every remote sensing image contains previously known categories, an auxiliary dataset was constructed. And for the auxiliary dataset Learning from previously known categories, the training dataset contains tasks. Remote sensing image dataset ,Task Remote sensing image dataset and tasks Auxiliary dataset Among them: (1) , , , Indicates task of Zhang remote sensing image, This indicates the label for the corresponding remote sensing image. Indicates task of One picture, task There is a set of target categories (2) , This represents the auxiliary dataset for Task 1. This represents the auxiliary dataset for Task 2. Indicates task Auxiliary datasets; specifically: when the task At the time, training remote sensing image dataset Remote sensing image dataset The model only includes newly known categories, and the loss function is... New known categories are learned from the labeled truth values, and the new known branches are derived from the labeled truth values (loss is...). Learn new known categories in the task. At the time, training remote sensing image dataset Remote sensing image dataset Includes both old and new known categories for remote sensing image datasets. We learn new known categories using labeled ground truth values and old known categories using a pseudo-label training strategy. The loss function for the old known categories is... Furthermore, due to the remote sensing image dataset Not every image in the dataset contains old known classes, therefore, the embodiment of the application learns old known classes in a way of constructing auxiliary dataset , , represents the dataset obtained by randomly sampling from the task corresponding remote sensing image dataset , represents the dataset obtained by randomly sampling from the task corresponding remote sensing image dataset , represents the dataset obtained by randomly sampling from the task corresponding remote sensing image dataset . It should be noted that the auxiliary dataset does not contain annotation information, therefore, the embodiment of the application learns old known classes in an auxiliary training strategy for the auxiliary dataset .
[0081] More specifically, the loss function training of the embodiment of the application includes: the total loss calculation expression of the remote sensing image dataset is as follows:
[0082]
[0083] In the formula, represents the weighted sum of new known class loss and old known class loss, represents the new known class loss, represents the weight for balancing the importance of old known class loss in the remote sensing image dataset , represents the loss function of old known class, , represents the batch size, represents the regression loss of the classification-regression model, represents the regression branch of the classification-regression model, represents the query output of the i-th remote sensing image in the remote sensing image dataset obtained by the encoder-decoder module, represents the pseudo label of the i-th image in the task , represents the classification loss of the classification-regression model, represents the classification branch of the classification-regression model;
[0084] More specifically, the loss function calculation expression of the auxiliary dataset of the embodiment of the application is as follows:
[0085]
[0086] In the formula, Representing auxiliary datasets Batch size in , Indicates batch size. Represents a constant. This represents the regression loss of the classification-regression model. This represents the regression branch of a classification-regression model. Representing auxiliary datasets The Middle The query output of Zhang's image. Indicates task pseudo-tags in This represents the classification loss of a classification-regression model. This represents the regression branch of a classification-regression model.
[0087] For more details, please continue reading. Figure 2 In the training phase of the New-Known model in this embodiment of the invention, the New-Known model learns from the corresponding task. Remote sensing image dataset In the process of identifying unseen targets, classifying unseen targets as unknown categories, and detecting previously known categories encountered in previous tasks; specifically: (1) the New-Known model for the task During detection, first detect the new set of known categories. The categories in the set are then checked against the previously known set of categories. The previously known categories; where, ; Object detection model for the task The detection also includes the detection of unknown category sets. The categories in; (2) New-Known model for tasks The test is complete; continue with the task. In remote sensing image datasets Target detection training is performed on the target, and the task is performed on the target. The New-Known model continues training by inheriting the weights from task t. Detecting new sets of known categories The categories in; (3) The New-Known model continues to train the target detection of the task until the New-Known model is fully trained.
[0088] The target detection model in this embodiment of the invention adopts an Old-New known framework. During training, it not only learns the New known categories while maintaining the memory of the Old known categories through the training dataset, but also uses an auxiliary dataset to assist in training the Old known categories. Thus, after the model training is completed, the target detection model in this embodiment of the invention improves the performance of the detection task in the process of open-world target detection of remote sensing images.
[0089] This invention also provides an open-world target detection system for remote sensing images. Based on the disclosed method for open-world target detection in remote sensing images, specifically, the open-world target detection system for remote sensing images provided by this invention includes an input unit, a processing unit, and an output unit; wherein: the input unit receives a remote sensing image dataset. The processing unit is equipped with a New-Known model for open-world object detection, and the remote sensing image dataset is also included. Input the New-Known model; the New-Known model processes remote sensing image datasets. The output unit outputs category information and location information.
[0090] The technical effects of the embodiments of the present invention are further illustrated by the following experiments:
[0091] (1) Remote Sensing Image Dataset: In this embodiment of the invention, the target detection model is trained and tested on the DOTA dataset. The remote sensing image dataset uses a 200-pixel overlap, dividing the dimensions of the remote sensing images into 1024×1024. An experimental setup for open-world target detection in remote sensing images is implemented, grouping the target detection task categories into a set of non-overlapping tasks. , In the process of open-world target detection in remote sensing images, when the target is reached... Introducing tasks under conditions The categories within. Object detection models in learning tasks. At that time, open-world object detection in remote sensing images All categories are considered known. All categories were considered unknown. The experimental procedure adopted... Figure 3 The two partitioning methods shown are used to study the performance of open-world object detection methods in remote sensing images. Please refer to [link / reference]. Figure 3 For the DOTA 1.0 dataset, this experiment divides the categories generated during the open-world object detection process of remote sensing images into two groups, among which... Figure 3 Task 1, as shown, contains 3498 images and 73703 instances. Figure 3Task 2 contains 2640 pictures and 61447 instances.
[0092] (2) Evaluation index: For known categories, the standard mean average precision (mAP) is used. For unknown categories, recall is used as the evaluation index. Experiment setting: This experiment uses python3.10.4, Pytorch2.0.0, and cuda11.8 to build the experimental environment, and uses a GeForce RTX 4090 GPU with 24GB of video memory for running. This experiment uses the DDETR of the Transformer model, the number of queries is , and the feature dimension of the corresponding feature map is . This experiment uses the ADAM optimizer to train the target detection model, the batch size is 4, and the learning rate is . During the training of the target detection model, the fine-tuning task of the model parameters is uniformly fine-tuned with 50 labeled instances per class.
[0093] (3) Result comparison: Due to the small proportion of real labels in remote sensing images, it is difficult for the target detection model to converge. This experiment first constructs the architecture based on the D-DETR model using the two-stage strategy, one-to-many strategy, and box REFINEMENT strategy. Please refer to Figure 4 , after adding the two-stage strategy, one-to-many strategy, and box REFINEMENT strategy to the D-DETR model, the model has obvious improvement in Task1 and Task2 compared with the D-DETR model, and the above strategies are effective for remote sensing image target detection. Therefore, this experiment selects the model as the baseline method to experimentally verify the open-world target detection of remote sensing images.
[0094] Please refer to Figure 5 , Figure 5 table shows the comparison results of the model and the target detection model on DOTA1.0. The comparison results show that for the DOTA1.0 dataset, the constructed model has experimental results of 5.29% and 62.56 in the unknown target recall (U-Recall) and the standard mean average precision (mAP) on Task 1, respectively. The experimental results of the previously known target mAP, the currently known target mAP, and the overall mAP on Task 2 are 1.3%, 47.71%, and 24.43%, respectively. After fine-tuning, The previous known target mAP, the current known target mAP and the overall mAP of Task 2 are 59.23%, 50.43% and 51.13% respectively. During the experiment, the fine-tuning strategy of the target detection model is very effective for the identification of the previous known target mAP category of Task 2. If the fine-tuning strategy is not used, the identification of the Old known category by the target detection model is disastrous. In contrast, the experimental results of the target detection model in the U-Recall and mAP of Task 1 are 5.84% and 58.01 respectively. The experimental results of the target detection model in the previous known target mAP, the current known target mAP and the overall mAP of Task 2 are 53.55%, 53.89% and 50.37% respectively. After fine-tuning, the experimental results of the target detection model in the previous known target mAP, the current known target mAP and the overall mAP of Task 2 are 57.61%, 55.95% and 53.18% respectively. It is verified that the target detection model retains the memory of the Old known category when learning the New known category. In addition, the target detection model of the embodiment of the present application has little difference in results compared with the architecture that does not use the fine-tuning strategy. Therefore, through the above experiments, it is verified that the technical effect of the target detection model of the embodiment of the present application on the open world target detection task of remote sensing images is effective, which helps to reduce the forgetting of the Old Known category and improves the performance of the target detection model. The above described is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.
[0095] The above described is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.
Claims
1. A method for remote sensing image open-world target detection, characterized in that, The application relates to a remote sensing image open world target detection system and a target detection method thereof. Step one, remote sensing image dataset processing: processing remote sensing image dataset wherein, , denotes the i-th remote sensing image, denotes the 1st remote sensing image, denotes the Nth remote sensing image, denotes the label of the corresponding remote sensing image, denotes the label of the corresponding 1st remote sensing image, denotes the label of the corresponding Nth remote sensing image; Step two, remote sensing image dataset detection: inputting the remote sensing image dataset into a New-Known model for open world target detection, the New-Known model outputs the category information and positioning information of the remote sensing image dataset ; specifically, the remote sensing image dataset detection process comprises: step (21) inputting the remote sensing image dataset into a backbone network of the New-Known model to extract picture features, the backbone network convolves the picture features to obtain a feature map, and dimensionally reduces, flattens and splices the feature map to obtain a feature ; step (22) inputting the feature into an encoder-decoder module of the New-Known model to obtain a query embedding of the remote sensing image; and step (23) inputting the query embedding into a classification-regression model of the New-Known model to obtain the category information and positioning information of the open world target of the remote sensing image. The model built in step two is a target detection model for target detection of a remote sensing image open world, which comprises a New-Known model and an Old-Known model. The New-Known model is used for target detection of a remote sensing image, and the New-Known model and the Old-Known model are trained together. (1) The New-Known model and the Old-Known model have the same network structure and comprise a backbone network, an encoder-decoder module and a classification-regression model; during training of the target detection model, the Old-Known model is initialized by using the learned weight from a previous task, the New-Known model and the Old-Known model are trained at the same time, and the parameters of the classification-regression model in the Old-Known model are frozen; (2) The data output by the Old-Known model is used for updating the parameters of the classification-regression model in the New-Known model through back propagation of the New-Known model; The New-Known model and the Old-Known model are trained at the same time, and the model training comprises pseudo-label training, auxiliary data set training and loss function training. The pseudo-label training comprises: Freezing the parameters of the classification-regression model in the Old-Known model, updating the branch parameters of the New-Known model through back propagation on the remote sensing image dataset The Old-Known class is hidden, and the branch parameters of the New-Known model are updated through back propagation. Pseudo-label training is performed on the New-Known class and the Old-Known class. The auxiliary data set training comprises: randomly sampling remote sensing images from the remote sensing image dataset of the previous task to construct an auxiliary dataset , freezing the parameters of the classification-regression model in the Old-Known model, inputting the auxiliary dataset into the Old-Known model to assist the Old-Known model in learning the Old-Known class, and labeling the auxiliary dataset with the Old-Known class; the classification-regression model of the New-Known model is trained based on the pseudo-labels of the Old-Known class; The loss function training comprises: The new known branch of the New-Known model is trained on the New-Known class and the Old-Known class, and the remote sensing image dataset The total loss of the New-Known model is the weighted sum of the New-Known class loss and the Old-Known class loss; the Old-Known model freezes the parameters of the Old-Known branch, and the New-Known class is trained based on the pseudo-label of the Old-Known class, and the auxiliary training calculates the known class loss through the loss function of the auxiliary dataset The total loss of the New-Known model is the weighted sum of the New-Known class loss and the Old-Known class loss; the Old-Known model freezes the parameters of the Old-Known branch, and the New-Known class is trained based on the pseudo-label of the Old-Known class, and the auxiliary training calculates the known class loss through the loss function of the auxiliary dataset 2.The method of claim 1, wherein, The step one in processing a remote sensing image dataset of: (1) Remote sensing image dataset There is a set of target classes ; (2) label corresponding to the Nth remote sensing image is represented as wherein, represents a first vector containing boundary coordinates and object class labels, represents a second vector containing boundary coordinates and object class labels, represents a third vector containing boundary coordinates and object class labels, represents an Nth vector containing boundary coordinates and object class labels; (3) based on remote sensing image dataset The unknown class set encountered by target detection in the detection process based on the New-Known model is expressed as , represents unknown class. 3.The method of claim 1, wherein, The backbone network adopts a ResNet-50 network; The encoder-decoder module adopts a Transformer model architecture of a multi-head self-attention mechanism; The classification-regression model adopts a 3-layer convolution with a convolution kernel of 3 and a linear function. 4.The method of claim 1, wherein, The pseudo-label training specifically comprises: The Old-Known branch in the classification-regression model of the Old-Known model with the frozen parameters generates prediction categories and image positioning information; specifically: (1) The training process adopts threshold Non-maximum suppression operation is performed to eliminate the overlapping of the prediction boxes when the targets of the remote sensing image are closely connected. (2) according to the confidence threshold Screening category information, filtering category score to get high score foreground category prediction result; (3) The corresponding positioning information is found by using the category prediction result of the high-score foreground, and the positioning information is selected as the pseudo-label of the Old-Known category.
5. The method of claim 1, wherein, The auxiliary data set training specifically comprises: For a remote sensing image dataset , considering that not every remote sensing image contains old known classes, an auxiliary dataset is constructed , and the old known classes are learned by the auxiliary dataset , so that the dataset for training contains the remote sensing image dataset of the task , the remote sensing image dataset of the task , and the auxiliary dataset of the task ; wherein: , , , representing a task of zooming a remote picture, representing a label of the corresponding remote picture, representing all pictures from task 1 to task , task there is a set of target classes ; , auxiliary dataset representing task 1, auxiliary dataset representing task 2, auxiliary dataset representing task . Specifically: When the task , the training remote sensing image dataset , the remote sensing image dataset contains only new known classes, the New-Known model learns new known classes from the labeled ground truth with loss function When the task , the remote sensing image dataset is trained , the remote sensing image dataset contains old known classes and new known classes, the remote sensing image dataset is trained , the new known classes are learned by using the labeled true value, and the old known classes are learned by using the pseudo-label training strategy, and the loss function of the old known classes is . 6.The method of claim 1, wherein, The loss function training specifically comprises: Remote sensing image dataset The total loss calculation expression of the dataset is: In the formula, This represents the weighted sum of the losses for the new known class and the old known class. Represents the loss of the new known class. Represents the use of balancing remote sensing image datasets The weights of the importance of known categories in the loss. The loss function representing the previously known categories. , Indicates batch size. This represents the regression loss of the classification-regression model. This represents the regression branch of a classification-regression model. Represents a remote sensing image dataset The Middle The query output obtained from the remote sensing image through the encoder-decoder module Indicates task The Middle Pseudo-labels for the image This represents the classification loss of a classification-regression model. This represents the classification branch of a classification-regression model; auxiliary data set The loss function calculation expression of the auxiliary data set is: wherein denotes the batch size in the auxiliary dataset , , denotes the batch size, denotes a constant, denotes the regression loss of the classification-regression model, denotes the regression branch of the classification-regression model, denotes the query output for the th image in the auxiliary dataset , denotes the pseudo label in the task , denotes the classification loss of the classification-regression model, denotes the classification branch of the classification-regression model.
7. The method of claim 1, wherein, The New-Known model is trained in the New-Known model training stage from a remote sensing image dataset that identifies an unmet target, classifies the category of the unmet target as an unknown category, and detects an old known category encountered by a previous task; specifically: The New-Known model for the task detects classes in the new known class set before detecting old known classes in the old known class set ; wherein The New-Known model for the task also detects classes in the unknown class set when detecting. The New-Known model for task Detection is finished, continue training for task On the remote sensing image dataset Target detection training, for task Inheritance of task t weight continues training, the New-Known model for task Detect new known class set In the class; The New-Known model continuously trains the target detection task until the New-Known model is fully trained.
8. A remote sensing image open-world target detection system based on the remote sensing image open-world target detection method of any one of claims 1-7, characterized in that, The remote sensing image open world target detection system comprises an input unit, a processing unit and an output unit. The input unit has inputted therein a remote sensing image data set ; The processing unit is configured with a New-Known model for open world object detection, a remote sensing image dataset The New-Known model is inputted, which processes a remote sensing image dataset ; The output unit outputs category information and positioning information.
Citation Information
Patent Citations
Target detection method and device based on annealing type label transfer learning
CN116152721A
Remote sensing target detection method for open world
CN118196616A