A Weakly Supervised Target Detection Method for High-Resolution Remote Sensing Images Based on Segmentation Model and Pseudo-Fully Supervised Learning

By using a segmentation model and pseudo-fully supervised learning, and adjusting the loss function with pseudo-ground instances, the problem of low target candidate box localization accuracy in high-resolution remote sensing images is solved, achieving more efficient target detection results.

CN118397345BActive Publication Date: 2026-11-10ZHENGZHOU UNIVERSITY OF LIGHT INDUSTRY
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410500327.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-24
Publication Date
2026-11-10
Estimated Expiration
2044-04-24

AI Technical Summary

Technical Problem

In existing weakly supervised target detection methods for high-resolution remote sensing images, the target candidate box localization accuracy is low, and the reliance on bounding box annotation increases the cost and time of data annotation, making it difficult to effectively cover the complete target in the image.

Method used

We employ a method based on the segmentation-all model and pseudo-fully supervised learning. By matching the bounding boxes guided by the segmentation-all model with the intersection-union ratio of seed instances, we can discover high-quality seed instances. We then use pseudo-ground instance to replace the original pseudo-ground instance, adjust the loss function, and improve the accuracy of object detection.

Benefits of technology

It significantly improves the accuracy and precision of weakly supervised target detection in high-resolution remote sensing images, reduces the dependence on annotation costs, and can more accurately cover and identify targets of interest in images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118397345B_ABST
    Figure CN118397345B_ABST
Patent Text Reader

Abstract

This invention proposes a weakly supervised target detection method for high-resolution remote sensing images based on a segmentation-everything model and pseudo-fully supervised learning. The steps include: S1: Constructing a weakly supervised deep detection network and obtaining its loss function, instance classification optimization branch loss function, and bounding box regression branch loss function; S2: Constructing a pseudo-fully supervised target detection head based on an online instance optimizer and bounding box regression branch, and mining seed instances; S3: Updating the seeds and propagating labels; S4: Replacing the original pseudo-ground value instances generated by the original method with pseudo-ground value instances to obtain the loss function of the PFSOD Head; S5: Establishing the overall loss function for training, resulting in a trained weakly supervised target detection model. This application improves the detection accuracy of weakly supervised target detection by designing a pseudo-ground value mining strategy based on a segmentation-everything model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning. Background Technology

[0002] High-resolution remote sensing image target detection is one of the most important tasks in the field of remote sensing image processing, aiming to identify and locate high-value ground features in high-resolution remote sensing images. In the military field, this technology can be used for reconnaissance of enemy military targets, such as monitoring important facilities like ports, airports, and military bases, providing crucial intelligence support for military decision-making. In the civilian field, remote sensing image target detection can be used in urban planning, including land use, transportation planning, and urban expansion monitoring, providing a scientific basis for urban planning. Therefore, remote sensing image target detection plays an extremely important role in both military and civilian applications.

[0003] Fully supervised object detection methods for high-resolution remote sensing images require class and location labels for every object instance in the image to supervise model training. This relies on a large number of fully labeled samples, making the labeling process both time-consuming and labor-intensive. In contrast, weakly supervised object detection methods based on deep learning only require image-level class labeling, significantly reducing labeling costs and thus attracting increasing attention.

[0004] Despite the advancements in weakly supervised object detection methods for remote sensing images, driven by the development of deep learning, two major challenges remain: First, most existing methods rely solely on class confidence scores to mine seed instances. However, given the significant intra-class diversity and inter-class similarity in remote sensing images, seed instances obtained using class confidence scores are unreliable. Furthermore, target candidate boxes with high class confidence scores typically only cover the most salient parts of the target, leading the model to overemphasize the most salient region. Second, existing methods largely generate target candidate boxes through selective search or edge boxes. However, remote sensing images have more complex backgrounds than natural scene images. Additionally, considering that selective search or edge boxes are early non-deep learning methods, the resulting target candidate boxes have relatively low localization accuracy, which limits the development of weakly supervised object detection to some extent.

[0005] For example, invention patent application number 202310845480.0 discloses a weakly supervised pig image instance segmentation method based on BoxInst. This method uses a trained BoxInst segmentation model to segment pig image instances. The training of the BoxInst segmentation model further includes: constructing a weakly supervised dataset; training the weakly supervised dataset using the constructed BoxInst segmentation model to obtain the trained BoxInst segmentation model; the constructed BoxInst segmentation model consists of a CotSENet backbone extraction network, an FPN layer, a ControllerHead layer, and a MaskBranch layer. The feature maps extracted by the CotSENet backbone extraction network are passed to the FPN layer for multi-scale fusion, object detection is performed through the ControllerHead layer, and final pixel prediction is performed through the MaskBranch layer. Gradient backpropagation is performed through ProjectionLoss and PairwiseaffinityLoss to finally achieve weakly supervised instance segmentation based on bounding boxes. This invention is used to segment pigs in pig images, reducing the cost of creating datasets while effectively segmenting pigs from images. However, compared to this application, a major drawback of BoxInst segmentation in that patent is its reliance on bounding box annotations as supervisory information. While this annotation method is simpler than pixel-level annotation, it still requires some manual intervention, increasing the cost and time of data annotation. Furthermore, bounding box annotations may not provide sufficient detail, limiting the accuracy of the segmentation results. In contrast, the SegmentAll model offers higher accuracy and flexibility when handling image segmentation tasks. Summary of the Invention

[0006] To address the low accuracy of target candidate box localization in existing target detection methods, this invention proposes a weakly supervised target detection method for high-resolution remote sensing images based on a segmentation-all model and pseudo-fully supervised learning. The pseudo-ground value mining strategy based on the segmentation-all model can obtain more reliable pseudo-ground value instances, thereby supervising the training of the pseudo-fully supervised target detection head and effectively improving the accuracy of weakly supervised target detection in high-resolution remote sensing images.

[0007] To achieve the above objectives, the technical solution of this invention is as follows: a method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning, comprising the following steps:

[0008] S1: Construct a weakly supervised deep detection network to obtain a weakly supervised object detection model. Add a classification optimization branch and a bounding box regression branch to the weakly supervised object detection model to obtain the loss function of the weakly supervised deep detection network, the instance classification optimization branch loss function, and the bounding box regression branch loss function.

[0009] S2: Construct a pseudo-fully supervised object detection head based on the online instance optimizer and bounding box regression branch, calculate the object integrity score and object quality score, and mine seed instances according to the seed mining strategy of MIST;

[0010] S3: Use the bounding box guided by the segmentation model to match the seed instance with the intersection-union ratio to update the seed and propagate the label to obtain pseudo-true value instances with the same seed category.

[0011] S4: Replace the original pseudo-true value instances generated by the original method with pseudo-true value instances, adjust the loss function of the instance classification optimization branch and the bounding box regression branch to obtain the loss function of PFSOD Head;

[0012] S5: Establish the overall loss function. Train the loss function of the weakly supervised deep detection network and the loss function of PFSOD Head to obtain the trained weakly supervised object detection model. Feed the image to be detected into the trained weakly supervised object detection model to obtain the category and location of the target of interest in the image.

[0013] The specific steps for obtaining the weakly supervised target detection model in step S1 are as follows:

[0014] S11: Generate target candidate boxes using a selective search algorithm;

[0015] S12: Use the backbone network of the detection network to process the high-resolution remote sensing image and the corresponding target candidate box to obtain the feature vector of the target candidate box;

[0016] S13: Add a classification optimization branch and a bounding box regression branch to the weakly supervised object detection model. By feeding the feature vector of the target candidate box into the two parallel branches of the detection network, the loss function of the weakly supervised deep detection network, the instance classification optimization branch loss function, and the bounding box regression branch loss function are obtained.

[0017] The specific method for obtaining the instance classification optimization branch loss function and the bounding box regression branch loss function in step S13 is as follows:

[0018] The target candidate bounding box vectors are fed into two parallel branches of the detection network. Each branch consists of a fully connected layer and a softmax classifier, resulting in two score matrices x. c ,

[0019] Category confidence score matrix It can be obtained through the following formula:

[0020] x = σ c (x c)⊙σ d (x d );

[0021] Where, σ c σ d Represents the softmax operation along the category and target candidate box dimensions, respectively; ⊙ represents the element-wise dot product operation; N represents the number of target candidate boxes; and φ represents the image-level prediction score for category c. c The calculation method is as follows:

[0022]

[0023] Where C represents the number of categories, x c,n ∈x represents the target candidate box r n The confidence score on category c is used to supervise the training of the weakly supervised deep detection network using image-level labels, which indicate whether a specific target category label exists in the image. The loss function L of the weakly supervised deep detection network is calculated using the image-level labels. w for:

[0024]

[0025] Among them, y c =1 or 0 is the image-level label of the input image, indicating whether the input image contains a target belonging to category c; K classification optimization branches are added to the weakly supervised deep detection network, each of which includes a fully connected layer and a softmax operation in the category direction;

[0026] Each classification optimization branch outputs a target candidate box category confidence score matrix, labeled as... Where k is the number of branches, k∈{1,2,...,K}, Represents the target candidate box r n The target candidate box category confidence score in category c, where class (C+1) represents the background class.

[0027] Optimize the branch loss function using instance classification. Training the k-th classification optimization branch:

[0028]

[0029] in, This indicates that r is under the k-th classification optimization branch category c. n Category confidence score, This represents the pseudo-label of the k-th classification optimization branch, when the target candidate box r n When a pseudo-true instance belongs to category c in the k-th instance classification optimization branch, otherwise Represents the target candidate box r n loss weights, This represents the highest target candidate box class confidence score in class c in the (k-1)th classification optimization branch;

[0030] Add a bounding box regression branch to each classification optimization branch, and use pseudo-ground values ​​to supervise the bounding box regression branch to obtain the bounding box regression branch loss function. As shown in the following formula:

[0031]

[0032] in, Denotes the set of positive samples. t represents the number of positive samples, Smooth L1(·) represents the Smooth L1 loss function, and t n and These represent the predicted offset and the actual offset of the position and size of the target candidate box n in the k-th bounding box regression branch, respectively.

[0033] The specific method for mining seed instances as described in step S2 is as follows:

[0034] S21: Construct a pseudo-fully supervised object detection head based on the online instance optimizer and bounding box regression branch to obtain the object integrity score;

[0035] S22: Dynamically weight and fuse the target integrity score and the traditional category confidence score to obtain the target quality score;

[0036] S23: Use target quality scores instead of traditional category confidence scores to mine seed instances according to MIST’s seed mining strategy.

[0037] The specific method for constructing a pseudo-fully supervised object detection head based on the online instance optimizer and bounding box regression branch in step S21 is as follows:

[0038] High-resolution remote sensing images are input into the all-encompassing segmentation model to obtain segmentation maps for each image. Rectangular bounding boxes enclosing each segmented region are obtained using the pixel coordinates of its top, bottom, leftmost, and rightmost points. These bounding boxes, guided by the all-encompassing model, tightly cover each segmented region. The intersection-union ratio (IUR) between the IUR-guided bounding boxes and the target candidate boxes is calculated. For each target candidate box, the IUR-guided bounding box with the highest IUR is selected, yielding the target integrity score (OCS). n As shown in the following formula:

[0039]

[0040] Where B represents the set of all bounding box samples guided by the segmentation model, |B| represents the number of bounding box samples guided by the segmentation model, and b j Let r represent the j-th bounding box in set B that is part of the model-guided partitioning of everything. n Represents the nth target candidate box in R, max{·} represents the operation of taking the maximum value, IoU(·,·) represents the intersection-union ratio operation between the target candidate box and the bounding box guided by the segmentation model, and OCS n This represents the target integrity score of the nth target candidate box.

[0041] The specific method for obtaining the target quality score in step S22 is as follows:

[0042] The target quality score is obtained by dynamically fusing the target integrity score and the traditional category confidence score, as shown in the following formula:

[0043]

[0044] Where α is the weighting coefficient, and K represents the number of PFSOD Heads. This represents the confidence score that the nth target candidate box in the kth PFSOD Head belongs to category c. When k>1, Represents the target candidate box r in the (k-1)th PFSOD Head. n The category confidence score for the c-th category, when k=1. Represents the target candidate box r in the basic classification network n The category confidence score in the c-th category, The weighting coefficient α represents the target quality score of the target candidate box in the k-th PFSOD Head in the c-th category, and is defined as follows:

[0045]

[0046] Where n1, 30000, and 200000 represent the current iteration number, the threshold iteration number, and the total iteration number, respectively.

[0047] The specific method for mining seed instances according to the MIST seed mining strategy described in step S23 is as follows:

[0048] Seed instance mining is performed using target quality score instead of traditional category confidence score. According to MIST's seed mining strategy, if a target quality score is in the top p percent, then the target candidate box r... nIt will be selected as the initial seed instance belonging to category c in the k-th instance classification optimization branch; if the target candidate box r n If the intersection-union ratio (CUC) between the target candidate box r and other initial seed pseudo-ground instance is less than the threshold TH, then r is a target candidate box r. n It will be selected as the final seed instance.

[0049] The specific method for obtaining the pseudo-truth instance with the same seed category as described in step S3 is as follows:

[0050] The bounding boxes guided by the segmentation-all model are matched with seed instances, and the intersection-union ratio (IU) between them is calculated. When the IU is less than a set threshold, it means that the seed instance has not matched the corresponding segmentation-all model-guided bounding box. When the IU is greater than the set threshold and multiple seed instances match the same segmentation-all model-guided bounding box, the seed instance with the largest IU with the segmentation-all model-guided bounding box is selected for matching. Finally, the category information of the seed instance is assigned to the segmentation-all model-guided bounding box, completing the seed instance update. Label propagation is performed using the updated seed instances. Target candidate boxes with IU in the range [0.5, 1] ​​with seed instances are marked as pseudo-ground value instances with the same category as the seed instance.

[0051] The specific method for replacing the original pseudo-truth instance generated by the original method with a pseudo-truth instance in step S4 is as follows:

[0052] Since pseudo-truth instances are replaced with pseudo-truth instances generated by the original method, the corresponding loss function should also be adjusted as follows:

[0053]

[0054]

[0055] Where, when the target candidate box r n belong At that time, otherwise Loss function of the k-th PFSOD Head for:

[0056]

[0057] The overall loss function described in step S5 includes: the training loss L of the weakly supervised deep detection network. w Training loss of K PFSODHeads

[0058]

[0059] The beneficial effects of this invention are as follows: A series of target candidate boxes are generated for the image using a selective search algorithm. The category scores of the target candidate boxes and the image's category prediction score are calculated sequentially to train a weakly supervised deep detection network. The target candidate boxes are then subjected to intersection-union (OUI) operations with the segmentation boxes guided by the "all-inclusive" segmentation model to obtain target integrity and target quality scores. Seed instances for pseudo-fully supervised target detection heads are mined based on the target quality scores. These seed instances are then updated by performing OUI operations with the bounding boxes guided by the "all-inclusive" segmentation model. Subsequently, label propagation is performed to obtain pseudo-ground truth instances. The training of the pseudo-fully supervised target detection head is supervised based on these pseudo-ground truth instances. This invention, through its designed seed instance mining strategy guided by the "all-inclusive" segmentation model, can mine high-quality seed instances. By designing a novel pseudo-ground truth mining strategy based on the "all-inclusive" segmentation model, it can obtain high-quality pseudo-ground truth instances and supervise the training of the pseudo-fully supervised target detection head, thereby effectively improving the detection accuracy of weakly supervised target detection in high-resolution remote sensing images. This invention can effectively improve the robustness of pseudo-label mining and significantly enhance the accuracy of weakly supervised target detection in images. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram of the target detection process of the present invention.

[0062] Figure 2 This is a visualization of the detection results of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] like Figure 1 As shown, the specific steps of the weakly supervised target detection method for high-resolution remote sensing images based on the segmentation model and pseudo-fully supervised learning are as follows:

[0065] S1: Construct a weakly supervised deep detection network to obtain a weakly supervised object detection model. Add a classification optimization branch and a bounding box regression branch to the weakly supervised object detection model to obtain the loss function of the weakly supervised deep detection network, the instance classification optimization branch loss function, and the bounding box regression branch loss function.

[0066] S11: Generate target candidate boxes using a selective search algorithm.

[0067] Specifically, the selective search algorithm proposed in the paper "Selective search for object recognition" is used to generate N target candidate boxes R = {r1,...,r...} for each high-resolution remote sensing image. n ,...,r N}, where N represents the number of target candidate boxes, and the number of target candidate boxes N is approximately 2000.

[0068] The high-resolution remote sensing image dataset used in this invention is the DIOR dataset, which contains a total of 20 target classes. Each image is 800×800 pixels in size. The training set, validation set, and test set contain 5862, 5863, and 11738 images, respectively. In this embodiment, the training set and validation set are used to train the weakly supervised target detection model, and the test set is used to test the weakly supervised target detection model.

[0069] S12: Use the backbone network of the detection network to process the high-resolution remote sensing image and the corresponding target candidate box to obtain the feature vector of the target candidate box.

[0070] High-resolution remote sensing images and corresponding target candidate boxes are fed into the backbone of the detection network to obtain feature maps of the high-resolution remote sensing images. Then, region-of-interest (ROI) pooling is used to obtain feature maps of the target candidate boxes. Finally, two fully connected layers are used to obtain feature vectors of the target candidate boxes. Specifically, the backbone network is the convolutional part of the weakly supervised object detection VGG-16 network architecture. Utilizing the feature maps obtained through the backbone network significantly improves the accuracy and efficiency of object detection. Utilizing RIO pooling effectively improves the accuracy and robustness of object detection while reducing computational cost and increasing detection efficiency.

[0071] S13: Add a classification optimization branch and a bounding box regression branch to the weakly supervised object detection model. By feeding the feature vector of the target candidate box into the two parallel branches of the detection network, the loss function of the weakly supervised deep detection network, the instance classification optimization branch loss function, and the bounding box regression branch loss function are obtained.

[0072] The target candidate bounding box vectors are fed into two parallel branches of the detection network. Each branch consists of a fully connected layer and a softmax classifier, resulting in two score matrices x. c ,

[0073] Category confidence score matrix It can be obtained through equation (1):

[0074] x = σ c (x c )⊙σ d (x d (1)

[0075] Where, σ c σ d φ represents the softmax operation along the category and target candidate box dimensions, respectively, and ⊙ represents the element-wise dot product operation. The image-level prediction score φ for category c is... c The calculation method is as follows:

[0076]

[0077] Where C represents the number of categories, x c,n ∈x represents the target candidate box r n Confidence score on category c. Training of the weakly supervised deep detection network is supervised using image-level labels. Image-level labels are used to indicate the presence of a specific target category label in the image.

[0078] Using image-level labels can reduce data annotation costs. The loss function calculated using image-level labels can be used to optimize the detection model and improve its detection accuracy. The loss function L used in this training is... w The definition is as follows:

[0079]

[0080] Among them, y c =1 or 0 is the image-level label of the input image, indicating whether the input image contains a target belonging to category c.

[0081] Specifically, K instance classifier refinement (ICR) branches are added to the weakly supervised deep detection network. Each ICR branch includes a fully connected layer and a softmax operation in the class direction. Utilizing instance classifier refinement branches can further improve the accuracy and robustness of object detection, enhance the network's adaptability to object variations, and reduce the model's sensitivity to outlier data to some extent. Each ICR branch outputs a target candidate box class confidence score matrix, labeled as... Where k is the number of branches, k∈{1,2,...,K}, Represents the target candidate box r n The confidence score of the target candidate box in category c, where the (C+1)th class represents the background class.

[0082] Using the ICR loss function Training the k-th ICR branch:

[0083]

[0084] in, This indicates that r is under the k-th ICR branch category c. n Category confidence score, This represents the pseudo-label of the k-th ICR branch, when the target candidate box r n When a pseudo-true instance belongs to category c in the k-th instance classification optimization branch, otherwise Represents the target candidate box r n loss weights, This represents the highest target candidate box category confidence score in category c within the (k-1)th ICR branch.

[0085] A bounding box regression (BBR) branch is added to each ICR branch to further improve detector performance. The pseudo-ground truth instances generated by the above method are used to supervise the training of the BBR branch, and the loss function is obtained by using the pseudo-ground truth to supervise the BBR branch. The model is optimized through backpropagation of the loss function to improve the localization accuracy of object detection. BBR branch loss function. As shown in the following formula:

[0086]

[0087] in, Denotes the set of positive samples. t represents the number of positive samples, Smooth L1(·) represents the Smooth L1 loss function, and t n and These represent the predicted offset and the actual offset of the position and size of the target candidate box n in the k-th BBR branch, respectively.

[0088] S2: Construct a pseudo-fully supervised object detection head based on the online instance optimizer and bounding box regression branch, calculate the object integrity score and object quality score, and mine seed instances according to the seed mining strategy of MIST.

[0089] S21: Construct a pseudo-fully supervised target detection head composed of OICR+BBR to obtain the target integrity score.

[0090] like Figure 1 As shown, this application adopts a pseudo-fully-supervised object detection head (PFSOD Head) composed of "OICR+BBR" (Online Instance Classifier Refinement; Bounding Box Regression) as the basic framework of this invention. Utilizing the pseudo-fully-supervised object detection head composed of OICR+BBR can improve localization accuracy, optimize instance representation, enhance detection performance, and reduce annotation costs, providing an effective and practical solution for object detection tasks in real-world applications.

[0091] First, high-resolution remote sensing images are input into the Segment Anything Model to obtain segmentation maps for each image. The Segment Anything Model is publicly available in the IEEE / CVF International Conference on Computer Vision proceedings (Kirillov, Alexander, et al. "Segment Anything." Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023). Next, using the top, bottom, leftmost, and rightmost pixel coordinates of each segmented region, a bounding box enclosing each region is obtained, resulting in a Segment Anything Model-Induced Bounding Box (SAMB) that tightly covers each segmented region. Finally, the intersection-union ratio (OCS) between the SAMB and the target candidate boxes is calculated. For each target candidate box, the SAMB with the largest OCS is selected to obtain the target integrity score (OCS) of that candidate box. n As shown in the following formula:

[0092]

[0093] Where B represents the set of all SAMBs, |B| represents the number of SAMBs, and b j Represents the j-th SAMB in set B, r n Represents the nth target candidate box in R, max{·} represents the operation of taking the maximum value, IoU(·,·) represents the intersection-union ratio operation between the target candidate box and SAMB, and OCS nThis represents the target integrity score of the nth target candidate box. Specifically, SAMB belongs to SAMBs, where SAMB represents a bounding box guided by a segmentation-all model, and SAMBs represents a sample set of bounding boxes guided by a segmentation-all model.

[0094] S22: Dynamically weight and fuse the target integrity score and the traditional category confidence score to obtain the target quality score.

[0095] The Object Quality Score (OQS) is obtained by fusing the Objective Integrity Score (OCS) and the traditional Class-Confidence Score (CS) with dynamic weighting, as shown in the following formula:

[0096]

[0097] Where α is the weighting coefficient, and K represents the number of PFSOD Heads. This represents the confidence score that the nth target candidate box in the kth PFSOD Head belongs to category c. When k>1, Represents the target candidate box r in the (k-1)th PFSOD Head. n The category confidence score for the c-th category, when k=1. Represents the target candidate box r in the basic classification network n The category confidence score in the c-th category, This represents the target quality score of the target candidate box in the k-th PFSOD Head in the c-th category. The target quality score (OQS) obtained by dynamically weighting and fusing the target integrity score (OCS) and the traditional category confidence score (OCS) yields seeds that possess both accurate category information and complete target coverage.

[0098] The weighting coefficient α is defined as:

[0099]

[0100] Where n1, 30000, and 200000 represent the current iteration number, the threshold iteration number, and the total iteration number, respectively.

[0101] S23: Utilizing Target Quality Score Instead of the original category confidence score, seed instances are mined according to MIST's seed mining strategy.

[0102] The target quality score generated using the above method Instead of the original category confidence score, seed instance mining is performed using the MIST seed mining strategy proposed in the paper "Instance-aware, context-focused, and memory-efficient weakly supervised object detection". If the target quality score... If the target candidate box is in the top p percent, then r n It will be selected as the initial seed instance belonging to category c in the k-th instance classification optimization branch; if the target candidate box r n If the intersection-union ratio (CUC) between the target candidate box r and other initial seed pseudo-ground instance is less than the threshold TH, then r is a target candidate box r. n It will be selected as the final seed instance.

[0103] According to the seed mining strategy proposed by MIST, if the target quality score If it is in the top 15%, then the target candidate box r n It will be selected as the initial seed instance belonging to category c in the k-th instance classification optimization branch; if the target candidate box r n If the intersection-union ratio (IUU) between the target candidate box r and other initial seed pseudo-ground instance is less than a threshold of 12%, then the target candidate box r is considered to be... n It will be selected as the final seed instance.

[0104] S3: Use the bounding box guided by the segmentation model to match the seed instance with the intersection-union ratio to update the seed, and propagate the label to obtain pseudo-ground value instances with the same seed category.

[0105] Seed instances mined by traditional weakly supervised object detection methods have the advantage of containing accurate category information, but their localization ability is insufficient, focusing only on the most salient part of the target and failing to cover the entire target. SAMB obtained through SAM segmentation has accurate localization ability and can contain the complete target, but lacks category information. This paper proposes to combine the advantages of both methods through Intersection over Union (IoU) matching. First, SAMB is used to match seed instances, and the IoU is calculated. When the IoU is less than a set threshold, it means that the seed instance has not matched the corresponding SAMB. When the IoU is greater than the set threshold and multiple seed instances match the same SAMB, the seed instance with the highest IoU is selected for matching. Finally, the category information of the seed instance is assigned to SAMB, completing the seed instance update. Seed instances generated by traditional weakly supervised object detection algorithms usually only contain the most salient part of the target and cannot completely cover the target. The method proposed in this application leverages the advantage of the segmentation model's ability to completely cover the target, combining seed instances and SAMB to obtain seed instances that can cover the entire target.

[0106] Label propagation is performed using the updated seed instance; that is, target candidate boxes with an intersection-union ratio (IU) with the seed instance in the interval [0.5, 1] ​​are labeled as pseudo-ground value instances of the same seed category.

[0107] S4: Replace the original pseudo-true value instances generated by the original method with pseudo-true value instances, adjust the loss function of the instance classification optimization branch and the bounding box regression branch to obtain the loss function of PFSOD Head.

[0108] The pseudo-truth instances used when calculating the loss functions of the ICR and BBR branches have changed; that is, the pseudo-truth instances generated by this method are now used instead. The pseudo-ground value instances generated by the proposed method, compared to traditional weakly supervised object detection methods, can more accurately locate and enclose complete targets, significantly improving the model's localization and detection accuracy. The corresponding loss functions should also be adjusted; that is, equations (4) and (5) are adjusted to equations (8) and (9), respectively.

[0109]

[0110]

[0111] Where, when the target candidate box r n belong At that time, otherwise Loss function of the k-th PFSOD Head for:

[0112]

[0113] S5: Establish the overall loss function. Train the loss function of the weakly supervised deep detection network and the loss function of PFSOD Head to obtain the weakly supervised object detection model. Feed the image to be detected into the trained weakly supervised object detection model to obtain the category and location of the target of interest in the image.

[0114] The overall training loss function L of the weakly supervised object detection model proposed in this application consists of two parts: the training loss L of the weakly supervised deep detection network. w The training loss for K PFSOD Heads is as follows:

[0115]

[0116] The training of the weakly supervised deep detection network and the training of K PFSOD Heads are performed simultaneously to complete the training of the entire weakly supervised object detection model. During the inference phase, the image to be detected is fed into the trained weakly supervised object detection model to obtain the category and location of the target of interest in the image. By backpropagating the overall loss function, the model is continuously optimized to improve its detection and localization accuracy.

[0117] The hardware configuration for implementing this invention is as follows: a workstation with an E5-2650V4 CPU (2.2GHz 12x2 cores), 512GB of memory, and 8 NVIDIA RTX Titan graphics cards was used for the experiment. The software platform configuration is as follows: Ubuntu 16.04, Python 3.7, and PyTorch 1.7.

[0118] To better demonstrate the performance of this invention, as shown in Table 1, this invention is compared with 15 popular algorithms on the DIOR dataset. The 15 algorithms are: WSDDN, OICR, PCL, MELM, DCL, MIST, PCIR, TCA, MIG, SAE, SPG, MHQ-PSL, SGPLM-IR, RINet, and AE-IS. The comparison results are shown in Table 1.

[0119] Among them, the paper "Weakly supervised deep detection networks" proposes the WSDDN algorithm; the paper "Multiple instance detection network with online instance classifier refinement" proposes the OICR algorithm; the paper "PCL: proposal cluster learning for weakly supervised object detection" proposes the PCL algorithm; the paper "Min-entropy latent model for weakly supervised object detection" proposes the MELM algorithm; the paper "Automatic weakly supervised object detection from high spatial resolution remote sensing images via dynamiccurriculum learning" proposes the DCL algorithm; the paper "Instance-aware, context-focused, and memory-efficient weakly supervised object detection" proposes the MIST algorithm; and the paper "Progressive contextual instance refinement" proposes the MIST algorithm. The PCIR algorithm is proposed in the paper "Progressive Context Instance Refinement for Weakly Supervised Object Detection in Remote Sensing Images"; the TCA algorithm is proposed in the paper "Tcanet: Triple Context-Aware Network for Weakly Supervised Object Detection in Remote Sensing Images".The MIG algorithm is proposed in the paper "Multiple instance graph learning for weakly supervised remote sensing object detection"; the SAE algorithm is proposed in the paper "Saenet: Self-supervised adversarial and equivariant network for weakly supervised object detection in remote sensing images"; the SPG algorithm is proposed in the paper "Self-guided proposal generation for weakly supervised object detection"; the MHQ-PSL algorithm is proposed in the paper "Mining high-quality pseudoinstance soft labels for weakly supervised object detection in remote sensing images"; and the MHQ-PSL algorithm is proposed in the paper "Semantic segmentation guided pseudo label mining and instance re-detection for weakly supervised object detection in remote sensing". The SGPLM-IR algorithm is proposed in the paper "Images (Application of pseudo-label mining and instance re-detection based on semantic segmentation in weakly supervised object detection of remote sensing images)"; the RINet algorithm is proposed in the paper "Weakly supervisedrotation-invariant aerial object detection network"; and the AE-IS algorithm is proposed in the paper "Attention Erasing and Instance Sampling for Weakly Supervised Object Detection".

[0120] Table 1 compares the average accuracy and positioning accuracy of this invention with 15 popular algorithms on the DIOR dataset.

[0121] method mAP CorLoc WSDDN 13.3 32.4 OICR 16.5 34.8 PCL 18.2 41.5 MELM 18.7 43.3 DCL 20.2 42.2 MIST 22.2 43.6 PCIR 24.9 46.1 TCA 25.8 48.4 MIG 25.1 46.8 SAE 27.1 49.4 SPG 25.8 48.3 MHQ-PSL 28.6 53.2 SGPLM-IR 28.5 53.2 RINet 28.3 52.8 AE-IS 29.1 55.9 This invention 30.9 56.4

[0122] As can be seen from Table 1, compared with the other 15 weakly supervised target detection methods, the method of the present invention can more accurately identify and locate ground targets in remote sensing images.

[0123] Figure 2 To utilize the detection results of the weakly supervised object detection method proposed in this application on the DIOR dataset, by Figure 2 It can be seen that this invention can accurately locate ground targets. The benchmark method adopts "OICR+BBR", which takes advantage of the ability of the segmentation model to accurately locate targets and integrates it into weakly supervised target detection. By comparing with the benchmark method, the visualization results more intuitively show that this method can achieve complete coverage of ground targets and effectively improve the performance of weakly supervised target detection algorithms.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A weakly supervised target detection method for high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning, characterized in that, The specific steps include: S1: Construct a weakly supervised deep detection network to obtain a weakly supervised object detection model. Add a classification optimization branch and a bounding box regression branch to the weakly supervised object detection model to obtain the loss function of the weakly supervised deep detection network, the instance classification optimization branch loss function, and the bounding box regression branch loss function. S2: Construct a pseudo-fully supervised object detection head based on the online instance optimizer and bounding box regression branch, calculate the object integrity score and object quality score, and mine seed instances according to the seed mining strategy of MIST; S3: Use the bounding box guided by the segmentation model to match the seed instance with the intersection-union ratio to update the seed and propagate the label to obtain pseudo-ground value instances with the same seed category. S4: Replace the original pseudo-true value instances generated by the original method with pseudo-true value instances, adjust the loss function of the instance classification optimization branch and the bounding box regression branch to obtain the loss function of PFSOD Head; S5: Establish the overall loss function, train the loss function of the weakly supervised deep detection network and the loss function of PFSOD Head to obtain the trained weakly supervised object detection model, and feed the image to be detected into the trained weakly supervised object detection model to obtain the category and location of the target of interest in the image; The specific method for obtaining the pseudo-truth instance with the same seed category as described in step S3 is as follows: The bounding boxes guided by the segmentation-all model are matched with seed instances, and the intersection-union ratio (IU) between them is calculated. When the IU is less than a set threshold, it means that the seed instance has not matched the corresponding segmentation-all model-guided bounding box. When the IU is greater than the set threshold and multiple seed instances match the same segmentation-all model-guided bounding box, the seed instance with the largest IU with the segmentation-all model-guided bounding box is selected for matching. Finally, the category information of the seed instance is assigned to the segmentation-all model-guided bounding box, completing the seed instance update. Label propagation is performed using the updated seed instances. Target candidate boxes with IU in the range [0.5, 1] ​​with seed instances are marked as pseudo-ground value instances with the same category as the seed instance. .

2. The method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning as described in claim 1, characterized in that, The specific steps for obtaining the weakly supervised target detection model in step S1 are as follows: S11: Generate target candidate boxes using a selective search algorithm; S12: Use the backbone network of the detection network to process the high-resolution remote sensing image and the corresponding target candidate box to obtain the feature vector of the target candidate box; S13: Add a classification optimization branch and a bounding box regression branch to the weakly supervised object detection model. By feeding the feature vector of the target candidate box into the two parallel branches of the detection network, the loss function of the weakly supervised deep detection network, the instance classification optimization branch loss function, and the bounding box regression branch loss function are obtained.

3. The method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning as described in claim 2, characterized in that, The specific method for obtaining the instance classification optimization branch loss function and the bounding box regression branch loss function in step S13 is as follows: The target candidate bounding box vectors are fed into two parallel branches of the detection network. Each branch consists of a fully connected layer and a softmax classifier, resulting in two score matrices. ; Category confidence score matrix It can be obtained through the following formula: ; in, , These represent the softmax operations along the category and target candidate box dimensions, respectively. This represents the element-wise dot product operation, where N represents the number of target candidate boxes, and c represents the image-level prediction score for category c. The calculation method is as follows: ; Where C represents the number of categories, Represents the target candidate box The confidence score on category c is used to supervise the training of the weakly supervised deep detection network using image-level labels, which indicate whether a specific target category exists in the image. The loss function of the weakly supervised deep detection network is then calculated using the image-level labels. for: ; in, It is the image-level label of the input image, indicating whether the input image contains a target belonging to category c; K classification optimization branches are added to the weakly supervised deep detection network, each of which includes a fully connected layer and a softmax operation in the category direction; Each classification optimization branch outputs a target candidate box category confidence score matrix, labeled as... Where k is the number of branches, , Represents the target candidate box The target candidate box category confidence score in category c, where class (C+1) represents the background class. Optimize the branch loss function using instance classification. Training the k-th classification optimization branch: ; in, This indicates that under the k-th classification optimization branch category c Category confidence score, This represents the pseudo-label of the k-th classification optimization branch, when the target candidate box... When a pseudo-true instance belongs to category c in the k-th instance classification optimization branch, =1, otherwise =0, Represents the target candidate box loss weights, This represents the highest target candidate box class confidence score in class c in the (k-1)th classification optimization branch; Add a bounding box regression branch to each classification optimization branch, and use pseudo-ground values ​​to supervise the bounding box regression branch to obtain the bounding box regression branch loss function. As shown in the following formula: ; in, Denotes the set of positive samples. Indicates the number of positive samples. Describes the Smooth L1 loss function. and These represent the predicted offset and the actual offset of the position and size of the target candidate box n in the k-th bounding box regression branch, respectively.

4. The method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning as described in claim 3, characterized in that, The specific method for mining seed instances as described in step S2 is as follows: S21: Construct a pseudo-fully supervised object detection head based on the online instance optimizer and bounding box regression branch to obtain the object integrity score; S22: Dynamically weight and fuse the target integrity score and the traditional category confidence score to obtain the target quality score; S23: Use target quality scores instead of traditional category confidence scores to mine seed instances according to MIST’s seed mining strategy.

5. The method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning as described in claim 4, characterized in that, The specific method for constructing a pseudo-fully supervised object detection head based on the online instance optimizer and bounding box regression branch in step S21 is as follows: High-resolution remote sensing images are input into the all-encompassing segmentation model to obtain segmentation maps for each image. Rectangular bounding boxes enclosing each segmented region are obtained using the pixel coordinates of its top, bottom, leftmost, and rightmost points. These bounding boxes, guided by the all-encompassing model, tightly cover each segmented region. The intersection-union ratio (IUR) between the IUR-guided bounding boxes and the target candidate boxes is calculated. For each target candidate box, the IUR-guided bounding box with the highest IUR is selected, resulting in a target integrity score. As shown in the following formula: ; Where B represents the set of all bounding box samples that segment the entire model. This represents the number of bounding box sample sets that segment all models guided by the model. This represents the j-th bounding box in set B that is guided by the model that divides everything. This represents the nth target candidate box in R. This represents the operation of retrieving the maximum value. This represents the intersection-union (IoU) operation between the target candidate bounding box and the bounding box guided by the segmentation model. This represents the target integrity score of the nth target candidate box.

6. The method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning as described in claim 5, characterized in that, The specific method for obtaining the target quality score in step S22 is as follows: The target quality score is obtained by dynamically fusing the target integrity score and the traditional category confidence score, as shown in the following formula: ; in, It is a weighting coefficient, where K represents the number of PFSOD Heads. This represents the confidence score that the nth target candidate box in the kth PFSOD Head belongs to category c. When k>1, This represents the target candidate box in the (k-1)th PFSOD Head. The class confidence score in the c-th category, when k=1. Represents the target candidate boxes in the basic classification network. The category confidence score in the c-th category, This represents the target quality score of the target candidate box in the k-th PFSOD Head in the c-th category, with weighting coefficients. Defined as: ; in, 30000 and 200000 represent the current iteration number, the threshold iteration number, and the total iteration number, respectively.

7. The method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning according to any one of claims 4-6, characterized in that, The specific method for mining seed instances according to the MIST seed mining strategy described in step S23 is as follows: Seed instance mining is performed using target quality score instead of traditional category confidence score. According to MIST's seed mining strategy, if a target quality score is in the top p percent, then the target candidate box is considered a candidate instance. It will be selected as the initial seed instance belonging to category c in the k-th instance classification optimization branch; if the target candidate box If the intersection-union ratio (CUC) between the target candidate box and other initial seed pseudo-ground instance instances is less than the threshold TH, then... It will be selected as the final seed instance.

8. The method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning as described in claim 7, characterized in that, The specific method for replacing the original pseudo-truth instance generated by the original method with a pseudo-truth instance in step S4 is as follows: Since pseudo-truth instances are replaced with pseudo-truth instances generated by the original method, the corresponding loss function should also be adjusted as follows: ; ; Among them, when the target candidate box belong At that time, =1, otherwise =0, the loss function of the k-th PFSOD Head for: 。 9. The method for weakly supervised target detection in high-resolution remote sensing images based on a segmentation model and pseudo-fully supervised learning as described in claim 8, characterized in that, The overall loss function described in step S5 includes: the training loss of the weakly supervised deep detection network. Training loss of K PFSOD Heads : 。

Citation Information

Patent Citations

  • BoxInst-based weak supervision pig image instance segmentation method, device and equipment

    CN117058369A