Weakly supervised object detection method based on multi-scale image cutting and instance difficulty

Abstract

The application provides a weakly supervised target detection method based on multi-scale image cutting and instance difficulty, and steps are as follows: input images are subjected to multi-scale cutting and sent to a shared backbone network to extract features and obtain feature maps; target candidate boxes are projected to the feature maps to obtain feature matrices of the target candidate boxes; the feature matrices are subjected to full connection layers to obtain feature vectors under each cutting scale, the feature vectors are input into a benchmark weakly supervised target detection module to obtain a score matrix and a class confidence score matrix; the feature vectors are respectively sent into multiple instance classification optimization branches to obtain class confidence scores and determine positive instances; the most class confidence score of the target candidate boxes on a background class is obtained to obtain difficult negative examples; a weakly supervised target detection model is obtained through training; and images to be detected are sent into the weakly supervised target detection model to obtain the class and position of the target. The application can effectively improve the detection accuracy of weakly supervised target detection of high-resolution remote sensing images.

Description

Technical Field

[0001] This invention relates to the technical field of object detection in deep learning, and in particular to a weakly supervised object detection method based on multi-scale image segmentation and instance difficulty. Background Technology

[0002] Object detection is an important task in the field of computer vision. Its main goal is to accurately identify and locate the position of a specific target from an image or video. For example, in the field of autonomous driving, it can automatically identify vehicles, pedestrians, zebra crossings, etc., to provide accurate information for autonomous driving systems. In the field of security monitoring, it can automatically identify sensitive targets in real time to reduce the probability of accidents. It also has wide applications in many other fields such as face detection and military reconnaissance.

[0003] With the continuous advancement of artificial intelligence, object detection based on deep learning has also developed rapidly. In fully supervised object detection, each training sample requires manual annotation of all objects in its image, using bounding boxes or pixel-level labels to indicate the location of the objects. This annotation process is often very time-consuming, labor-intensive, and costly. Weakly supervised object detection only requires image category annotation, which can save a significant amount of manpower and annotation time, thus attracting widespread attention.

[0004] With the development of deep learning, weakly supervised object detection methods based on deep learning have made great progress, but two problems still exist. The first problem is that existing weakly supervised object detection models often do not pay attention to the non-salient features of the target, causing the model's detection results to tend to locate the most salient region of the target rather than the target as a whole, especially in remote sensing images with complex backgrounds. The second challenge is the imbalance between easy and difficult samples in existing methods, i.e., the number of easy instances is much greater than the number of difficult instances. Although the loss for easy instances is smaller, the accumulation of losses from a large number of easy instances causes the network to focus too much on easy instances and ignore difficult instances, limiting further improvements in the performance of weakly supervised object detection.

[0005] Patent application number 202011481683.9 discloses a weakly supervised object detection method based on positive and negative sample balancing, including the following steps: 1) Acquire scene images for object detection, each with corresponding labels, forming a training set mainly composed of the scene images and their corresponding labels; 2) Input the training set into a target candidate box filtering module, which obtains all target candidate boxes of the scene images through a selective search method, calculates the environmental coefficients of all target candidate boxes based on the weakly supervised semantic segmentation result M corresponding to the scene images, sorts all the environmental coefficients of the target candidate boxes, and selects several target candidate boxes with the highest environmental coefficients, using the selected target candidate boxes as the initial target candidate boxes corresponding to the scene images; 3) Establish a weakly supervised object detection network, simultaneously inputting the training set and the corresponding initial target candidate boxes into the weakly supervised object detection network for training, using the optimal target box update method during training to obtain the trained weakly supervised object detection network; 4) Input the scene images to be tested into the target candidate box filtering module and the trained weakly supervised object detection network respectively, and classify and locate the targets in the scene images to be tested. The aforementioned invention only requires a dataset containing image-level category labels corresponding to the image and exhibits good object detection capabilities even without bounding box level labels. However, the network model of the aforementioned invention is relatively complex, and this weakly supervised object detection model typically does not focus on the non-salient region features of the target, causing the model's detection results to tend to locate the most salient region of the target rather than the target as a whole, especially in remote sensing images with complex backgrounds. On the other hand, the weakly supervised object detection method based on positive and negative sample balancing proposed in this patent cannot solve the problem of imbalance between the number of easy and difficult samples. Summary of the Invention

[0006] To address the technical problems of existing weakly supervised object detection methods, such as models tending to focus on the most salient parts of the target and an imbalance between the number of easy and difficult samples, this invention proposes a weakly supervised object detection method based on multi-scale image segmentation and instance difficulty. By introducing a novel feature enhancement model based on multi-scale image segmentation and a perceptual loss strategy, this method can effectively highlight the overall target region and give more attention to difficult negative examples during training, thereby effectively improving the localization accuracy of the weakly supervised object detection model.

[0007] To achieve the above objectives, the technical solution of this invention is implemented as follows: a weakly supervised target detection method based on multi-scale image segmentation and instance difficulty, comprising the following steps:

[0008] Step 1: Feature Enhancement Based on Multi-Scale Image Segmentation: The input image is segmented at multiple scales. The input image and corresponding image patches at at least two randomly selected segmentation scales are fed into a shared backbone network for feature extraction and then concatenated according to their spatial relationships to obtain feature maps at each segmentation scale. Multiple target candidate boxes are obtained from the input image. The target candidate boxes are projected onto the feature maps at each segmentation scale, and region-of-interest pooling is performed to obtain the feature matrix of the target candidate boxes at each segmentation scale. Based on the feature matrix of the target candidate boxes at each segmentation scale, the maximum spatial attention map of all positive instances on the feature maps at each segmentation scale is obtained, and the loss function of the feature enhancement module based on multi-scale image segmentation is obtained. The feature enhancement module based on multi-scale image segmentation is then trained.

[0009] Step 2: Pass the feature matrix of the target candidate box at each cutting scale through two fully connected layers to obtain the feature vector of the target candidate box at each cutting scale. Input the feature vector into the benchmark weakly supervised object detection module to obtain the class confidence score matrix. Based on the class confidence score matrix, obtain the prediction score of each class at each cutting scale. Calculate the loss function of the benchmark weakly supervised object detection module and train the benchmark weakly supervised object detection module.

[0010] Step 3: Feed the feature vectors obtained in Step 2 into multiple instance classification optimization branches to obtain the class confidence scores of the target candidate boxes at each cutting scale, and use the class confidence scores to determine the set of positive instances;

[0011] Step 4: Obtain the set of hard negative examples of the target candidate box based on the highest class confidence score among the categories that do not exist in the image. Calculate the instance difficulty score of the target candidate box at all cutting scales as the weight of the loss function. Calculate the loss function of multiple instance classification optimization branches and train multiple instance classification optimization branches.

[0012] Step 5: Repeat steps 1 to 4 to train the weakly supervised object detection model on the dataset; feed the image to be detected into the trained weakly supervised object detection model to obtain the category and location of the target of interest in the image.

[0013] Preferably, the method for extracting features from the shared backbone network in step one and concatenating them according to spatial position relationships to obtain feature maps is as follows: The input image is segmented at Z scales, where the z-th scale segmentation divides the image into z×z image blocks where z∈[1,Z]; N-1 scales are randomly selected from the 2nd to the Zth scales, where N≤Z; the input image and the image blocks corresponding to the N-1 scales are respectively fed into the shared backbone network for feature extraction to obtain feature maps of corresponding sizes; the feature maps of all image blocks at the nth segmentation scale are concatenated according to spatial position relationships to obtain F. n, n∈[1,N], F n This refers to the feature map of the input image at the nth cutting scale.

[0014] Preferably, a selective search algorithm is used to generate M target candidate boxes for the input image, and the r-th target candidate box is represented as p. r And project it onto F n Perform region-of-interest pooling operations to obtain feature maps of target candidate boxes. r∈[1,M], where H, W, and L represent feature maps respectively. Height, width, and number of channels.

[0015] The loss function of the feature enhancement module based on multi-scale image segmentation is:

[0016]

[0017] Where N is the total number of cutting scales, For positive instance t q Maximize the spatial attention graph, where max(·) represents the element-wise maximum value operation; For positive instance t q Spatial attention map at the nth cut scale Represents the positive instance t at the nth cutting scale. q The i-th channel of the feature map, i∈[1,L], where H, W, and L represent the feature map respectively. The height, width, and number of channels are given, q∈[1,|Q|] is the index of the positive instance, Q represents the set of positive instances in all instance classification optimization branches, |Q| represents the number of positive instances, and Sigmoid represents the activation function.

[0018] Preferably, the baseline weakly supervised target detection module includes two parallel branches, each consisting of a fully connected layer and a softmax classifier;

[0019] The method for obtaining the category confidence score matrix in step two is as follows:

[0020] Feature map After passing through two fully connected layers, the target candidate box p is obtained. r Feature vector at the nth cutting scale And feed them into two parallel branches in the benchmark weakly supervised object detection module; when all feature vectors Two matrices are obtained after passing through two fully connected layers with two parallel branches. Where C represents the number of target categories; and the category confidence score matrix of all target candidate boxes at the nth cut scale.

[0021] Where, σ c (.),σ d (.) represent the softmax operation along the category and target candidate box dimensions, respectively, and ⊙ represents the Hadman product.

[0022] Preferably, the loss function of the weakly supervised target detection network is...

[0023] The loss function at the nth cutting scale is

[0024] Among them, y c = 1 or 0 indicates whether the input image contains a target of the c-th category; the prediction score of the input image belonging to the c-th category at the n-th cut scale. in This represents the target candidate box p at the nth cutting scale. r Confidence score for category c.

[0025] Preferably, the method for obtaining the category confidence score of the target candidate box at the cutting scale in step three is as follows: [The text abruptly ends here, so the translation stops.] r Feature vector at the nth cutting scale The corresponding class confidence score is obtained by feeding it into the classification optimization branch of the k-th instance. k∈[1,K]; when k=1 The supervision signal for the k-th instance classification optimization branch is the class confidence score of all target candidate boxes in the (k-1)-th instance classification optimization branch. The algorithm utilizes a pseudo-label mining strategy; where rcat(·,·) represents the concatenation operation along the row direction, K is the total number of instance classification optimization branches, and ZM∈R M The vector represents all zeros; the (C+1)th dimension represents the background category.

[0026] Preferably, the method for mining using the pseudo-label mining strategy is as follows: target candidate box p r The mean score is obtained by averaging the class confidence scores across all cut scales in the (k-1)th instance classification optimization branch. and:

[0027]

[0028] Following the pseudo-label mining strategy proposed by MIST, based on the average score Obtain the instance-level pseudo-label of the k-th instance classification optimization branch. If the target candidate box p r If the target belongs to the c-th category, then otherwise like Then the target candidate box p r ∈Q.

[0029] Preferably, the target candidate box p is determined. r The method for determining whether an example belongs to a difficult burden case is as follows: In the k-th ICR branch at the n-th cut scale, label p r The highest category confidence score in category C+1 is If j≤C and And y j =0, then the target candidate box p r For example, j represents the target candidate box p. r The category that achieves the highest category confidence score, y j p r Pseudo-labels on category j.

[0030] Preferably, the overall loss function of the K instance classification optimization branches

[0031] The loss function for the classification optimization branch of the k-th instance is:

[0032]

[0033] in, Indicates category confidence score The c-th element in p represents the target candidate box r The weight, Optimize the instance-level pseudo-labels of the branch for classifying the k-th instance; For the target candidate box p r The instance difficulty score at the k-th ICR branch and the n-th cutting scale, and:

[0034]

[0035] Among them, R hn Let R be the set of all hard-to-distribute cases. o Let h represent the set of all candidate boxes p except for the hard-to-handle cases. r The true category, Indicates category confidence score The h-th element in;

[0036] The overall loss function of the weakly supervised target detection model is L = L M +L B +L I .

[0037] Compared with existing technologies, the beneficial effects of this invention are as follows: By designing a novel feature enhancement model based on multi-scale image segmentation, the overall target region in positive instances can be effectively highlighted; by designing an instance difficulty-aware loss strategy, difficult negative examples can be effectively given more attention during training, thereby effectively improving the detection accuracy of weakly supervised target detection in high-resolution remote sensing images. Simulation results show that this invention can more accurately identify and locate ground objects in remote sensing images. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the process of the present invention.

[0040] Figure 2 The figure shows the simulation results of this invention.

[0041] Figure 3 This is a comparison chart of the simulation results of the present invention and the benchmark algorithm. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, a weakly supervised object detection method based on multi-scale image segmentation and instance difficulty includes the following steps:

[0044] Step 1: Feature Enhancement Based on Multi-Scale Image Segmentation: The input image is segmented at multiple scales. The input image and corresponding image patches at at least two randomly selected segmentation scales are fed into a shared backbone network for feature extraction and then concatenated according to their spatial relationships to obtain feature maps. Multiple target candidate boxes are obtained from the input image. The target candidate boxes are projected onto the feature maps, and region-of-interest pooling is performed to obtain feature maps of the target candidate boxes. The maximum spatial attention map of positive instances is obtained from the feature maps of the target candidate boxes, and the loss function of the feature enhancement module based on multi-scale image segmentation is obtained. The feature enhancement module based on multi-scale image segmentation is then trained.

[0045] (1) The training and testing samples used in this embodiment are both high-resolution remote sensing image datasets with only category labels: the DIOR dataset. The DIOR dataset contains a total of 20 target classes, and each image is 800×800 pixels in size. The training set, validation set, and test set contain 5862, 5863, and 11738 images, respectively. In this embodiment, the samples in the training and validation sets of the DIOR dataset are used to train the weakly supervised target detection model of this invention, and the test set is used to test the weakly supervised target detection model.

[0046] like Figure 1 As shown, each input high-resolution remote sensing image is first segmented at Z = 6 scales, where the z-th scale segment divides the image into z × z image patches, and z ∈ [1, 6]. Then, N-1 = 2 scales are randomly selected from the 2nd to the 6th scales, and N ≤ Z. The image patches corresponding to scale 1 (the original image) and the N-1 scales are respectively fed into a shared backbone network for feature extraction to obtain feature maps of corresponding sizes. The feature maps of all image patches at the nth scale are then concatenated according to their spatial position to obtain feature map F. n , n∈[1,3], F n This refers to the feature map of the input image at the nth cutting scale.

[0047] Input images of different sizes are fed into the feature enhancement module based on multi-scale image segmentation to obtain feature maps of corresponding sizes. After an image is segmented into several image blocks, each image block is fed into the feature enhancement module based on multi-scale image segmentation to obtain a corresponding feature map. Then, the feature maps are stitched together according to the original segmentation order, i.e., the spatial relationship, to obtain the stitched feature map F. n The weakly supervised object detection network uses VGG16 as its backbone, and all weakly supervised object detection networks use VGG16. Z=6 was determined experimentally; when Z is greater than 6, the detection performance decreases, and the number of cuts z∈[1,6]. Randomly selecting N-1 means randomly selecting N-1 scales from the five cutting scales from 2 to 6 (N=3 in the experiment). It does not mean that all image patches formed at all cutting scales are sent to the backbone network, but rather that N-1 scale corresponding image patches are randomly selected and input into the backbone network. The features extracted from the cut image patches tend to highlight the salient regions of the local target, while the features extracted from the uncut original image tend to highlight the salient regions of the entire image. Subsequently, a spatial attention map approximation method is used to obtain the complete features of the target. For example, when z=2, the original image will be cut into 4 image patches. These four image patches are then fed into the backbone network to obtain 4 features. Finally, these 4 feature matrices need to be concatenated to obtain the features of the entire image. Subsequent steps in the backbone network rely on features from the entire image for processing.

[0048] The Selective Search algorithm [J.R.R.J.L ... r And project it onto the feature map F n Perform region-of-interest pooling operations to obtain feature maps of target candidate boxes. r∈[1, 2000], where 7, 7, and 512 represent the feature maps of the target candidate boxes, respectively. The height, width, and number of channels. The purpose of region-of-interest pooling is to extract the features of each target candidate box based on the features of the entire image. Target candidate boxes are as follows: Figure 1 The small squares in the image are mapped onto the feature map to obtain the features of each white square. Since the target candidate boxes vary in size, the height and width are fixed at 7x7 during pooling. The number of channels is determined by the number of channels in the feature map; the image after passing through the backbone network has 512 feature channels.

[0049] Since weak supervision lacks the true location of each instance, generating 2000 object candidate boxes aims to capture regions that may contain objects at different locations and scales. In object detection tasks, because the location and size of object objects are uncertain, generating a large number of candidate boxes increases the model's coverage of object objects and improves detection accuracy. Different positive instances acquire their own maximum spatial attention maps within their respective series of spatial attention maps, encouraging different instances' spatial attention maps to approximate their respective maximum spatial attention maps.

[0050] Obtain the Spatial Attention Map (SAM) for positive instances: label the q-th positive instance as t. q q∈[1,|Q|], positive instance t q The SAM at the nth cutting scale is denoted as Obtain it through the following methods:

[0051]

[0052] in, Represents the positive instance t at the nth cutting scale. qThe i-th channel of the feature map, Q represents the set of positive instances in all ICR (Instance Classifier Refinement) branches, and |Q| represents the number of positive instances. Sigmoid represents the activation function.

[0053] Mark positive instance t q The Maximum Spatial Attention Map (MSAM) is It can be obtained through the following formula:

[0054]

[0055] Here, max(·) represents the operation of taking the maximum value element by element. These are spatial attention maps at the 1st, 2nd, and 3rd cutting scales, respectively.

[0056] Loss function L based on feature enhancement of multi-scale image segmentation M It can be obtained through the following formula:

[0057]

[0058] Step 2: Pass the feature map of the target candidate box through two fully connected layers to obtain the feature vector of the target candidate box at the cutting scale. Input the feature vector into the benchmark weakly supervised object detection module to obtain two score matrices. Obtain the class confidence score matrix based on the two score matrices. Obtain the prediction score of each class at the cutting scale based on the class confidence score matrix. Calculate the loss function of the benchmark weakly supervised object detection module and train the benchmark weakly supervised object detection module.

[0059] like Figure 1 The feature map of the target candidate box is shown in the upper right corner. After passing through two fully connected layers (FC), the r-th target candidate box p is obtained. r Feature vector at the nth cutting scale The data is then fed into two parallel branches of the baseline weakly supervised object detection module, each consisting of a fully connected layer and a softmax classifier. When all feature vectors... Two matrices are obtained after passing through two fully connected layers II of two branches respectively. Where C=20 represents the number of target categories. This invention was conducted on the DIOR dataset, which contains 20 different target categories; therefore, C=20 represents the number of target categories.

[0060] At this point, the class confidence score matrix X for all target candidate boxes at the nth cutting scale is... nIt can be obtained through the following formula:

[0061]

[0062] Where, σ c (·),σ d (·) represents the softmax operation along the category and target candidate box dimensions, respectively, and ⊙ represents the Hadman product.

[0063] The predicted score of the input image belonging to the c-th category at the n-th cut scale. Obtained using the following formula:

[0064]

[0065] in, This represents the target candidate box p at the nth cutting scale. r The confidence score for category c. This is expressed using the category confidence score matrix X at the nth cut scale. n The score for each category at the nth cutting scale is obtained, and then the target candidate box p is selected from all category scores. r The score for category c.

[0066] At this point, the loss function of the benchmark weakly supervised target detection module at the nth cutting scale is... This can be expressed as follows:

[0067]

[0068] Among them, y c = 1 or 0 indicates whether the input image contains a target of the c-th category. The overall loss function L of the baseline weakly supervised target detection module is... B As shown below:

[0069]

[0070] The benchmark weakly supervised object detection module provides useful classification scores and object candidate box information for the subsequent instance-level inference in the ICR branch, laying an important foundation for object detection performance. The weakly supervised object detection network only contains category information for the entire image. The benchmark weakly supervised object detection module calculates the loss function by comparing the predicted object category information with the true object category and then backpropagates.

[0071] Step 3: Input the feature vectors obtained in Step 2 into multiple instance classification optimization branches to obtain the class confidence scores of the target candidate boxes at the cutting scale, and use the class confidence scores to update the set of positive instances.

[0072] like Figure 1 As shown in the bottom right corner, the target candidate box pr Feature vector at the nth cutting scale The data is fed into the k-th ICR branch, k∈[1,3], to obtain the corresponding category confidence score. The 21st dimension represents the background category. The supervision signal for the k-th ICR branch is mined from the category confidence scores of all target candidate boxes in the (k-1)-th ICR branch, where, when k=1, rcat(·,·) represents a concatenation operation along the row direction, ZM∈R 2000 The vector represents all zeros, and T represents the matrix transpose. The ICR branch obtains the eigenvectors. After passing through a fully connected layer (FC) and a softmax operation in the category direction, a target candidate box category confidence score matrix can be obtained.

[0073] Specifically as follows:

[0074] Set the target candidate box p r The average of the category confidence scores across all cut scales in the (k-1)th ICR branch is obtained.

[0075]

[0076] Following the pseudo-label mining strategy proposed by MIST [Z.Ren et al., “Instance-aware, context-focused, and memory-efficient weakly supervised object detection,” in Proc. IEEE / CVFConf.Comput.Vis.Pattern Recognit., 2020, pp.10598–10607.], based on We can obtain the instance-level pseudo-label of the k-th ICR branch. If the target candidate box p r If the target belongs to the c-th category, then otherwise like Then the target candidate box p r ∈Q.

[0077] Step 4: Obtain the set of hard negative examples of the target candidate box based on the highest class confidence of the target candidate box in the background class, calculate the instance difficulty score of the target candidate box under the cutting scale, calculate the loss function of multiple instance classification optimization branches, and train multiple instance classification optimization branches.

[0078] Determine the target candidate box p rIs it a difficult-to-bear example? In the k-th ICR branch at the n-th cut scale, label the target candidate box p. r The highest category confidence score across 21 categories is: Where j represents the target candidate box p r The category that achieves the highest class confidence score, if j≤20 and And y j =0, then the target candidate box p r It is difficult to set an example.

[0079] Calculate the target candidate box p r The instance difficulty score (IDS) at the k-th ICR branch and the n-th cutting scale, i.e. The calculation method is as follows:

[0080]

[0081] Among them, R hn Let R be the set of all hard-to-distribute cases. o Let h represent the set of all candidate boxes p except for the hard-to-handle cases. r The true category, Indicates category confidence score The h-th element in the graph represents the k-th ICR branch, the n-th cutting scale, and the r-th target candidate box p. r The score in the h-th category. |R o |and|R hn | represent the aggregation R respectively o and R hn The quantity.

[0082] Calculate the loss function of the k-th ICR branch

[0083]

[0084] in, Let be the loss function of the k-th ICR branch at the n-th cutting scale. Indicates category confidence score The c-th element in. p represents the target candidate box rThe weights, and their calculation method, can be found in the literature OICR [P. Tang, X. Wang, X. Bai, and W. Liu, “Multiple instance detection network with online instance classifier refinement,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2843–2851.]. The total loss L of the three ICR branches is... I as follows:

[0085]

[0086] Step 5: Repeat steps 1 to 4, following the predetermined number of training rounds (iterations). Training ends after each iteration. During the iteration process, the weakly supervised object detection model is saved every 1000 iterations, resulting in multiple weakly supervised object detection models after training. The image to be detected is then fed into the trained weakly supervised object detection model to obtain the category and location of the object of interest in the image.

[0087] The weakly supervised object detection model comprises a feature enhancement module based on multi-scale image segmentation, two fully connected layers, a baseline weakly supervised object detection module, and multiple instance classification optimization branches. Each candidate bounding box contains its own location information. Elements in the class confidence score matrix are called class confidence scores. The network predicts all candidate bounding boxes for C classes in the dataset to obtain a class confidence score matrix (M×(C+1) dimensional). The prediction score of a candidate bounding box for a specific class is called its class confidence score. The final target of interest is the location information selected from the candidate bounding boxes, and the target's class information is determined based on the class corresponding to the highest class confidence score. Training ends after a predetermined number of training epochs (iterations). During iteration, the weakly supervised object detection model is saved every 1000 iterations, resulting in multiple weakly supervised object detection models after training. The test set in the dataset is then used to test the weakly supervised object detection models generated throughout the iteration process, and the best model is selected based on the test results.

[0088] The overall loss function L of the weakly supervised target detection model proposed in this invention is as follows:

[0089] L = L M +L B +L I

[0090] The overall loss function L is used to train the entire weakly supervised object detection model. During the inference phase, the image to be detected is fed into the trained weakly supervised object detection model to obtain the category and location of the target of interest in the image.

[0091] The hardware configuration for implementing this invention is as follows: a workstation with an E5-2650V4 CPU (2.2GHz 12x2 cores), 512GB of memory, and 8 NVIDIA RTX Titan graphics cards was used for the experiment. The software platform configuration is as follows: Ubuntu 16.04, Python 3.7, and PyTorch 1.7.

[0092] To better demonstrate the performance of this invention, as shown in Table 1, this invention is compared with nine popular algorithms on the DIOR dataset. The nine algorithms are: WSDDN [H. Bilen, A. Vedaldi, Weakly supervised deepdetection networks, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 2846–2854], OICR [P. Tang, X. Wang, X. Bai, W. Liu, Multiple instance detection network with online instance classifier refinement, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 3059–295 3067], PCL [P. Tang, X. Wang, S. Bai, W. Shen, X. Bai, W. Liu, AL Yuille, PCL: proposal cluster learning for weakly supervised object detection, IEEE Trans. Pattern Anal.Mach.Intell.42(1)(2020)176–191], MELM [F.Wan, P.Wei, J.Jiao, Z.Han, and Q.Ye, “Min-entropy latent model for weakly supervised object detection,” in Proc.IEEE Conf.Comput.Vis.PatternRecognit.,Jun.2018,pp.1297–1306.], DCL [X.Yao, Sens.59(1)(2021)675–685],MIST[Z.Ren,Z.Yu,X.Yang,M.-Y.Liu,YJLee,AGSchwing,and J.Kautz, “Instance-aware, context-focused, and memory-efficient weakly supervised object detection,” in Proc.IEEE / CVFConf.Comput.Vis.Pattern Recognit., Jun.2020, pp.10 598–10 607.], PCIR [X.Feng, J.Han, X.Yao, G.Cheng, Progressive contextual instance refinement for weakly supervised object detection in remote sensing images, IEEE Trans.Geosci.RemoteSens.58(11)(2020)8002–8012], TCA [X.Feng, J.Han, X.Yao, and G.Cheng, “Tcanet: Triplecontext-aware network for weakly supervised object detection in remotesensing images,” IEEE Trans.Geosci.Remote Sens., vol. 59, no. 8, pp. 6946–6955, Oct. 2021., MIG [B. Wang, Y. Zhao, X. Li, Multiple instance graph learning for weakly supervised remote sensing object detection, IEEE Trans. Geosci. Remote Sens. 60 (2022) 1–12]. mAP and Corloc represent the average accuracy and localization accuracy, respectively.

[0093] Table 1 compares the average accuracy and positioning accuracy of this invention with 9 popular algorithms on the DIOR dataset.

[0094] method mAP CorLoc WSDDN 13.3 32.4 OICR 16.5 34.8 PCL 18.2 41.5 MELM 18.7 43.3 DCL 20.2 42.2 MIST 22.2 43.6 PCIR 24.9 46.1 TCA 25.8 48.4 MIG 25.1 46.8 This invention 26.4 49.2

[0095] As can be seen from Table 1, compared with the other nine weakly supervised target detection methods, the method of the present invention has higher average accuracy and positioning accuracy than other target detection methods, and can more accurately identify and locate ground objects in remote sensing images.

[0096] Figure 2The detection results of this invention are presented, demonstrating that this invention can accurately locate ground targets. Figure 3 The results are shown in the comparison charts between the present invention and the benchmark method. The benchmark method uses OICR. The present invention, by introducing a novel feature enhancement model based on multi-scale image segmentation and a perceptual loss strategy, can effectively highlight the target region as a whole in positive instances and give more attention to difficult negative instances during training. Compared with the OICR benchmark method, the first two images show that the present invention can highlight the overall target region, and the last three images show that the present invention can effectively focus on difficult negative instances and hard instances.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A weakly supervised object detection method based on multi-scale image cut and instance difficulty, characterized in that, The steps are as follows: Step one: feature enhancement based on multi-scale image cutting: the input image is cut in multiple scales, and the input image and the image blocks corresponding to at least two randomly selected cutting scales are respectively sent into a shared backbone network for feature extraction and spliced according to the spatial position relationship to obtain feature maps under each cutting scale; a plurality of target candidate boxes are obtained from the input image, the target candidate boxes are projected onto the feature maps under each cutting scale to perform region of interest pooling to obtain feature matrices of the target candidate boxes under each cutting scale, the maximum spatial attention maps of all positive instances on the feature maps under each cutting scale are obtained according to the feature matrices of the target candidate boxes under each cutting scale, and a loss function of a feature enhancement module based on multi-scale image cutting is obtained, and the feature enhancement module based on multi-scale image cutting is trained; Step two: the feature matrices of the target candidate boxes under each cutting scale are sent through two fully connected layers to obtain feature vectors of the target candidate boxes under each cutting scale, the feature vectors are input into a benchmark weakly supervised target detection module to obtain a class confidence score matrix, the prediction scores of each class under the cutting scale are obtained according to the class confidence score matrix, a loss function of the benchmark weakly supervised target detection module is calculated, and the benchmark weakly supervised target detection module is trained; Step three: the feature vectors obtained in step two are respectively sent into a plurality of instance classification optimization branches to obtain class confidence scores of the target candidate boxes under each cutting scale, and the class confidence scores are used to determine a positive instance set; Step four: the highest class confidence score in the class in which the target candidate box does not exist in the image is obtained to obtain a hard negative instance set of the target candidate box, the instance difficulty scores of the target candidate boxes under each cutting scale are calculated as weights of the loss function, the loss functions of the plurality of instance classification optimization branches are calculated, and the plurality of instance classification optimization branches are trained; Judgment target candidate frame p r The method for judging whether it belongs to the hard negative example is: in the kth instance classification optimization branch under the nth cutting scale, marking the target candidate frame p r The highest class confidence score on C+1 classes is If j≤C and And y j =0, the target candidate frame p r is a hard negative example; wherein, j represents the target candidate frame p r The class obtaining the highest class confidence score, y j , represents the target candidate frame p r The pseudo label on the class j; The class confidence score obtained in the kth instance classification optimization branch; Loss function for k-instance classification optimization branch loss function of the kth instance classification optimization branch The loss function of the kth instance classification optimization branch under the nth cutting scale is: where N is the total number of cutting scales; a class confidence score the c-th element in a target bounding box p r a weight of an instance-level pseudo label for the k-th instance classification optimization branch; a target bounding box p r an instance difficulty score under the k-th instance classification optimization branch and the n-th cutting scale, and: where R hn represents the set of all hard negatives, R o represents the set of all other target candidate boxes except hard negatives, h represents the true class of target candidate box p r represents the h-th element in the class confidence score ​​ Step five: steps one to four are cycled to train a weakly supervised target detection model in a data set; and an image to be detected is input into the trained weakly supervised target detection model to obtain the class and position of the target of interest in the image.

2. The weakly supervised object detection method based on multi-scale image cut and instance difficulty according to claim 1, characterized in that, The method for feature extraction and splicing of the feature map according to the spatial position relationship by the shared backbone network in the step one is: cutting the input image in Z scales, wherein the cutting in the zth scale cuts the image into z×z image blocks and z∈[1, Z]; randomly selecting N-1 scales in the 2nd to Zth scales, and N≤Z; sending the input image and the image blocks corresponding to the N-1 scales into the shared backbone network respectively for feature extraction, obtaining the feature maps of corresponding sizes, and splicing the feature maps of all the image blocks in the nth cutting scale to obtain F n , n∈[1, N], F n , which is the feature map of the input image in the nth cutting scale. 3.The weakly supervised object detection method based on multi-scale image cut and instance difficulty according to claim 2, characterized in that, The selective search algorithm is used to generate M target candidate boxes for the input image, and the r-th target candidate box is represented as p. r And project it onto F n Perform region-of-interest pooling operations to obtain feature maps of target candidate boxes. Where H, W, and L represent feature maps respectively. Height, width, and number of channels.

4. The weakly supervised object detection method based on multi-scale image cut and instance difficulty according to any one of claims 1-3, characterized in that, The loss function of the feature enhancement module based on multi-scale image cutting is where N is the total number of cut scales, is a positive instance t q The maximum spatial attention map of the positive instance t is a positive instance t q The spatial attention map under the nth cut scale, is the ith channel of the feature map of the positive instance t q under the nth cut scale, i∈[1, L], H, W, L represent the height, width and channel number of the feature map respectively, q∈[1, |Q|] is the index of the positive instance, Q represents the set of positive instances in all instance classification optimization branches, |Q| represents the number of positive instances, and Sigmoid represents the activation function.

5. The weakly supervised object detection method based on multi-scale image cut and instance difficulty according to claim 4, characterized in that, The benchmark weakly supervised target detection module includes two parallel branches, each parallel branch is composed of a fully connected layer and a softmax classifier; The method for obtaining the class confidence score matrix in step two is: feature map G r n after two fully connected layers to obtain the target candidate frame p r feature vector at the nth cutting scale and sent into two parallel branches in the benchmark weakly supervised target detection module; when all feature vectors two matrices are obtained after two fully connected layers of two parallel branches respectively wherein C represents the number of target categories; the category confidence score matrix of all target candidate frames at the nth cutting scale where σ c (.) and σ d (.) denote softmax operation along category and target bounding box dimension respectively, and ⊙ denotes Hadamard product.

6. The weakly supervised object detection method based on multi-scale image cut and instance difficulty according to claim 5, characterized in that, The loss function of the baseline weakly supervised object detection module The loss function under the nth cutting scale is where y c = 1 or 0 indicates whether the input image contains a target of the c-th class; the predicted score of the input image belonging to the c-th class at the n-th cut scale where denotes the confidence score of the target candidate box p r belonging to the c-th class at the n-th cut scale.

7. The weakly supervised object detection method based on multi-scale image cut and instance difficulty according to claim 4 or 5, characterized in that, The method for obtaining the class confidence score of the target candidate box at the cutting scale in step three is: taking the target candidate box p r Feature vector at the nth cutting scale The corresponding class confidence score is obtained by inputting into the kth instance classification optimization branch When k=1, The supervision signal of the kth instance classification optimization branch is the class confidence score of all target candidate boxes in the k-1th instance classification optimization branch The pseudo-label mining strategy is used for mining in the middle; wherein, rcat(·,·) represents the splicing operation along the row direction, K is the total number of instance classification optimization branches, ZM∈R M represents a full 0 vector; the (C+1)th dimension represents the background class.

8. The weakly supervised object detection method based on multi-scale image cut and instance difficulty according to claim 7, characterized in that, The pseudo-label mining strategy mines a method, which is a target candidate frame p r In the k-1th instance classification optimization branch, the class confidence scores under all cutting scales are averaged to obtain a score average And: According to the pseudo label mining strategy proposed by MIST, the score mean value Get the instance-level pseudo label of the kth instance classification optimization branch If the target candidate box p r belongs to the cth class, then Otherwise If then the target candidate box p r ∈Q.

9. The weakly supervised object detection method based on multi-scale image cut and instance difficulty according to claim 8, characterized in that, The overall loss function L of the weakly supervised object detection model = L M + L B + L I .

Images