Weight redistribution and task alignment-based remote sensing image weak supervision target detection method

Abstract

The invention discloses a remote sensing image weak supervision target detection method based on weight redistribution and task alignment. The method mainly solves the problems that neighborhood targets are misclassified, only the most significant region of the targets is concerned, and classification branches are inconsistent with regression branches in an existing detection scheme. According to the scheme, the method comprises the steps of introducing high-quality positive instance mining to obtain seed instances, and determining neighborhood instances and background instances according to space and feature similarity between the instances; by calculating a spatial relation matrix between positive instances, screening instances covering a complete target, and completing weight redistribution of the positive instances; meanwhile, the initial instability of a weak supervision target detection model is considered, task alignment frame regression loss is introduced, the weight is dynamically adjusted by designing a task alignment modulation factor, and then the modulation factor and the difficulty degree of an instance are combined to serve as the weight of a regression branch. According to the method, misclassification of neighborhood instances can be avoided, the whole target is concerned, the coupling of classification and regression branches is improved, and the detection performance is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and further relates to a remote sensing image weakly supervised target detection technology, specifically a remote sensing image weakly supervised target detection method based on weight redistribution and task alignment, which can be used in military strategy, urban planning and geographic information system construction, etc. Background Art

[0002] Weakly supervised object detection uses image-level category labels to locate and classify high-value targets. Compared with instance-level labels used in fully supervised object detection, weakly supervised object detection saves a lot of annotation costs. This technology is widely used in military, civil, environmental monitoring, and agricultural fields. Existing weakly supervised object detection is implemented through multi-instance learning. Specifically, the input image is regarded as a set of object instances, and the image-level labels are used to supervise the instance classifier under the multi-instance learning framework. With the development of weakly supervised object detection, iteratively optimizing instance classifiers using instance-level pseudo-labels (including positive and negative instances) has attracted much attention from researchers. First, seed instances are mined as positive instances using prediction scores. Second, during the label propagation process, when there is a large spatial similarity between the neighborhood instances and the seed instances, positive labels are assigned to the neighborhood instances, otherwise negative labels are assigned. Accordingly, instance-level pseudo-labels supervise instance classifiers.

[0003] In the prior art document WSDDN [H.Bilen and A.Vedaldi, "Weakly supervised deep detection networks," in Proc.IEEE Conf.Comput.Vis.Pattern Recognit.(CVPR),Jun.2016,pp.2846–2854.], WSDDN first combines multi-instance learning and deep learning to complete weakly supervised target detection. First, a selective search algorithm is used to generate target candidate boxes for the input image. Secondly, the target candidate boxes and the input image are input into the backbone network, the region of interest pooling and two fully connected layers to obtain the feature vector of each target candidate box. Next, the feature vector of the target candidate box is input into the classification branch and the detection branch in the multi-instance detection network. Finally, the prediction scores of the classification branch and the detection branch are dot-multiplied and summed along the instance dimension to obtain the image-level prediction score. The image-level prediction score and the image-level category label calculate the binary cross entropy loss for back propagation. The prior art document OICR [P. Tang, X. Wang, X. Bai, and W. Liu, "Multiple instance detection network with online instance classifier refinement," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 2843–2851.] adds multiple parallel instance classification optimization branches based on WSDDN. The supervision signal of each branch uses the prediction score of the previous branch to generate instance-level pseudo labels. Specifically, the instance with the highest prediction score is selected as the seed instance. Secondly, during the label propagation process, when the neighborhood instance and the seed instance have a large spatial relationship, a positive label is assigned to the neighborhood instance, otherwise a negative label is assigned. Based on this, the generated instance-level pseudo label supervises the instance classification optimization branch.

[0004] At present, the main problems of weakly supervised remote sensing image target detection include the following: 1) Misclassification of neighborhood instances: The existing weakly supervised target detection only relies on spatial similarity to assign pseudo labels to neighborhood instances during label propagation, resulting in spatially adjacent targets being incorrectly assigned pseudo labels, which in turn causes neighborhood instances to be incorrectly classified. 2) The problem of locating the most salient area of ​​the target: The weakly supervised target detection model assigns high loss weights to instances covering the most salient area of ​​the target, resulting in the model only focusing on the most salient area of ​​the target. 3) The incompatibility of classification and regression tasks: In weakly supervised target detection, the classification and regression tasks use shared features, and there is no coupling between the two, resulting in some target candidate boxes with high classification scores not necessarily being the most accurate regression. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention aims to propose a remote sensing image weakly supervised target detection method based on weight redistribution and task alignment, which solves the problems of misclassification of neighborhood targets, focusing only on the most prominent area of ​​the target, and inconsistency between the classification branch and the regression branch in the existing detection scheme. By introducing high-quality positive instances mining (HPIM) into the existing instance classifier online optimization model OICR, the spatial similarity and feature similarity between instances are considered in the label propagation process, thereby avoiding the misclassification of neighborhood instances; at the same time, by calculating the spatial relationship matrix between positive instances, the instances covering the target are screened out, and the positive instance weights re-assignment (PIWR) is used to encourage the weakly supervised remote sensing image target detection model to focus on the target as a whole; finally, considering the initial instability of the weakly supervised target detection model, the task-aligned bounding box regression loss (TABRL) is introduced, and the task alignment modulation factor is designed to enable the model to dynamically adjust the weight according to the number of iterations. Then, the modulation factor and the difficulty of the instance are combined as the weight of the regression branch, which improves the consistency of the classification branch and the regression branch. The present invention can effectively overcome the defects of the prior art and improve the detection effect of weakly supervised targets in remote sensing images.

[0006] The specific steps of the present invention to achieve the above-mentioned purpose include the following:

[0007] (1) Given a remote sensing image Where H, W and B represent the width, height and number of channels of the image respectively. The selective search algorithm is used to generate M target candidate frames for the remote sensing image I, forming a target candidate frame set PR = {pr1, pr2, ...pr m ,pr M}, where pr m Represents the mth target candidate box;

[0008] (2) The remote sensing image I and the target candidate box set PR are input into the instance classifier online optimization model OICR, which includes a backbone network, a multi-instance detection network, an instance classification optimization branch, and a task alignment bounding box regression branch; the feature vectors of all target candidate boxes are obtained through the backbone network, the region of interest pooling, and two fully connected layers. And feed it into the multi-instance detection network, where f m Represents the feature vector of the mth target candidate box;

[0009] (3) Target candidate box feature vector Input two parallel branches in the multi-instance detection network, and obtain two score matrices in category c after passing through the two branches respectively. Obtain the category confidence score matrix of the target candidate box in category c according to the following formula:

[0010]

[0011] Where ⊙ represents the Hadamard product, c is any one of all categories C;

[0012] The image-level category prediction score φ is obtained according to the following formula c :

[0013]

[0014] in, Represents the category confidence score of the mth target candidate box in category c;

[0015] (4) Add K parallel instance classification optimization branches to the baseline weakly supervised object detection model, and Input the kth instance classification optimization branch to obtain the category confidence score of the target candidate box Among them, C+1 represents the background category;

[0016] (5) The prediction score of the k-1th branch As the input of high-quality positive instance mining, Sort in descending order and select the first p percent of instances as the initial seed instance set of the cth category in the kth branch. Secondly, the seed instance set is obtained by performing non-maximum suppression operation on the initial seed instance Implement seed instance mining;

[0017] (6) Mining the initial neighborhood instances through spatial similarity, setting the spatial similarity judgment threshold ζ, when the jth seed instance When the spatial similarity IoU between the instance and other instances is greater than the threshold ζ, the instance is determined to be the initial neighborhood instance, otherwise it is determined to be the background instance; the seed instance is obtained. The initial neighborhood instance set

[0018] (7) By using the feature similarity between instances, we can eliminate the misclassified initial neighborhood instances and set the feature similarity threshold δ. When the jth seed instance When the feature similarity between the initial neighborhood instance and the background instance is less than δ, the initial neighborhood instance is judged as a background instance and the jth seed instance is obtained. The set of neighborhood instances Get the neighborhood instance set of all seed instances and obtain the instance-level supervision information y of the kth instance classification optimization branch k ;

[0019] (8) The positive instance weight redistribution strategy is used to assign loss weights to positive instances that cover the entire target. First, a positive instance subset SN is constructed for the seed instance and its neighboring instances. Then, the spatial relationship between instances in SN is calculated to construct a spatial distance matrix Then sum up G along the row direction to get the spatial distance vector Find the maximum value in V and mine the complete instance r covering the target z , and find the instance r with the highest prediction category confidence score in the positive instance subset SN p , in the positive instance subset SN, instance r z and r p The loss weights are exchanged to achieve positive instance weight redistribution;

[0020] (9) Construct the loss function of the kth instance classification optimization branch

[0021]

[0022] in, represents the loss weight of the target candidate box m in the kth branch, Represents y k The element in the cth row and mth column of When it is equal to 1, it means that the target candidate box m contains the target of category c; When it is equal to 0, it means that the target candidate box m does not contain the target of category c; Represents the predicted category confidence score of the target candidate box m on category c in the kth branch;

[0023] (10) The task-aligned bounding box regression branch performs bounding box regression on the positive instance, taking into account the difficulty of the positive instance and the number of model iterations, and increases the coupling between the model classification task and the regression task, as follows:

[0024] (10.1) Design a modulation factor μ that changes dynamically with model iteration to ensure the stability of training:

[0025]

[0026] Where t represents the current number of iterations, T represents the total number of model iterations; σ represents the hyperparameter used to control the growth rate;

[0027] (10.2) Align the task of the k-th instance classification optimization branch with the bounding box regression loss The definition is as follows:

[0028]

[0029]

[0030] The number of elements in represents the predicted category confidence score of the corresponding category of the lth positive instance in the kth instance classification optimization branch; and g l denote the target offset and predicted offset of the lth positive instance respectively;

[0031] (11) Construct the overall loss L of the weakly supervised remote sensing image target detection model ALL :

[0032]

[0033] Among them, L WSDDN represents the loss function of the baseline weakly supervised object detection network, K represents the number of instance classification optimization branches;

[0034] (12) Using L ALL The entire weakly supervised remote sensing image target detection model is trained, and the remote sensing image to be tested and its target candidate box are input into the trained model to obtain the category and position of the target of interest in the image, thereby realizing weakly supervised target detection in remote sensing images.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] First, since the present invention introduces high-quality positive instance mining, in the process of label propagation, by considering the feature similarity between seed instances and neighborhood instances, misclassified neighborhood instances are eliminated, thereby alleviating the misclassification problem of neighborhood instances and effectively improving the mining effect.

[0037] Second, the present invention proposes a positive instance weight redistribution strategy, which mines instances that completely cover the target by calculating the spatial relationship matrix between positive instances, assigns large loss weights to instances that completely cover the target, and encourages the model to focus on the target as a whole.

[0038] Third, since the present invention introduces the task-aligned bounding box regression loss, in the bounding box regression branch, the instability of the model in the early stage of iteration and the difficulty of the positive instance are considered at the same time, thereby significantly improving the coupling of the regression branch and the classification branch. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the overall implementation flow chart of the method of the present invention;

[0040] Figure 2A schematic diagram of the application process of using the present invention to perform target detection;

[0041] Figure 3 This is a partial detection result diagram obtained by using the present invention on the DIOR dataset. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0043] Example 1: Reference Figure 1 The present invention proposes a remote sensing image weakly supervised target detection method based on weight redistribution and task alignment; the specific steps are as follows:

[0044] Step 1. Given a remote sensing image Where H, W and B represent the width, height and number of channels of the image respectively. The selective search algorithm is used to generate M target candidate frames for the remote sensing image I, forming a target candidate frame set PR = {pr1, pr2, ...pr m ,...pr M}, where pr m Represents the mth target candidate box;

[0045] Step 2. Input the remote sensing image I and the target candidate box set PR into the instance classifier online optimization model OICR, which includes a backbone network, a multi-instance detection network, an instance classification optimization branch, and a task alignment bounding box regression branch; obtain the feature vectors of all target candidate boxes through the backbone network, region of interest pooling, and two fully connected layers And feed it into the multi-instance detection network, where f m Represents the feature vector of the mth target candidate box;

[0046] Step 3. Target candidate box feature vector Input two parallel branches in the multi-instance detection network. In this embodiment, both parallel branches are composed of a fully connected layer and a Softmax classifier. After passing through the two branches, two score matrices in category c are obtained. Obtain the category confidence score matrix of the target candidate box in category c according to the following formula:

[0047]

[0048] Where ⊙ represents the Hadamard product, c is any one of all categories C;

[0049] The image-level category prediction score φ is obtained according to the following formula c :

[0050]

[0051] in, Represents the category confidence score of the mth target candidate box in category c;

[0052] Step 4. Add K parallel instance classification optimization branches to the baseline weakly supervised object detection model, by Input the kth instance classification optimization branch to obtain the category confidence score of the target candidate box Among them, C+1 represents the background category;

[0053] Step 5. Take the category confidence score of the k-1th branch As the input of high-quality positive instance mining, Sort in descending order and select the first p percent of instances as the initial seed instance set of the cth category in the kth branch. Then perform non-maximum suppression operations on it to remove small and redundant initial seed instances and obtain a set of seed instances. Implement seed instance mining;

[0054] Step 6. Mining the initial neighborhood instances through spatial similarity, setting the spatial similarity judgment threshold ζ, when the jth seed instance When the spatial similarity IoU between the instance and other instances is greater than the threshold ζ, the instance is determined to be the initial neighborhood instance, otherwise it is determined to be the background instance; the seed instance is obtained. The initial neighborhood instance set

[0055] Step 7. Eliminate misclassified initial neighborhood instances by comparing feature similarity between instances, set feature similarity threshold δ, and when the jth seed instance When the feature similarity between the initial neighborhood instance and the background instance is less than δ, the initial neighborhood instance is judged as a background instance and the jth seed instance is obtained. The set of neighborhood instances Get the neighborhood instance set of all seed instances and obtain the instance-level supervision information y of the kth instance classification optimization branch k .

[0056] The jth seed instance and the initial neighborhood instance ini i The feature similarity between them is obtained according to the following formula:

[0057]

[0058] in, for and its initial neighborhood instance The feature similarity between them, sim(·,·) represents the dot product operation, express The characteristic vector of Represents the initial neighborhood instance ini i The characteristic vector of When the initial neighborhood instance ini i are identified as neighborhood instances, whereas the others are identified as background instances.

[0059] Step 8. Assign loss weights to the positive instances that cover the target completely through the positive instance weight redistribution strategy. First, construct a positive instance subset SN for the seed instance and its neighboring instances, then calculate the spatial relationship between the instances in SN and construct a spatial distance matrix Then sum up G along the row direction to get the spatial distance vector Find the maximum value in V and mine the complete instance r covering the target z , and find the instance r with the highest prediction category confidence score in the positive instance subset SN p , in the positive instance subset SN, instance r z and r p The loss weights are exchanged to achieve positive instance weight redistribution.

[0060] The positive instance weight redistribution strategy is used to allocate loss weights to positive instances that fully cover the target. The specific implementation steps are as follows:

[0061] (8.1) For the jth seed instance and its neighborhood instances Construct a positive instance subset SN = {r1,...r n ,...,r N}, where r n represents the nth positive instance in the positive instance subset;

[0062] (8.2) By calculating the spatial relationship between instances in the positive instance subset SN, a spatial distance matrix is ​​constructed. The spatial relationship between the instances is established according to the following formula:

[0063]

[0064] Among them, g n,n′ ∈G is a positive instance r n and r n' The spatial distance between them, ∩ and ∪ represent the intersection and union of two target candidate boxes, respectively, r n′ represents the n'th positive instance in the positive instance subset, and n'≠n.

[0065] (8.3) For the spatial distance matrix Sum along the row direction to get the spatial distance vector

[0066]

[0067] (8.4) Find the maximum value in the spatial distance vector V and mine the complete instance r covering the target z , whose index is obtained by:

[0068] z=argmaxV,

[0069] Find the instance r with the highest prediction category confidence score in the positive instance subset SN p , whose index is obtained by:

[0070]

[0071] (8.5) In the positive instance subset SN, instance r z and r p The loss weights between are exchanged to achieve positive instance weight redistribution, as follows:

[0072]

[0073] in, Respectively represent the instance r p and r z The predicted category confidence score of category c in the k-1th instance classification optimization branch;

[0074] Step 9. Construct the loss function for the k-th instance classification optimization branch

[0075]

[0076] in, represents the loss weight of the target candidate box m in the kth branch, Represents y k The element in the cth row and mth column of When it is equal to 1, it means that the target candidate box m contains the target of category c; When it is equal to 0, it means that the target candidate box m does not contain the target of category c; Represents the predicted category confidence score of the target candidate box m on category c in the kth branch;

[0077] Step 10. The task alignment bounding box regression branch performs bounding box regression on the positive instance, taking into account the difficulty of the positive instance and the number of model iterations, and increases the coupling between the model classification task and the regression task, as follows:

[0078] (10.1) Design a modulation factor μ that changes dynamically with model iteration to ensure the stability of training:

[0079]

[0080] Where t represents the current number of iterations, T represents the total number of model iterations; σ represents the hyperparameter used to control the growth rate;

[0081] (10.2) Align the task of the k-th instance classification optimization branch with the bounding box regression loss The definition is as follows:

[0082]

[0083] Among them, P k represents the set of positive instances of the kth instance classification optimization branch, |P k | represents the number of elements in the positive instance set; represents the predicted category confidence score of the corresponding category of the lth positive instance in the kth instance classification optimization branch; and g l denote the target offset and predicted offset of the lth positive instance respectively;

[0084] Step 11. Construct the overall loss L of the weakly supervised remote sensing image target detection model ALL :

[0085]

[0086] Among them, L WSDDN represents the loss function of the baseline weakly supervised object detection network, K represents the number of instance classification optimization branches;

[0087] The loss function L of the baseline weakly supervised object detection network is WSDDN , as follows:

[0088]

[0089] Among them, C represents the number of target categories, y c =1 or 0 means that the input image contains or does not contain the target of the cth category.

[0090] Step 12. Use L ALL The entire weakly supervised remote sensing image target detection model is trained, and the remote sensing image to be tested and its target candidate box are input into the trained model to obtain the category and position of the target of interest in the image, thereby realizing weakly supervised target detection in remote sensing images.

[0091] Embodiment 2: The overall implementation steps of the method proposed in this embodiment are the same as those in Embodiment 1. Figure 2, the implementation process of the present invention is further described in detail from the model construction, high-quality positive instance mining, positive instance weight redistribution and task alignment frame regression loss in the present invention:

[0092] Step 1: Construction of online optimization model OICR for instance classifier

[0093] First, as Figure 2 As shown, given a remote sensing image Where H, W and B represent the width, height and number of channels of the image respectively. The selective search algorithm [JRUijlings, KEvan de Sande, T.Gevers, and A.W.Smeulders. Selective search for object recognition. IJCV, 104(2): 154–171, 2013. 1, 3, 6] is used to generate M target candidate boxes PR = {pr1, ...pr m ,...pr M}, where pr m Represents the mth target candidate box.

[0094] Second, the remote sensing image I and the target candidate box set PR are input into the backbone network, the region of interest pooling and two fully connected layers to obtain the feature vectors of all target candidate boxes. where f m Represents the feature vector of the mth target candidate box.

[0095] Third, the target candidate box feature vector Input two parallel branches in the multi-instance detection network, each branch consists of a fully connected layer and a Softmax classifier. After passing through the two branches, we get two score matrices in category c. So far, the category confidence score matrix of all target candidate boxes in category c Obtained through:

[0096]

[0097] Where ⊙ represents the Hadamard product. Image-level category prediction score φ c Obtained through:

[0098]

[0099] in, represents the category confidence score of the mth target candidate box in category c. So far, the loss function L of the baseline weakly supervised target detection network is WSDDN It is expressed as follows:

[0100]

[0101] Among them, C represents the number of target categories, y c =1 or 0 indicates whether the input image contains the target of the cth category.

[0102] Fourth, if Figure 2 As shown in the lower right corner, K parallel instance classification optimization branches are added to the baseline weakly supervised object detection model. Input the kth instance classification optimization branch to obtain the category confidence score of the target candidate box Among them, the C+1th dimension represents the background category.

[0103] Fifth, the supervision signal y of the k-th instance classification optimization branch k ,pass Generate (see step 2). The loss function of the k-th instance classification optimization branch is as follows:

[0104]

[0105] in, Represents the loss weight of the target candidate box m in the kth branch (see step 3). Represents y k The element in the cth row and mth column of . When it is equal to 1, it means that the target candidate box m contains the target of category c. When it is equal to 0, it means that the target candidate box m does not contain the target of category c. represents the predicted category confidence score of the target candidate box m on category c in the kth branch.

[0106] Step 2: High-quality positive instances mining (HPIM)

[0107] Instance-level pseudo labels include positive instances and negative instances. Positive instances include seed instances and neighborhood instances, and negative instances include background instances. The HPIM module first mines seed instances. After the seed instance is determined, in the label propagation process, the neighborhood instances and background instances are determined based on the spatial similarity and feature similarity between the seed instance and other instances.

[0108] First, through Mining seed instances. In the category c where the image exists, sort the target candidate boxes in descending order by score. Select the first p percent of instances as the initial seed instance set of the cth category in the kth branch. Secondly, the non-maximum suppression operation [J.Hosang, R.Benenson, and B.Schiele, "Learning non-maximum suppression," in Proc.IEEEConf.Comput.Vis.Pattern Recognit., Jul.2017, pp.4507–4515.] is performed on the initial seed instances to remove small and redundant initial seed instances and obtain a seed instance set.

[0109] Second, we mine neighborhood instances by spatial similarity and feature similarity. First, the jth seed instance When the spatial similarity between the instance and other instances is large (i.e., IoU>0.5), the instance is determined to be the initial neighborhood instance, otherwise it is determined to be the background instance. The initial neighborhood instance set Seed Example and its initial neighborhood instance The similarity of features between It is expressed as follows:

[0110]

[0111] where sim(·,·) represents the dot product operation, Represents a seed instance The characteristic vector of Represents the initial neighborhood instance ini i The eigenvector of When the initial neighborhood instance ini i is identified as a neighborhood instance. On the contrary, the initial neighborhood instance is identified as a background instance. Based on this, the seed instance is obtained. The set of neighborhood instances Perform the above operations for all seed instances, and divide all target candidate boxes into seed instances, neighborhood instances, and background instances. Seed instances and neighborhood instances constitute positive samples, and background instances constitute negative samples. Positive samples and negative samples constitute the supervision information y of the kth instance classification optimization branch k .

[0112] Step 3: Positive instances weights re-assigning (PIWR)

[0113] First, for the seed instance and its neighborhood instances Construct a positive instance subset SN = {r1,...r n ,...,r N}, where r n represents the nth positive instance in the subset.

[0114] Second, by calculating the spatial relationship between instances in the positive instance subset SN, a spatial distance matrix is ​​constructed Positive Example n and r n' The spatial distance between n,n′ ∈G is represented as follows:

[0115]

[0116] Third, the spatial distance vector By using the spatial distance matrix The sum along the row direction is obtained, and the expression is as follows:

[0117]

[0118] Fourth, by finding the maximum value in the spatial distance vector V, we can mine instances r that cover the target more completely. z . Its index is obtained as follows:

[0119] z=argmaxV <8>

[0120] Fifth, find the instance r with the highest prediction category confidence score in the positive instance subset SN p , whose index is obtained by:

[0121]

[0122] Sixth, in the positive instance subset SN, the target complete instance r z and the instance r with the highest class confidence score p The loss weights are exchanged between , encouraging the model to focus on the overall target. The details are as follows:

[0123]

[0124] in, Respectively represent the instance r p 、r z The predicted class confidence score for class c in the k-1th instance classification optimization branch.

[0125] Step 4: Task-aligned bounding box regression loss (TABRL)

[0126] First, as Figure 2As shown, a bounding box regression branch is added to each instance classification regression branch, and each bounding box regression branch contains a fully connected layer.

[0127] Second, the weakly supervised target detection model is unstable in the early stage of iteration. A modulation factor μ is designed that changes dynamically with the model iteration. Its definition is as follows:

[0128]

[0129] Among them, t represents the current number of iterations, T represents the total number of model iterations; σ represents a hyperparameter used to control the growth rate.

[0130] Third, the task-aligned bounding box regression loss combines the predicted class confidence score and the modulation factor to increase the coupling between the model classification task and the regression task. The definition is as follows:

[0131]

[0132] Among them, P k represents the set of positive instances of the kth instance classification optimization branch, |P k | represents the number of elements in the positive instance set. represents the predicted category confidence score of the corresponding category of the lth positive instance in the kth instance classification optimization branch, Used to measure the difficulty of the instance; the modulation factor μ is used to ensure the stability of training. and g l denote the target offset and predicted offset of the lth positive instance respectively;

[0133] Step 5: Total model loss

[0134] The overall loss L of the weakly supervised remote sensing image target detection model proposed in this paper is ALL The definition is as follows:

[0135]

[0136] Where K represents the number of instance classification optimization branches. ALL The entire weakly supervised remote sensing image target detection model is trained. In the inference stage, the remote sensing image to be tested and its target candidate box are input into the trained model to obtain the category and location of the target of interest in the image.

[0137] The effect of the present invention will be further described below in conjunction with experiments.

[0138] 1. Experimental conditions:

[0139] The hardware configuration for the implementation of the present invention is: E5-2650V4 CPU (2.2GHz 12x2 cores), 512GB memory, and a workstation with 8 NVIDIA RTX Titan graphics cards for experiments. The platform configuration of the software is: Ubuntu16.04, Python3.7, and Pytorch1.7.

[0140] 2. Experimental content:

[0141] The detection performance of the present invention on the DIOR dataset is compared with the following 10 popular algorithms in the prior art: WSDDN [H.Bilen, A.Vedaldi, Weakly supervised deep detection networks, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 2846–2854], OICR [P.Tang, X.Wang, X.Bai, W.Liu, Multiple instance detection network with online instanceclassifier refinement, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 3059–295 3067], PCL [P.Tang, X.Wang, S.Bai, W.Shen, X.Bai, W.Liu, A.L.Yuille, PCL: proposal cluster learning for weakly supervised object detection, IEEE Trans. Pattern Recognit., 2017, pp. 3067–3071], and PCL [P.Tang, X.Wang, S.Bai, W.Shen, X.Bai, W.Liu, A.L.Yuille, PCL: proposal cluster learning for weakly supervised object detection, IEEE Trans. Pattern Recognit., 2017, pp. 3067–3071]. Anal.Mach.Intell.42(1)(2020)176–191], MELM [F.Wan, P.Wei, J.Jiao, Z.Han, and Q.Ye, “Min-entropy latent model for weakly supervised object detection,” in Proc.IEEE Conf.Comput.Vis.Pattern Recognit.,Jun.2018,pp.1297–1306.], DCL [X.Yao, Sens.59(1)(2021)675–685],MIST[Z.Ren,Z.Yu,X.Yang,M.-Y.Liu,YJLee,AGSchwing,and J.Kautz, “Instance-aware, context-focused, and memory-efficient weakly supervised object detection,” in Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2020, pp. 10598–10607.],PCIR[X. Feng, J. Han, X. Yao, G. Cheng, Progressive contextual instance refinement for weakly supervised object detection in remote sensing images, IEEE Trans. Geosci. Remote Sens. 58(11)(2020)8002–8012], TCA[X. Feng, J. Han, X. Yao, and G. Cheng, “Tcanet: Triple context-aware network for weakly supervised object detection in remote sensing images,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 8, pp. 6946–6955, Oct. 2021.],MIG[B. Wang, Y. Zhao, X. Li, Multiple instance graph learning for weakly supervised remote sensing object detection, IEEE Trans. Geosci. Remote Sens. 60(2022)1–12],SPG[G. Cheng, X. Xie, W. Chen, X. Feng, X. Yao, and J. Han, “Self guided proposal generation for weakly supervised object detection,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no. 5625311.]. mAP and Corloc represent the mean average precision and the localization precision respectively.

[0142] 3. Simulation results and analysis:

[0143] On the DIOR dataset, the present invention is compared with 10 popular algorithms in the prior art, and the comparison results of mean average precision (mAP) and correct localization (CorLoc) are shown in Table 1:

[0144] Table 1

[0145] method mAP CorLoc WSDDN 13.3 32.4 OICR 16.5 34.8 PCL 18.2 41.5 MELM 18.7 43.3 DCL 20.2 42.2 MIST 22.2 43.6 PCIR 24.9 46.1 TCA 25.8 48.4 MIG 25.1 46.8 SPG 25.8 48.3 The present invention 28.9 53.9

[0146] The above simulation analysis proves the correctness and effectiveness of the method proposed in the present invention. It can be seen from Table 1 that the average precision (mAP) of the present invention is 15.6%, 12.4%, 10.7%, 10.2%, 8.7%, 6.7%, 4.0%, 3.1%, 3.8%, and 3.1% higher than that of WSDDN, OICR, PCL, EELM, DCL, MIST, PCIR, TCA, MIG, and SPG respectively. The positioning accuracy (CorLoc) is 21.5%, 19.1%, 12.4%, 10.6%, 11.7%, 10.3%, 7.8%, 5.5%, 7.1%, and 5.6% higher than that of WSDDN, OICR, PCL, EELM, DCL, MIST, PCIR, TCA, MIG, and SPG respectively. It can be seen that the model proposed in the present invention can more accurately identify and locate the ground objects in the remote sensing image than the prior art, and effectively improves the detection effect of the weakly supervised targets in the remote sensing image.

[0147] Weakly supervised remote sensing image target detection has shown great potential in the fields of military, urban planning, disaster detection, transportation construction and ecological research. The present invention can accurately detect specific targets in remote sensing images containing rich information and unique perspectives, solving the problem of a large amount of manpower and material resources required for supervised learning that relies on complete annotated information, making remote sensing image target detection more efficient and economical, which is crucial for decision support and practical operations in various fields. The method of the present invention provides more accurate and efficient support for the application field of weakly supervised remote sensing image target detection.

[0148] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A weakly supervised target detection method for remote sensing images based on weight redistribution and task alignment, characterized in that: The implementation steps include the following: (1) Given a remote sensing image Where H, W and B represent the width, height and number of channels of the image respectively. The selective search algorithm is used to generate M target candidate frames for the remote sensing image I, forming a target candidate frame set PR = {pr1, pr2, ...pr m ,pr M }, where pr m Represents the mth target candidate box; (2) The remote sensing image I and the target candidate box set PR are input into the instance classifier online optimization model OICR, which includes a backbone network, a multi-instance detection network, an instance classification optimization branch, and a task alignment bounding box regression branch; the feature vectors of all target candidate boxes are obtained through the backbone network, the region of interest pooling, and two fully connected layers. And feed it into the multi-instance detection network, where f m Represents the feature vector of the mth target candidate box; (3) Target candidate box feature vector Input two parallel branches in the multi-instance detection network, and obtain two score matrices in category c after passing through the two branches respectively. Obtain the category confidence score matrix of the target candidate box in category c according to the following formula: Where ⊙ represents the Hadamard product, c is any one of all categories C; The image-level category prediction score φ is obtained according to the following formula c : in, Represents the category confidence score of the mth target candidate box in category c; (4) Add K parallel instance classification optimization branches to the baseline weakly supervised object detection model, and Input the kth instance classification optimization branch to obtain the category confidence score of the target candidate box Among them, C+1 represents the background category; (5) The category confidence score of the k-1th branch As the input for high-quality positive instance mining, we get the initial seed instance set Then perform non-maximum suppression operation on it to obtain a set of seed instances Implement seed instance mining; (6) Mining the initial neighborhood instances through spatial similarity, setting the spatial similarity judgment threshold ζ, when the jth seed instance When the spatial similarity IoU between the instance and other instances is greater than the threshold ζ, the instance is determined to be the initial neighborhood instance, otherwise it is determined to be the background instance; the seed instance is obtained. The initial neighborhood instance set (7) By using the feature similarity between instances, we can eliminate the misclassified initial neighborhood instances and set the feature similarity threshold δ. When the jth seed instance When the feature similarity between the initial neighborhood instance and the background instance is less than δ, the initial neighborhood instance is judged as a background instance and the jth seed instance is obtained. The set of neighborhood instances Get the neighborhood instance set of all seed instances and obtain the instance-level supervision information y of the kth instance classification optimization branch k ; (8) The positive instance weight redistribution strategy is used to assign loss weights to positive instances that cover the entire target. First, a positive instance subset SN is constructed for the seed instance and its neighboring instances. Then, the spatial relationship between instances in SN is calculated to construct a spatial distance matrix Then sum up G along the row direction to get the spatial distance vector Find the maximum value in V and mine the complete instance r covering the target z , and find the instance r with the highest prediction category confidence score in the positive instance subset SN p , in the positive instance subset SN, instance r z and r p The loss weights are exchanged to achieve positive instance weight redistribution; (9) Construct the loss function of the kth instance classification optimization branch in, represents the loss weight of the target candidate box m in the kth branch, Represents y k The element in the cth row and mth column of When it is equal to 1, it means that the target candidate box m contains the target of category c; When it is equal to 0, it means that the target candidate box m does not contain the target of category c; Represents the predicted category confidence score of the target candidate box m on category c in the kth branch; (10) The task-aligned bounding box regression branch performs bounding box regression on the positive instance, taking into account the difficulty of the positive instance and the number of model iterations, and increases the coupling between the model classification task and the regression task, as follows: (10.1) Design a modulation factor μ that changes dynamically with model iteration to ensure the stability of training: Where t represents the current number of iterations, T represents the total number of model iterations; σ represents the hyperparameter used to control the growth rate; (10.2) Align the task of the k-th instance classification optimization branch with the bounding box regression loss The definition is as follows: Among them, P k represents the set of positive instances of the kth instance classification optimization branch, |P k | represents the number of elements in the positive instance set; represents the predicted category confidence score of the corresponding category of the lth positive instance in the kth instance classification optimization branch; and g l denote the target offset and predicted offset of the lth positive instance respectively; (11) Construct the overall loss L of the weakly supervised remote sensing image target detection model ALL : Among them, L WSDDN represents the loss function of the baseline weakly supervised object detection network, K represents the number of instance classification optimization branches; (12) Using L ALL The entire weakly supervised remote sensing image target detection model is trained, and the remote sensing image to be tested and its target candidate box are input into the trained model to obtain the category and position of the target of interest in the image, thereby realizing weakly supervised target detection in remote sensing images.

2. The method according to claim 1, characterized in that: The two parallel branches in the multi-instance detection network described in step (3) are both composed of a fully connected layer and a Softmax classifier.

3. The method according to claim 1, characterized in that: The initial seed instance set described in step (5) Specifically, through Sort in descending order, and then select the first p percent of instances as the initial seed instance set of the cth category in the kth branch 4. The method according to claim 1, characterized in that: The jth seed instance described in step (7) and the initial neighborhood instance ini i The feature similarity between them is obtained according to the following formula: in, for and its initial neighborhood instance The feature similarity between them, sim(·,·) represents the dot product operation, express The characteristic vector of Represents the initial neighborhood instance ini i The characteristic vector of When the initial neighborhood instance ini i are identified as neighborhood instances, whereas the others are identified as background instances.

5. The method according to claim 1, characterized in that: In step (8), the loss weights are assigned to the positive instances that completely cover the target through the positive instance weight redistribution strategy. The specific implementation steps are as follows: (8.1) For the jth seed instance and its neighborhood instances Construct a positive instance subset SN = {r1,...r n ,...,r N }, where r n represents the nth positive instance in the positive instance subset; (8.2) By calculating the spatial relationship between instances in the positive instance subset SN, a spatial distance matrix is ​​constructed. (8.3) For the spatial distance matrix Sum along the row direction to get the spatial distance vector (8.4) Find the maximum value in the spatial distance vector V and mine the complete instance r covering the target z , whose index is obtained by: z=argmaxV, Find the instance r with the highest prediction category confidence score in the positive instance subset SN p , whose index is obtained by: (8.5) In the positive instance subset SN, instance r z and r p The loss weights between are exchanged to achieve positive instance weight redistribution, as follows: in, Respectively represent the instance r p and r z The predicted class confidence score for class c in the k-1th instance classification optimization branch.

6. The method according to claim 5, characterized in that: The spatial relationship between the instances described in step (8.2) is established according to the following formula: Among them, g n,n′ ∈G is a positive instance r n and r n' The spatial distance between them, ∩ and ∪ represent the intersection and union of two target candidate boxes, respectively, r n′ represents the n'th positive instance in the positive instance subset, and n'≠n.

7. The method according to claim 1, characterized in that: The loss function L of the baseline weakly supervised target detection network described in step (11) is WSDDN , as follows: Among them, C represents the number of target categories, y c =1 or 0 means that the input image contains or does not contain the target of the cth category.

Images