Weakly supervised remote sensing object detection method based on feature consistency and localization update

Abstract

The present application relates to the technical field of image processing, and particularly relates to a weakly supervised remote sensing target detection method based on feature consistency and positioning update, comprising: through a spatial voting multi-instance mining module, spatial voting is carried out in combination with instance scores and spatial relationships, and boundary regularization processing is carried out on the voting results, to obtain pseudo labels of each input image respectively; through a feature consistency learning module, non-maximum suppression is carried out on the pseudo labels under different angles of the input image, to obtain comprehensive pseudo labels; a target positioning update module is trained by using the comprehensive pseudo labels, and in each round of training, the prediction results meeting the preset conditions are combined with the original comprehensive pseudo labels and then participate in the next round of training, and the iteration training is carried out, to obtain the final target positioning update module. The present application can avoid target instance loss and improve target positioning accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a weakly supervised remote sensing target detection method based on feature consistency and localization update. Background Technology

[0002] Object detection, as an important task in the field of remote sensing image interpretation, has always received attention from researchers. Recently, due to the availability of large amounts of finely labeled data and the rapid development of convolutional neural networks, object detection has advanced rapidly. However, for high-resolution remote sensing images, due to the large number of targets in the images, labeling each object is both time-consuming and labor-intensive. To reduce the workload of fine labeling, weakly supervised target mapping has been extensively studied.

[0003] Weakly supervised target detection in remote sensing images plays an important role in remote sensing image interpretation.

[0004] However, due to technological limitations, several challenges remain: First, in remote sensing images, many instances of the same category can appear in any direction, inevitably leading to inconsistencies in the features of similar objects. Current popular CNN-based methods lack the ability to actively classify these inconsistent features into the same category, easily resulting in instance loss. Second, images often contain multiple objects of the same category, and current methods typically mine the most reliable instances without considering the number of instances. For images with multiple instances of the same category, lower-scoring instances may be treated as background. Furthermore, the most reliable suggestions tend to focus on the most discriminative parts of the object, leading the model to detect only a small portion of the object. Finally, most current methods tend to consistently use the same scheme when selecting pseudo-labels in the second stage. However, as the detector continues to train, the pseudo-labels selected in the second stage are still not accurate enough for target localization, limiting the detector's detection performance.

[0005] Therefore, how to avoid the loss of target instances and improve the accuracy of target location is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a weakly supervised remote sensing target detection method based on feature consistency and localization update, which can avoid target instance loss and improve target localization accuracy.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A weakly supervised remote sensing target detection method based on feature consistency and location update includes:

[0009] A target detection model is constructed, which includes a basic network, a spatial voting multi-instance mining module, a feature consistency learning module, and a target localization and update module.

[0010] During the training phase, the original sample images are randomly rotated and then input into the base network along with the original sample images.

[0011] The basic network is used to extract features and perform preliminary detection on each input image to obtain preliminary detection results for each input image.

[0012] The spatial voting multi-instance mining module analyzes the preliminary detection results, combines instance scores and spatial relationships to perform spatial voting, and performs boundary regularization on the voting results to obtain pseudo-labels for each input image.

[0013] The feature consistency learning module performs non-maximum suppression on the pseudo-labels of each input image to obtain the comprehensive pseudo-label H. ref ;

[0014] Using the aforementioned comprehensive pseudo-label H ref The target localization update module is trained, and in each training round, the prediction results that meet the preset conditions are retained. The prediction results that meet the preset conditions are then merged with the original comprehensive pseudo-labels and participated in the next training round. This process is repeated to obtain the final target localization update module.

[0015] Furthermore, the basic network includes a backbone network and a weakly supervised network; the backbone network extracts features from each input image to obtain feature maps, and uses the feature maps and proposals to perform region pooling, and after passing through a fully connected layer, all proposals are transformed into proposal feature vectors of the same size;

[0016] The weakly supervised network performs preliminary detection on the proposal feature vector, obtaining preliminary detection results φ. 0 .

[0017] Furthermore, in the target detection stage, the trained target localization and update module is combined with the backbone network to obtain the final target detection model, and target detection is performed on the image to be processed based on the final target detection model.

[0018] Furthermore, the spatial voting multi-instance mining module performs the same pseudo-label mining process for each input image, wherein the pseudo-label mining process for the original sample image includes:

[0019] The proposal set B = {b1,...b1} in the preliminary test results r ,...b R Scoring will be conducted, and those scoring below the threshold T will be... scoreAfter filtering out proposals, we obtain proposal sets B for each category. c ={b r |p cr >T score}, where p cr Let b represent the score of the r-th proposal. r This indicates that the score is higher than the score threshold T. score The r-th proposal, where c represents the category;

[0020] Constructing Proposal Set B c Spatial probability score matrix Where H and W represent the height and width of the image, respectively;

[0021] The spatial probability score matrix M c The elements in the image are binarized, and all connected components within the binarized image are selected using a minimum bounding rectangle to obtain the result. Among them, g m N represents the m-th rectangle. c Represents the number of all connected components;

[0022] For each proposal cluster S under each category m Construct an undirected spatial graph and calculate the bounding box g. m and Proposal Cluster S m Intersection over Union (IOU) among the proposals; proposal cluster S m Proposal b in r As vertices of the undirected graph, edges are used as the IOU relationship between proposals. If the IOU is greater than a specified threshold, it is 1; otherwise, it is 0. The proposal with the most connections in the undirected graph is selected as the pseudo-label of the proposal cluster.

[0023] By combining the pseudo-labels under all categories, we obtain the pseudo-label H of the original sample image. s .

[0024] Furthermore, Proposal B c Spatial probability score matrix M c The construction process is as follows:

[0025] The score matrix M c Initialize to 0, and assign the score of each proposal to the score matrix M according to the following formula. c Spatial accumulation is performed:

[0026]

[0027] Where (i,j)∈b r , indicating b r Internal pixels; Representation matrix M cThe sum of scores for the middle pixel (i,j); B c The final score p of one element cr This represents the probability that category c appears at that position.

[0028] Furthermore, the proposal clusters under all categories are as follows:

[0029]

[0030] Among them, S m ={b r |iou(g m B) > T iou} represents the cluster of proposals for calculating the intersection-union ratio with the m-th rectangle; T iou The crossover ratio (CR) is set to a specified threshold.

[0031] Furthermore, the original sample image I is randomly rotated twice to obtain images I1 and I2 and corresponding proposals B1 and B2. The original sample image I and images I1 and I2 are then sequentially passed through the basic network and the spatial voting multi-instance mining module to obtain the pseudo-label H. s , The feature consistency learning module is used to process the three pseudo-labels H. s , By merging, we obtain the comprehensive pseudo-label H. ref .

[0032] Furthermore, in each training round of the target localization update module, the original comprehensive pseudo-label H is calculated. ref Compared with the prediction result H pef The Intersection over Union (IoU) is used to retain prediction results with an IoU greater than 0.3 with the original composite pseudo-label. And With the original composite pseudo-label H ref The final pseudo-label is obtained by merging. pseudo-tags Participate in the next round of training; pseudo-labels The expression is as follows:

[0033]

[0034] in, Indicates the prediction result One of the elements.

[0035] As can be seen from the above technical solution, compared with the prior art, this invention discloses a weakly supervised remote sensing target detection method based on feature consistency and localization update. First, the same image under different states is simultaneously input into a weakly supervised deep basic detection network. By combining different detection results, similar instances from different angles are captured as supervision, which can effectively reduce feature inconsistencies between similar objects. Then, a spatial voting multi-instance mining module uses instance scores and spatial relationships to perform spatial voting, and the voting results are regularized at the boundary, enabling more instances to serve as supervision. This helps the target localization update module detect more objects and alleviates the problem of the target localization update module only focusing on the target portion. Finally, during training, intermediate predictions are used to supplement pseudo-labels, greatly improving the problem of inaccurate localization in the target localization update module. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 A flowchart of the weakly supervised remote sensing target detection method based on feature consistency and location update provided by the present invention;

[0038] Figure 2 This is a schematic diagram of an undirected space graph provided by the present invention;

[0039] Figure 3 The feature consistency learning flowchart provided by this invention;

[0040] Figure 4 The operation flowchart of the target positioning and update module provided by the present invention;

[0041] Figure 5 The comparison chart of detection results of the present invention and the prior art is provided for the present invention;

[0042] Figure 6 This is a schematic diagram illustrating the detection results of the present invention in other aspects. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] like Figure 1 As shown in the figure, this invention discloses a weakly supervised remote sensing target detection method based on feature consistency and localization update, including:

[0045] A target detection model is constructed, which includes a basic network, a spatial voting multi-instance mining module, a feature consistency learning module, and a target localization and update module.

[0046] During the training phase, the original sample images are randomly rotated and then input into the base network along with the original sample images.

[0047] The base network performs feature extraction and preliminary detection on each input image to obtain preliminary detection results for each input image. The base network includes a backbone network and a weakly supervised network. The backbone network extracts features from each input image to obtain feature maps, and uses these feature maps and proposals for region pooling. After passing through a fully connected layer, all proposals are transformed into proposal feature vectors of the same size. The weakly supervised network performs preliminary detection on the proposal feature vectors to obtain preliminary detection results φ. 0 The preliminary detection results include the image and its proposal B = {b1,...b}. r ,...b R Proposal B was generated using a selective search algorithm.

[0048] The spatial voting multi-instance mining module analyzes the preliminary detection results, combines instance scores and spatial relationships to perform spatial voting, and performs boundary regularization on the voting results to obtain pseudo-labels for each input image.

[0049] The feature consistency learning module performs non-maximum suppression on the pseudo-labels of each input image to obtain the comprehensive pseudo-label H. ref ;

[0050] Using the aforementioned comprehensive pseudo-label H ref The target localization update module is trained, and in each training round, the prediction results that meet the preset conditions are retained. The prediction results that meet the preset conditions are then merged with the original comprehensive pseudo-labels and participated in the next training round. This process is repeated to obtain the final target localization update module.

[0051] In one embodiment, during the target detection stage, the trained target localization update module is combined with the backbone network to obtain the final target detection model, and target detection is performed on the image to be processed based on the final target detection model.

[0052] In one specific embodiment, the spatial voting multi-instance mining module performs the same process for mining pseudo-labels for each input image. Specifically, taking the original sample image as an example, the process for mining pseudo-labels for the original sample image includes:

[0053] The proposal set B = {b1,...b1} in the preliminary test results r ,...b R Scoring will be conducted, and those scoring below the threshold T will be... score After filtering out proposals, we obtain proposal sets B for each category. c ={b r |p cr >T score}, where p cr Let b represent the score of the r-th proposal. r This indicates that the score is higher than the score threshold T. score The r-th proposal, where c represents the category; in this step, to save training time, T is set... score This allows you to filter out proposals with lower scores.

[0054] Constructing Proposal Set B c Spatial probability score matrix Where H and W represent the height and width of the image, respectively; the scoring matrix M c Initialize to 0, and assign the score of each proposal to the score matrix M according to the following formula. c Spatial accumulation is performed:

[0055]

[0056] Where (i,j)∈b r , indicating b r Internal pixels; Representation matrix M c The sum of scores for the middle pixel (i,j); B c The final score p of one element cr This represents the probability that category c appears at that position.

[0057] The spatial probability score matrix M c The elements in the image are scaled to [0, 255], and a threshold is set for binarization. Then, all connected components within the binarized image are selected using a minimum bounding rectangle to obtain the desired result. Among them, g m N represents the m-th rectangle. c This represents the number of all connected components.

[0058] For each proposal cluster S under each category m Construct an undirected spatial graph and calculate the bounding box g. mand Proposal Cluster S m The spatial relationship between the various proposals, i.e., g m and B c The intersection-union ratio (IOU) among all proposals; where the proposal clusters under all categories are:

[0059]

[0060] Among them, S m ={b r |iou(g m B) > T iou} represents the intersection-union ratio with the m-th rectangle being greater than T. iou The proposal cluster; T iou The crossover ratio (CR) is set to a specified threshold.

[0061] With proposal cluster S m Proposal b in r As vertices of the undirected graph, edges are used to represent the Intersection over Union (IoU) between proposals. If the IoU is greater than a specified threshold, it is 1 (connected); otherwise, it is 0 (unconnected). The proposal with the most connections in the undirected graph is selected as the pseudo-label for that proposal cluster. The pseudo-labels for all members of category C are then H. c .

[0062] like Figure 2 As shown, solid lines represent connections of 1, and dashed lines represent connections of 0. Among them, b1 has the most connections, so b1 is selected as the pseudo-label for this proposal cluster.

[0063] Finally, the pseudo-labels under all categories are combined to obtain the pseudo-label H of the original sample image. s .

[0064] In one embodiment, since objects in remote sensing images are often distributed in the image at random angles, the present invention rotates the image at different angles and sends it into the same network for detection, and then couples the different detection results into pseudo-labels, forcing the detector (target localization update module) to make the same prediction for the same instance from different angles.

[0065] like Figure 3 The diagram illustrates the process of pseudo-label consistency learning by the feature consistency learning module. Specifically, the original sample image I undergoes two random rotations to obtain images I1 and I2, and corresponding proposals B1 and B2. The original sample image I and images I1 and I2 are then sequentially passed through the BaseNet network for preliminary detection and the spatial voting multi-instance mining module for instance mining to obtain the pseudo-label H. s , The feature consistency learning module is used to process the three pseudo-labels H.s , Non-maximum suppression (NMS) is performed to obtain the comprehensive pseudo-label H. ref .

[0066] In one embodiment, during training, it was found that the model's prediction accuracy at certain stages was better than the pseudo-label localization performance selected by the original method. However, consistently using the same method to select pseudo-labels throughout training would eventually lead to inaccurate predictions. Therefore, this invention considers fusing the model's stage-specific prediction results with the original pseudo-labels to improve the model's detection performance. In each training iteration of the target localization update module, the original comprehensive pseudo-label H is calculated. ref Compared with the prediction result H pre The Intersection over Union (IoU) is used to retain prediction results with an IoU greater than 0.3 with the original composite pseudo-label. And With the original composite pseudo-label H ref The final pseudo-label is obtained by merging. pseudo-tags Participate in the next round of training; pseudo-labels The expression is as follows:

[0067]

[0068] in, Indicates the prediction result One of the elements.

[0069] like Figure 4 The diagram illustrates the operation flow of the target localization update module. (a) represents the original pseudo-label, (b) represents the current prediction result, and (c) represents the pseudo-label selected by the target localization update module. The line between (a) and (b) represents the Interchange of Units (IOU) calculation; a solid line indicates an IOU greater than the threshold, and a dashed line indicates an IOU less than the threshold. The results show that the original pseudo-label (a) did not accurately locate the target. The target localization update module of this invention obtains the pseudo-label (c), which provides more accurate target localization, thereby improving target detection performance.

[0070] To further verify the performance of the target detection model trained in this invention, the following experiments were conducted.

[0071] This invention relates to weakly supervised remote sensing image target detection. It utilizes the commonly used target detection evaluation metric mAP to assess detection performance, and simultaneously compares qualitative detection results to verify the effectiveness of the proposed method. The experimental dataset is the Northwest University of Technology (NWPU) VHR-10.v2 dataset, an optical remote sensing image (DIOR) dataset. Higher mAP indicates better detection performance. Detection results are shown in Table 1.

[0072] Table 1. Performance comparison of different methods on the NWPU VHR-10.v2 and DIOR datasets (mAP (%))

[0073]

[0074] This invention first compares its detection performance with some state-of-the-art methods on the NWPU VHR-10.v2 dataset. As shown in the second column of Table 1, the method of this invention achieves the best detection results on the NWPU VHR-10.v2 dataset. Furthermore, to demonstrate the robustness of the method, validation experiments were conducted on the larger and more challenging remote sensing dataset DIOR, and the experimental results also prove the effectiveness of the method.

[0075] To verify the effectiveness of the three modules proposed in this invention, ablation comparison experiments were conducted on the DIOR dataset. The experimental results are shown in Table 2, where FCL represents the feature consistency learning module, SVIM spatial voting multi-instance mining module, and OLA represents the target localization and update module. The experimental results in Table 2 demonstrate the effectiveness of each module of this invention, as well as the effectiveness of the combination of modules.

[0076] Table 2 Ablation experiments of this invention on the DIOR dataset.

[0077] Baseline √ √ √ √ √ √ +FCL √ √ √ +SVIM √ √ √ +OLA √ √ mAP(mAP) 16.5 18.0 19.3 19.0 21.9 26.0

[0078] The baseline detection results and the detection results of this invention were visualized. Figure 5 The image shows three sets of images, with the baseline results for each set on the left and the results of this invention on the right. The method of this invention alleviates the problem of instance loss: (1) The feature consistency learning module mines more similar objects ( Figure 5 (1) Left side); (2) Spatial voting multi-instance mining module helps the network find multiple objects ( Figure 5 (Central region). Meanwhile, the target location update module also alleviates the problem of inaccurate network positioning to some extent. Figure 5 (right side).

[0079] Figure 6 This paper presents more qualitative results of the present invention on two datasets. Compared to the baseline, the method can effectively address the issues of missing instances and inaccurate target localization in some complex remote sensing scenarios. The DIOR dataset is more challenging to detect on than NWPU VHR-10.v2, therefore the DIOR dataset contains more instances and categories, and the detection scenarios are more complex, yet the method of the present invention still performs well in these scenarios.

[0080] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0081] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A weakly supervised remote sensing target detection method based on feature consistency and localization update, characterized in that, include: A target detection model is constructed, which includes a basic network, a spatial voting multi-instance mining module, a feature consistency learning module, and a target localization and update module. During the training phase, the original sample images are randomly rotated and then input into the base network along with the original sample images. The basic network performs feature extraction and preliminary detection on each input image to obtain preliminary detection results for each input image. The basic network includes a backbone network and a weakly supervised network. The backbone network extracts features from each input image to obtain feature maps, and uses these feature maps and proposals for region pooling. After passing through a fully connected layer, all proposals are transformed into proposal feature vectors of the same size. The weakly supervised network performs preliminary detection on the proposal feature vectors to obtain preliminary detection results φ. 0 ; The spatial voting multi-instance mining module analyzes the preliminary detection results, combines instance scores and spatial relationships to perform spatial voting, and performs boundary regularization on the voting results to obtain pseudo-labels for each input image. The feature consistency learning module performs non-maximum suppression on the pseudo-labels of each input image to obtain the comprehensive pseudo-label H. ref The process of consistency learning for pseudo-labels of an input image includes: performing two random rotations on the original sample image I to obtain images I1 and I2 and corresponding proposals B1 and B2; and then passing the original sample image I and images I1 and I2 sequentially through the basic network and the spatial voting multi-instance mining module to obtain the pseudo-label H. S , The feature consistency learning module is used to process the three pseudo-labels H. s , Nonmaximum suppression is performed to obtain the comprehensive pseudo-label H. ref ; Using the aforementioned comprehensive pseudo-label H ref The target localization update module is trained, and in each training round, the prediction results that meet the preset conditions are retained. The prediction results that meet the preset conditions are then merged with the original comprehensive pseudo-labels and participated in the next training round. This process is repeated to obtain the final target localization update module.

2. The weakly supervised remote sensing target detection method based on feature consistency and localization update according to claim 1, characterized in that, In the target detection stage, the trained target localization and update module is combined with the backbone network to obtain the final target detection model, and target detection is performed on the image to be processed based on the final target detection model.

3. The weakly supervised remote sensing target detection method based on feature consistency and localization update according to claim 1, characterized in that, The spatial voting multi-instance mining module employs a consistent process for mining pseudo-labels for each input image. Specifically, the process for mining pseudo-labels for the original sample images includes: The proposal set B = {b1,...b1} in the preliminary test results r ,...b R Scoring will be conducted, and those scoring below the threshold T will be... score After filtering out proposals, we obtain proposal sets B for each category. c ={b r |p cr >T score }, where p cr Let b represent the score of the r-th proposal. r This indicates that the score is higher than the score threshold T. score The r-th proposal, where c represents the category; Constructing Proposal Set B c Spatial probability score matrix Where H and W represent the height and width of the image, respectively; The spatial probability score matrix M c The elements in the image are binarized, and all connected components within the binarized image are selected using a minimum bounding rectangle to obtain the result. Among them, g m N represents the m-th rectangle. c Represents the number of all connected components; For each proposal cluster S under each category m Construct an undirected spatial graph and calculate the bounding box g. m and Proposal Cluster S m Intersection over Union (IOU) among the proposals; proposal cluster S m Proposal b in r As vertices of the undirected graph, edges are used as the IOU relationship between proposals. If the IOU is greater than a specified threshold, it is 1; otherwise, it is 0. The proposal with the most connections in the undirected graph is selected as the pseudo-label of the proposal cluster. By combining the pseudo-labels under all categories, we obtain the pseudo-label H of the original sample image. s .

4. The weakly supervised remote sensing target detection method based on feature consistency and localization update according to claim 3, characterized in that, Proposal B c Spatial probability score matrix M c The construction process is as follows: The score matrix M c Initialize to 0, and assign the score of each proposal to the score matrix M according to the following formula. c Spatial accumulation is performed: Where (i,j)∈b r , indicating b r Internal pixels; Representation matrix M c The sum of scores for the middle pixel (i,j); B c The final score p of one element cr This represents the probability that category c appears at that position.

5. The weakly supervised remote sensing target detection method based on feature consistency and localization update according to claim 3, characterized in that, The proposal clusters under all categories are: Among them, S m ={b r |iou(g m ,B)>T iou } represents the intersection-union ratio with the m-th rectangle being greater than T. iou The proposal cluster; T iou The crossover ratio (CR) is set to a specified threshold.

6. The weakly supervised remote sensing target detection method based on feature consistency and localization update according to claim 1, characterized in that, In each training round of the target localization update module, the original comprehensive pseudo-label H is calculated. ref Compared with the prediction result H pre The Intersection over Union (IoU) is used to retain prediction results with an IoU greater than 0.3 with the original composite pseudo-label. And With the original composite pseudo-label H rep The final pseudo-label is obtained by merging. pseudo-tags Participate in the next round of training; pseudo-labels The expression is as follows: in, Indicates the prediction result One of the elements.

Images