A method, apparatus, device, and medium for detecting targets in remote sensing images.

By optimizing the horizontal bounding boxes of remote sensing images through point label annotation and multi-instance learning algorithms, and combining weakly supervised learning to train the rotating bounding box detector, the problems of high annotation cost and insufficient accuracy in target detection of remote sensing images are solved, and low-cost and high-precision target detection is achieved.

CN118644658BActive Publication Date: 2026-11-17NORTHWESTERN POLYTECHNICAL UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410753489.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-12
Publication Date
2026-11-17
Estimated Expiration
2044-06-12

AI Technical Summary

Technical Problem

Existing remote sensing image target detection technologies suffer from high dataset annotation costs and insufficient detection accuracy, especially in dense target scenes where it is difficult to effectively distinguish and annotate rotated bounding box targets.

Method used

Point labels are used for target annotation. The bounding horizontal boxes of the mask image are generated by the image segmentation model as the initial boxes. The target horizontal box generator is trained by combining the multi-instance learning algorithm and the weakly supervised learning algorithm. The initial horizontal boxes are optimized to generate the target horizontal boxes. Finally, the rotated box detector is used for target detection.

Benefits of technology

It effectively reduces the training cost of target detection in remote sensing images, improves detection accuracy, enhances the utilization of the dataset, and achieves higher detection accuracy in dense target scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118644658B_ABST
    Figure CN118644658B_ABST
Patent Text Reader

Abstract

The target detection method in remote sensing images disclosed in this application includes: acquiring a remote sensing image with target annotation using point labels, inputting it into an image segmentation model to obtain a mask image corresponding to the point labels, using the bounding horizontal box of the mask image as the initial horizontal box, and acquiring an initial remote sensing image with target annotation using the initial horizontal box; training a target horizontal box generator using several unannotated remote sensing images and corresponding initial remote sensing images; using the target horizontal box generator, optimizing the initial horizontal box in the initial remote sensing image using the unannotated remote sensing image to generate target horizontal boxes, and obtaining a target remote sensing image with target annotation using the target horizontal box; using several target remote sensing images as training data, jointly training a rotated box detector using a weakly supervised learning algorithm and a self-supervised learning algorithm, and using the trained rotated box detector to detect targets in the remote sensing image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a method, apparatus, device and medium for detecting targets in remote sensing images. Background Technology

[0002] In recent years, as the horizontal bounding box target detection technology has matured, the horizontal bounding box detection in the field of remote sensing has also encountered many bottlenecks due to the special characteristics of targets in remote sensing images: (1) the size and aspect ratio of the bounding box cannot reflect the true shape of the target object; (2) the object and background pixels are not effectively separated; (3) densely arranged objects are difficult to distinguish intuitively; (4) the use of non-maximum suppression in the post-processing stage will produce serious erroneous filtering of dense target groups.

[0003] At this time, rotated bounding box (ROBC) object detection technology has gradually come into focus. Currently, fully supervised ROBC detection algorithms have been extensively studied and have achieved high performance. These fully supervised detection algorithms all rely on detailed ROBC annotations. However, due to the diverse target scales and complex dense scenes in remote sensing images, many existing datasets only use horizontal bounding boxes for annotation, making it impossible to train ROBC object detectors. In such cases, the original dataset needs to be re-annotated; however, annotating ROBCs requires significantly more labor and resources than annotating horizontal bounding boxes, almost doubling the workload. Therefore, the ease of obtaining training corpora and the demand for ROBC object detection have become an implicit contradiction under these factors. Against this backdrop, it is essential to seek a feasible method to reduce the dependence of ROBC object detection on dataset annotation.

[0004] However, current methods only employ weakly supervised orientation detection algorithms at the horizontal bounding box level. These algorithms leverage the correlation between matching horizontal bounding boxes and the bounding boxes of rotated bounding boxes to supervise the training of orientation target detectors, thereby reducing annotation costs. However, horizontal bounding boxes are not the most suitable annotation objects for remote sensing images. When dealing with densely packed targets, such as ships in a port, horizontal bounding box annotation remains difficult. Furthermore, applying the matching of horizontal bounding boxes and the bounding boxes of rotated bounding boxes is unreasonable for remote sensing targets with diverse scales and poses. Therefore, there is currently no method that can simultaneously solve the problem of balancing dataset annotation costs and the accuracy of rotated bounding box target detection. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device, and medium for detecting targets in remote sensing images, which improves the accuracy of target detection in remote sensing images while reducing the cost of annotation.

[0006] To address the aforementioned technical problems, embodiments of this application provide a method for detecting targets in remote sensing images, comprising the following steps:

[0007] Obtain a remote sensing image with point labels for target annotation, input the remote sensing image into an image segmentation model to obtain a mask image corresponding to the point labels in the remote sensing image, and use the outer horizontal bounding box of the mask image as the initial horizontal bounding box to obtain an initial remote sensing image with target annotation using the initial horizontal bounding box.

[0008] Based on the multi-instance learning algorithm, a target horizontal box generator is obtained by training the backbone network and the feature pyramid network with several remote sensing images without target annotation and several corresponding initial remote sensing images.

[0009] A target horizontal bounding box generator is used to optimize the initial horizontal bounding boxes in the initial remote sensing image from the remote sensing image without target annotation to generate target horizontal bounding boxes, and then obtain the target remote sensing image with target horizontal bounding boxes for target annotation.

[0010] Using several target remote sensing images as training data, a rotating frame detector is jointly trained using a weakly supervised learning algorithm and a self-supervised learning algorithm, and the trained rotating frame detector is used to detect targets in remote sensing images.

[0011] In some optional embodiments, the step of inputting the remote sensing image into an image segmentation model to obtain a mask image corresponding to the point labels in the remote sensing image includes:

[0012] The remote sensing image is input into the image segmentation model to obtain multiple mask images corresponding to different objects in the remote sensing image;

[0013] Determine the type of object indicated by the point labels in the remote sensing image;

[0014] Based on the object type indicated by the point labels in the remote sensing image, select the mask image corresponding to the point labels in the remote sensing image from multiple mask images.

[0015] In some optional embodiments, the objects indicated by the point labels in the remote sensing image are either non-scene objects or scene objects;

[0016] The step of selecting a mask image corresponding to the point labels in the remote sensing image from multiple mask images based on the object type indicated by the point labels in the remote sensing image includes:

[0017] If the target labeled by the point in the remote sensing image is not a scene object, then the mask image with a relative mask distance less than a first preset threshold will be used as the mask image corresponding to the point label in the remote sensing image.

[0018] If the target marked by the point label in the remote sensing image is a scene object, then obtain the Euclidean distance between the point label in the remote sensing image and the center point of each mask image, and select at least two mask images from multiple mask images whose center point and point label have an Euclidean distance less than or equal to a preset distance.

[0019] At least two mask images are combined to generate a combined mask image, and the mask image whose relative mask distance is less than a second preset threshold is selected from the combined mask image and the at least two mask images as the mask image corresponding to the point label in the remote sensing image.

[0020] The mask relative distance is the ratio of the Euclidean distance between the point label in the remote sensing image and the center point of the mask image to the length of the diagonal of the mask image.

[0021] In some optional embodiments, the multi-instance learning algorithm, which uses several unlabeled remote sensing images and corresponding initial remote sensing images to train the backbone network and feature pyramid network to obtain the target horizontal bounding box generator, includes:

[0022] The unlabeled remote sensing image is input into the backbone network and the feature pyramid network to obtain the first feature image. The first feature image is then aligned with the corresponding initial remote sensing image to obtain the second feature image of the initial remote sensing image.

[0023] The scores of the second feature images are obtained through a classification branch consisting of a first fully connected layer and a first softmax activation function, and an instance branch consisting of a second fully connected layer and a second softmax activation function.

[0024] By using the scores of the second feature images corresponding to several initial remote sensing images, a rough horizontal box generator is trained to generate remote sensing images with target annotation using rough horizontal boxes.

[0025] Fine-tune the coarse horizontal boxes in the remote sensing image that uses coarse horizontal boxes for target annotation to obtain a remote sensing image that uses positive horizontal boxes for target annotation. Then, use randomly generated horizontal boxes to annotate the targets in the remote sensing image to obtain a remote sensing image that uses negative horizontal boxes for target annotation.

[0026] By using the scores of the second feature images corresponding to remote sensing images labeled with target horizontal boxes using positive example horizontal boxes and remote sensing images labeled with target horizontal boxes using negative example horizontal boxes, a target horizontal box generator is trained to generate remote sensing images labeled with target horizontal boxes.

[0027] In some alternative embodiments, the rotating frame detector is constructed based on a backbone network and a multi-scale feature fusion network.

[0028] In some optional embodiments, before jointly training the rotated box detector using a combination of weakly supervised and self-supervised learning algorithms with several target remote sensing images as training data, the method further includes:

[0029] Acquire several remote sensing images of targets labeled with horizontal bounding boxes;

[0030] Based on the Kmeans clustering algorithm, the target horizontal bounding box with a cluster ratio less than a preset ratio in each target remote sensing image is obtained;

[0031] Target horizontal bounding boxes in each target remote sensing image with a cluster ratio less than a preset ratio are removed, and the target remote sensing images after removing the target horizontal bounding boxes are used as training data for the rotating box detector.

[0032] In some optional embodiments, after acquiring the initial remote sensing image with target annotations using initial horizontal bounding boxes, the method further includes:

[0033] Dithering augmentation is performed on the initial remote sensing image to obtain initial remote sensing images of different scales and sizes.

[0034] Embodiments of this application also provide a target detection device in a remote sensing image, comprising:

[0035] The image acquisition module is used to acquire remote sensing images with point labels for target annotation, input the remote sensing images into the image segmentation model to obtain the mask images corresponding to the point labels in the remote sensing images, and use the outer horizontal bounding box of the mask image as the initial horizontal bounding box to acquire the initial remote sensing image with target annotation using the initial horizontal bounding box.

[0036] The model training module is used to train the backbone network and feature pyramid network to obtain the target horizontal box generator based on the multi-instance learning algorithm, using several remote sensing images without target annotation and several corresponding initial remote sensing images.

[0037] The image optimization module is used to optimize the initial horizontal bounding boxes in the initial remote sensing image using a target horizontal bounding box generator, thereby generating a target remote sensing image with target horizontal bounding boxes and obtaining a target remote sensing image with target horizontal bounding boxes and target annotations.

[0038] The image detection module is used to train a rotating frame detector using several target remote sensing images as training data through a weakly supervised learning algorithm and a self-supervised learning algorithm, and then use the trained rotating frame detector to detect targets in the remote sensing images.

[0039] Embodiments of this application also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for detecting targets in remote sensing images.

[0040] Embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting targets in remote sensing images.

[0041] The target detection method in remote sensing images provided by the embodiments of this application has at least the following beneficial effects:

[0042] Using point labels to annotate targets in remote sensing images effectively reduces the training cost of target detection in remote sensing images compared to rotating bounding boxes, and improves the utilization of current remote sensing datasets. Simultaneously, remote sensing images annotated with point labels are converted into remote sensing images with target annotations using the bounding boxes (i.e., initial bounding boxes) of the mask images corresponding to the point labels. The trained target bounding box generator is then optimized to achieve the conversion from remote sensing images annotated with point labels to those annotated with target bounding boxes. The final generated remote sensing images with target bounding boxes are of higher quality than the original images annotated with point labels. Therefore, the rotating bounding box detector trained using these remote sensing images with target bounding boxes as training data not only ensures low training costs for target detection in remote sensing images but also guarantees high detection accuracy. Attached Figure Description

[0043] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0044] Figure 1 This is a flowchart of a method for detecting targets in remote sensing images according to an embodiment of this application;

[0045] Figure 2 This is a schematic diagram of an initial horizontal bounding box acquisition process according to an embodiment of this application;

[0046] Figure 3 This is a schematic diagram illustrating the implementation process of a target horizontal bounding box according to an embodiment of this application;

[0047] Figure 4This is a schematic diagram illustrating the implementation process of a rough horizontal frame according to an embodiment of this application;

[0048] Figure 5 This is a schematic diagram illustrating the implementation process of a rotating detector according to an embodiment of this application;

[0049] Figure 6 This is a schematic diagram illustrating the detection of a target in a remote sensing image according to an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of a target detection device in a remote sensing image according to an embodiment of this application;

[0051] Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0053] One embodiment of this application relates to a method for detecting targets in remote sensing images. The implementation details of the method for detecting targets in remote sensing images in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0054] The specific process of the target detection method in the remote sensing image in this embodiment can be as follows: Figure 1 As shown, it includes:

[0055] Step 101: Obtain the remote sensing image with point labels for target annotation, input the remote sensing image into the image segmentation model to obtain the mask image corresponding to the point labels in the remote sensing image, and use the outer horizontal bounding box of the mask image as the initial horizontal bounding box to obtain the initial remote sensing image with the initial horizontal bounding box for target annotation.

[0056] Specifically, the method for obtaining the mask images corresponding to point labels in remote sensing images is as follows: First, acquire the remote sensing image with target annotations using point labels. Then, input the remote sensing image into the image segmentation model to obtain multiple mask images corresponding to different objects in the remote sensing image. Determine the object type indicated by the targets annotated by the point labels in the remote sensing image, and based on the object type indicated by the targets annotated by the point labels in the remote sensing image, select the mask image corresponding to the point labels in the remote sensing image from the multiple mask images, and use it as the mask image corresponding to the point labels in the remote sensing image. The image segmentation model (Segment Anything Model, SAM) selects the vit_h model weights.

[0057] In practical implementation, the objects indicated by the point labels in the remote sensing image can be of two types: non-scene objects or scene objects. Therefore, when selecting a mask image corresponding to the point labels from multiple mask images based on the object type indicated by the point labels in the remote sensing image, the Euclidean distance between the point labels in the remote sensing image and the center point of each mask image is first obtained.

[0058] Specifically, if the target labeled by the point tag in the remote sensing image is a non-scene object, then the mask image with a relative mask distance less than a first preset threshold is used as the mask image corresponding to the point tag in the remote sensing image. If the target labeled by the point tag in the remote sensing image is a scene object, then the Euclidean distance between the point tag in the remote sensing image and the center point of each mask image is obtained, and at least two mask images are selected from multiple mask images whose center point and point tag have an Euclidean distance less than or equal to a preset distance. The at least two mask images are combined to generate a combined mask image, and the mask image with a relative mask distance less than a second preset threshold is selected from the combined mask image and the at least two mask images as the mask image corresponding to the point tag in the remote sensing image. The first and second preset thresholds can be set to 0.35. The relative mask distance is specifically the ratio of the Euclidean distance between the point tag in the remote sensing image and the center point of the mask image to the diagonal length of the mask image, calculated using the following formula:

[0059]

[0060] In the formula, Thre mask Represents the relative distance between the masked images, (x) p y p ) and (x m y m ) represent the coordinates of the point label and the center point of the mask image, respectively. m and h mrepresents the width and height of the bounding horizontal box of the mask image, respectively, and λ represents the preset scale correlation coefficient, which takes a value of 1 when the target marked by the point label is a composite object formed by a non-scene object and a scene object, and otherwise takes a value of 0.25.

[0061] Given the mask image corresponding to the point labels in the acquired remote sensing image, the outer horizontal bounding box of the mask image is used as the initial horizontal bounding box to obtain the initial remote sensing image with target annotation using the initial horizontal bounding box. Figure 2 An initial horizontal box is shown. Figure 2 The process of obtaining the suggestion box in the middle.

[0062] In one example, after acquiring the initial remote sensing image with target annotation using the initial horizontal bounding box, a dithering augmentation operation is performed on the initial remote sensing image with target annotation using the initial horizontal bounding box to obtain initial remote sensing images with target annotation using the initial horizontal bounding box at different scales and sizes. For example, a total of 42 different operations are used to augment the number of the above-mentioned remote sensing images with target annotation using the initial horizontal bounding box, using six scales (0.9, 0.95, 1, 1.02, 1.05, 1.1) and seven scales (1, 4 / 5, 3 / 4, 2 / 3, 5 / 4, 4 / 3, 3 / 2).

[0063] Step 102: Based on the multi-instance learning algorithm, a target horizontal box generator is obtained by training the backbone network and the feature pyramid network using several remote sensing images without target annotation and several corresponding initial remote sensing images.

[0064] Specifically, the process involves inputting unlabeled remote sensing images into a backbone network and a feature pyramid network to obtain a first feature image. This first feature image is then aligned with the corresponding initial remote sensing image to obtain a second feature image of the initial remote sensing image. Scores for the second feature image are obtained through a classification branch consisting of a first fully connected layer and a first softmax activation function, and an instance branch consisting of a second fully connected layer and a second softmax activation function. Using the scores of the second feature images corresponding to several initial remote sensing images, a coarse horizontal box generator is trained to generate remote sensing images with coarse horizontal boxes labeled with targets. The coarse horizontal boxes in the remote sensing images with coarse horizontal boxes labeled with targets are fine-tuned to obtain remote sensing images with positive horizontal boxes labeled with targets. Randomly generated horizontal boxes are then used to label targets on these remote sensing images to obtain remote sensing images with negative horizontal boxes labeled with targets. Finally, using the scores of the second feature images corresponding to the remote sensing images with positive and negative horizontal boxes labeled with targets, a target horizontal box generator is trained to generate remote sensing images with target horizontal boxes labeled with targets.

[0065] In practical implementation, since the optimization of remote sensing images is actually the optimization of the horizontal bounding boxes used to annotate targets on the remote sensing images, the horizontal bounding boxes will be used directly to refer to the remote sensing images below. The initial horizontal bounding boxes have been obtained in the steps above. The following steps optimize the initial horizontal bounding boxes to obtain the target horizontal bounding boxes. The process of obtaining the target horizontal bounding boxes includes obtaining a coarse horizontal bounding box and refining it to obtain the final target horizontal bounding box.

[0066] Figure 3 The implementation process of the target horizontal bounding box is shown. Figure 4 The diagram illustrates the implementation process of the rough horizontal bounding box. The implementation process of the target horizontal bounding box is explained in detail below:

[0067] (1) Input the training set images (remote sensing images without target annotations) into the backbone network and feature pyramid network to extract features and obtain feature maps. Then, the feature maps and the initial horizontal bounding boxes B i (i = 1, 2, ..., T, where T is the number of targets) The initial horizontal bounding box features X are obtained through the region of interest alignment (ROIalign) operation and two fully connected layers. i ;

[0068] (2) Input the features of the initial horizontal bounding box into the classification branch and obtain the classification score through a fully connected layer and a softmax activation function [F]. i cls ] nc (n = 1, 2, ..., N, c = 1, 2, ..., C); the input to the instance branch is processed by a fully connected layer and a softmax activation function to obtain the instance score [F]. i det ] nc The calculation process is as follows:

[0069]

[0070] In the formula, [·] nc This represents the score of the c-th class in the nth initial horizontal bounding box, where C represents the number of classes, and f cls (·) and f det (·) represent the fully connected layer operations for the categorical branch and the instance branch, respectively. i This indicates that the initial horizontal bounding box features are obtained through two fully connected layers.

[0071] (3) The score F of each initial horizontal box is obtained by performing the Hadamard product of the classification score and the instance score. i And for each target's initial horizontal bounding box, F is applied in the N dimension. i Perform initial horizontal bounding box classification and obtain the initial horizontal bounding box score P for each target. i The calculation method is as follows:

[0072] [F i ] nc =[F i cls ] nc ⊙[F i det ] nc ,

[0073] And the multi-instance learning loss is calculated using the cross-entropy loss function as the loss for the coarse horizontal box generator:

[0074]

[0075] In the formula, y ic The sign function is 1 when c is the true class of the i-th target, and 0 otherwise; P ic denoted as the initial horizontal box score in class c, μ1 represents the weight coefficient of the coarse horizontal box generator, set to 0.25, and T represents the number of targets.

[0076] Finally, the initial horizontal bounding boxes of the highest scores exceeding the set threshold for each target are weighted and averaged to obtain pseudo-bounding boxes, calculated as follows:

[0077]

[0078] In the formula, P represents the score of the nth initial horizontal box among all initial horizontal boxes of the i-th target on the correct category. i true δ represents the score of the initial horizontal bounding box for the target in the correct category, which is the sum of the scores of all N initial horizontal bounding boxes in the correct category. n This represents the weighted weight of the nth initial horizontal box in the correct category, where (x, y) and (w, h) represent the center point coordinates and width and height of the initial horizontal box, respectively.

[0079] (4) Fine-tune the rough horizontal frame obtained in step (3) using the length and width coefficients and the center point offset to obtain the positive horizontal frame B. i * (i = 1, 2, ..., T, where T is the target number), and set an intersection-union threshold to randomly obtain the horizontal bounding box B of negative examples. Neg The feature X of the positive example horizontal box is obtained by repeating step (1) for both sets of horizontal boxes. i * (Dimension is N×D, where N represents the number of positive example horizontal boxes and D represents the feature dimension) and the feature X of the negative example horizontal boxes. Neg (Dimensions are M×D, where M represents the number of negative example horizontal boxes);

[0080] (5) Input the features of the positive example horizontal bounding boxes into the classification branch and obtain the classification score through a fully connected layer and a sigmoid activation function. C represents the number of categories, and the calculation process is as follows:

[0081]

[0082] In the formula, This represents the classification score of the positive example horizontal box.

[0083] Simultaneously, the input to the instance branch is processed through a fully connected layer and a softmax activation function to obtain the instance score. The calculation is the same as step (2); the negative example horizontal bounding box is input into the classification branch and the classification score is obtained through a fully connected layer and a sigmoid activation function. The calculation process is as follows:

[0084]

[0085] In the formula, This represents the classification score of the horizontal box representing a negative example.

[0086] (6) The score F of each horizontal box is obtained by performing the Hadamard product of the classification score and the instance score of the positive example horizontal box. i * And for each target's horizontal bounding box, perform F on the N-dimensional plane. i * Perform horizontal bounding box classification and obtain the horizontal bounding box score P for each target. i * (Dimension is 1×C, each score includes the score in each category), and the improved horizontal bounding box is obtained by averaging the highest score horizontal bounding boxes of each target that exceed the set threshold. The calculation process is the same as step (3). At the same time, the multi-instance learning loss L is calculated using the Focal loss function. Pos The calculation method is as follows:

[0087]

[0088] In the formula, y ic The sign function is 1 when c is the true class of the i-th target, and 0 otherwise; P ic * P represents i * In the horizontal box score for class c, μ2 represents the positive example weight coefficient of the target horizontal box generator, set to 0.25. γ is set to 2 according to the Focal loss. Furthermore, to enhance robustness, the target's horizontal box score P in the target horizontal box generator on the correct class is used. i true As the weight of the target in the Focal loss.

[0089] And calculate the negative example loss L based on the negative example classification score. Neg :

[0090]

[0091] In the formula, P is the score P of the target's horizontal box score in the target horizontal box generator for the correct category. i true The average value θ is used as the weight of the target in the Focal loss, μ3 is the negative example weight coefficient of the target horizontal box generator, which is set to 0.75, and γ is set to 2 according to the Focal loss.

[0092] The final total loss L2 of the target horizontal bounding box generator is as follows:

[0093] L2 = L Pos +L Neg

[0094] Step 103: Using a target horizontal bounding box generator, the initial horizontal bounding boxes in the initial remote sensing image are optimized using the remote sensing image without target annotation to generate target horizontal bounding boxes, and the target remote sensing image with target annotation using the target horizontal bounding boxes is obtained.

[0095] Step 104: Using several target remote sensing images as training data, the rotated box detector is jointly trained by a weakly supervised learning algorithm and a self-supervised learning algorithm, and the trained rotated box detector is used to detect targets in the remote sensing images.

[0096] The rotated box detector is constructed based on a backbone network and a multi-scale feature fusion network. The training process of the rotated box detector is as follows: Figure 5 As shown, it includes the following steps:

[0097] (1) The weakly supervised branch uses FCOS as the baseline model and introduces angle parameters into the regression subnetwork to detect the training set images (i.e., several remote sensing images with target horizontal boxes labeled with targets). The predicted rotated boxes are taken as the largest inscribed ellipse, and its equation is calculated as follows:

[0098]

[0099] Furthermore, by critically solving the equation, the x and y coordinates of the ellipse's circumcircle can be obtained, and by center point mapping, the VOC (Voice of Origin) bounding box representation label information {x1, y1, x2, y2} of the predicted bounding box can be obtained. The calculation process is as follows:

[0100]

[0101] (2) In the classification sub-network, the feature map input to the detector head is passed through four convolutional layers with 256 channels each (3×3 kernel, stride and padding are both 1), and then through the convolutional layers of the classification branch and the centrality branch respectively. The Sigmoid activation function is used to obtain the class score P (dimension H×W×C, where H and W are the dimensions of the input feature map and C represents the number of classes) and the centrality score Q (dimension H×W×1). The Focal loss function is used to calculate the class loss L on the class score. cls As shown below:

[0102]

[0103] In the formula, α and γ are used to adjust the sample weights and the overall loss, respectively, and their values ​​are set to 0.25 and 2, respectively. pos P represents the number of positive samples. c Let i be the class c score for all (x,y) positions of P. c The sign function is 1 when c is the true class of the i-th target, and 0 otherwise.

[0104] After the detector head obtains a positive sample, it calculates the distances from the positive sample to the corresponding ground truth bounding box in all directions (up, down, left, right) and calculates the true centrality value Q. t The calculation process is as follows:

[0105]

[0106] In the formula, l*, t*, r*, and b* represent the distances from the point to the top, bottom, left, and right sides of the bounding box, respectively.

[0107] Then, loss is calculated using the predicted centrality score. Since the centrality value ranges from 0 to 1, the cross-entropy loss function is used to calculate the centrality loss L. cn The calculation is as follows:

[0108]

[0109] In the formula, Q t Q and Q are respectively Q t and all positive samples of Q The true centrality value and the predicted centrality score of the location are used to calculate the centrality loss. Only the cross-entropy loss of the positive samples is calculated and averaged.

[0110] (3) In the regression subnetwork, the predicted distances from the center to the four sides of the bounding box and the rotation angle are decoded to obtain the rotation box prediction result rbox (center point, width, height, and angle are {x,y,w,h,a}), and compared with the ground truth box tbox (VOC format is {x,y,w,h,a}). t ,y t ,x t,y t The regression loss L is calculated using the IoU loss function. reg The calculation is as follows:

[0111]

[0112] In the formula, IoU() represents the intersection-union ratio calculation, reh() represents the elliptical circumcircle moment transformation of the predicted rotated box, and the regression loss is calculated only for all positive samples using the intersection-union ratio and the true centrality value of the positive sample points. As weights, then for all Q t The sum is used to make a weighted average.

[0113] Finally, the loss L for the entire weakly supervised branch is obtained. w As shown below:

[0114] L w =L reg +L cn +L cls

[0115] (4) Randomly rotate the training set images and process the black border regions. Map the predicted rbox (center point, width, height, and angle {x, y, w, h, a}) of the weakly supervised branch onto the rotated image in the self-supervised branch through affine transformation as the target bounding box rbox of the self-supervised branch. w ({x w ,y w ,w w ,h w ,a w The mapping process is as follows:

[0116]

[0117] In the formula, θ represents the rotation center of the image, and θ represents the rotation angle of the image.

[0118] (5) The self-supervised branch and the weakly supervised branch share the backbone network and the multi-scale feature fusion network to perform inference on the rotated image to obtain the predicted rotation box (rbox). s ({x s ,y s ,w s ,h s ,a s The predicted results are compared with the original predicted results and the image rotation angle to calculate the rotation box difference loss L. rbox The calculation process is as follows:

[0119]

[0120] In the formula, hbox w The VOC format coordinates are {-w w ,-h w ,w w ,h w}, hbox s Coordinates are {-w s ,-h s ,w s ,h s}, hbox s ′ is the horizontal bounding box after swapping lengths, with coordinates {-h}. s ,-w s ,h s ,w s}

[0121] Simultaneously, calculate the center point difference loss L. xy The prediction of the center point is self-supervised, and the calculation process is as follows:

[0122]

[0123] Then the regression loss L of the self-supervised branch is obtained. reg as follows:

[0124] L reg =0.1L xy +L rbox

[0125] Finally, the true value of the centrality Q at each location in the weakly supervised branch is used. txy As L reg The weights are then used to calculate the true centrality Q of all centers. t The weighted average of the sums yields the loss L for the entire self-supervised branch. s The calculation process is as follows:

[0126]

[0127] In the formula, (x,y) represents the self-supervised branch (x s ,y s The position is at the original position of the weakly supervised branch mapping.

[0128] In one example, before jointly training the bounding box detector using a weakly supervised learning algorithm and a self-supervised learning algorithm with several target remote sensing images as training data, several target remote sensing images labeled with target horizontal boxes are first acquired; based on the K-means clustering algorithm, target horizontal boxes with a cluster ratio less than a preset ratio are obtained in each target remote sensing image; target horizontal boxes with a cluster ratio less than the preset ratio in each target remote sensing image are removed, and the target remote sensing images after removing the target horizontal boxes are used as training data for the bounding box detector.

[0129] Specifically, firstly, each remote sensing image labeled with target horizontal boxes undergoes image preprocessing operations such as scaling, tensor transformation, and normalization. Then, feature extraction is performed on the processed remote sensing images, and K-means clustering with a K value of 2 is performed according to the category. That is, the target labeling method used in the remote sensing images—target horizontal boxes—is clustered. For each category of target horizontal boxes, target horizontal boxes with a cluster ratio less than a preset ratio are considered low-quality target horizontal boxes and are removed to obtain remote sensing images labeled with higher-quality target horizontal boxes. These are used as training datasets, and the rotation box detector trained based on this dataset has higher detection accuracy.

[0130] In a specific example, the implementation process of the target detection method in the remote sensing image of this embodiment can be as follows: Figure 6 As shown.

[0131] The rotation box detection results of this embodiment are evaluated using the {mAP50:95, AP50, AP75} index. At the same time, the mean intersection-union ratio (mIoU) is used to evaluate the quality of the target horizontal box. Experimental results on the DIOR and DIOR-R datasets show that the above method improves the AP50 accuracy by 20.4% compared with the optimal point-horizontal-rotation box paradigm, and improves the mIoU by 0.1813 compared with the optimal point-supervised method. Its performance is significantly better than the existing point-supervised methods.

[0132] In this embodiment, point labels are used to annotate targets in remote sensing images. Compared to rotating bounding boxes, this effectively reduces the training cost of target detection in remote sensing images and improves the utilization of the current remote sensing dataset. Simultaneously, the remote sensing image annotated with point labels is converted into one annotated with the outer horizontal bounding boxes (i.e., initial horizontal bounding boxes) of the mask images corresponding to the point labels. The trained target horizontal bounding box generator is then optimized to achieve the conversion from point-labeled to target-bounding images. The resulting remote sensing image annotated with target horizontal bounding boxes is of higher quality than the original point-labeled image. Therefore, the rotating bounding box detector trained using this target-bounding image not only ensures low training cost for target detection in remote sensing images but also guarantees high detection accuracy.

[0133] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0134] Another embodiment of this application relates to a target detection device in a remote sensing image. The implementation details of the target detection device for remote sensing images in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution. A schematic diagram of the target detection device for remote sensing images in this embodiment can be seen as follows: Figure 7 As shown, it includes: an image acquisition module 701, a model training module 702, an image optimization module 703, and an image detection module 704.

[0135] Specifically, the image acquisition module 701 is used to acquire remote sensing images with point labels for target annotation, input the remote sensing images into the image segmentation model to obtain the mask images corresponding to the point labels in the remote sensing images, and use the outer horizontal bounding box of the mask images as the initial horizontal bounding box to acquire the initial remote sensing images with target annotation using the initial horizontal bounding box.

[0136] The model training module 702 is used to train the backbone network and feature pyramid network to obtain a target horizontal box generator based on the multi-instance learning algorithm, using several remote sensing images without target annotation and several corresponding initial remote sensing images.

[0137] Image optimization module 703 is used to optimize the initial horizontal bounding box in the initial remote sensing image using a target horizontal bounding box generator, thereby generating a target remote sensing image with target horizontal bounding boxes and obtaining a target remote sensing image with target horizontal bounding boxes.

[0138] The image detection module 704 is used to train the rotated frame detector using several target remote sensing images as training data through a weakly supervised learning algorithm and a self-supervised learning algorithm, and to perform target detection in the remote sensing images using the trained rotated frame detector.

[0139] It is not difficult to see that this embodiment is a device embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0140] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0141] Another embodiment of this application relates to a computer device, such as... Figure 8 As shown, it includes: at least one processor 801; and a memory 602 communicatively connected to the at least one processor 801; wherein the memory 802 stores instructions executable by the at least one processor 801, the instructions being executed by the at least one processor 801 to enable the at least one processor 801 to perform the target detection method in the remote sensing image in the above embodiments.

[0142] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0143] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0144] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0145] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method of detecting a target in a remote sensing image, characterized in that, The method includes: Obtain a remote sensing image with point labels for target annotation, input the remote sensing image into an image segmentation model to obtain a mask image corresponding to the point labels in the remote sensing image, and use the outer horizontal bounding box of the mask image as the initial horizontal bounding box to obtain an initial remote sensing image with target annotation using the initial horizontal bounding box. Based on the multi-instance learning algorithm, a target horizontal box generator is obtained by training the backbone network and the feature pyramid network with several remote sensing images without target annotation and several corresponding initial remote sensing images. A target horizontal bounding box generator is used to optimize the initial horizontal bounding boxes in the initial remote sensing image from the remote sensing image without target annotation to generate target horizontal bounding boxes, and then obtain the target remote sensing image with target horizontal bounding boxes for target annotation. Using several target remote sensing images as training data, the rotating frame detector is jointly trained by a weakly supervised learning algorithm and a self-supervised learning algorithm, and the trained rotating frame detector is used to detect targets in remote sensing images. The multi-instance learning algorithm, which uses several unlabeled remote sensing images and corresponding initial remote sensing images to train the backbone network and feature pyramid network to obtain a target horizontal bounding box generator, includes: The unlabeled remote sensing image is input into the backbone network and the feature pyramid network to obtain the first feature image. The first feature image is then aligned with the corresponding initial remote sensing image to obtain the second feature image of the initial remote sensing image. The scores of the second feature images are obtained through a classification branch consisting of a first fully connected layer and a first softmax activation function, and an instance branch consisting of a second fully connected layer and a second softmax activation function. By using the scores of the second feature images corresponding to several initial remote sensing images, a rough horizontal box generator is trained to generate remote sensing images with target annotation using rough horizontal boxes. Fine-tune the coarse horizontal boxes in the remote sensing image that uses coarse horizontal boxes for target annotation to obtain a remote sensing image that uses positive horizontal boxes for target annotation. Then, use randomly generated horizontal boxes to annotate the targets in the remote sensing image to obtain a remote sensing image that uses negative horizontal boxes for target annotation. By using the scores of the second feature images corresponding to remote sensing images labeled with target horizontal boxes using positive example horizontal boxes and remote sensing images labeled with target horizontal boxes using negative example horizontal boxes, a target horizontal box generator is trained to generate remote sensing images labeled with target horizontal boxes.

2. The method for detecting targets in remote sensing images according to claim 1, characterized in that, The step of inputting the remote sensing image into the image segmentation model to obtain the mask image corresponding to the point labels in the remote sensing image includes: The remote sensing image is input into the image segmentation model to obtain multiple mask images corresponding to different objects in the remote sensing image; Determine the type of object indicated by the point labels in the remote sensing image; Based on the object type indicated by the point labels in the remote sensing image, select the mask image corresponding to the point labels in the remote sensing image from multiple mask images.

3. The method for detecting targets in remote sensing images according to claim 2, characterized in that, The target indicated by the point label in the remote sensing image is either a non-scene object or a scene object; The step of selecting a mask image corresponding to the point labels in the remote sensing image from multiple mask images based on the object type indicated by the point labels in the remote sensing image includes: If the target labeled by the point in the remote sensing image is not a scene object, then the mask image with a relative mask distance less than a first preset threshold will be used as the mask image corresponding to the point label in the remote sensing image. If the target marked by the point label in the remote sensing image is a scene object, then obtain the Euclidean distance between the point label in the remote sensing image and the center point of each mask image, and select at least two mask images from multiple mask images whose center point and point label have an Euclidean distance less than or equal to a preset distance. At least two mask images are combined to generate a combined mask image, and the mask image whose relative mask distance is less than a second preset threshold is selected from the combined mask image and the at least two mask images as the mask image corresponding to the point label in the remote sensing image; The mask relative distance is the ratio of the Euclidean distance between the point label in the remote sensing image and the center point of the mask image to the length of the diagonal of the mask image.

4. The method for detecting targets in remote sensing images according to claim 1, characterized in that, The rotating frame detector is constructed based on a backbone network and a multi-scale feature fusion network.

5. The method for detecting targets in remote sensing images according to any one of claims 1 to 4, characterized in that, Before jointly training the rotated box detector using a combination of weakly supervised and self-supervised learning algorithms with several target remote sensing images as training data, the method further includes: Acquire several remote sensing images of targets labeled with horizontal bounding boxes; Based on the Kmeans clustering algorithm, the target horizontal bounding box with a cluster ratio less than a preset ratio in each target remote sensing image is obtained; Target horizontal bounding boxes in each target remote sensing image with a cluster ratio less than a preset ratio are removed, and the target remote sensing images after removing the target horizontal bounding boxes are used as training data for the rotating box detector.

6. The method for detecting targets in remote sensing images according to any one of claims 1 to 4, characterized in that, After acquiring the initial remote sensing image with target annotations using initial horizontal bounding boxes, the method further includes: Dithering augmentation is performed on the initial remote sensing image to obtain initial remote sensing images of different scales and sizes.

7. A device for detecting targets in remote sensing images, characterized in that, The device includes: The image acquisition module is used to acquire remote sensing images with point labels for target annotation, input the remote sensing images into the image segmentation model to obtain the mask images corresponding to the point labels in the remote sensing images, and use the outer horizontal bounding box of the mask image as the initial horizontal bounding box to acquire the initial remote sensing image with target annotation using the initial horizontal bounding box. The model training module is used to train the backbone network and feature pyramid network to obtain the target horizontal box generator based on the multi-instance learning algorithm, using several remote sensing images without target annotation and several corresponding initial remote sensing images. The image optimization module is used to optimize the initial horizontal bounding boxes in the initial remote sensing image using a target horizontal bounding box generator, thereby generating a target remote sensing image with target horizontal bounding boxes and obtaining a target remote sensing image with target horizontal bounding boxes and target annotations. The image detection module is used to train a rotating frame detector by using several target remote sensing images as training data, through a weakly supervised learning algorithm and a self-supervised learning algorithm, and then to perform target detection in the remote sensing images using the trained rotating frame detector. Specifically, the model training module is used for: The unlabeled remote sensing image is input into the backbone network and the feature pyramid network to obtain the first feature image. The first feature image is then aligned with the corresponding initial remote sensing image to obtain the second feature image of the initial remote sensing image. The scores of the second feature images are obtained through a classification branch consisting of a first fully connected layer and a first softmax activation function, and an instance branch consisting of a second fully connected layer and a second softmax activation function. By using the scores of the second feature images corresponding to several initial remote sensing images, a rough horizontal box generator is trained to generate remote sensing images with target annotation using rough horizontal boxes. Fine-tune the coarse horizontal boxes in the remote sensing image that uses coarse horizontal boxes for target annotation to obtain a remote sensing image that uses positive horizontal boxes for target annotation. Then, use randomly generated horizontal boxes to annotate the targets in the remote sensing image to obtain a remote sensing image that uses negative horizontal boxes for target annotation. By using the scores of the second feature images corresponding to remote sensing images labeled with target horizontal boxes using positive example horizontal boxes and remote sensing images labeled with target horizontal boxes using negative example horizontal boxes, a target horizontal box generator is trained to generate remote sensing images labeled with target horizontal boxes.

8. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method for detecting targets in a remote sensing image as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for detecting targets in remote sensing images as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Multi-head weak supervision target detection method based on cross attention mechanism

    CN113920302A

  • Weak supervision remote sensing image rotating target detection method based on view consistency network

    CN118097447A