Remote sensing image target detection method and device based on ellipse intersection and union ratio

Abstract

The present invention discloses a remote sensing image target detection method and device based on ellipse intersection-and-union ratio, belonging to the field of computer vision technology, the method comprises: obtaining an original image, converting the annotation information of the target in the original image into an ellipse annotation and then inputting it into a target detector, the target detector comprises a backbone network, an RRPN network and a RoIHead network connected in sequence, the regression loss functions of the RRPN network and the RoIHead network both adopt an ellipse intersection-and-union ratio loss function; extracting a multi-scale feature map of the input image using the backbone network; using the multi-scale feature map as the input of the RRPN network to obtain a rotation candidate frame of the target; using the multi-scale feature map and the rotation candidate frame as the input of the RoIHead network to obtain the detection result of the original image. The present invention converts the annotation information of the target in the original image into an ellipse annotation, and proposes a calculation method of the ellipse intersection-and-union ratio to calculate the overlap between the prediction frame and the target frame, thereby improving the effect of remote sensing image target detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a remote sensing image target detection method and device based on ellipse intersection-union ratio. Background Art

[0002] As an important research field in computer vision, object detection has attracted the attention of many scholars and researchers. At the same time, the rise of machine learning and deep learning has further promoted the development of this field. Object detection technology has been successfully applied to various fields such as military and civilian. As one of the typical research branches of object detection, remote sensing image object detection has a variety of application scenarios, including resource exploration, urban planning, ship and aircraft detection, etc.

[0003] Traditional target detection algorithms are mainly aimed at images in natural scenes, aiming to use horizontal bounding boxes to represent the location of the target and give the category of the target in the bounding box. The targets in this scene are mostly axis-aligned. The two-stage target detector based on R-CNN and the single-stage detector based on YOLO and SSD have shown good detection performance in this scene. However, with the rapid development of target detection technology, more and more fields have put forward new application requirements for this technology, such as resource exploration, urban planning, remote sensing image target detection, etc. Unlike natural images, remote sensing images come from drones and remote sensing platforms. The targets in the images have arbitrary directions, dense arrangements, cluttered backgrounds, large aspect ratios, and small sizes. In recent years, a large number of excellent algorithms have emerged in this field, which are mainly divided into two categories: those based on rotated rectangular box representation and those based on arbitrary quadrilateral representation. Among them:

[0004] The main idea of ​​remote sensing image target detection based on rotated rectangular boxes is to add angle information θ to the original horizontal bounding box (x, y, w, h). RRPN proposes a new way to generate a proposal box with angle information, and implements rotated bounding box regression on this basis; RoI Transformer proposes a RoI converter to convert the horizontal anchor box into a rotated anchor box, and then realizes the regression of the rotated bounding box; SCRDet proposes a fusion framework from the aspects of anchor box sampling, feature fusion, result fusion, etc., and designs a multi-dimensional attention network to reduce background noise; CSL transforms the angle prediction from a regression task to a classification task, further improving the accuracy of angle prediction.

[0005] Remote sensing image target detection based on arbitrary quadrilaterals directly regresses the coordinates of the four corner points of the target through the network, and uses an arbitrary quadrilateral (x1, y1, x2, y2, x3, y3, x4, y4) to represent the location information of the target. TextBoxes++ uses corner regression on SSD; RRD decouples the classification task and bounding box regression task based on the rotation invariance and rotation sensitivity of the feature, making the regression of long text more accurate; Gliding_vertex realizes arbitrary quadrilateral target detection by regressing the offsets of the four vertices of the horizontal bounding box.

[0006] The above various cutting-edge algorithms have good performance for remote sensing image target detection, but the only drawback is that most of the above algorithms use the L1 series loss as the loss function. IoU is defined as the intersection over union ratio between the predicted box and the target box. It has long been used as a metric for target detection, used for the selection of positive and negative samples and the evaluation of the distance between the predicted box and the true box. In the field of natural image target detection, attempts have been made to use IoU loss as a loss function to guide network training, but there are two problems with using IoU as a loss function: (1) When the predicted box and the true box have no intersection, by definition, IoU is equal to zero and cannot reflect the distance between the two (the degree of overlap). At the same time, because loss = 0, there is no gradient backpropagation, and learning and training cannot be performed; (2) IoU cannot accurately reflect the degree of overlap between the predicted box and the true box. When the IoU value is the same, the degree of overlap between the predicted box and the target box may be completely different.

[0007] Based on this, some scholars proposed the concept of GIoU to solve the problem when there is no overlap between the predicted box and the real box. Unlike IoU, GIoU not only focuses on the overlapping area, but also on other non-overlapping areas, which can better reflect the overlap between the two. However, when the target box is completely wrapped with the predicted box, the values ​​of IoU and GIoU are the same. At this time, GIoU degenerates into IoU, and its relative position relationship cannot be distinguished. On this basis, some scholars proposed DIoU, which adds the normalized distance of the center point to GIoU to better optimize such problems. Furthermore, CIoU adds an aspect ratio term to DIoU to measure the similarity of the target aspect ratio, and its convergence accuracy is higher.

[0008] In the related art, the invention patent application with application number 201510475992.8 discloses a method for detecting docked ships in port remote sensing images based on scanning of coastal abrupt areas, and the implementation steps are: 1) image input; 2) rapid scanning of the coastal area; 3) determination of suspected docked ship targets; 4) removal of false targets and target marking; 5) image output. In the absence of GIS and prior information on ships, it is possible to quickly scan coastal ships, identify suspected ships, remove false targets, and finally mark the ships using a rectangular frame. However, the ship is still marked using a rectangular frame, so the target frame will still contain a lot of background areas.

[0009] In summary, the annotation form of the data set in the above methods is a rectangular box or an arbitrary quadrilateral, which cannot accurately represent the characteristics of the target, and the remote sensing image target detection effect is poor. Summary of the invention

[0010] The technical problem to be solved by the present invention is how to improve the target detection effect of remote sensing images.

[0011] The present invention solves the above technical problems through the following technical means:

[0012] On the one hand, the present invention proposes a remote sensing image target detection method based on ellipse intersection-union ratio, the method comprising:

[0013] Acquire an original image, and convert the annotation information of the target in the original image into an ellipse annotation as an input image of a pre-trained target detector, wherein the target detector includes a backbone network, a RRPN network, and a RoIHead network connected in sequence, and both the RRPN network and the RoIHead network use an ellipse intersection-over-union ratio and an ellipse intersection-over-union ratio loss function;

[0014] Extracting a multi-scale feature map of the input image using the backbone network;

[0015] Using the multi-scale feature map as the input of the RRPN network to obtain a rotation candidate box of the target;

[0016] The multi-scale feature map and the rotated candidate box are used as inputs of the RoIHead network to obtain a detection result of the original image.

[0017] The present invention converts the annotation information of the target in the original image into elliptical annotation, and on the basis of the original IoU, considering that the existing IoU is not applicable to various challenges existing in the remote sensing image target detection task, such as large target aspect ratio and arbitrary direction, an ellipse intersection-over-union ratio calculation method is proposed to calculate the overlap between the prediction box and the target box, and the scale, position, orientation, aspect ratio, etc. of the target are jointly optimized, thereby improving the effect of remote sensing image target detection.

[0018] Furthermore, converting the annotation information of the target in the original image into an ellipse annotation includes:

[0019] Based on the annotation information of the target in the original image, the positions of the two foci of the ellipse are determined to obtain the focal line of the ellipse, wherein the annotation information of the target is a rotated rectangular annotation box or a minimum circumscribed rectangle of a quadrilateral;

[0020] The center of the focal line is taken as the center point of the ellipse, and a straight line perpendicular to the focal line is drawn through the center point as the minor axis of the ellipse;

[0021] The acute angle formed by the long side of the rotated rectangular annotation frame or the minimum circumscribed rectangle of the quadrilateral and the horizontal line is taken as the inclination angle of the ellipse.

[0022] Furthermore, the backbone network includes a ResNet101 network and an FPN network, and the FPN network uses C2-C5 layer features.

[0023] Furthermore, the RRPN network includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, and the third convolutional layer includes two parallel convolutions; the multi-scale feature map is used as the input of the RRPN network to obtain a rotation candidate box of the target, including:

[0024] The multi-scale feature map is used as the input of the first convolutional layer to generate an anchor frame, and the angle of the anchor frame includes The aspect ratio is (1:2, 1:5, 1:8) and the size is (8, 16, 32, 64);

[0025] The anchor frame is used as the input of the second convolutional layer and passes through the third convolutional layer to predict the foreground score, coordinate offset and angle of the anchor frame;

[0026] The NMS algorithm is used to process the foreground score, coordinate offset and angle of the anchor frame to obtain a rotation candidate frame of the target.

[0027] Furthermore, the RoIHead network includes an RRoI pooling layer and two multi-layer perceptron layers, and the multi-scale feature map and the rotated candidate box are used as inputs of the RoIHead network to obtain the detection result of the original image, including:

[0028] The multi-scale feature map and the rotated candidate box are used as inputs of the RRoI pooling layer, and the rotated candidate boxes are projected onto corresponding layers of the multi-scale feature map to obtain corresponding feature matrices;

[0029] Input the feature matrix into two multi-layer perceptron layers to obtain target categories and regression parameters;

[0030] The target category and the regression parameter are processed using the NMS algorithm to obtain a detection result of the original image.

[0031] Furthermore, the formula of the ellipse intersection and union ratio is as follows:

[0032]

[0033] The formula of the elliptic intersection-over-union loss function is as follows:

[0034] EllipseIoU Loss=1-EllipseIoU

[0035] Among them, EllipseIoU is the ellipse intersection-union ratio, L1 and L2 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are closer, L3 and L4 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are farther away, θ is the angle between the two focal lines of the real ellipse and the predicted ellipse, b and b′ are the minor axis lengths of the real ellipse and the predicted ellipse, respectively.

[0036] Furthermore, the method further comprises:

[0037] Inputting a sample image and an ellipse annotation of an object in the sample image into the object detector;

[0038] The multi-task loss composed of the RRPN network and the RoIHead network of the target detector is jointly trained, and when the regression loss function converges or iterates a set number of times, the pre-trained target detector is obtained.

[0039] On the other hand, the present invention also proposes a remote sensing image target detection device based on ellipse intersection-union ratio, the device comprising:

[0040] A preprocessing module, used to obtain an original image, and convert the annotation information of the target in the original image into an ellipse annotation as an input image of a pre-trained target detector, wherein the target detector includes a backbone network, a RRPN network and a RoIHead network connected in sequence, and both the RRPN network and the RoIHead network use an ellipse intersection-over-union ratio and an ellipse intersection-over-union ratio loss function;

[0041] A feature extraction module, used to extract a multi-scale feature map of the input image using the backbone network;

[0042] A candidate box generation module, used to use the multi-scale feature map as the input of the RRPN network to obtain a rotation candidate box of the target;

[0043] A detection module is used to use the multi-scale feature map and the rotated candidate box as inputs of the RoIHead network to obtain a detection result of the original image.

[0044] Furthermore, the preprocessing module comprises:

[0045] A focus determination unit, configured to determine the positions of two foci of an ellipse based on the annotation information of the target in the original image, and obtain the focal line of the ellipse, wherein the annotation information of the target is a rotated rectangular annotation box or a minimum circumscribed rectangle of a quadrilateral;

[0046] An ellipse annotation determination unit is used to take the center of the focal line as the center point of the ellipse, draw a straight line through the center point that is perpendicular to the focal line as the minor axis of the ellipse, and take the acute angle formed by the long side of the minimum circumscribed rectangle of the rotated rectangular annotation box or the quadrilateral and the horizontal line as the inclination angle of the ellipse.

[0047] Furthermore, the formula of the ellipse intersection and union ratio is as follows:

[0048]

[0049] The formula of the elliptic intersection-over-union loss function is as follows:

[0050] EllipseIoU Loss=1-EllipseIoU

[0051] Among them, EllipseIoU is the ellipse intersection-union ratio, L1 and L2 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are closer, L3 and L4 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are farther away, θ is the angle between the two focal lines of the real ellipse and the predicted ellipse, b and b′ are the minor axis lengths of the real ellipse and the predicted ellipse, respectively.

[0052] The advantages of the present invention are:

[0053] (1) The present invention converts the annotation information of the target in the original image into an ellipse annotation. Based on the original IoU, the present invention considers that the existing IoU is not applicable to various challenges in the remote sensing image target detection task, such as large target aspect ratio and arbitrary direction. An ellipse intersection-over-union ratio calculation method is proposed to calculate the overlap between the prediction box and the target box, and the scale, position, orientation, aspect ratio, etc. of the target are jointly optimized, thereby improving the effect of remote sensing image target detection.

[0054] (2) The present invention proposes an elliptic intersection-union ratio loss function, which achieves good remote sensing image target detection effect.

[0055] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a remote sensing image target detection method based on ellipse intersection-union ratio in a first embodiment of the present invention;

[0057] Figure 2 is a flow chart of remote sensing image target detection based on ellipse intersection-union ratio in the first embodiment of the present invention;

[0058] Figure 3 It is a schematic diagram of a target ellipse marking method in a remote sensing image in the present invention;

[0059] Figure 4 It is a network structure diagram of Obb-FasterRCNN, a two-stage detector based on a rotation bounding box in the present invention;

[0060] Figure 5 It is a schematic diagram illustrating each term in the ellipse intersection and union ratio formula in the present invention;

[0061] Figure 6 It is a schematic diagram comparing the test results of the single-stage rotating rectangular frame detector in the present invention with the test results of other detectors on the mAP evaluation standard;

[0062] Figure 7 It is a schematic diagram comparing the test results of the two-stage rotating rectangular frame detector in the present invention and other detectors on the mAP evaluation standard;

[0063] Figure 8 It is a schematic diagram comparing the convergence speed of the single-stage rotation target detector Obb-RetinaNet in the present invention and other IoU-based loss functions;

[0064] Fig. 9 It is a structural diagram of a remote sensing image target detection device based on ellipse intersection-union ratio in the second embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] like Figure 1 As shown, an embodiment of the present invention proposes a remote sensing image target detection method based on ellipse intersection-union ratio, and the method includes the following steps:

[0067] S10, obtaining an original image, and converting the annotation information of the target in the original image into an ellipse annotation as an input image of a pre-trained target detector, wherein the target detector includes a backbone network, a RRPN network and a RoIHead network connected in sequence, and both the RRPN network and the RoIHead network use an ellipse intersection-over-union ratio and an ellipse intersection-over-union ratio loss function;

[0068] S20, extracting a multi-scale feature map of the input image using the backbone network;

[0069] S30, using the multi-scale feature map as the input of the RRPN network to obtain a rotation candidate box of the target;

[0070] S40: Using the multi-scale feature map and the rotated candidate box as inputs of the RoIHead network to obtain a detection result of the original image.

[0071] In this embodiment, the annotation information of the target in the original image is converted into an ellipse annotation, and on the basis of the original IoU, considering that the existing IoU is not applicable to various challenges existing in the remote sensing image target detection task, such as large target aspect ratio and arbitrary direction, an ellipse intersection-over-union ratio calculation method is proposed to calculate the overlap between the prediction box and the target box, and the scale, position, orientation, aspect ratio, etc. of the target are jointly optimized, thereby improving the effect of remote sensing image target detection.

[0072] In one embodiment, the step S10 includes the following steps:

[0073] S11, based on the annotation information of the target in the original image, determine the positions of the two foci of the ellipse to obtain the focal line of the ellipse, wherein the annotation information of the target is a spiral rectangular annotation box or a minimum circumscribed rectangle of a quadrilateral;

[0074] S12, taking the center of the focal line as the center point of the ellipse, and drawing a straight line perpendicular to the focal line through the center point as the minor axis of the ellipse;

[0075] S13: taking the acute angle formed by the long side of the rotated rectangular annotation box or the minimum circumscribed rectangle of the quadrilateral and the horizontal line as the inclination angle of the ellipse.

[0076] It should be noted that the annotation form of the original dataset is based on a rotated rectangle (HRSC2016 dataset) or an arbitrary quadrilateral (DOTA dataset). Since this solution uses the ellipse intersection and union ratio, the dataset needs to be processed first, and the rotated rectangle annotation or arbitrary quadrilateral annotation of the target is converted into the ellipse annotation of the target through a formula. Specifically, as follows Figure 3 As shown in the figure, for the rotation rectangle annotation of the HRSC2016 dataset, first determine the positions of the two foci of the ellipse, A and B. Connecting points A and B is the focal line of the ellipse. The center point of the focal line is the center point of the ellipse. A straight line perpendicular to the focal line is drawn through this point, which is the short axis. The acute angle θ formed by the long side of the rotation rectangle and the horizontal line is the angle information of the target. For the DOTA dataset, because its true value is a quadrilateral annotation, this experiment first uses the minimum circumscribed rectangle of the quadrilateral as its rotation bounding box annotation, and then uses the same processing method as HRSC2016 to obtain the ellipse annotation of the corresponding target.

[0077] In a method for detecting ships docked in a port remote sensing image based on scanning of a coastal abrupt zone disclosed in the invention patent application with application number 201510475992.8, the ellipse mark is determined by first determining the average value of the horizontal coordinate and the vertical coordinate in the suspected target link component to determine its major axis and minor axis distance, and then the rectangular mark frame of the ship target is obtained by combining morphological expansion and corrosion. In this embodiment, the inscribed ellipse of the rotating rectangular frame is first obtained, the two foci of the ellipse are determined, the focal line, the major axis distance, the center point and the minor axis distance are determined, and thus the scale, position and direction information of the target can be obtained, which is convenient and effective.

[0078] This embodiment is different from the previous remote sensing image target detection method of using a rotating rectangular box or an arbitrary quadrilateral to represent the target. In this embodiment, an ellipse is used to represent the target. The inscribed ellipse is obtained by marking the rotating rectangular box of the target. First, its focus is determined, and then the position of its center point, the distance of the major axis and the minor axis, and the direction are determined. When predicting, the focus position is also predicted, avoiding the problem of sensitivity to angle prediction and sudden increase in loss due to the periodicity of the angle when the target is represented by a rotating rectangular box (x, y, w, h, θ) in the past.

[0079] By printing the heat map during the training process, it is found that most of them are elliptical or elliptical in shape. Therefore, using elliptical annotation to represent the target can better extract the characteristics of the target on the one hand. On the other hand, for the phenomenon that the targets in the remote sensing image are densely arranged and have a large length and width ratio, the use of ellipse can represent the target in a more compact way. Compared with using horizontal bounding boxes or rotated rectangular boxes to represent the target, the interference of background information is reduced. Especially for the remote sensing image dataset HRSC2016, a ship dataset, the target length and width ratio is large, and its advantage is more obvious.

[0080] In one embodiment, if Figure 4 As shown, the backbone network includes a ResNet101 network and an FPN network, and the FPN network uses C2-C5 layer features.

[0081] It should be noted that the image features extracted in the early stage of image processing are similar. This embodiment uses the weight initialization model pre-trained by ResNet101 for transfer learning, which can speed up the convergence of the model.

[0082] In addition, using the C2-C5 layers of FPN as prediction feature layers solves the problem of large differences in predicted target sizes. Feature maps with lower layers have high resolution but lack sufficient semantic information, and can be used to predict small targets; feature maps with higher layers have lower resolution but stronger semantics. The FPN structure enables feature maps of different scales in each layer to have strong semantic information. Under the premise of increasing less computational effort, it fuses feature maps with strong low-resolution semantic information and feature maps with weak high-resolution semantic information but rich spatial information, which can handle the multi-scale change problem in target detection well.

[0083] In one embodiment, the RRPN network includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence, and the third convolutional layer includes two parallel convolutions; the step S30 includes:

[0084] The multi-scale feature map is used as the input of the first convolutional layer to generate an anchor frame, and the angle of the anchor frame includes The aspect ratio is (1:2, 1:5, 1:8) and the size is (8, 16, 32, 64);

[0085] The anchor frame is used as the input of the second convolutional layer and passes through the third convolutional layer to predict the foreground score, coordinate offset and angle of the anchor frame;

[0086] The NMS algorithm is used to process the foreground score, coordinate offset and angle of the anchor frame to obtain a rotation candidate frame of the target.

[0087] It should be noted that the features extracted by the backbone network are fed into the RRPN network to generate target candidate boxes with angles. The working process is as follows: the first 3*3 convolutional layer is used on each layer of the FPN network to generate anchor boxes of different scales, aspect ratios, and angles. Compared with the traditional anchor box design, RRPN combines the direction convergence speed and computational efficiency to select 6 directions, namely The aspect ratio of the anchor box is modified to (1:2, 1:5, 1:8), and the size of the anchor box is set to four sizes (8, 16, 32, 64). Then, the extracted features are first passed through a 3*3 second convolution layer, and then two 1*1 convolutions in parallel in the third convolution layer are used to predict the foreground score of the anchor box, as well as the coordinate offset and angle of the anchor box. Finally, the candidate box of the target is obtained through post-processing such as NMS.

[0088] In one embodiment, if Figure 2 As shown, the step S40 includes:

[0089] The multi-scale feature map and the rotated candidate box are used as inputs of the RRoI pooling layer, and the rotated candidate boxes are projected onto corresponding layers of the multi-scale feature map to obtain corresponding feature matrices;

[0090] Input the feature matrix into two multi-layer perceptron layers to obtain target categories and regression parameters;

[0091] The target category and the regression parameter are processed using the NMS algorithm to obtain a detection result of the original image.

[0092] It should be noted that if Figure 4 As shown in the figure, the RoIhead network includes an RRoI pooling layer and two multi-layer perceptron layers. The RRoI pooling layer is used to project the rotation candidate boxes generated by RRPN onto the corresponding layers of the feature map to obtain the corresponding feature matrix, and then the features are uniformly scaled to 7*7 size through the maximum pooling layer, and then the feature map is flattened, and then two 1024*1024 fully connected layers are connected. The two fully connected layers are used as classification branch networks and regression branch networks, respectively, for the prediction of target categories and the regression of five parameters.

[0093] The output dimension of the classification branch is N*number of categories, and the output dimension of the regression branch is N*5, where N is the number of targets in the image. The obtained results are then processed through NMS and other post-processing processes to obtain the rotated rectangular box detection result as the final target detection result of the input image.

[0094] In some embodiments, in both the RRPN network and the RoIhead network, the proposed Ellipse IoU (EllipseIoU) is used instead of the traditional IoU (IoU) to select positive and negative samples. The formula of the Ellipse IoU is as follows:

[0095]

[0096] Among them, IoU is the intersection over union ratio between the predicted rotation box and the true rotation box.

[0097] It should be noted that, from the heat map in the middle of the output network training, it can be seen that its shape is basically elliptical or quasi-elliptical, so it is reasonable and effective to use the inscribed ellipse of the rotating rectangular frame to calculate the intersection-union ratio in this embodiment, which improves the convergence speed of the network and the accuracy of feature extraction.

[0098] After derivation, the L term can be further converted into a form that contains both focal length and angle:

[0099] L1 2 +L2 2 =L3 2 +L4 2 -2FF'·cosθ

[0100] Among them, the following Figure 5 As shown in, Figure 5 -(1) and Figure 5 -(2) Two cases where the two focal lines of the target's real ellipse annotation and the predicted ellipse intersect are shown respectively. A and B are the two foci of the real ellipse, and the straight line connecting the two foci A and B is the focal line F of the real ellipse; A' and B' are the two foci of the predicted ellipse, and the straight line connecting A' and B' is the focal line F' of the predicted ellipse. L1 and L2 are the straight lines formed by connecting two pairs of foci that are close to each other on the real ellipse and the predicted ellipse (such as the shorter line L1 formed by a focus A of the real ellipse and a focus A' of the predicted ellipse, and the shorter line L2 formed by a focus B of the real ellipse and a focus B' of the predicted ellipse). L3 and L4 are the straight lines formed by connecting two pairs of foci that are far away from each other on the real ellipse and the predicted ellipse (such as the shorter line L4 formed by a focus A of the real ellipse and a focus B' of the predicted ellipse, and the shorter line L3 formed by a focus B of the real ellipse and a focus A' of the predicted ellipse). θ is the angle formed between the two focal lines of the real ellipse and the predicted ellipse (acute angle) and b' is the length of the minor axis of the real ellipse and the length of the minor axis of the predicted ellipse respectively.

[0101] The purpose of target detection is to make the focus (focal line) of the predicted ellipse coincide with the corresponding focus (focal line) of the real ellipse as much as possible. The final effect is that A' coincides with point A, B' coincides with point B, the value of F', L3, L4 are equal to F, and the values ​​of L1 and L2 are close to 0.

[0102] The existing IoU series of methods calculate the intersection over union ratio between the prediction box and the target box for targets in natural image scenes, without considering the angle information of the target. However, in remote sensing images, the angle information is a very important factor because the targets are small, densely arranged, the background is cluttered, and the aspect ratio is high. When there is no overlap between the prediction box and the target box, there is no way to optimize the distance and angle at the same time. When the prediction box completely overlaps with the true box and all the prediction boxes are the same size, it is impossible to select the prediction box that is most consistent with the target direction from all the prediction boxes to guide network training.

[0103] This embodiment addresses the problem that gradient backpropagation training cannot be performed when there is no intersection between the prediction box and the target box in the existing IoU series methods, and the problem that the position and angle cannot be optimized simultaneously when the prediction box and the target box completely overlap. By taking into account the size, direction, aspect ratio, angle and other information of the target into the calculation formula of the ellipse intersection and union ratio, the inscribed ellipse of the prediction box and the real box is calculated, and then the ellipse distance between the two is calculated based on the focal distance, and the minor axis distance of the ellipse is introduced, so that the proposed calculation formula of the ellipse intersection and union ratio takes into account the size, direction, aspect ratio, angle and other information of the target at the same time, so that the network is optimized in the right direction.

[0104] In one embodiment, the loss functions used in the RRPN network and the RoIhead network include a classification loss function and a regression loss function. The classification loss function uses a cross entropy loss function, and the regression classification function uses an elliptic intersection-over-union loss function, which is expressed as follows:

[0105]

[0106] Among them, EllipseIoU is the ellipse intersection-union ratio, L1 and L2 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are closer, L3 and L4 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are farther away, θ is the angle between the two focal lines of the real ellipse and the predicted ellipse, b and b′ are the minor axis lengths of the real ellipse and the predicted ellipse, respectively.

[0107] This embodiment considers that the existing IoU is not applicable to various challenges in remote sensing image target detection tasks, such as large target aspect ratio and arbitrary direction, and designs an ellipse intersection over union (EllipseIoU) to jointly optimize the scale, position, orientation, aspect ratio, etc. of the target, further improving the effect of remote sensing image target detection. And using the proposed ellipse intersection over union as a discrimination condition can more accurately reflect the overlap between the target frame and the prediction frame. At the same time, based on the ellipse intersection over union loss for training, the network converges faster and the network converges towards the expected goal.

[0108] In one embodiment, the method further comprises:

[0109] Inputting a sample image and an ellipse annotation of an object in the sample image into the object detector;

[0110] The multi-task loss composed of the RRPN network and the RoIHead network of the target detector is jointly trained, and when the regression loss function converges or iterates a set number of times, the pre-trained target detector is obtained.

[0111] Specifically, the training process of the target detector is as follows:

[0112] (1) The image and the pre-processed ellipse annotation of the target are input into the network for training. First, the multi-scale features are extracted by inputting the feature extraction module composed of ResNet101+FPN. The 91 layers from ResNet101conv1 to conv4-x are used as shared convolutional layers, and then bifurcates from the output of conv4-x. One branch is passed through the RRPN network for region selection, and the other branch is directly connected to an RROI Pooling layer.

[0113] (2) The feature map obtained above is sent to the AnchorsGenrators of the RRPN network to generate anchor frames. AnchorsGenrators generates 6*4=24 anchor frames of different sizes and ratios for each layer of FPN. For each feature layer, the anchor frames that exceed the image range are first removed, and then the EllipseIoU value between the anchor frame and the true value frame is calculated. According to the set positive and negative sample thresholds, 128 positive samples and 128 negative samples for training are selected at each true value frame, where the threshold of positive samples is set to 0.7 and the threshold of negative samples is set to 0.3.

[0114] (3) The generated positive and negative samples are combined with the RPNHead module to generate the candidate box with angle of the target. The RRPNHead includes a shared 3*3 convolution and two parallel 1*1 convolutions, which are used for the classification of the target foreground and background and the regression of the bounding box and angle respectively. Finally, after NMS and other processing, the candidate box with angle is obtained. In this stage, the classification and regression losses of the RRPN stage can be calculated based on the predicted candidate box scores and regression offsets. The classification loss used in this stage is the cross entropy loss, and the regression loss is the elliptical intersection loss proposed based on the elliptical intersection loss.

[0115] (4) The multi-scale features extracted by the backbone network and the angled candidate boxes generated by the RRPN are sent to the RoIHead. The RoIHead includes the RoIpooling layer and two fully connected layers. First, the candidate boxes are projected onto the corresponding feature maps through the RoIpooling layer, and the feature maps are scaled to 7*7 through maximum pooling, and then flattened and passed through two 1024*1024 fully connected layers.

[0116] (5) The features obtained in the previous step are connected in parallel to two fully connected layers for classification and regression. The selection of positive and negative samples in this stage is the same as (2). The output dimension of the classification branch is N*the number of target categories, and the output dimension of the regression branch is N*5, where N is the number of predicted targets. In this stage, the classification loss and regression loss of this stage can be calculated by predicting the classification score of the target and the regression parameters. The classification loss used in this stage is the cross entropy loss, and the regression loss is the elliptic intersection-over-union loss proposed based on the elliptic intersection-over-union ratio.

[0117] (6) The multi-task loss composed of the RRPN stage and the ROIHead part is jointly trained. Through forward propagation, back propagation, gradient descent and other algorithms, when the loss converges or after a certain number of iterations, the trained network model is obtained.

[0118] Furthermore, this embodiment tests the single-stage rotating rectangular box detector Obb-RetinaNet and the two-stage rotating rectangular box detector Obb-FasterRCNN on the public aerial photography datasets DOTA and HRSC2016, and compares the test results with other detectors on the mAP evaluation standard, such as Figure 6 to Figure 7As shown in the figure, Obb-RetinaNet (on baseline 1) represents the result based on the single-stage detector Obb-RetinaNet in the embodiment of the present invention, and ours (on baseline 1) represents the result based on the two-stage detector Obb-FasterRCNN in the embodiment of the present invention. It can be seen that compared with other existing methods, both the single-stage and two-stage methods have improved their detection performance and convergence speed to a certain extent.

[0119] The embodiment of the present invention is based on the single-stage rotation target detector Obb-RetinaNet and other IoU-based loss functions in terms of convergence speed. Figure 8 As shown, the training process of the HRSC2016 dataset shows that the convergence speed of the elliptic intersection-over-union loss proposed in the embodiment of the present invention is significantly higher than the IoU-based loss of other existing methods.

[0120] This embodiment uses an ellipse shape to more compactly represent the target and can better improve the discriminability of the extracted features. Using the proposed ellipse intersection-over-union ratio as a discrimination condition can more accurately reflect the overlap between the target box and the prediction box. At the same time, based on the ellipse intersection-over-union loss for training, the network converges faster and converges toward the expected target. Finally, the minimum enclosing rectangle of the ellipse is used to represent the target, which contains less background information.

[0121] It should be noted that the remote sensing image target detection method based on ellipse intersection-to-union ratio proposed in this embodiment has general applicability, not only for ship images, but also for targets in other remote sensing images.

[0122] In addition, if Fig. 9 As shown, the embodiment of the present invention also proposes a remote sensing image target detection device based on ellipse intersection-union ratio, the device comprising:

[0123] A preprocessing module 10 is used to obtain an original image, and convert the annotation information of the target in the original image into an ellipse annotation as an input image of a pre-trained target detector, wherein the target detector includes a backbone network, a RRPN network and a RoIHead network connected in sequence, and both the RRPN network and the RoIHead network use an ellipse intersection-over-union ratio and an ellipse intersection-over-union ratio loss function;

[0124] A feature extraction module 20, configured to extract a multi-scale feature map of the input image using the backbone network;

[0125] A candidate box generation module 30 is used to use the multi-scale feature map as an input of the RRPN network to obtain a rotation candidate box of the target;

[0126] The detection module 40 is used to use the multi-scale feature map and the rotated candidate box as inputs of the RoIHead network to obtain a detection result of the original image.

[0127] In one embodiment, the preprocessing module 10 includes:

[0128] A focus determination unit, configured to determine the positions of two foci of an ellipse based on the annotation information of the target in the original image, and obtain the focal line of the ellipse, wherein the annotation information of the target is a rotated rectangular annotation box or a minimum circumscribed rectangle of a quadrilateral;

[0129] An ellipse annotation determination unit is used to take the center of the focal line as the center point of the ellipse, draw a straight line through the center point that is perpendicular to the focal line as the minor axis of the ellipse, and take the acute angle formed by the long side of the minimum circumscribed rectangle of the rotated rectangular annotation box or the quadrilateral and the horizontal line as the inclination angle of the ellipse.

[0130] In one embodiment, the formula of the elliptic intersection-over-union loss function is expressed as follows:

[0131] EllipseIoU Loss=1-EllipseIoU

[0132]

[0133] Among them, EllipseIoU is the ellipse intersection-union ratio, L1 and L2 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are closer, L3 and L4 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are farther away, θ is the angle between the two focal lines of the real ellipse and the predicted ellipse, b and b′ are the minor axis lengths of the real ellipse and the predicted ellipse, respectively.

[0134] It should be noted that other embodiments of the remote sensing image target detection device based on ellipse intersection-to-union ratio or implementation methods of the present invention can refer to the above-mentioned method embodiments, which are not repeated here.

[0135] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0136] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0137] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0138] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0139] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A remote sensing image target detection method based on ellipse intersection-union ratio, characterized in that: The method comprises: Acquire an original image, and convert the annotation information of the target in the original image into an ellipse annotation as an input image of a pre-trained target detector, wherein the target detector includes a backbone network, a RRPN network, and a RoIHead network connected in sequence, and both the RRPN network and the RoIHead network use an ellipse intersection-over-union ratio and an ellipse intersection-over-union ratio loss function; Extracting a multi-scale feature map of the input image using the backbone network; Using the multi-scale feature map as the input of the RRPN network to obtain a rotation candidate box of the target; Using the multi-scale feature map and the rotated candidate box as inputs of the RoIHead network to obtain a detection result of the original image; The RRPN network includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, and the third convolutional layer includes two parallel convolutions; the multi-scale feature map is used as the input of the RRPN network to obtain a rotation candidate box of the target, including: The multi-scale feature map is used as the input of the first convolutional layer to generate an anchor frame, and the angle of the anchor frame includes The aspect ratio is (1:2, 1:5, 1:8) and the size is (8, 16, 32, 64); The anchor frame is used as the input of the second convolutional layer and passes through the third convolutional layer to predict the foreground score, coordinate offset and angle of the anchor frame; The NMS algorithm is used to process the foreground score, coordinate offset and angle of the anchor frame to obtain a rotation candidate frame of the target; The RoIHead network includes an RRoI pooling layer and two multi-layer perceptron layers, and the multi-scale feature map and the rotated candidate box are used as inputs of the RoIHead network to obtain the detection result of the original image, including: The multi-scale feature map and the rotated candidate box are used as inputs of the RRoI pooling layer, and the rotated candidate boxes are projected onto corresponding layers of the multi-scale feature map to obtain corresponding feature matrices; Input the feature matrix into two multi-layer perceptron layers to obtain target categories and regression parameters; The target category and the regression parameter are processed using the NMS algorithm to obtain a detection result of the original image.

2. The remote sensing image target detection method based on ellipse intersection-to-union ratio as claimed in claim 1, characterized in that: The step of converting the annotation information of the target in the original image into an ellipse annotation comprises: Based on the annotation information of the target in the original image, the positions of the two foci of the ellipse are determined to obtain the focal line of the ellipse, wherein the annotation information of the target is a rotating rectangular annotation box or a minimum circumscribed rectangle of a quadrilateral; The center of the focal line is taken as the center point of the ellipse, and a straight line perpendicular to the focal line is drawn through the center point as the minor axis of the ellipse; The acute angle formed by the long side of the rotated rectangular annotation frame or the minimum circumscribed rectangle of the quadrilateral and the horizontal line is taken as the inclination angle of the ellipse.

3. The remote sensing image target detection method based on ellipse intersection and union ratio as claimed in claim 1, characterized in that: The backbone network includes a ResNet101 network and an FPN network, and the FPN network uses C2-C5 layer features.

4. The remote sensing image target detection method based on ellipse intersection-over-union ratio as claimed in claim 1, characterized in that: The formula of the ellipse intersection and union ratio is as follows: The formula of the elliptic intersection-over-union loss function is as follows: EllipseIoU Loss=1-EllipseIoU Among them, EllipseIoU is the ellipse intersection-union ratio, L1 and L2 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are closer, L3 and L4 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are farther away, θ is the angle between the two focal lines of the real ellipse and the predicted ellipse, b and b′ are the minor axis lengths of the real ellipse and the predicted ellipse, respectively.

5. The remote sensing image target detection method based on ellipse intersection-over-union ratio according to any one of claims 1 to 4, characterized in that: The method further comprises: Inputting a sample image and an ellipse annotation of an object in the sample image into the object detector; The multi-task loss composed of the RRPN network and the RoIHead network of the target detector is jointly trained, and when the regression loss function converges or iterates a set number of times, the pre-trained target detector is obtained.

6. A remote sensing image target detection device based on ellipse intersection-combination ratio, characterized in that: The device comprises: A preprocessing module, used to obtain an original image, and convert the annotation information of the target in the original image into an ellipse annotation as an input image of a pre-trained target detector, wherein the target detector includes a backbone network, a RRPN network and a RoIHead network connected in sequence, and both the RRPN network and the RoIHead network use an ellipse intersection-over-union ratio and an ellipse intersection-over-union ratio loss function; A feature extraction module, used to extract a multi-scale feature map of the input image using the backbone network; A candidate box generation module, used to use the multi-scale feature map as the input of the RRPN network to obtain a rotation candidate box of the target; A detection module, used to use the multi-scale feature map and the rotated candidate box as inputs of the RoIHead network to obtain a detection result of the original image; The RRPN network includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, and the third convolutional layer includes two parallel convolutions; the multi-scale feature map is used as the input of the RRPN network to obtain a rotation candidate box of the target, including: The multi-scale feature map is used as the input of the first convolutional layer to generate an anchor frame, and the angle of the anchor frame includes The aspect ratio is (1:2, 1:5, 1:8) and the size is (8, 16, 32, 64); The anchor frame is used as the input of the second convolutional layer and passes through the third convolutional layer to predict the foreground score, coordinate offset and angle of the anchor frame; The NMS algorithm is used to process the foreground score, coordinate offset and angle of the anchor frame to obtain a rotation candidate frame of the target; The RoIHead network includes an RRoI pooling layer and two multi-layer perceptron layers, and the multi-scale feature map and the rotated candidate box are used as inputs of the RoIHead network to obtain the detection result of the original image, including: The multi-scale feature map and the rotated candidate box are used as inputs of the RRoI pooling layer, and the rotated candidate boxes are projected onto corresponding layers of the multi-scale feature map to obtain corresponding feature matrices; Input the feature matrix into two multi-layer perceptron layers to obtain target categories and regression parameters; The target category and the regression parameter are processed using the NMS algorithm to obtain a detection result of the original image.

7. The remote sensing image target detection device based on ellipse intersection-over-union ratio according to claim 6, characterized in that: The preprocessing module comprises: A focus determination unit, configured to determine the positions of two foci of an ellipse based on the annotation information of the target in the original image, and obtain the focal line of the ellipse, wherein the annotation information of the target is a spiral rectangular annotation box or a minimum circumscribed rectangle of a quadrilateral; An ellipse annotation determination unit is used to take the center of the focal line as the center point of the ellipse, draw a straight line through the center point that is perpendicular to the focal line as the minor axis of the ellipse, and take the acute angle formed by the long side of the minimum circumscribed rectangle of the rotated rectangular annotation box or the quadrilateral and the horizontal line as the inclination angle of the ellipse.

8. The remote sensing image target detection device based on ellipse intersection-over-union ratio according to claim 6, characterized in that: The formula of the ellipse intersection and union ratio is as follows: The formula of the elliptic intersection-over-union loss function is as follows: EllipseIoU Loss=1-EllipseIoU Among them, EllipseIoU is the ellipse intersection-union ratio, L1 and L2 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are closer, L3 and L4 are the straight lines formed by connecting the foci on the real ellipse and the predicted ellipse that are farther away, θ is the angle between the two focal lines of the real ellipse and the predicted ellipse, b and b′ are the minor axis lengths of the real ellipse and the predicted ellipse, respectively.

Images