Directional remote sensing target detection method based on label assignment and feature alignment

Abstract

The application provides a directional remote sensing target detection method. The method comprises the following steps: step 1: a backbone network is used to extract feature maps C3, C4 and C5 of different sizes of an input image; step 2: a feature pyramid network is used to fuse the feature maps C3, C4 and C5 to obtain pyramid feature maps P3, P4, P5, P6 and P7 of different sizes; step 3: a cross-grid label assignment strategy is used to assign positive and negative samples to a true value frame, and then the assigned samples are used to train a feature refinement detection head; and step 4: the pyramid feature maps P3, P4, P5, P6 and P7 of different sizes are input into the trained feature refinement detection head to obtain a target detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing target detection, and particularly relates to a directional remote sensing target detection method based on label assignment and feature alignment. BACKGROUND

[0002] With the development of deep learning, the target detection task in remote sensing images has made rapid progress and is widely used in many fields, such as maritime rescue, port management and urban planning, etc. The task aims to identify the location and category of the object of interest. Different from natural images, most objects in remote sensing images have different sizes and aspect ratios, are densely arranged and are displayed in arbitrary directions. Therefore, the target detection task on remote sensing images still faces great challenges.

[0003] Due to the different differences between natural images and remote sensing images, such as object size ratio, aspect ratio, dense arrangement and arbitrary direction, the target detection task on remote sensing images still faces great challenges: (1) Most existing target detection methods are based on the intersection over union between different scale pre-defined anchor boxes and ground truth boxes to assign positive samples and negative samples. However, the targets in remote sensing images are usually distributed in large aspect ratios, dense arrangements and arbitrary directions, resulting in fewer positive samples assigned to the ground truth boxes, thereby increasing the possibility of missed detection. (2) The objects on remote sensing images are often arranged in arbitrary directions, while the detection method often has misalignment between anchor boxes and convolutional features during the training process, resulting in an inconsistent suboptimal learning process between classification scores and positioning accuracy, further increasing the difficulty of accurate detection. SUMMARY

[0004] In order to solve at least one of the problems of missed detection and false detection existing in the existing remote sensing image target detection method, the present application provides a directional remote sensing target detection method based on label assignment and feature alignment.

[0005] The directional remote sensing target detection method provided by the present application comprises:

[0006] Step 1: a backbone network is used to extract three different scale size feature maps C3, C4 and C5 of the input image;

[0007] Step 2: a feature pyramid network is used to fuse the feature maps C3, C4 and C5 to obtain five different scale size pyramid feature maps P3, P4, P5, P6 and P7;

[0008] Step 3: a cross-grid label assignment strategy is used to assign positive and negative samples to the ground truth boxes, and then the assigned samples are used to train the feature refinement detection head;

[0009] Step 4: input the five pyramid feature maps P3, P4, P5, P6 and P7 of different sizes into the trained feature refinement detection head to obtain the target detection result.

[0010] Further, the cross-grid label assignment strategy specifically comprises:

[0011] Traverse the feature maps output by each layer of the feature pyramid network, and match all anchor boxes on the feature maps of each layer into candidate positive samples, candidate negative samples and ignored samples based on the maximum IoU rule;

[0012] Traverse the feature maps output by each layer of the feature pyramid network, and judge whether there is a candidate positive sample on the current layer feature map If there is, select the k nearest neighbor grids to the target center from the neighbor grid set of the target center, and take the anchor boxes corresponding to the k neighbor grids as the preliminary candidate positive sample set, denoted as For each sample s in , if the center point of the sample s is in the true value box and the candidate positive sample and the grid where the center point is located is responsible for detecting the target, then the sample s is taken as the positive sample of the true value box ; if not, traverse the next layer; until all layers of feature maps are traversed; wherein i represents the i-th layer of the feature pyramid network.

[0013] Further, the target detection process of the feature refinement detection head specifically comprises:

[0014] The input pyramid feature map pi passes through two convolution layers in series to obtain a classification feature map P cls , and a refinement feature map X is generated based on the classification feature map P cls according to formula (1):

[0015]

[0016] Wherein, i = 3, 4, 5, 6, 7.

[0017] The input pyramid feature map pi passes through two convolution layers in series to obtain a regression feature map P reg , and 9 sampling points are selected on the regression feature map P reg ; for each sampling point p0, an alignment feature map Y(p0) is generated according to formula (2):

[0018] Y(p0) = AlignConv(X, offset(p0)) (2)

[0019] wherein AlignConv represents an alignment convolution, and offset(p0) represents a 5D offset vector generated by convolution for each sampling point p0 on the regression feature map P reg

[0020] According to the alignment feature map Y(p0), classification and boundary regression are respectively performed, and the category and directional frame parameters of the target are predicted.

[0021] The beneficial effects of the present application are as follows:

[0022] (1) The present application introduces a cross-grid label assignment strategy to increase the number of high-quality positive samples, so that the anchor frame can cover the real target as much as possible, thereby reducing the missed detection rate. At the same time, since the label assignment strategy does not have additional computational overhead, it can be well applied to existing models.

[0023] (2) The present application designs a feature refinement detection head to align the receptive field of the convolution kernel and the corresponding features of the rotated region, thereby alleviating the misalignment problem between the alignment convolution features and the rotated target, to generate high-quality anchor frames, which to some extent improves the positioning accuracy.

[0024] (3) The present application can obtain any directional target in a remote sensing image by directly predicting five parameters of each spatial position, and can realize better training sample selection and accurate positioning in a dense scene in a remote sensing image. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The overall network architecture schematic diagram provided for the embodiment of the present application;

[0026] Figure 2 The cross-grid label assignment strategy provided for the embodiment of the present application;

[0027] Figure 3 The effect schematic diagram of the cross-grid label assignment strategy provided for the embodiment of the present application;

[0028] Figure 4 The structure schematic diagram of the feature refinement detection head provided for the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0030] ​The embodiment of the application provides a directional remote sensing target detection method based on label assignment and feature alignment, uses deep learning to replace a traditional method, and the overall network structure of a deep learning network model used mainly comprises four parts: a backbone network for feature extraction, a feature pyramid network for feature fusion, a cross-grid label assignment strategy for training sample selection, and a feature refinement detection head for feature alignment. Figure 1 As shown in the overall network structure, the method mainly comprises the following steps: Figure 1

[0031] S101: pre-process a remote sensing image to generate a plurality of feature maps with a size of HxW as input images; input the input images with the size of HxW into a backbone network, and extract three feature maps C3, C4 and C5 with different sizes from the input images by using the backbone network;

[0032] S102: fuse the feature maps C3, C4 and C5 by using a feature pyramid network to obtain five pyramid feature maps P3, P4, P5, P6 and P7 with different sizes; wherein, C is the channel dimension of Pi; s i = 2 i is the input remote sensing image corresponding to the s i down sampling rate.

[0033] S103: a cross-grid label assignment strategy is used to assign positive and negative samples to the ground truth box, and then the feature refinement detection head is trained by using the assigned samples;

[0034] Specifically, most target detectors achieve target detection by presetting a large number of dense anchor boxes on input images. However, for remote sensing images, a large number of dense anchor boxes are difficult to achieve good spatial alignment with directional targets with large aspect ratios and densely arranged features, resulting in fewer positive samples assigned to the ground truth box.

[0035] In order to solve the above problems, the embodiment of the application introduces a cross-grid label assignment strategy, which increases the number of high-quality positive samples to alleviate the imbalance between the foreground (i.e. the region of interest in the image) and the background (i.e. the image region irrelevant to the target) in the remote sensing image training process. The operation process of the cross-grid label assignment strategy specifically comprises:

[0036] Traverse the feature maps output by each layer of the feature pyramid network, and match all anchor boxes on each layer of feature maps into candidate positive samples, candidate negative samples and ignored samples based on the maximum IoU rule;

[0037] ​Specifically, for the feature maps output by each layer, maximum IoU rule-based matching is adopted to ensure that each ground truth box can correspond to at least one anchor box. If the IoU of an anchor box and a ground truth box is greater than the foreground threshold T pos (0.5 in this embodiment), and the grid where the anchor point is located is responsible for detecting the ground truth box, the anchor box is assigned as a candidate positive sample; if the IoU is less than the background threshold T neg (0.4 in this embodiment), the anchor box is assigned as a candidate negative sample. The rest of the samples are assigned as ignored samples. It should be noted that the positive sample is also referred to as a positive sample, and the negative sample is also referred to as a negative sample.

[0038] The feature maps output by each layer of the feature pyramid network are traversed to determine whether there is a candidate positive sample on the feature map of the current layer If there is, the k nearest neighbor grids of the target center are selected from the neighbor grid set of the target center The anchor boxes corresponding to the k neighbor grids are taken as the preliminary candidate positive sample set, denoted as For each sample s in If the center point of the sample s is in the ground truth box and the candidate positive sample and the grid where the center point is located is responsible for detecting the target, the sample s is taken as the positive sample of the ground truth box If not, the next layer is traversed; until all layers of feature maps are traversed; wherein i represents the i-th layer of the feature pyramid network, i = 3, 4, 5, 6, 7.

[0039] Specifically, in this embodiment, the positive sample selection process of this step is referred to as cross-grid rule matching. For the feature maps output by each layer, cross-grid rule matching is adopted to ensure that the number of positive samples in the feature map output by the current layer is increased.

[0040] In this embodiment, the neighbor grid set of the target center is composed of 5 grids including the grid where the target center is located and the grids adjacent to the grid in the up, down, left and right directions First, the two nearest neighbor grids of the target center are found, at this time, at most three grids exist, i.e., the grid where the target center is located and the two nearest neighbor grids of the target center; then, the anchor boxes are generated by taking the top-left corner points of the three grids as anchor points, and the generated anchor boxes (as shown by the yellow boxes in Figure 3 ) are taken as the preliminary candidate positive samples. Next, the anchor boxes whose center coordinates are outside the target bounding box and the candidate positive samples are removed to ensure that the increased positive samples are responsible for the detection of the current target; finally, for the remaining anchor boxes, they are considered as high-quality positive samples that can be used for training and loss calculation.

[0041] AsFigure 3 The left subgraph of FIG. 1 shows that the green dot represents the center point of the ground truth box (i.e., the target center), the red dot represents the center point of the three neighbor grids found, and the yellow dot represents the anchor point (i.e., the upper left corner point of one of the grids) assigned as a positive sample. The grid where the center point of the ground truth box is located is responsible for detecting the target. If the cross-grid rule matching is met, the two neighbor networks closest to the target center need to be considered as positive samples of the ground truth box. Through this assignment strategy, the present application can assign more positive training samples to each ground truth box, not only suppressing redundant positive samples, but also ensuring a stable training process.

[0042] S104: input the pyramid feature maps P3, P4, P5, P6 and P7 of five different sizes into the trained feature refinement detection head to obtain a target detection result.

[0043] Specifically, due to the misalignment problem between anchor boxes and aligned convolution features in the rotation target detection, inconsistency between classification scores and positioning accuracy is caused. Therefore, in order to efficiently extract the rotation features from the rotation region, the present application improves the baseline detection head based on the aligned convolution module, which can capture the geometric information of the oriented bounding box (OBB) and the context information around it.

[0044] Optionally, the model structure of the feature refinement detection head is as shown in FIG. 2. Figure 4 The feature refinement detection head is a network with two parallel branches: a classification subnetwork for predicting categories and a regression subnetwork for regressing bounding boxes. The classification subnetwork classifies anchor boxes into different categories, and the regression subnetwork optimizes the horizontal anchor box into a high-quality rotation anchor box. Both the classification subnetwork and the regression subnetwork are stacked by multiple convolution layers, and the two subnetworks share feature map parameter weights to reduce the amount of calculation.

[0045] Specifically, the feature refinement detection head takes the feature map Pi output by the feature pyramid network as input.

[0046] First, the feature map Pi is processed by the above two subnetworks respectively. Among them, the feature map obtained after the input feature map Pi in the classification subnetwork passes through two convolution layers in series is the classification feature map P cls ; the feature map obtained after the input feature map Pi in the regression subnetwork passes through two convolution layers in series is the regression feature map P reg . Among them, s i =2 i is the input remote sensing image corresponding to s i times down-sampling rate.

[0047] Then, from the regression feature map P regThe sampling points P∈{0, 1, ..., H-1}×{0, 1, ..., W-1} are selected. In this embodiment, the two concatenated convolutional layers in the regression subnet and the classification subnet are both 3×3 convolutional layers. Based on this, 9 sampling points are usually selected; preferably, the sampling positions are the 4 vertices, the midpoints of the 4 edges, and the center point of the OBB, such as... Figure 4 The nine orange dots represent nine sampling points. These nine locations are mapped onto the feature map, and the features at the projection points (corresponding to...) are processed by alignment convolution. Figure 4 The nine blue dots in the image are convolved.

[0048] Then, based on the classification feature map P cls Generate the refined feature map X according to formula (1):

[0049]

[0050] Where i = 3, 4, 5, 6, 7; the σ function is the sigmoid function.

[0051] Next, feature alignment is achieved based on the refined feature map X, specifically: for each sampling point p0∈P, an aligned feature map Y(p0) is generated according to formula (2):

[0052] Y(p0)=AlignConv(X,offset)p0)) (2)

[0053] Where AlgnConv represents aligned convolution, and offset(p0) represents the regression feature map P. reg Each sampling point p0 on the vector is generated by convolution into a 5D offset vector.

[0054] In this process, the present invention uses aligned convolution to align the rotational features in the rotational region. Compared with conventional 2D convolution, aligned convolution simulates the shape of the target by adding an additional offset Δp to each sampling point p0, transforming the sampling points into convolutional features with arbitrary rotation. The aligned convolution process can be further represented by formula (3):

[0055]

[0056] Where R = (-1, -1), (-1, 0), ..., (0, 1), (1, 1) represents a standard grid with a convolution kernel size of 3×3 and an expansion rate of 1. Δp k = (k = 1, ..., |R|) is the offset, which is the difference between the anchor frame sampling position and the regular sampling position.

[0057] Finally, the aligned feature maps Y(p0) are respectively classified and boundary-regressed to predict the category and orientation box parameters of the target.

[0058] It is worth noting that K in the final feature maps in the classification subnetwork refers to K categories of each anchor box in each feature map, where there are A anchor boxes in each feature map. Considering the density of object arrangement in remote sensing images, preferably, the present embodiment only sets a group of anchor boxes for each position in the feature map.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting a remote sensing target of interest, characterized in that, The method comprises the following steps: Step 1: extracting feature maps C3, C4 and C5 of three different sizes of the input image by using a backbone network; Step 2: fusing the feature maps C3, C4 and C5 by using a feature pyramid network to obtain pyramid feature maps P3, P4, P5, P6 and P7 of five different sizes; Step 3: assigning positive and negative samples to the true value frame by using a cross-grid label assignment strategy, and then training a feature refinement detection head by using the assigned samples; The cross-grid label assignment strategy specifically comprises: traversing the feature maps output by each layer of the feature pyramid network, and matching all anchor frames on each layer of the feature maps into candidate positive samples, candidate negative samples and ignored samples based on a maximum IoU rule; Traverse the feature maps output by each layer of the feature pyramid network to determine whether there are candidate positive samples in the feature map of the current layer. If it exists, then select from the set of neighborhood grids of the target center. Select the one closest to the target center Each neighborhood grid will The anchor frames corresponding to each neighborhood grid are used as the initial candidate positive sample set, denoted as... ;against Each sample in If the sample The center point is in the truth box and candidate positive samples If the grid containing the center point is responsible for detecting the target, then the sample... As a truth box If no positive sample is found, the next layer is traversed until all feature maps of all layers have been traversed; where i represents the i-th layer of the feature pyramid network. Step 4: inputting the pyramid feature maps P3, P4, P5, P6 and P7 of five different sizes into the trained feature refinement detection head to obtain a target detection result; and the target detection process of the feature refinement detection head specifically comprises: input pyramid feature map classification feature map obtained through two convolution layers in series , based on the classification feature map generate a refined feature map according to formula (1) : （1） wherein = 3,4,5,6,7; input pyramid feature map obtained by two convolution layers in series , the regression feature map select 9 sampling points; for each sampling point , the alignment feature map is generated according to formula (2) （2） wherein, denotes an aligned convolution, denotes a regression feature map over each sampling point 5D offset vector generated by a convolution; According to the alignment feature map The classification and the boundary regression are respectively performed, and the category and the directional frame parameter of the target are predicted.

Images