High-density foreground object detection method in optical remote sensing images based on semantic fusion and box alignment

Abstract

The present invention discloses a method for detecting high-density foreground targets in optical remote sensing images based on semantic fusion and frame alignment. The execution steps of the method are: (1) preprocessing data; (2) using a backbone network to extract features; (3) inputting the extracted features into a prediction network to generate prediction results; (4) determining whether it is a training mode; (5) in the training mode, the model will calculate the training loss and optimize the network parameters; (6) in the non-training mode, the prediction results are output. The method includes two feature re-identification convolution modules that can effectively distinguish internal texture features, thereby improving the accuracy of small targets. At the same time, an auxiliary point balance IoU (ABIoU) loss is also proposed. The ABIoU loss can not only avoid the local optimal solution of the distance term in the single-point regression, but also avoid the non-convergence of the IoU loss of the local aspect ratio, thereby achieving the stability of the loss value and the direct measurement of the side length, thereby reducing the problem of target misdetection and missed detection in a complex texture feature environment.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a method for detecting high-density foreground objects in optical remote sensing images based on semantic fusion and frame alignment. Background Art

[0002] With the iteration of optical sensor technology, satellite images have produced more spectral channels and spatial detail specifications, which can accommodate multi-scale objects. Aerial photos have different projection directions and coverage areas, resulting in targets with random scales and complex internal texture features, such as H-shaped ports and strip airports. When non-target tasks using detectors or feature extraction modules directly, relevant targets may be mistakenly lost. In order to enable faster classification and positioning of complex targets, transfer learning is widely used in image classification and detection in convolutional neural networks. Automatically generating features by loading a pre-trained model makes convergence easy, but it urgently requires a robust pre-trained model as a prerequisite for target detectors.

[0003] Modern object detectors are a method of statistically comparing data, allowing category labels to be stored in RGB units through many convolutional layers. This prior knowledge learning method allows for a wide range of visual effects, ultimately resulting in models that significantly improve device detection accuracy, and deeper or wider convolutional neural networks are widely accepted for outputting richer semantic features. Summary of the invention

[0004] The purpose of the present invention is to provide a method for detecting high-density foreground targets in optical remote sensing images based on semantic fusion and frame alignment, so as to solve the problem of how to reduce false detection and missed detection of targets in remote sensing images under complex texture feature environments.

[0005] The present invention adopts the following technical solutions:

[0006] A method for detecting high-density foreground objects in optical remote sensing images based on semantic fusion and frame alignment comprises the following steps:

[0007] Step 1: Unify the images in the dataset, set their size to 512×512, and randomly use the standard horizontal flipping and four-image stitching method for data augmentation;

[0008] Step 2, using the backbone feature extraction network to perform feature extraction;

[0009] Step 3: Input the feature information of the top layer of the feature pyramid into the Atrous Spatial Pyramid Pooling Lengthening (ASPPL) module to obtain a new multi-channel feature layer, and then use 1×1 convolution to change the number of channels;

[0010] ASPPL module: a variant of the Atrous Spatial Pyramidal Pooling (ASPP) module, including a CBL module with a 1×1 convolution kernel and three consecutive CBL modules with a 3×3 convolution kernel. The dilation rates of the convolution layers in these three CBL modules are 1, 2, and 3, respectively.

[0011] First, the CBL module performs a convolution operation on the feature layer, then normalizes the convolution result, and finally uses the Leaky ReLU function as the activation function;

[0012]

[0013] Where a is a constant

[0014] Step 4: Process the feature map obtained in step 3 through the IR-SimAM module and perform upsampling. Then, the upsampled feature map and the processed feature map are fused with the feature map of the penultimate layer of the feature pyramid using the concat method. Finally, a 1×1 convolution layer is used to change the number of channels.

[0015] IR-SimAM module refers to the reverse residual attention module, in which the SimAM attention mechanism is integrated, so that the attention block can achieve better feature selection and expression ability improvement. Specifically, it replaces the global average pooling layer with the SimAM attention mechanism, and considers the relationship between different moments through dynamic convolution. The reverse residual block and attention block structure remain basically unchanged in the reverse residual attention module. Through the reverse residual block, the gradient vanishing problem of the network is alleviated, thereby accelerating the convergence speed of the model; and through the attention block, the model's ability to focus on features is enhanced, further improving the performance of the model;

[0016] Step 5: The feature map obtained in step 4 is processed again by the IR-SimAM module and upsampled, and then the concat method is used to fuse it with the feature information of the third-to-last layer of the feature pyramid, and finally a 1×1 convolution layer is used to change the number of channels;

[0017] Step 6, combining the feature information in step 3, step 4, and step 5 to obtain the output result, then calculating the loss according to the true value, and using the optimization algorithm to update the weight parameters of the target feature extraction network to reach a convergence state;

[0018] lossfunction=Error class +Error obj +Error iou (1)

[0019]

[0020] BCE = plog(p) + (1-p)log(1-p) (5) where I is the prediction target, and the jth bounding box is responsible for the cell belonging to the 1st index. If satisfied, the value range is 1, otherwise 0. Z is the number of cells divided, T is the total number of categories, b x , b y , b w , b h , and are the horizontal and vertical coordinates, width and height margins of the box label;

[0021] The method of the present invention proposes an auxiliary-point balancing IoU (ABIoU). ABIoU optimizes the original IoU (Intersection over Union) loss function and is a new type of IoU. The loss function of ABIoU is:

[0022] L ABIoU* =1-IoU+P ABIoU (6)

[0023]

[0024] Where L ABIoU* Indicates L ABIoU The overall formula is, IoU is the intersection over union ratio of the predicted box and the true box, P ABIoU The calculation formula is:

[0025]

[0026] Among them B gt and B denote the centroids of TB and PB, ρ is the distance between two points in the 2D plane, C is the diagonal inside the smallest covered box that encloses the two boxes, v measures the consistency of the angles at corresponding positions, and a is a non-negative trade-off variable based on v and IoU. v and a are defined as:

[0027]

[0028] Where w is the width of the prediction box, h is the height of the prediction box, and w gt is the width of the real box, h gt is the height of the real frame, P ABIoU There are three sub-penalties, It is used to measure the consistency of the distance of the center point relative to the reference point, β and e are the trade-off parameters of the synchronous phase difference, and γ constitutes the way to measure the length of each side. They are defined as:

[0029]

[0030] Where Δ(·) represents two reference points A 1 and A 2 The difference in distance to the centroid of PB and TB, is a scale normalization formula that calculates the difference Δ(·) and Δ(·) between the length and width of the real box and the predicted box respectively. Defined as:

[0031]

[0032] Where W C is the width of the outer frame, H C is the height of the outer frame;

[0033] Step 7: Take the remote sensing image and rectangular box annotation that have not participated in the training as input, use the trained converged target feature extraction network to perform forward operation on the input to generate the detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a network structure diagram of the method of the present invention;

[0035] Figure 2 It is the structural diagram of ASPPL module and IR-SimAM module; DETAILED DESCRIPTION

[0036] The present invention proposes a method for detecting high-density foreground targets in optical remote sensing images based on semantic fusion and frame alignment. The specific implementation methods and basic principles of the present invention are further described below in conjunction with the accompanying drawings. Figure 1 As shown, the specific process includes the following:

[0037] Step 1: Data processing. Taking the DIOR dataset as an example, the dataset is first divided into a training set and a test set in a ratio of 8:2. Then the image size is adjusted to a fixed size, and the images in the dataset are randomly enhanced using the standard horizontal image flipping and four-image stitching method.

[0038] Step 2: Feature extraction, using the network and feature pyramid as the backbone feature network, saving the highest layer, the second to last layer, and the third to last layer of the feature pyramid as the feature extraction results;

[0039] Step 3: The output result of the highest level of the feature pyramid is a feature map with 512 channels. It is input into the ASPPL module to obtain a feature map with 256 channels. Then, a 1×1 convolution layer is used to expand its channels to 512.

[0040] like Figure 1 ,The ASPPL module consists of a CBL module with a convolution kernel of 1×1 and three consecutive CBL modules with a convolution kernel of 3×3. In order not to lose resolution and still expand the receptive field, the last three CBL modules in the ASPPL module use dilated convolutions with dilation rates of 1, 2, and 3 respectively. This method obtains multi-scale information by adjusting the dilation rate;

[0041] like Figure 2 ,A CBL module consists of a convolution layer, normalization, and activation function. After a lot of experiments, Leaky ReLU has better convergence and robustness than other activation functions;

[0042]

[0043] Step 4: Input the feature map obtained in step 3 into the IR-SimAM module, then use the Upsample method for upsampling, then use the concat method to fuse the upsampled result with the penultimate layer in the feature pyramid, and finally use 1×1 convolution to compress the number of channels of the concat result to 128;

[0044] Step 5: Input the feature map obtained in step 4 into an IR-SimAM module again and upsample it again. Use the concat method to fuse the output result with the feature map of the third-to-last layer of the feature pyramid.

[0045] Step 6, combining the characteristic information in steps 3, 4, and 5 to obtain the output result, calculating the loss according to the true value, and updating the weight parameters of the target feature extraction network by minimizing the loss function through the optimization algorithm until convergence;

[0046] Step 7: Use the remote sensing image that has not participated in the training as input, use the trained and converged target feature extraction network to perform forward operation on the input to generate the detection result.

[0047] Experimental setup:

[0048] The experiment was run for 150 epochs each time, using the stochastic gradient descent (SGD) optimizer and a cosine decay learning rate schedule with an initial learning rate of 0.01.

[0049] The experimental results are shown in Table 1-2. In YOLOv3-U, the mAP50 and mAP50:95 using ABIoU loss are 0.25% and 4.25% higher than those using IoU loss, respectively. Since YOLOv4 has sufficient improvement on the neck, there is no significant difference in mAP50 between YOLOv4 based on ABIoU loss and IoU loss. However, P and mAP50:95 of ABIoU loss are 2.1% and 0.91% higher than P and mAP50-95 of CIoU loss, respectively. In summary, the results show that ABIoU loss is robust in remote sensing image target detection tasks and can be widely integrated into mainstream target detection algorithms to improve detection efficiency and accuracy. In the comparative experiment with other methods, the mAP0.5 of this method is much higher than Faster R-CNN, SSD and ObjectBox, which proves the superiority of the method in accuracy. In addition, the original YOLOv3-U is an advanced single-stage method that concentrates on current data enhancement techniques and loss functions, and its mAP0.5 reaches an advanced level of 84.3%, but the recall rate lags behind YOLOv4's 87.1% and YOLOR's 86.5%, spending only 0.87×107 parameters while ensuring that AP0.5 and R are 2.7% and 1.7% higher than YOLOv4, respectively, thereby further improving the detection accuracy. The above experiments show that this method achieves a better balance between accuracy and model complexity.

[0050] Table 1 Comparison of results using ABIoU and IoU

[0051]

[0052] Table 2 Comparison of the results of the method of the present invention with other methods

[0053]

Claims

1. A method for detecting high-density foreground objects in optical remote sensing images based on semantic fusion and frame alignment, characterized in that: The method comprises: Step 1: Unify the images in the dataset, set their size to 512×512, and randomly use the standard horizontal flipping and four-image stitching method for data augmentation; Step 2, use the backbone feature extraction network to extract features; Step 3: Input the feature information of the top layer of the feature pyramid into the ASPPL module to obtain a new multi-channel feature layer, and then use a 1×1 convolution layer to change the number of channels. The ASPPL module is an ASPP extension module. Step 4: Process the feature map obtained in step 3 through the IR-SimAM module and perform upsampling. Then, the upsampled feature map and the processed feature map are fused with the feature map of the penultimate layer of the feature pyramid using the concat method. Finally, a 1×1 convolution layer is used to change the number of channels. The IR-SimAM module uses a combination of the reverse residual block and the SimAM attention block. Step 5: The feature map obtained in step 4 is processed again by the IR-SimAM module and upsampled, and then the concat method is used to fuse it with the feature information of the third-to-last layer of the feature pyramid, and finally a 1×1 convolution layer is used to change the number of channels; Step 6, combining the feature information in steps 3, 4, and 5 to obtain the output result, then calculating the loss based on the true value, and using the optimization algorithm to update the weight parameters of the rotation target feature extraction network to reach a convergence state; Step 7: Take the remote sensing image and rectangular box annotation that have not participated in the training as input, use the rotation target feature extraction network that has converged in the training to perform forward operation on the input to generate the detection result.

2. The method for detecting high-density foreground objects in optical remote sensing images based on semantic fusion and frame alignment according to claim 1, characterized in that: In step 3, an ASPPL module is proposed to obtain multi-scale information. The ASPPL module consists of a CBL (Convolution Batch-normalization LeakyRelu) module with a convolution kernel of 1×1 and three consecutive CBL modules with a convolution kernel of 3×3, hereinafter referred to as CBL module 1, CBL module 2, CBL module 3, and CBL module 4; CBL module: contains a convolution layer, a normalization layer and an activation function LeakyReLU(X); the specific formula of LeakyReLU(X) is: in is a constant; The convolution kernel of the convolution layer in CBL module 1 is 1×1, the convolution kernel of the convolution layer in CBL module 2 is 3×3, and the dilation rate is 1, the convolution kernel of the convolution layer in CBL module 3 is 3×3, and the dilation rate is 2, and the convolution kernel of the convolution layer in CBL module 4 is 3×3, and the dilation rate is 3. Their normalization processing layers and activation functions are consistent.

3. The method for detecting high-density foreground objects in optical remote sensing images based on semantic fusion and frame alignment according to claim 2, characterized in that: The IR-SimAM module is introduced in steps 4 and 5 to better preserve the details in the shallow feature layer. The workflow is as follows: s1, use the CBL module with a convolution kernel of 1×1 to expand the number of channels and reduce the dimension of the input feature map; s2, and then add residual and attention mechanisms through the residual attention module and SimAM attention mechanism. The feature map first input by the residual attention module first passes through a convolution layer and batch normalization, and then is divided into two branches: one is through 1×1 convolution and batch normalization and then passes through a ReLU activation function, and the other is also through 1×1 convolution and batch normalization. Finally, the two are added to get the residual output; In addition, the feature map first input by the SimAM attention mechanism is obtained by passing through three convolution layers respectively. θ and g, where The feature maps of θ and θ are expanded into two-dimensional vectors. The similarity matrix sim obtained by a matrix multiplication is normalized by Softmax, and then the weight matrix attn of the feature map is obtained by matrix multiplication. Finally, the input feature map is output by 1×1 convolution and batch normalization, and then added to get the output after the attention mechanism. s3, the final output feature map is obtained through the CBL module with a convolution kernel of 1×1.

4. The method for detecting high-density foreground objects in optical remote sensing images based on semantic fusion and frame alignment according to claim 3, characterized in that: In step 6, an auxiliary-point balancing IoU (ABIoU) is proposed. ABIoU optimizes the original IoU (Intersection over Union) loss function and is a new type of IoU. The loss function of ABIoU is: L ABIoU* =1-IoU+P ABIoU (1) Where L ABIoU* Indicates L ABIoU The overall formula is, IoU is the intersection over union ratio of the predicted box and the true box, P ABIoU The calculation formula is: Among them B gt and B denote the centroids of TB and PB, ρ is the distance between two points in the 2D plane, C is the diagonal inside the smallest covered box that encloses the two boxes, v measures the consistency of the angles at corresponding positions, and a is a non-negative trade-off variable based on v and IoU. v and a are defined as: Where w is the width of the prediction box, h is the height of the prediction box, and w gt is the width of the real box, h gt is the height of the real frame, P ABIoU There are three sub-penalties, It is used to measure the consistency of the distance of the center point relative to the reference point, β and e are the trade-off parameters of the synchronous phase difference, and γ constitutes the way to measure the length of each side. They are defined as: where Δ(·) represents the difference in distance from the two reference points A1 and A2 to the centroids of PB and TB, is a scale normalization formula that calculates the difference Δ(·) and Δ(·) between the length and width of the real box and the predicted box respectively. Defined as: Where W C is the width of the outer frame, H C The height of the outer frame.

Images