Remote sensing image rotation target detection method with aggregated frequency domain information context awareness

Abstract

The application discloses a kind of polymeric frequency domain information context-aware remote sensing image rotating target detection methods, specifically including the following steps: step 1, obtain the image X after pre-processing;Step 2, the image X obtained in step 1 is extracted, fused and decoupled with feature information, and a polymeric frequency domain information context-aware remote sensing image rotating target detection network model is constructed;Step 3, the model constructed in step 3 is trained using the data set preprocessed in step 1, to obtain the trained polymeric frequency domain information context-aware remote sensing image rotating target detection model.Step 4, the test set image processed in step 1 is put into the model trained in step 3, i.e.This application overcomes the complex background interference of aerial remote sensing image and the inter-class similarity problem of ground target, and then accurately detects the ground rotating target.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to a remote sensing image rotating target detection method based on aggregation of frequency domain information and context perception. BACKGROUND

[0002] Aerial remote sensing image target detection aims to give the category and location of the target of interest in the aerial remote sensing image, and provides evidence for further information interpretation and reasoning, and is a key technology for aerial remote sensing image interpretation, and has important applications in intelligence reconnaissance, target monitoring, disaster rescue and the like.

[0003] A traditional remote sensing image target detection method uses a manually designed feature to match the target, and this way is difficult to adapt to massive data and has poor generalization performance, and cannot meet the requirements of accurate and efficient target detection tasks in complex and changeable application scenarios; a data-driven deep learning method has strong feature expression capability, and has become a development trend in the field of remote sensing image target detection. Since the angle direction of the ground target is arbitrary, and the scales and textures of different ground targets are similar, and the background is complex, the existing method has the problems of inaccurate positioning and identification of the target to be detected when capturing the rotating target in the aerial remote sensing image.

[0004] Fusion of the context information of the target in the image can obtain key information related to the target to be detected from a large amount of complex image background information, and then improve the detection capability of the model for the rotating target, and is an effective method for solving the problem of remote sensing image rotating target detection. However, in the aspect of using image context information, most of the previous aerial remote sensing image target detection methods aggregate different context information with fixed weights, and do not consider the influence of target scale, angle and different frequency signals of the image itself on the target feature, and therefore have deficiencies in extracting robust target features and suppressing background redundant information. In summary, under the influence of the complex background interference of the high-resolution image and the similarity between the target classes, it is still a severe challenge to realize accurate detection of the rotating target in the remote sensing image. SUMMARY

[0005] The purpose of the present application is to provide a remote sensing image rotating target detection method based on aggregation of frequency domain information and context perception, which overcomes the problems of complex background interference of the aerial remote sensing image and the similarity between the ground target classes, and accurately detects the rotating ground target.

[0006] The technical scheme adopted by the present application is a remote sensing image rotating target detection method based on aggregation of frequency domain information and context perception, which specifically comprises the following steps:

[0007] Step 1, pre-process the images in the training set and the test set which have been divided in the HRSC2016 data set, and obtain the pre-processed training set and test set images X;

[0008] Step 2, feature extraction, fusion and feature information decoupling are performed on the image X obtained in step 1, and an aggregated frequency domain information context-aware remote sensing image rotating target detection network model is constructed;

[0009] Step 3, the model constructed in step 2 is trained by using the data set preprocessed in step 1, and a trained aggregated frequency domain information context-aware remote sensing image rotating target detection model is obtained;

[0010] Step 4, the test set image processed in step 1 is put into the model trained in step 3, and finally an aerial remote sensing image after detecting the target is obtained.

[0011] The characteristics of the present application are:

[0012] The image preprocessing process in step 1 is: the image is sequentially subjected to random flipping, random cropping and color transformation, then the picture is reduced to one scale in [0.25, 0.5, 0.75] with a probability of 30% and filled to 608x608 with 0 pixels, and finally the preprocessed training set and test set images X are obtained.

[0013] The specific process of step 2 is:

[0014] Step 2.1, ResNet152 is used as the backbone network, and the network weight pre-trained on the ImageNet data set is loaded, then the preprocessed image X is input into the backbone network to extract multiple image features of different scales {C1, C2, C3, C4, C5}, as shown in formula (1); the above different scale image features {C2, C3, C4, C5} are aggregated based on PAFPN to obtain multi-scale aggregated features {P2, P3}, as shown in formula (2):

[0015] {C1, C2, C3, C4, C5}=ResNet152(X)(1);

[0016] {P2, P3}=PAFPN({C2, C3, C4, C5})(2);

[0017] Step 2.2, the C1 scale feature is subjected to a high-low frequency feature enhancement module to obtain an aggregated context information feature {Q2, Q3}, as shown in formula (3), the high-low frequency feature enhancement module includes a high frequency branch and a low frequency branch, which respectively extract different frequency image information for feature enhancement:

[0018] {Q2, Q3}=HLFE(C1) (3);

[0019] Step 2.3, according to the multi-scale aggregated feature {P2, P3} obtained in step 2.1 and the aggregated context information feature {Q2, Q3} obtained in step 2.2, the features {F2, F3} obtained by splicing the corresponding indexes are as shown in the following formula (4):

[0020] {F2, F3} = Cat({P2, P3}, {Q2, Q3}) (4);

[0021] Step 2.4, taking the features {F2, F3} obtained in step 2.3 as the input of the channel attention module, learning the aggregated features {M2, M3} by modeling the relationship between different feature channels, as shown in the following formula (5):

[0022] {M2, M3} = CA({F2, F3}) (5);

[0023] Step 2.5, sending the aggregated features {M2, M3} obtained in step 2.4 into the decoupling head to decouple the information, wherein the aggregated feature M i will be converted into four branches, as shown in the following formula (6): heat map H, offset O, box parameter B and azimuth map a, the four conversion branches are all realized by two convolution layers, the first convolution layer is composed of 3x3 convolution operation, batch normalization and activation function ReLU, and the second convolution layer is composed of 1x1 convolution operation; the overall operation process of the global information aggregation module is as follows: for the input aggregated feature M i , the deep separable large convolution operation is used to realize the acquisition of super large receptive field, and then the information gain feature M i ' is obtained, as shown in the following formula (7):

[0024] H, O, B, a = Head(M i ) (6);

[0025] M i ' = ReLU(BN(ReLU(DWBC 31×31 (M i ))) (7).

[0026] The specific process of step 2.2 is as follows:

[0027] Step 2.2.1, for the input feature map C1, the low-frequency feature LF is extracted by the combination of N1 steps of 3x3 convolution operation with a step of 2, batch normalization and activation function ReLU on the low-frequency branch, as shown in the following formula (8):

[0028] LF = <ReLU(BN(Conv 3×3 (C1))) > x N1 (8);

[0029] Step 2.2.2, on the high-frequency branch, the global context information path mainly realizes the aggregation of context information through a large convolution kernel to obtain context aggregation features CI, as shown in equation (9); the maximum pooling path extracts high-frequency features HF of image information through the combination of N2 maximum pooling operations and 1x1 convolution adjustment channels, as shown in equation (10); then the high-frequency features and the context aggregation features are spliced in the channel dimension, and after 1x1 convolution adjustment channels, the real high-frequency feature map is obtained; finally, the high-frequency features and the low-frequency features are spliced as the input of channel attention, as shown in equation (11), and after realizing the channel importance weight screening, the output aggregated context information feature Q is obtained, as shown in equation (12):

[0030] CI = BC 31×31 (C1) (9) ;

[0031] HF = <Conv 1×1 (MaxPool(C1))> x N2 (10) ;

[0032]

[0033] Q = SE(Cat(Conv 1×1 (Cat(HF, CI)), LF)) (12).

[0034] The loss function used in step 3 is as follows:

[0035] Smooth L1 loss function: Smooth L1 loss proposes to set the loss function as a piecewise function, which is more robust to outliers, where x is the numerical difference between the predicted frame and the real frame:

[0036]

[0037] BCE loss function: BCELoss is a binary cross-entropy loss, suitable for 0 / 1 distribution binary classification problems, where y is the label, y∈{0,1}, x is the probability of the predicted class output by sigmoid, x∈[0,1], and the function learning goal is to let the output x approximate the label y:

[0038] L BCE =-(1-y)log(1-x)-ylog(x) (14) ;

[0039] Focal Loss loss function: when γ = 0, Focal Loss degenerates into ordinary cross-entropy function; while when γ > 0 and 1-p t ≥ 0, the loss of Focal Loss is less than the standard cross-entropy loss, as follows:

[0040] FL(p t )=-α t (1-p t ) γ log(p t )(15);

[0041] In formula (15), when y=1, p t =p and a t =a, and when y≠1, p t =1-p and a t =1-a; y is a predicted label value;

[0042] The total loss function is:

[0043] Loss=L h +L o +L b +L α (16);

[0044] In formula (16), L h , L o , L b and L a respectively represent the loss of the heat map branch, the loss of the offset branch, the loss of the frame parameter branch and the loss of the orientation map branch.

[0045] The beneficial effects of the present application are that the present application proposes a single-stage, anchor-free paradigm efficient aggregated context information rotating target detection method. On the basis of the box boundary perception vector detector, the high-low frequency feature enhancement module and the global information aggregation module proposed by the present application can consider the basic feature information of the image, better aggregate the context information to accurately model the target feature, and perform better in capturing the rotating boundary box. The verification on the public data set proves that the method of the present application has higher indicators and excellent performance. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 Fig. 1 is a flowchart of the remote sensing image rotating target detection method of the present application;

[0047] Figure 2 Fig. 3 is a high-low frequency feature enhancement module diagram of the remote sensing image rotating target detection method of the present application;

[0048] Figure 3 Fig. 4 is a global information aggregation module diagram of the remote sensing image rotating target detection method of the present application;

[0049] Figure 4A schematic diagram of the overall structure of the polymeric frequency domain information context-aware remote sensing image rotating target detection method of the present application;

[0050] Figure 5 A heat map result comparison chart of the polymeric frequency domain information context-aware remote sensing image rotating target detection method of the present application;

[0051] Figure 6 A detection result comparison chart of the polymeric frequency domain information context-aware remote sensing image rotating target detection method of the present application. DETAILED DESCRIPTION

[0052] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0053] The polymeric frequency domain information context-aware remote sensing image rotating target detection method of the present application, as shown in Figure 1 , specifically includes the following steps:

[0054] Step 1, input the HRSC2016 dataset, which contains 1061 RGB color aerial remote sensing images and a total of 2976 ship targets, the image resolution is 0.4-2m, the image size ranges from 300x300 pixels to 1500x900 pixels, and most of them are 1000 pixels. The training data set is used for training, and the test set is used for testing.

[0055] Step 2, the images in the training set and test set in the HRSC2016 dataset are preprocessed, the images are sequentially subjected to random flipping, random cropping and color transformation, then the pictures are reduced to one scale in [0.25, 0.5, 0.75] with a probability of 30% and filled with 0 pixels to 608x608, and finally the preprocessed training set and test set images X are obtained.

[0056] Step 3, according to Figure 2 , the image is subjected to feature extraction (Backbone), fusion (Neck) and feature information decoupling (Head) to construct a polymeric frequency domain information context-aware remote sensing image rotating target detection network model. The specific steps are as follows:

[0057] Step 3.1, using ResNetl52 as the backbone network, loading the pre-trained network weight on the ImageNet dataset, then taking the pre-processed image X as the input of the backbone network, extracting multiple image features of different scales {C1, C2, C3, C4, C5} (as formula 1), the size of which is 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size, respectively. Based on the PAFPN, the above different scale image features {C2, C3, C4, C5} are aggregated to obtain multi-scale aggregated features {P2, P3} (as formula 2), and the feature map size is consistent with the corresponding feature map size of the previous index.

[0058] {C1, C2, C3, C4, C5} = ResNetl52(X) (1);

[0059] {P2, P3} = PAFPN({C2, C3, C4, C5}) (2);

[0060] Step 3.2, at the same time as step 3.1, as shown in Figure 3 , the C1 scale feature passes through the high-low frequency feature enhancement module (HLFE) to obtain the aggregated context information feature {Q2, Q3} (as formula 3). The high-low frequency feature enhancement module includes high frequency branch and low frequency branch, which respectively extracts image information of different frequencies for feature enhancement. The specific steps of the module are as follows:

[0061] {Q2, Q3} = HLFE(C1) (3);

[0062] Step 3.2.1, for the input feature map C1, in the low frequency branch, the combination of N1 steps of 3x3 convolution operation (Conv 3×3 ), batch normalization (BN) and activation function ReLU is used to realize the extraction of low frequency feature, and the low frequency feature LF is obtained, as formula (4).

[0063] LF = <ReLU(BN(Conv 3×3 (C1)))> x N1 (4);

[0064] Step 3.2.2, in the high frequency branch, the global context information path mainly realizes the aggregation of context information through a large convolution kernel (BC 31×31 ), and obtains the context aggregation feature CI, as shown in formula (5); the maximum pooling path realizes the aggregation of context information through N2 maximum pooling operations (MaxPool) and 1x1 convolution (Conv 1×1) to extract high frequency features HF, as shown in equation (6). Then the high frequency features and the context aggregated features are concatenated in the channel dimension, and after 1x1 convolution adjustment of the channel, the real high frequency feature map is obtained. Finally, the high frequency features and the low frequency features are concatenated as the input of the channel attention (SE) (as shown in equation 7), and after realizing the channel importance weight screening, the output aggregated context information feature Q is obtained, as shown in equation (8). Wherein, N1 and N2 in equations (4) and (6) represent the number of times of convolution and pooling required to realize the final feature map size on the low frequency branch and the high frequency branch, respectively; X in 、X out in equation (7) represent the feature input and output of the channel attention respectively, GAP represents the global average pooling layer, FC represents the full connection layer, and Sigmoid represents the activation function; Cat() in equation (8) represents the concatenation of the feature maps in the channel dimension.

[0065] CI=BC 31×31 (C1)(5);

[0066] HF=<Conv 1×1 (MaxPool(C1))>×N2(6);

[0067]

[0068] Q=SE(Cat(Conv 1×1 (Cat(HF,CI)),LF))(8);

[0069] Step 3.3, according to the multi-scale aggregated features {P2, P3} obtained in step 3.1 and the aggregated context information features {Q2, Q3} obtained in step 3.2, the features {F2, F3} obtained by concatenating them with the corresponding indexes are obtained (as shown in equation 9).

[0070] {F2,F3}=Cat({P2,P3},{Q2,Q3}) (9);

[0071] Step 3.4, taking the features {F2, F3} obtained in step 3.3 as the input of the channel attention module (CA), learning the aggregated features {M2, M3} by modeling the relationship between different feature channels, as shown in equation (10).

[0072] {M2,M3}=CA({F2,F3})(10);

[0073] Step 3.5, the extracted final features are decoupled, specifically, the aggregated features {M2, M3} obtained in step 3.4 are sent to the decoupling head for information decoupling, wherein the aggregated feature M iThe conversion is divided into four branches (as formula 11): heat map (H), offset (O), box parameter (B) and azimuth map (a). The four conversion branches are realized by two convolutional layers, the first convolutional layer is composed of 3x3 convolution operation (Conv 3×3 ), batch normalization (BN) and activation function ReLU, and the second convolutional layer is composed of 1x1 convolution operation. In the information decoupling process, the first convolutional layer in the heat map conversion branch is replaced by a global information aggregation module, and a heat map feature with information gain is further constructed. The global information aggregation module mainly uses a large-scale convolution kernel design, as shown in Figure 4 . The large receptive field provided by the large-scale convolution combines the global information of the target obtained by the heat map feature, and realizes the feature enhancement of the heat map by fusing the context information. The overall operation process of the global information aggregation module is as follows: for the input aggregation feature M i , the deep separable large convolution operation (DWBC 31×31 ) is used to realize the acquisition of super large receptive field, and then the feature M i ' with information gain is obtained, as shown in formula (12). The gain results of the heat map obtained by the benchmark detector and the method of the present application are visualized, as shown in Figure 5 . Figure 5 The first row is the input image, Figure 5 The second row and Figure 5 The third row is the heat map effect obtained by the benchmark detector, Figure 5 The fourth row and Figure 5 The fifth row is the heat map effect obtained by the method of the present application. From the comparison of the heat maps in Figure 5 , the method of the present application can more accurately select the center point of the target on the heat map (as shown in columns a-c in Figure 5 ), and reduce the false alarm in the benchmark method (as shown in columns d and e in Figure 6 ).

[0074] H,O,B, a=Head(M i )(11);

[0075] M i '=ReLU(BN(ReLU(DWBC 31×31 (M i ))))(12);

[0076] Step 3.6, after steps 3.1 to 3.5, the remote sensing image rotating target detection network model with aggregated frequency domain information context awareness is constructed.

[0077] Step 4, training the model using the preprocessed data set from Step 2: using Smooth L1 loss function, BCE loss function and Focal Loss loss function to constrain the network model obtained in Step 3, then backpropagation for parameter update, through 100 rounds of training, here 1 round refers to training all preprocessed images, finally get the trained aggregated frequency domain information context-aware remote sensing image rotation target detection model. The loss function used is as follows:

[0078] Smooth L1 loss function (offset branch, box parameter branch): Smooth L1 loss proposes to set the loss function as a piecewise function, which is more robust to outliers, where x is the numerical difference between the predicted box and the true box:

[0079]

[0080] BCE loss function (azimuth map branch): BCELoss is a binary cross entropy loss, which is suitable for 0 / 1 distribution of binary classification problem, where y is the label (y∈{0,1}), x is the probability of sigmoid output of predicted class (x∈[0,1]), the function learning goal is to make the output x to approximate the label y:

[0081] L BCE =-(1-y)log(1-x)-ylog(x)(14);

[0082] Focal Loss loss function (heat map branch): in order to solve the problem of positive and negative sample imbalance and serious difficulty sample imbalance in one-stage target detection, Focal Loss increases a prediction probability p t , hyperparameter γ and hyperparameter α t in the original cross entropy loss. Hyperparameter α t is to solve the class imbalance problem, where class 1 introduces a weight factor α∈[0,1], and the weight for class-1 is 1-α; Hyperparameter γ is to solve the difficulty sample problem, when γ = 0, Focal Loss degenerates into ordinary cross entropy function; When γ > 0 and 1-p t ≥ 0, the loss of Focal Loss is less than the standard cross entropy loss, at this time the function will focus on the proportion of difficult sample loss in the total loss. The overall formula is as follows:

[0083] FL(p t )=-α t (1-p t ) γ log(pt (15);

[0084] In formula (15), when y = 1, p t =p and α t =α, and when y≠1, p t =1-p and α t =1-α; y is the predicted label value.

[0085] Therefore, the total loss function of this invention is:

[0086] Loss = L h +L o +L b +L α (16);

[0087] In formula (16), L h L o L b and L a These represent the branch loss of the heatmap, the branch loss of the offset, the branch loss of the box parameter, and the branch loss of the orientation map, respectively.

[0088] Step 5: Input the test set images processed in Step 2 into the model trained in Step 4 to obtain the final aerial remote sensing image after target detection. The comparison results are as follows: Figure 6 As shown. Figure 6 The first row shows the detected target image obtained by the baseline detector. Figure 6 The second line shows the detected target image obtained by the method of this invention. From Figure 6 Based on the comparison results of target detection bounding box capture, the method of the present invention can capture the target bounding box more accurately. Figure 6 (columns a-c) and reduce false alarms in the baseline method ( ​ (Column d) and missed detections (Column e).

Claims

1. A method for polymeric frequency domain information context-aware remote sensing image rotation target detection, characterized in that: Specifically comprising the following steps: Step 1, pre-process the images in the training set and test set divided in HRSC2016 dataset, the pre-processed training set and test set images X; Step 2, feature extraction, fusion and feature information decoupling are performed on the images X obtained in step 1, and a remote sensing image rotation target detection network model with aggregated frequency domain information context perception is constructed; Step 3, the model constructed in step 2 is trained by using the pre-processed dataset in step 1, and a trained remote sensing image rotation target detection model with aggregated frequency domain information context perception is obtained; The specific process of step 2 is as follows: Step 2.1, using ResNet152 As the backbone network, while loading the pre-trained network weight on the ImageNet dataset, then taking the preprocessed image X as the input of the backbone network, extracting multiple image features of different scales As shown in the following formula (1); based on PAFPN aggregating the image features of different scales , to obtain multi-scale aggregated features as shown in the following formula (2): （1） （2） Step 2.2, The scale feature after passing through the high-low frequency feature enhancement module obtains the aggregated context information feature As shown in the following formula (3), the high-low frequency feature enhancement module includes a high frequency branch and a low frequency branch, which respectively extracts image information of different frequencies to perform feature enhancement: （3） Step 2.

3. The multi-scale aggregated features obtained from Step 2.1 and the aggregated context information features obtained from Step 2.2 The features obtained by concatenating the corresponding indices as shown in the following equation (4): （4） Step 2.

4. The features obtained in step 2.3 are input into a channel attention module As input to the channel attention module, the relationship between different feature channels is learned to aggregate features As shown in the following formula (5): （5） Step 2.5, aggregate the features obtained in step 2.

4. The information is decoupled by sending it into the decoupling head, where aggregated features are included. This will be converted into four branches, as shown in the following formula (6): Heatmap Offset , box parameters and orientation map All four transformation branches are implemented through two convolutional layers, with the first convolutional layer consisting of... Convolution operations, batch normalization, and activation functions ReLU The second convolutional layer consists of... The convolution operation constitutes the overall computation flow of the global information aggregation module as follows: For the input aggregated features... By employing depthwise separable large convolutional operations, an ultra-large receptive field can be acquired, thereby obtaining features with information gain. As shown in formula (7): （6） （7） The specific process of step 2.2 is as follows: Step 2.2.1, for the input feature map on the low-frequency branch by N 1 step of 2 The combination of convolution operation, batch normalization and activation function realizes the extraction of low-frequency features, and obtains low-frequency features ReLU , as shown in formula (8): ​ （8） Step 2.2.2: On high-frequency branches, the global context information path mainly uses large convolutional kernels to aggregate context information, obtaining context aggregation features. As shown in formula (9); the maximum pooling path passes through N The combination of two max pooling operations and 1×1 convolution to adjust the number of channels is used to extract high-frequency features of image information. HF As shown in formula (10); then, the high-frequency features and context aggregation features are concatenated along the channel dimension, and after adjusting the channels through a 1×1 convolution, the true high-frequency feature map is obtained; finally, the high-frequency features and low-frequency features are concatenated as the input for channel attention as shown in formula (11), and the output features of the aggregated context information are obtained after implementing channel importance weight filtering. As shown in formula (12): （9） （10） （11） （12） Step 4, the test set image processed in step 1 is put into the model trained in step 3, and finally the aerial remote sensing image after detecting the target is obtained.

2. The method of claim 1, wherein the method further comprises: The image pre-processing process in step 1 is as follows: the images are sequentially subjected to random flipping, random cropping and color transformation, then the images are reduced to one scale in [0.25, 0.5, 0.75] with a probability of 30% and filled to 608*608 with 0 pixels, and finally the pre-processed training set and test set images X are obtained.

3. The method of claim 2, wherein the method further comprises: The loss function used in step 3 is as follows: Smooth L1 loss function: Smooth L1 loss proposes to set the loss function as a piecewise function, let Loss be more robust to outliers, where, x For the numerical difference between the prediction box and the real box: （13） BCE loss function: BCELoss is a binary cross-entropy loss suitable for 0 / 1 distributed binary classification problems, where, y is the label , x is the probability of the predicted class of the sigmoid output, , the function learning goal is to make the output x approximate the label y : （14） Focal Loss loss function: when Focal Loss degenerates to the ordinary cross-entropy function; while when and the loss of Focal Loss is less than the standard cross-entropy loss, as follows: （15） In equation (15), when y = 1, when , ; y is the predicted label value; The total loss function is as follows: （16） In Equation (16), L h , L o , L b and L a denote the heat-map branch loss, the offset branch loss, the box parameter branch loss, and the orientation map branch loss, respectively.

Images