A remote sensing image target detection method

Abstract

The present application relates to a kind of remote sensing image target detection methods, belong to remote sensing image processing technical field.The present application is by constructing the target detection model comprising positive and negative feature guide module, global feature information complementary module, utilizes positive and negative feature guide module self-adaptively perceive the contribution degree of the spatial position and semantic information of useful feature, removes background noise information from the feature stream of model, to suppress the interference of noise feature in complex background and highlight refinement target feature;Utilize global feature information complementary module to compress deep abstract feature by pooling operation, improve the ability of model to resist feature displacement and rotation, and utilize hollow convolution to increase the receptive field of model under the condition that spatial resolution is unchanged, establish the spatial position relationship between features, improve the cognitive ability of model to long-distance dependent information;And through the strategy of double multi-scale feature fusion, improve the detection performance of different scale targets in the same image.

Description

TECHNICAL FIELD

[0001] The present application relates to a remote sensing image target detection method, belonging to the technical field of remote sensing image processing. BACKGROUND

[0002] Remote sensing technology is one of the most important ways of earth observation, which provides a broader perspective for human earth perception. Target detection based on remote sensing images has been applied in many fields. In recent years, with the promotion of deep learning technology, target detection methods have made important progress.

[0003] The target detection method based on deep learning technology mainly includes two types: two-stage series model based on region establishment and one-stage series model based on regression. The main difference between the two is that the former needs to generate candidate boxes that may contain objects in advance through heuristic methods, and then perform fine classification and regression on the candidate boxes. The latter directly samples densely on the feature map, samples regions of different scales and aspect ratios, and then directly classifies and regresses the extracted features. The representative algorithm of two-stage series includes R-CNN, Faster R-CNN, etc. This kind of algorithm needs to generate candidate boxes in advance and then perform accurate regression, which usually has high precision, but the calculation is complex and the running speed is slow. One-stage series model does not need to generate candidate boxes in advance, which saves a lot of computing resources and at the same time obtains the available accuracy. It is the commonly used model in the industry, and the classic algorithm includes YOLO series, SSD series, etc.

[0004] The general target detection algorithm based on natural images has achieved excellent performance and is widely used, but unlike images in natural scenes, remote sensing images have the characteristics of strong background information interference, large target scale variation, rich target appearance form, and many small target categories. These characteristics bring great challenges to the classification and regression of remote sensing image target detection. In view of the characteristics of remote sensing images, a large number of scholars have proposed corresponding solutions. However, the accuracy of these models on large remote sensing image data still has a lot of room for improvement.

[0005] The imaging field of view of remote sensing images is large, which leads to strong interference of background to the target detection task. To solve the problem of complex background, some algorithms have been proposed. For example, a relationship connection attention module is proposed, which obtains global information by stacking the features themselves and the relationship features between the features, so as to enhance the foreground information and weaken the background information, and make the features have better distinguishability. A target detection method based on time-frequency analysis is proposed, which makes the detector focus on the object area rather than the background area through a discrete wavelet multi-scale attention mechanism. Prior scene information and Bayesian rule are used to infer the relationship between scene and target, and various scene semantics are used as specific priors to improve the performance of target detection in remote sensing images. From the above methods, the current methods to solve the complex background of remote sensing images include: 1) using attention mechanism and other feature enhancement methods to improve the feature representation of the target, so as to indirectly weaken the background information, which is the current mainstream and has obvious effect on improving the accuracy, but this method needs to design corresponding modules and the calculation is relatively complex; 2) selectively eliminating background features or enhancing target features through the relationship between target and background. This method considers both target features and background features. However, a good strategy is needed to distinguish between beneficial background and interfering background, otherwise it will lead to confusion between target and background; 3) using prior information to artificially reduce the influence of complex background information on the performance of the detector. This method is simple and easy to use, but the improvement of accuracy is limited, and a large amount of manpower is needed to select background single data for pre-training, which increases the training cost.

[0006] The target scale difference in remote sensing images brings great challenges to the target detection regression task. In view of this difficulty, scholars have carried out extensive research from different angles. Lin et al. proposed an adaptive feature pyramid network to effectively solve the multi-scale and dense target detection, which contains a selective refinement module to realize selective refinement of different feature maps. Wu et al. proposed a feature refinement module to further refine the features and improve the feature discrimination at different scales by combining different branches to convolve multiple receptive fields for objects with sharp changes in scale and shape. Ma et al. proposed a segmentation aggregation (FSM) module to distribute objects of different sizes in a scene in multiple feature maps for subsequent detection, reducing feature confusion between multi-scale objects, and proposed an offset error correction (OER) module to correct the inconsistency of object spatial layout in multiple layers of feature maps. The above models have achieved advanced performance in current remote sensing image target detection, but still have the following shortcomings: 1) lack of more effective feature perception strategy for small-scale target objects; 2) unable to deeply mine and utilize the application of context information in target detection in the model; 3) the proposed modules are mostly for different scale targets on different images, and the consideration for the same scale targets of the same category in the same image is lacking. Therefore, the current remote sensing image target detection method has low detection accuracy and poor detection effect. SUMMARY

[0007] The purpose of the present application is to provide a remote sensing image target detection method to solve the problem of poor detection effect in current remote sensing image target detection.

[0008] The present application provides a remote sensing image target detection method to solve the above technical problems, which comprises the following steps:

[0009] 1) Construct a target detection model, wherein the target detection model comprises a backbone network, a positive and negative feature guidance module, a global feature information complementary module and a spatial decoupling prediction head;

[0010] The backbone network is used to generate N primary features of different scales through a deep convolutional network;

[0011] The positive and negative feature guidance module is used to process the obtained primary features of different scales as follows: generating a positive feature information stream and a negative feature information stream based on a dual attention module, and removing background noise information from the target feature according to the target feature and background noise information extracted by the global feature information complementary module, and fusing the removed information and the input feature;

[0012] The global feature information complementary module is used to extract beneficial target features from the positive feature information stream and perform multi-scale enhancement, and to explore background noise information from the negative feature information stream;

[0013] The spatial decoupling prediction head comprises N, and each prediction head is used for classifying and predicting the features output by the corresponding positive and negative feature guiding module.

[0014] 2) training the constructed target detection model;

[0015] 3) obtaining a remote sensing image to be detected, inputting the remote sensing image to the trained target detection model, and realizing target detection by the target detection model.

[0016] The target detection model comprises a positive and negative feature guiding module and a global feature information complementary module, the positive and negative feature guiding module adaptively perceives the contribution degree of the spatial position and semantic information of beneficial features, and background noise information is removed from the feature flow of the model, so as to suppress the interference of noise features in a complex background and highlight and refine target features; the global feature information complementary module extracts beneficial target features from the positive feature information flow and performs multi-scale enhancement, and excavates background noise information from the negative feature information flow, so as to improve the reliable noise information of the positive and negative feature guiding module and further improve the detection effect of the model.

[0017] Further, the global feature information complementary module comprises a target feature extraction block and a background feature extraction block, the target feature extraction block is used for performing same-level multi-scale feature information compression on the positive feature information flow by using maximum pooling, and using a hollow convolution to supplement the features lost in the maximum pooling process, and establishing the spatial position relationship between the global context features.

[0018] The background feature extraction block is used for extracting multi-scale global semantic information from the negative feature information flow by using average pooling, and using a hollow convolution to supplement the features lost in the average pooling process.

[0019] The global feature information complementary module of the present application compresses deep abstract features by a pooling operation, improves the ability of the model to resist feature displacement and rotation, and uses a hollow convolution to increase the receptive field of the model while keeping the spatial resolution unchanged, establishes the spatial position relationship between the features, and improves the cognitive ability of the model to long-distance dependent information.

[0020] Further, the target feature extraction block and the background feature extraction block each comprise a plurality of dilated convolution branches and pooling branches and a fusion module; input features are respectively input to each dilated convolution branch and pooling branch, each dilated convolution branch performs dilated convolution operation on the grouped features, the pooling branch performs pooling operation on the input features, and the pooling operation result is up-sampled and superimposed with the output of the corresponding dilated branch on the group to serve as the processing result of the branch; the fusion module is used to superimpose the processing results of each branch and fuse with the input features, and the fusion result is taken as the output feature of the target feature extraction block or the background feature extraction block, the output feature of the target feature extraction block is the target feature, and the output feature of the background feature extraction block is the background noise information.

[0021] The pooling branch in the target feature extraction block adopts maximum pooling operation, and the pooling branch in the background feature extraction block adopts average pooling operation; the dilated convolution in each dilated convolution branch of the target feature extraction block has different expansion coefficients; the dilated convolution in each dilated convolution branch of the background feature extraction block has different expansion coefficients from those of the target feature extraction block.

[0022] The present application adopts a target feature extraction block (OFEB) and a background feature extraction block (BFEB) which are completely consistent in structure, wherein the target feature extraction block extracts abstract features by maximum pooling, and the background feature extraction block compresses information by average pooling. The target feature extraction block extracts beneficial target features from positive feature information streams and performs multi-scale enhancement through multi-path dilated convolution processing with different expansion coefficients, and the background feature extraction block excavates background noise information which seriously interferes with the detection task from negative feature information streams.

[0023] Further, the target feature extraction block and the background feature extraction block are further provided with a CBR unit, each group is processed by the CBR unit before being input to the corresponding dilated convolution branch and pooling branch, and is processed by the CBR unit again after being processed by the dilated convolution branch and the pooling branch, so as to eliminate linear influence.

[0024] The processing of the CBR unit includes channel dimension reduction processing of the input features, normalization processing by Batch Normalization, and activation processing by using an activation function.

[0025] Through the channel dimension reduction processing, normalization processing and activation processing of the CBR unit, the data processing amount is reduced, and the data processing efficiency is improved.

[0026] Further, the positive and negative feature guiding module comprises N, and each positive and negative feature guiding module adopts a double attention module comprising channel attention and spatial attention, the channel attention is used for generating an attention weight matrix according to the input feature, and the spatial attention is used for generating a normalized weight matrix according to the input feature;

[0027] The generation process of the positive feature information flow is that the attention weight matrix generated by the channel attention is multiplied with the input feature, and the result is the positive feature information flow;

[0028] The generation process of the negative feature information flow is that the generated attention weight matrix and the normalized weight matrix are respectively subtracted from 1 element by element, and the result is multiplied with the input feature, and the result is the negative feature information flow.

[0029] Further, the process that the positive and negative feature guiding module eliminates background noise information from the target feature is as follows: the background noise information output by the background feature extraction block is multiplied with a learnable parameter, and the multiplication result is subtracted from the target feature output by the target feature extraction block.

[0030] Further, the backbone network adopts CSPDarkNet53.

[0031] Further, the spatial decoupling prediction head adopts the Decoupled Head of YOLOX.

[0032] Further, the target detection model adopts a loss function comprising a positioning loss function and a classification loss function during training, the positioning loss function adopts a CIOU loss function, and the calculation formula is as follows:

[0033] ;

[0034] Wherein, represent the Euclidean distance of the center points of the predicted frame and the real frame, C represents the diagonal distance of the smallest closed region capable of containing the predicted frame and the real frame, is a parameter for balancing, for measuring the consistency of the length-width ratio of the predicted frame.

[0035] The application introduces an influence factor in the positioning loss function, thereby improving the regression stability of the target prediction frame.

[0036] Further, the classification loss function adopts a Focal Loss function, and the calculation formula is as follows:

[0037] ;

[0038] Wherein, represents the probability that the sample belongs to a positive sample, A regulation factor introduced on the basis of a balanced cross-entropy loss function, Contribute to the weight of the loss value of positive and negative samples.

[0039] The application uses Focal Loss to calculate classification loss, effectively improves the training inaccuracy problem caused by positive and negative sample imbalance, and reduces the influence of easy classification samples on gradient descent in the model training process. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 It is the structure diagram of the target detection model (MFICDet) constructed by the application;

[0041] Figure 2a It is the structure diagram of DarkNet53;

[0042] Figure 2b It is the structure diagram of CSPDarkNet53;

[0043] Figure 3 It is the function image of SPDarkNet53 using Mish activation function;

[0044] Figure 4 It is the structure diagram of the global context information complementary module constructed by the application;

[0045] Figure 5 It is the channel attention structure diagram used by the application;

[0046] Figure 6 It is the spatial attention structure diagram used by the application;

[0047] Figure 7 It is the structure diagram of the positive and negative feature guide module constructed by the application;

[0048] Figure 8 It is the structure diagram of the Decoupled Head prediction head used by the application;

[0049] Figure 9 It is an example of the DIOR data set used in the experiment;

[0050] Figure 10 It is an mAP diagram of the model in the DIOR data set in the experiment;

[0051] Figure 11a It is an average log error rate diagram of the Baseline model in the experiment;

[0052] Figure 11b It is an average log error rate diagram of the Baseline+Dhead model in the experiment;

[0053] Figure 11c is the average log error rate diagram of the Baseline+GFIC+DHead model in the experiment;

[0054] Figure 11d is the average log error rate diagram of the MFICDet model in the experiment;

[0055] Figure 12a is the feature visualization result diagram of the typical target of DIOR Figure 1 ;

[0056] Figure 12b is the feature visualization result diagram two of the typical target of DIOR

[0057] Figure 13 is the feature visualization result diagram of the different scale targets of the same image

[0058] Figure 14a is the detection result diagram of the same target with large scale change in the experiment

[0059] Figure 14b is the detection result diagram of the inter-class similarity and intra-class difference in the experiment

[0060] Figure 14c is the detection result diagram under different imaging conditions in the experiment

[0061] Figure 14d is the detection result diagram of different seasons in the experiment DETAILED DESCRIPTION

[0062] The specific embodiments of the present application will be further described in combination with the accompanying drawings.

[0063] The present application constructs a target detection model comprising positive and negative feature guiding modules and a global feature information complementary module, the positive and negative feature guiding modules adaptively perceive the contribution degree of spatial position and semantic information of beneficial features, and background noise information is removed from the feature flow of the model, so as to suppress the interference of noise features in the complex background and highlight the refined target features; the global feature information complementary module compresses deep abstract features through pooling operation, improves the ability of the model to resist feature displacement and rotation, and uses a hollow convolution to increase the receptive field of the model while keeping the spatial resolution unchanged, establishes the spatial position relationship between features, and improves the cognitive ability of the model to long-distance dependent information.

[0064] Specifically, as Figure 1As shown, the target detection model (MFICDet) constructed by the present application includes a feature extraction backbone network, a positive and negative feature guidance module (PNFG), a global feature information complement module (GFIC), and a spatial decoupling prediction head (DHead).

[0065] Backbone network

[0066] The backbone network in the present embodiment adopts CSPDarkNet53, which is improved in two parts of activation function and network structure based on DarkNet53. First, CSPDarknet53 introduces the idea of CSPNet, which divides the input features in each Block into two branches on the basis of DarkNet53, one branch is used for dense calculation, and the other branch is used as a skip connection to splice with the branch after dense calculation. For example, the structure before introducing the idea of CSPNet is as shown in Figure 2a As shown, the structure after introducing the idea of CSPNet is as shown in Figure 2b As can be seen from the figure, on the basis of maintaining the original structure of ResNet, the input features are split into two parts in the channel dimension and input into Part1 and Part2. Part1 branch is used as a skip connection to splice with the Part2 branch after dense calculation in the channel dimension. Then the result is input into the PartialTransition layer to realize the difference of maximum gradient combination. As can be seen from the figure, the channel participating in dense calculation is reduced to half of the original, so there is no need for a bottleneck layer. The above design enables the backbone network to have more diverse gradient combinations to enhance the learning ability of the convolutional neural network, while reducing memory occupation and computation under the condition of constant FLOPS.

[0067] The activation function in CSPDarkNet53 is improved to Mish function with better high saliency level, and the Mish function image is as shown in Figure 3 As can be seen from Figure 3 , it has the characteristic of no positive boundary, so it effectively avoids gradient saturation. And this function is tolerant to negative values, allowing the model to have better gradient flow. At the same time, the Mish function also has infinite order continuity and smoothness, so it has better generalization ability and improves the precision of the model.

[0068] Global feature information complement module

[0069] In order to improve the model's perception of the global scene and enhance the expression of features, the application proposes a global feature information complementary module. This module is inspired by the pyramid pooling module in PSPNet, which effectively aggregates multi-scale context information from non-neighborhood, improving the model's ability to extract global features. The pyramid pooling module is implemented by the global average pooling method. Global average pooling can slow down the convergence speed of the model, and the operation of averaging the global feature map can lose a lot of detailed features, so it is not suitable for object detection tasks. The global feature information complementary module proposed in the application enhances the target detail feature information while considering the global semantic information, and extracts background noise features for downstream rejection.

[0070] The structure of the global feature information complementary module is shown in Figure 4 The target feature extraction block and the background feature extraction block are composed of a multi-path pooling branch and a dilated convolution branch, and the difference between the two is that the target feature extraction block uses maximum pooling to extract abstract features, while the background feature extraction block uses average pooling for information compression. Among them, the target feature extraction block is used to extract beneficial target features from the positive feature information stream and perform multi-scale enhancement, and the background feature extraction block is used to explore background noise information that seriously interferes with the detection task from the negative feature information stream. For this embodiment, as shown in Figure 4 The target feature extraction block and the background feature extraction block each include a multi-path pooling branch and a dilated convolution branch, each including a 4-path pooling branch and a dilated convolution branch. The dilation coefficients of the dilated convolution in each dilated convolution branch are different to obtain four different scales of features, which together with the multi-scale features obtained in the backbone network constitute double multi-scale features.

[0071] The background feature extraction block is used in the negative feature information stream, and the purpose is to improve the sensitivity of the negative feature information stream to background noise information. Specifically, first, use adaptive average pooling to extract multi-scale global semantic information of the feature map; average pooling can maintain the local smoothness of feature information, and can also improve the problem of increased variance of estimated values caused by limited field size, retaining more image information, and is more suitable for extracting background information. Average pooling has the disadvantage of confusing detailed features, which can cause the semantic information of the deep features after pooling to become blurred in the upsampling process. Therefore, a dilated convolution is introduced in the background feature extraction block to supplement the information. However, it should be noted that the dilation coefficient of the dilated convolution in the background feature extraction block is different from that in the target feature extraction block. The processing process is as follows Figure 4As shown, first through the CBR unit processing, output features into adaptive average pooling branch and up-sampling (UP) to get global semantic information; then with the context information extracted by the hollow convolution branch in the pixel space fusion to get complementary deep feature map; adaptive average pooling and hollow convolution respectively use different pool kernel size and inflation coefficient, so as to generate multi-scale deep feature map; then each scale feature map is processed through CBR and concatenated in the channel dimension; finally, the concatenated feature map is processed through CBR again and fused with the initial feature in the spatial dimension to obtain the output of the module. The processing of the CBR unit includes channel dimension reduction, normalization processing through Batch Normalization, and activation processing using the Relu function. The purpose of the CBR unit is to remove linearity, so that the network obtains a nonlinear property.

[0072] The target feature extraction block is applied to the positive feature information flow to extract the feature information of the target of interest and perform multi-scale semantic enhancement. This part first uses the maximum pooling method to compress the multi-scale feature information of the same level, increase the receptive field of the network, and the maximum pooling can resist the translation and rotation of the features, and can overcome the influence of spatial position transformation such as target displacement and rotation on detection. The maximum pooling effectively extracts the key features of the target of interest, and realizes the filtering of redundant noise in the feature information. The maximum pooling operation can reduce network information redundancy while realizing down-sampling, but it will also cause a large amount of detail information loss, resulting in excessive compression of features, so the hollow convolution is also introduced. The hollow convolution contributes a larger receptive field to each pixel output compared to the pooling operation without changing the image resolution. The input features lost in the pooling process are supplemented by using the hollow convolution to establish the spatial position relationship between the global context features. In addition, the hollow convolution with different inflation coefficients can capture multi-scale context information, further enhance the receptive field of multi-scale features, and improve the cognitive ability of the model to multi-scale information. The target feature extraction block is the same as the above process, only the adaptive average pooling is replaced by maximum pooling in the pooling branch, and the specific process is not described in detail.

[0073] Positive and negative feature guiding module

[0074] The complex background of remote sensing image can seriously affect the target classification and positioning, and it is particularly difficult to identify the target with strong similarity in geometric and radiation features. The spatial resolution of different sensors affects the size of the target imaging, so the remote sensing image with poor imaging quality brings challenges to the target detection task. In addition, the radiation resolution of different sensors also has great difference, resulting in the diversity of the detailed features of the same target in different images. In order to improve the influence of the above problems on the target detection of remote sensing image, the present application proposes a positive and negative feature guiding module (PNFG). The design of PNFG is inspired by the Focus Module used for refining prediction in PFNet, as shown in Figure 1 The backbone network in the embodiment generates three scale features, F1, F2 and F3, respectively, and the three positive and negative feature guiding modules corresponding to F1, F2 and F3, respectively, and the output signal of the upper positive and negative feature guiding module is fed back to the generation of the positive feature information flow of the lower positive and negative feature guiding module. PNFG mainly includes two steps: 1) generating positive and negative features by using spatial attention mechanism (SAM) and channel attention mechanism (CAM); 2) realizing positive and negative feature guidance by using parallel information flow.

[0075] 1) Positive and negative feature generation

[0076] In the present application, the positive and negative features are generated by spatial attention and channel attention, wherein the spatial attention is sensitive to the spatial position information of the target, and the channel attention pays more attention to the semantic information of the input image. Therefore, the combination of the two can effectively explore the key feature representation of the target and enhance the flow of beneficial information in the network.

[0077] The structure of the channel attention CAM is shown in Figure 5 For an input X with C channels, CAM first aggregates spatial information to generate a spatial context descriptor by using the maximum pooling and average pooling methods; then generates a channel attention map through a multi-layer perceptron (MLP); and finally fuses the attention map through element-wise addition and then passes it through a Sigmoid function. The formula is as follows:

[0078] (1)

[0079] Wherein represents the attention map output by the channel attention, represents the Sigmoid function, , MLP is a multi-layer perceptron, and the weights of the multi-layer perceptron to the maximum pooling and average pooling , are shared.

[0080] Spatial Attention (SAM) utilizes the spatial attention mechanism in PFNet, with the following structure: Figure 6 As shown, the input features are first processed through three 1×1 convolutional layers to generate three matrices Q, K, and V. Then, the transpose of Q is multiplied by K. To accelerate training convergence, the result of the matrix multiplication is normalized using softmax to obtain spatial attention. Figure X a The above process can be represented as:

[0081] (2)

[0082] In the formula, This represents the influence of position j on position i. This represents the i-th column of the matrix. Finally, matrix multiplication is performed between V and the transpose of the spatial attention map to obtain the output of the spatial attention. To balance the learning process, a learnable parameter is introduced into the above results. The input features are then fused with the output results via skip connections to obtain the final output of the spatial attention mechanism.

[0083] By complementing spatial attention and channel attention, the model's ability to mine deep semantic information and establish correlations between features along the channel dimension can be improved. Information beneficial to object detection tasks in the output features is significantly enhanced; this invention defines such features as positive features. Then, the absolute complement of the positive features is taken from the generated information space set as negative features to accurately identify background noise features. Specifically, the attention weight matrix generated by channel attention is multiplied by the input features, resulting in the positive feature information stream; the generated attention weight matrix and the normalized weight matrix are subtracted element-wise from 1, and the result is multiplied by the input features, resulting in the negative feature information stream.

[0084] 2) Positive and negative feature guidance

[0085] Spatial attention and channel attention are combined to gather spatial and semantic information for feature mapping, extracting positive and negative information of the target of interest. This information is then used to enable the model to effectively mine the spatial location and deep semantic features of the target and remove irrelevant background noise.

[0086] To achieve the above process, this invention proposes a positive and negative feature guidance strategy, such as... Figure 7 As shown, the positive and negative feature guidance strategy consists of parallel positive and negative feature information streams. Specifically, firstly, the attention weight matrix M output by CAM is... trcMultiply by the input feature X to produce a positive feature information stream. Normalize the attention map output by SAM (NM) to generate the weight matrix M. trs Then the attention weight matrix M output by CAM trc and the SAM-normalized weight matrix M trs Element-wise subtraction with 1 is performed, and the result is multiplied by the input to generate a negative feature information stream. To balance the learning process, learnable parameters are introduced at various stages of the above process. , The generated positive and negative feature information streams are input into the global feature information complementation module. Positive feature information guides the model to learn beneficial information strongly correlated with the target, while negative feature information is responsible for detecting background noise and removing useless features that interfere with the model. Finally, the negative feature information stream is removed from the positive feature information stream, and a learnable parameter is introduced. To prevent feature vanishing, the input features are fused with the above results and then processed by a BR (Breakthrough Processing) to obtain the output result. BR refers to removing the convolution in CBR and retaining only the normalization and activation function processing. Finally, the positional information generated by SAM is fed into the information stream to assist the model in locating the target.

[0087] By employing a positive and negative feature guidance strategy, the model adaptively determines which features are beneficial to the task and uses them for subsequent feature enhancement. For useless noise features with low contribution, the positive and negative feature guidance effectively discovers and removes them from the subsequent feature stream, increasing the proportion of effective information in the model's information flow. Therefore, the positive and negative feature guidance strategy can effectively enhance the target's own features while eliminating the negative impact of complex background information on the target detection task. This improves the model's regression accuracy and feature perception capability.

[0088] The above describes the structure and working process of a single positive and negative feature guidance module. For positive and negative feature guidance modules at different scales, there is also interaction between positive and negative feature guidance modules corresponding to two adjacent scales. The output information of the positive and negative feature guidance module corresponding to the higher scale is upsampled and input into the positive feature information stream of the positive and negative feature guidance module corresponding to the lower scale.

[0089] Spatial decoupling prediction head

[0090] like Figure 1 As shown, the spatial decoupling prediction head of this invention includes multiple heads, with one head corresponding to each scale of features. In this embodiment, the features extracted by the backbone network include three scales: F1, F2, and F3, thus corresponding to three prediction heads. This invention uses the Decoupled Head proposed in YOLOX as the prediction head, which not only improves the detection accuracy of the model but also accelerates the model convergence speed. The structure of the Decoupled Head is as follows: Figure 8As shown, specifically, for each feature layer of input, the channel is first reduced to 256 dimensions, and then a 3x3 convolution layer is used in two parallel branches to respectively implement the classification and positioning sub-tasks of the model, and the IoU calculation branch will be implemented in parallel in the regression branch.

[0091] Training of the model

[0092] The above target detection model (referred to as MFICDet) constructed based on the present application is trained, and the training process needs to determine the corresponding loss function. For the MFICDet model constructed based on the present application, the loss in the training process includes a positioning loss and a classification loss.

[0093] The positioning loss uses CIOU as the loss function, and CIOU considers the overlapping area, center point distance, and aspect ratio of the predicted frame and the real frame, and further optimizes the regression accuracy and improves the regression stability of the target prediction frame. Specifically, the penalty term of CIOU introduces an influence factor on the basis of DIOU , and the formula of the penalty term is as follows:

[0094] (3)

[0095] Among them, respectively represent the Euclidean distance of the center points of the predicted frame and the real frame, and C represents the diagonal distance of the smallest closed region that can contain the predicted frame and the real frame. In the formula, is a parameter for balancing, is used to measure the consistency of the aspect ratio of the predicted frame, and the formulas are as follows:

[0096] (4)

[0097] (5)

[0098] Therefore, the overall calculation formula of the CIOU loss function is as follows:

[0099] (6)

[0100] The one-stage target detection method has a serious class imbalance problem in the training process, which will lead to low training efficiency because useless easy classification information dominates the gradient. And negative samples will guide the training process and cause the model to degenerate. Therefore, the present application uses Focal Loss to calculate the classification loss. The calculation formula of Focal Loss is as follows:

[0101] (7)

[0102] In the formula, represents the probability that the sample belongs to a positive sample, is a regulation factor introduced on the basis of balanced cross-entropy loss function. is used to control the contribution weight of positive and negative samples to the loss value. The Focal Loss loss function effectively improves the training inaccuracy problem caused by the imbalance of positive and negative samples, and reduces the influence of easy-to-classify samples on the gradient descent in the model training process.

[0103] Through the two types of loss functions and the training data, the MFICDet model of the present application can be trained. After the training is completed, the trained model can be used to detect the target of the remote sensing image to be detected. During detection, the remote sensing image to be detected is only needed to be input into the trained MFICDet model.

[0104] Experimental verification

[0105] In order to further illustrate the target detection method (MFICDet) of the present application, experiments are carried out on two widely used remote sensing image target detection benchmark data sets, DIOR and NWPUVHR-10. The NWPUVHR-10 data set contains 10 categories, a total of 650 labeled images, and the NWPUVHR-10 data set is divided into a training set, a validation set and a test set in a ratio of 6:2:2. DIOR is one of the largest and most common open source data sets in the field of remote sensing image target detection, which contains a total of 23463 images, covering 20 common categories such as airplanes, airports, baseball fields, basketball fields, bridges, chimneys, dams, highway service areas, highway toll stations, ports, ships, golf courses, track and field stadiums, overpasses, stadiums, oil storage tanks, tennis courts, train stations, vehicles and wind power generation. Among them, the training set contains 5862 images, the validation set contains 5863 images, and the remaining 11738 images are used as the test set. The schematic diagram of each category is as shown in Figure 9 .

[0106] 1) Determine the evaluation index

[0107] The present experiment uses four commonly used indexes to evaluate the precision of the present application. The four evaluation indexes are precision, recall, harmonic mean (F1) and average precision mean (mAP). The precision refers to the ratio of the number of correctly detected positive samples to the total number of predicted positive samples in the test data set. The recall reflects the probability that all detected results are correctly identified, which measures the false detection of the detector on the real target. The formula is as follows:

[0108] (8)

[0109] (9)

[0110] wherein, represents the number of correctly classified positive samples, represents the number of incorrectly classified negative samples, represents the number of incorrectly classified positive samples.

[0111] The precision and recall are contradictory in practical use, that is, when the precision is high, the recall is usually low, and vice versa. Therefore, F1 is proposed to consider both indicators. The formula is as follows:

[0112] (10)

[0113] mAP is a commonly used comprehensive index to reflect the performance of the model in target detection, which represents the average of all class AP (Average Precision). The AP value of each class is calculated by the area under the PR curve composed of Precision and Recall. The formula is as follows:

[0114]

[0115] (11)

[0116] In this experiment, the log-average miss rate is used to evaluate the influence of different module detectors on the target detection ability. This index is calculated by the curve of False Positive Per Image (FPPI) as the horizontal coordinate and the logarithm of the miss rate as the vertical coordinate. Specifically, 9 FPPI ratios in the range of [0.01, 1] are uniformly selected in the logarithmic space, the corresponding logarithm of the miss rate is obtained and the average value is calculated, and finally the exponential operation is performed to restore the percentage form as the evaluation index.

[0117] 2) Set the training parameters

[0118] The experiment was completed under the PyTorch architecture, and the hardware environment for the training and testing stages was NVIDIA GeForce RTX 3090. During the model training stage, the Mosaic data augmentation strategy was used to improve data diversity, and the Label-Smoothing regularization method was used to effectively suppress model overfitting. The pre-weights of YOLOv4 were used to initialize the parameters of the backbone network of the model. The Adam optimizer combines the adaptive learning rate gradient descent algorithm with the momentum gradient descent algorithm, making it suitable for sparse gradients and effectively improving the gradient shock problem. Therefore, the Adam optimizer was used, with a weight decay of 1x10 -5 . The initial learning rate was set to 0.001, and the learning rate decay strategy used a combination of cosine annealing and equal interval methods.

[0119] 3) Ablation experiment

[0120] Detailed ablation experiments were conducted on the DIOR and NWPU VHR-10 datasets to evaluate the effectiveness of different modules. The ablation experiment in this paper uses CSPDarknet53 as the backbone network, uses the same pre-trained model for initialization of the backbone network, and sets the initial value of other parameters by normal distribution fitting. This paper introduces FPN as the Baseline based on the CSPDarknet53 backbone network, and compares and analyzes the effectiveness of each module by controlling the variable method.

[0121] a. Ablation experiment on the DIOR dataset

[0122] The influence of the positive and negative feature guidance module: As shown in Table 1, the overall precision of the model improved by 1.21%mAP through the positive and negative feature guidance strategy. This proves that positive and negative feature guidance can effectively promote the network to learn the features of the target of interest and weaken the negative impact of background noise information on the model. Through spatial attention mechanism and channel attention mechanism, target location and semantic content can be extracted from feature information flow. Figure 10 The AP values of each class on different models are intuitively displayed. From Figure 10 It is obvious that it has obvious improvement on the precision of targets with high feature similarity or serious background interference, such as bridges and dams, bridges and overpasses, highway toll stations and service areas, baseball fields, and airplanes. This proves that the positive and negative feature guidance module enables the network to effectively distinguish the target feature information and background noise information in the feature response area, suppresses the interference of background noise information, and adaptively locates the spatial position of the target.

[0123] The influence of the global information complementary module: The global information complementary module deeply fuses the global multi-scale feature information, increasing the model mAP from 66.77% to 70.87% and increasing the mF1 index by 4.25%. For the classification task, the model focuses on whether a certain feature appears or not, rather than the specific position of its appearance. The pooling operation desensitizes the network to the position of the target, obtaining strong priori. The dilated convolution can compensate for the loss of internal data structure and spatial hierarchical information in the upsampling process, while maintaining a relatively low parameter amount, obtaining a larger receptive field, and supplementing and enhancing the detailed feature information of the network. As can be seen from Figure 10 , the module has a relatively large improvement for targets with complex geometric and radiation features, such as stadiums and athletic fields. The internal features of the two are almost identical, only the edge features are different, and the sizes of the two on different images in the dataset vary greatly. Therefore, the larger receptive field of the global information complementary module and its advantage in dealing with different scales of the same target are fully utilized. For example, the improvement in the accuracy of targets such as highway toll stations and dams proves that the global feature information complementary module has strong detail capturing capability. Moreover, due to the supplement of detailed information, the model has better anti-interference performance for similar inter-class and diverse intra-class target objects.

[0124] The influence of the spatial decoupling detection head: In order to verify the effect of decoupling the two different tasks of classification and positioning, the spatial decoupling head and the shared head are compared in the experiment. As can be seen from Table 1, this module only increases the mAP by 0.6%, but the model obtains the highest precision after introducing the decoupling head on the baseline. Although the spatial decoupling head has a small contribution to the improvement of the model accuracy, it has a great influence on the convergence speed of the model. As shown in Figure 10 , after introducing the spatial decoupling head, the model has obvious performance improvement for targets with various and fuzzy boundaries such as airports, train stations and stadiums. After decoupling the classification and positioning tasks, the model has better feature alignment performance, so the boundary positioning accuracy of the target is improved.

[0125] Table 1

[0126]

[0127] According to the experimental results, the log-average miss rate index of each ablation experiment is calculated, and the results are shown in Figure 11a , Figure 11b , Figure 11c and Figure 11d . Among them, Figure 11dIt can be seen that the missed detection performance of the other 15 types of targets except for ships, train stations, oil tanks, and stadiums is improved after adding the positive and negative feature guidance module. This is because ships are closely related to the wharf and the appearance of the ship at the dock is smaller than the appearance of the wharf. Therefore, after introducing the positive and negative feature guidance strategy, the model excessively weakens the background information, which in turn leads to an increase in the missed detection rate of the ship. The appearance of the train station is similar to that of the building, and there is great similarity between the features of the target and the background, which leads to contradictions in the learning process of the model, thereby causing the missed detection of the train station. The targets such as oil tanks and stadiums are similar to the above analysis. Figure 11c It can be seen that for the global information complementary module, due to its strong deep feature mining capability, the network improves the discovery capability of all targets, proving the effectiveness of the information complementary strategy.

[0128] b. Ablation experiment on the NWPU VHR-10 dataset

[0129] In order to prove the generality and generalization of the model in the remote sensing image target detection task, the ablation experiment is repeated on the NWPU VHR-10 dataset, and the results are shown in Table 2. From the table, it can be found that after decoupling the classification and regression tasks in space, the mAP is improved by 1.03%. The GIC module and the PNFS strategy proposed in the present application increase the mAP to 95.35% and 96.41%, respectively. With the increase of the designed module, it can be found that the recall rate and the mF1 index of the model are stably improved, which shows that each module is still effective on different datasets. However, when Dhead replaces the detection head of the baseline, the precision rate decreases, which is because the decoupling of the classification and regression tasks leads to overfitting of the classification, resulting in an increase in the false detection rate. In general, the ablation experiment results on the NWPU VHR-10 are similar to the conclusions of DIOR.

[0130] Table 2

[0131]

[0132] 4) Comparison with other models

[0133] In order to verify the effectiveness and advancement of the target detection model proposed in the present application, experiments are performed on two commonly used remote sensing image datasets. The models widely used and recently proposed are selected as the control group. These models not only include the most commonly used general target detection methods, but also include models designed for remote sensing image target detection. At the same time, these methods include one-stage and two-stage as well as anchor-free three kinds of commonly used target detection architectures.

[0134] Comparison and analysis on the DIOR dataset

[0135] The detection model proposed in the application is tested in detail by using the most challenging DIOR dataset in remote sensing image target detection. Comparative experiments are carried out with the classic general target detection model and the latest remote sensing image target detection model, and the results are shown in Table 3.

[0136] Table 3

[0137]

[0138] As can be seen from Table 3, the Faster-RCNN model has low detection accuracy because the multi-scale feature fusion and other effective feature enhancement strategies are lacking, and the features of multi-scale targets cannot be effectively modeled. The commonly used one-stage detector YOLOv4 model has multi-scale feature fusion capability, but its feature enhancement method of stacking multiple convolution layers is only for the target's own features, so it cannot achieve advanced performance when facing remote sensing images with complex backgrounds, which also proves that there are still some deficiencies in directly applying general target detection models to remote sensing images. Compared with the advanced models recently proposed for the characteristics of remote sensing images (i.e. CF2PN, FENet, ASSD, CSFF), as can be seen from Table 3, because the positive and negative feature guiding modules and the global information complementary modules proposed in the application effectively solve the problems of complex background information and multi-scale target changes, the model proposed in the application achieves the highest detection accuracy. CF2PN only uses the loss function to solve the problem of complex background of remote sensing images, so the effect is not good. The model in the application leads the mAP by 4.83% than the CF2PN model. FENet uses attention mechanism to enhance target features and introduces context feature enhancement method, and achieves advanced performance in two-stage detectors. As can be seen from the table, the precision of the one-stage remote sensing image detector proposed in this paper is better than that of the two-stage detector.

[0139] As can be seen from the last four rows of Table 3, the anchor-free detector performs poorly when dealing with complex datasets like DIOR because it lacks more effective coupling semantic relationships and has problems such as semantic ambiguity. MSFC-Net achieves excellent performance in anchor-free detectors, but is still inferior to the latest anchor-based detectors in terms of precision due to the influence of the recall rate of anchor-free detectors.

[0140] Comparison and analysis on NWPU VHR-10 dataset

[0141] Generalization experiments were conducted on the NWPU VHR-10 dataset to verify the versatility of the proposed model in remote sensing image target detection tasks. Comparative experiments were also performed with the latest proposed remote sensing image target detectors, as shown in Table 4. Compared with the latest models, the proposed model achieved the highest detection accuracy. ABNet is an excellent detector based on Faster R-CNN, and the proposed model achieved a 2.2% higher mAP compared to ABNet. SMENet and MSGNet are the latest high-precision one-stage detectors proposed to address the problems of complex backgrounds and diverse target scales in remote sensing images. As shown in Table 4, the proposed model achieved higher accuracy than SMENet and MSGNet. MPFPNet is a weakly supervised detector for multi-scale targets, and the proposed model achieved a 1.84% higher mAP than MPFPNet. Furthermore, the proposed target detection model achieved the highest detection accuracy in all four categories. These results yielded similar conclusions to those on the DIOR dataset. These results demonstrate the effectiveness and versatility of the proposed model in multi-class remote sensing image target detection tasks.

[0142] Table 4

[0143]

[0144] 5) Visualization of test results

[0145] Specifically, for example, there is a significant difference in scale between chimneys and oil tanks, such as... Figure 12a and Figure 12b As shown, the model can accurately locate both. This result demonstrates that the model has good multi-scale detection capabilities. Airports typically have irregular aspect ratios, and the feature heatmap shows that the model in this paper can accurately identify the center location of the airport, verifying the model's adaptability to targets with irregular aspect ratios. Figure 12b The track and field stadium is located in an extremely complex environment with strong background interference, but the present invention still accurately recalled the data, demonstrating the advantages of the present invention in dealing with complex background information interference.

[0146] Remote sensing images typically have a large field of view, resulting in the presence of targets with significant scale differences within the same image. This experiment performs visualization analysis on such images, such as... Figure 13 As shown, this invention can accurately respond to each target in images containing targets of different sizes and types. Furthermore, it establishes a good coupling relationship between the semantic information of the scene and the targets, indicating that the model fully considers the contextual information of the region of interest.

[0147] The spatial resolution of remote sensing images varies significantly between different sensors, resulting in large scale differences for the same target and varying background interference information, posing a significant challenge to detectors. Figure 14a As shown, this invention achieves good detection results for targets with large scale variations, indicating that the detection model proposed in this invention has a strong generalization ability to target scale changes. Another challenge of remote sensing imagery is the strong inter-class interference and the significant differences in size and shape among targets of the same class, such as... Figure 14b As shown, the detection model proposed in this invention can effectively distinguish between bridges and dams, demonstrating that the invention can effectively extract salient details of targets, has a certain resistance to inter-class interference, and accurately identify chimneys with significant appearance differences, proving that the model accurately models the feature space of this type of target. Remote sensing imaging is affected by many factors such as sensor height, radiometric resolution, atmospheric conditions, imaging season, and ground object radiation, resulting in inconsistent image quality under different imaging conditions. Robustness to image quality is also an important indicator of network performance, such as... Figure 14c , Figure 14d As shown, this invention demonstrates strong robustness to image quality and the ability to resist interference from different imaging conditions.

[0148] First, addressing the issue of complex backgrounds in remote sensing images, this invention employs a positive and negative feature guidance module to extract beneficial features from coupled feature information and suppress ineffective noise. Then, a global feature information complementarity module addresses the problem of large object scale variations. This module combines pooling and dilated convolution. Pooling compresses and extracts abstract features, improving the model's resistance to object translation and rotation. Dilated convolution, by maintaining spatial resolution, compensates for the loss of detailed features during pooling while increasing the model's receptive field. Furthermore, to improve the low detection accuracy caused by multi-scale targets within the same image, this invention proposes a dual multi-scale feature fusion strategy. Extensive experiments were conducted on two large remote sensing image datasets, DIOR and NWPU VHR-10. The results show that this invention not only effectively overcomes the impact of complex backgrounds on performance but also solves the challenge posed by multi-scale targets, especially those within the same image. Compared to other models, this invention achieves superior detection performance.

Claims

1. A method for detecting a target in a remote sensing image, characterized in that, The target detection method comprises the following steps: 1) constructing a target detection model, wherein the target detection model comprises a backbone network, a positive-negative feature guiding module, a global feature information complementary module, and a spatial decoupling prediction head; the backbone network is used for generating N primary features of different scales through a deep convolutional network; the positive-negative feature guiding module is used for performing the following processing on the obtained primary features of different scales respectively: generating a positive feature information stream and a negative feature information stream based on a double attention module, and removing background noise information from the target feature according to target feature information and background noise information extracted by the global feature information complementary module, and fusing the removed information and the input feature; the global feature information complementary module is used for extracting beneficial target features from the positive feature information stream and performing multi-scale enhancement, and extracting background noise information from the negative feature information stream; the spatial decoupling prediction head comprises N prediction heads, and each prediction head is used for performing classification prediction on the feature output by the corresponding positive-negative feature guiding module; 2) training the constructed target detection model; 3) obtaining a remote sensing image to be detected, inputting the remote sensing image to the trained target detection model, and realizing target detection by the target detection model. 2.The method of claim 1, wherein, The global feature information complementary module comprises a target feature extraction block and a background feature extraction block, the target feature extraction block is used for performing same-level multi-scale feature information compression on the positive feature information stream by using maximum pooling, and supplementing the features lost in the maximum pooling process by using a hollow convolution, and establishing the spatial position relationship between the global context features; the background feature extraction block is used for extracting multi-scale global semantic information from the negative feature information stream by using average pooling, and supplementing the features lost in the average pooling process by using a hollow convolution. 3.The method of claim 2, wherein, The target feature extraction block and the background feature extraction block each comprise a multi-path hollow convolution branch, a pooling branch, and a fusion module; the input feature is input to the hollow convolution branch and the pooling branch of each path, the hollow convolution branch of each path performs a hollow convolution operation on the input feature, the pooling branch performs a pooling operation on the input feature, and the pooling operation result is up-sampled and superimposed with the output of the hollow branch on the corresponding group as the processing result of the path; the fusion module is used for superimposing the processing results of each path and fusing the input feature, and taking the fusion result as the output feature of the target feature extraction block or the background feature extraction block, the output feature of the target feature extraction block is the target feature, and the output feature of the background feature extraction block is the background noise information; the pooling branch in the target feature extraction block adopts a maximum pooling operation, the pooling branch in the background feature extraction block adopts an average pooling operation, the dilation coefficients of the hollow convolution in each hollow convolution branch in the target feature extraction block are different, and the dilation coefficients of the hollow convolution in each hollow convolution branch in the background feature extraction block are different from the dilation coefficients of the hollow convolution in each hollow convolution branch in the target feature extraction block. 4.The method of claim 1 or 2, wherein, The target feature extraction block and the background feature extraction block are also provided with a CBR unit, each of which is processed by the CBR unit before the input corresponding to the hole convolution branch and the pooling branch, and is processed by the CBR unit again after the hole convolution branch and the pooling branch are processed, so as to eliminate the linear effect. The processing of the CBR unit includes channel dimension reduction processing of the input feature, normalization processing through Batch Normalization, and activation processing by using an activation function.

5. The method of claim 1 or 2, wherein, The positive-negative feature guiding module includes N, and the dual attention module adopted by each positive-negative feature guiding module includes channel attention and spatial attention, the channel attention is used to generate an attention weight matrix according to the input feature, and the spatial attention is used to generate a normalization weight matrix according to the input feature. The generation process of the positive feature information flow is that the attention weight matrix generated by the channel attention is multiplied with the input feature, and the result is the positive feature information flow. The generation process of the negative feature information flow is that the generated attention weight matrix and the normalization weight matrix are respectively subtracted from 1 element by element, and the result is multiplied with the input feature, and the result is the negative feature information flow.

6. The remote sensing image target detection method of claim 2, wherein, The process of eliminating background noise information from the target feature by the positive-negative feature guiding module is as follows: the background noise information output by the background feature extraction block is multiplied with a learnable parameter, and the multiplication result is subtracted from the target feature output by the target feature extraction block.

7. The method of claim 1 or 2, wherein, The backbone network adopts CSPDarkNet53.

8. The method of claim 1 or 2, wherein, The spatial decoupling prediction head adopts the Decoupled Head of YOLOX. 9.The method of claim 8, wherein, The loss function adopted by the target detection model in training includes a positioning loss function and a classification loss function, the positioning loss function adopts a CIOU loss function, and the calculation formula is as follows: ； wherein respectively represent the Euclidean distance between the center points of the predicted and ground truth bounding boxes, C represents the diagonal distance of the minimum closed region that can contain both the predicted and ground truth bounding boxes, is a parameter for balancing, is used to measure the consistency of the aspect ratio of the predicted bounding boxes. 10.The method of claim 9, wherein, The classification loss function adopts a Focal Loss function, and the calculation formula is as follows: ； wherein represents a probability that the sample belongs to positive samples, is a regulation factor introduced on the basis of balanced cross-entropy loss function, is a weight used to control the contribution of positive and negative samples to the loss value.

Images