Remote sensing oriented object recognition method combining feature decoupling refinement and confusion minimization constraint

Abstract

The application provides a remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraint. A remote sensing directed target recognition network combining feature decoupling refinement and confusion minimization constraint is constructed, including a basic backbone network, a directed region proposal frame generation network, a regression branch, a coarse classification branch, a fine recognition branch, a confusion minimization loss and an interactive verification strategy. The fine recognition branch can decouple and recode target features, can extract discriminative representations of the target, and realizes more accurate target recognition. The fine recognition branch is trained by the confusion minimization loss, which can further improve the feature representation capability of the fine recognition branch. The interactive verification strategy can enhance the robustness of remote sensing directed target recognition inference. The application can solve the problem that the feature representation capability of the existing remote sensing directed target recognition method is limited, and realizes fine and accurate recognition of remote sensing image directed targets.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a remote sensing directional target recognition method with joint feature decoupling refinement and ambiguity minimization constraints, which can simultaneously and accurately locate remote sensing targets in various orientations and accurately identify the specific model or type of the target. Background Technology

[0002] Remote sensing directed target recognition is an important component of computer vision. With the increasing complexity of tasks and the growing demand for fine interpretation in remote sensing applications, coarse-grained interpretation methods, represented by remote sensing directed target detection, have gradually shown their limitations. They are unable to achieve more refined and accurate recognition of targets while locating targets with different orientations. Therefore, remote sensing directed target recognition has received widespread attention.

[0003] Current mainstream methods typically treat remote sensing directed target recognition as directed target detection, adjusting the output dimension of the classification branch of existing advanced directed detectors and retraining the improved detector on relevant datasets to achieve remote sensing directed target recognition. While these methods are direct and effective, their limitations in feature representation capabilities make it difficult to achieve precise and accurate target identification, especially when distinguishing between similar-looking targets. These mainstream remote sensing directed target recognition methods usually employ shared or independent feature representation. For shared feature representation, the recognition and localization branches use the same features. Although sharing features reduces the number of parameters, the recognition branch primarily focuses on discriminative features, while the regression branch emphasizes location-sensitive features. Using the same feature to simultaneously satisfy the needs of precise recognition and regression can easily lead to feature conflicts, hindering discriminative feature learning. For independent feature representation, the recognition and localization branches use separate features, effectively avoiding feature conflicts. However, since independent feature representation is usually implemented through simple fully convolutional networks, this method faces the challenge of feature misalignment with the target, making the target features susceptible to interference from the background or other targets, thus making high-precision target recognition difficult. Therefore, extracting discriminative feature representations is a key technology for breakthroughs in remote sensing oriented target recognition. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a remote sensing directed target recognition method that combines feature decoupling and refinement with confusion minimization constraints. A neural network model for directed target recognition in remote sensing images is constructed, including a basic backbone network, a directed region proposal generation network, a regression branch, a coarse classification branch, a fine recognition branch, a confusion minimization loss, and an interactive verification strategy. The directed target recognition head employs the fine recognition branch, enabling more accurate target recognition; training the fine recognition branch with confusion minimization loss improves feature representation capabilities; and the interactive verification strategy enhances the robustness of remote sensing directed target recognition inference. This invention addresses the limitations of existing remote sensing directed target recognition methods in feature representation capabilities, achieving precise and accurate recognition of directed targets in remote sensing images.

[0005] A remote sensing directed target recognition method with joint feature decoupling refinement and ambiguity minimization constraints, characterized by the following steps:

[0006] Step 1: Construct a remote sensing image dataset;

[0007] Step 2: Construct a remote sensing directed target recognition network with joint feature decoupling and refinement and confusion minimization constraints, including a basic backbone network, a directed region proposal box generation network, a regression branch, a coarse classification branch, a fine recognition branch, confusion minimization loss, and an interactive verification strategy;

[0008] The backbone network is used to extract features from the input image, and it employs a ResNet50 residual network with a feature pyramid structure.

[0009] The directed region proposal box generation network is used to generate directed candidate boxes for the target. It adopts a fully convolutional network based on the RPN network, takes the image features obtained by the backbone network as input, and after obtaining the directed candidate boxes of the target, it outputs the features corresponding to the target through a directed region proposal alignment operation.

[0010] The regression branch is used for the location regression of directed candidate boxes. It contains three fully connected layers, which take the target features obtained by the directed region proposal alignment operation as input and output the location information of the target.

[0011] The coarse classification branch is used to classify the target at a coarse level. It contains three fully connected layers and shares the first two fully connected layers with the regression branch. The coarse classification branch takes the target features obtained from the directed region proposal alignment operation as input and outputs the classification score of the target on K coarse categories, where the value of K is determined by the dataset.

[0012] The fine-grained recognition branch achieves fine-grained classification of targets. It consists of several convolutional layers and fully connected layers, and works in parallel with the coarse classification branch and the regression branch to output the classification score of the target in the subclass.

[0013] The aforementioned confusion minimization loss is used to train the fine recognition branch, and the calculation formula for the confusion minimization loss is as follows:

[0014]

[0015] Among them, L cm This represents the loss value minimized by confusion, where N represents the total number of samples in a mini-batch. pos and N neg C represents the number of positive and negative samples, respectively. i This represents the confidence distribution of the i-th positive sample across the Q+1 subclasses. Its corresponding real label is a binary vector; C j BCE(·) represents the prediction confidence of the j-th negative sample, where 0 represents the true label of the negative sample; BCE(·) represents the calculation of cross-entropy loss; w i This represents the weight factor of the i-th positive sample; Q is the total number of subclasses contained in the dataset, and the value of Q is determined by the dataset.

[0016] The weighting factor is calculated using the following formula:

[0017]

[0018] Where w represents the sample weighting factor, λ1 and λ2 are two hyperparameters with values ​​of 0.5 and 0.1 respectively; s represents the sample separability, determined by the following formula:

[0019]

[0020] Where C represents the score set of the sample on Q+1 subclasses, remove(C,α) represents the remaining score set after removing α from the score set, and α represents the confidence of the sample on the true subclass;

[0021] The interactive verification strategy described above will interactively verify the results output by the coarse classification branch and the fine recognition branch.

[0022] Step 3: Train the remote sensing directed target recognition network described in Step 2 using the image dataset from Step 1;

[0023] Step 4: Input the remote sensing image to be processed into the trained network model. The regression branch outputs the location information of the target, and the class confidence output by the coarse classification branch and the fine recognition branch after interactive verification is the directional target recognition result of the image.

[0024] Specifically, in step 1, the publicly available remote sensing directed target recognition datasets FAIR1M-1.0 and FAIR1M-2.0 are used to crop the original images in FAIR1M-1.0 and FAIR1M-2.0 to obtain image blocks of size 1024×1024. The sliding window size used for cropping is 1024×1024 with a step size of 824. The image blocks are then randomly flipped vertically and horizontally to obtain the constructed remote sensing image dataset.

[0025] Specifically, the ResNet50 residual network with a feature pyramid structure described in step 2 is initialized with weights by pre-training on the ImageNet dataset.

[0026] Specifically, the fully convolutional network based on the RPN network mentioned in step 2 includes one 3×3 convolutional layer and two parallel 1×1 convolutional layers. One 1×1 convolutional layer is used for front / background classification, and its output feature map has the same number of channels as the RPN network, which is 3. The other 1×1 convolutional layer is used for regression processing, and its output feature map has the number of channels changed from 12 in the RPN to 18.

[0027] Specifically, the detailed processing procedure for fine-grained branch identification described in step 2 is as follows:

[0028] Step a: Input the target feature X, and perform two independent 1×1 convolution operations to generate the feature map U and the spatial attention map A, respectively;

[0029] Step b: The spatial attention map A is normalized to the range [0,1] using the sigmoid activation function σ(·);

[0030] Step c: Multiply the feature map U and the normalized attention map A element-wise to generate the spatially weighted feature V;

[0031] Step d: Perform global average pooling on the spatially weighted features to extract the global channel descriptor V. gap , channel descriptor V gap Input the fully connected layer to generate channel modulation weights b; multiply the channel modulation weights b with the spatial weighted features V element-wise to generate features Y refined by double features; reduce the dimensionality of features Y to a 1024-dimensional vector z through the fully connected layer.

[0032] Step e: Using vector z as input, generate scores for Q+1 subclasses through a fully connected layer. Where c0 represents the background score, K is the number of coarse classes, and q k This indicates the number of subclasses contained in the k-th major category. Represents the q-th class under the k-th coarse class iThe scores of the subclasses, k = 1, ..., K, q i =1,...,q k Q = q1 + ... + q K ;

[0033] Select the subclass corresponding to the highest score as the target subclass and output it.

[0034] Specifically, the interactive verification strategy described in step 2 is as follows:

[0035] Compare the maximum confidence scores of coarse classification and fine classification. If the maximum confidence score of coarse classification is greater than that of fine classification, and the result of coarse classification corresponds to the background class, set the fine classification confidence scores of all non-background subclasses to 0, retaining only the confidence score of the background class, resulting in an updated subclass confidence vector M. If the maximum confidence score of coarse classification is greater than that of fine classification, and the result of coarse classification is not the background class, calculate the maximum value of the coarse classification confidence vector and find its corresponding coarse category index, denoted as i. Find the maximum value among all subclass fine classification confidence scores under coarse category i, retain only the subclass corresponding to the maximum value, and set the confidence scores of other subclasses to 0, resulting in an updated subclass confidence vector M. The coarse classification confidence score refers to the result obtained through the coarse classification branch; the fine classification confidence score refers to the result obtained through the fine recognition branch.

[0036] Specifically, in step 3, the loss function L of the network is set during training as follows:

[0037] L = L orpn +L cm +L reg +L cls (4)

[0038] Among them, L orpn For the loss of the directed region suggestion generation network, L cm To train the confusion minimization loss for the fine recognition branch, L reg For the loss of the regression branch, L cls The loss for the coarse classification branch;

[0039] Training was conducted end-to-end using the AdamW optimizer, with an initial learning rate of 0.0001, a batch size of 2, and other training hyperparameters using the default configurations in mmrotate.

[0040] The beneficial effects of this invention are as follows: Firstly, by designing a fine-grained recognition branch that operates in parallel with the localization branch, this branch functionally decouples the fine-grained classification features and regression features, and refines the target features in the spatial and channel dimensions, re-encoding them to generate more discriminative feature representations, thus achieving more accurate target recognition. Secondly, by designing a confusion-minimizing loss for training the fine-grained recognition branch, this loss dynamically adjusts the loss contribution of each sample during training based on the separability of each sample, further improving the discriminative power of the feature representations and better handling easily misidentified targets. Thirdly, by introducing an interactive verification strategy that combines the results of the fine-grained recognition branch and the coarse-grained classification branch, the robustness of remote sensing directed target recognition inference is enhanced by fusing the information from both. Finally, this invention addresses the limitations of existing remote sensing directed target recognition methods in feature representation capabilities, demonstrating significant advantages in remote sensing directed target recognition tasks. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the remote sensing directed target recognition network structure with joint feature decoupling refinement and confusion minimization constraints of the present invention;

[0042] Figure 2 The image is the result of remote sensing directional target recognition processing using the method of this invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0044] This invention provides a remote sensing directed target recognition method with joint feature decoupling refinement and ambiguity minimization constraints, the specific implementation process of which is as follows:

[0045] 1. Construct a remote sensing image dataset

[0046] This invention constructs a remote sensing image dataset for training a network model based on the publicly available remote sensing directed target recognition datasets FAIR1M-1.0 and FAIR1M-2.0. The bounding boxes are formatted as directed rectangles. The original images from FAIR1M-1.0 and FAIR1M-2.0 are cropped to obtain image patches of size 1024×1024. The cropping uses a sliding window of size 1024×1024 with a stride of 824. The image patches are then randomly flipped vertically and horizontally to reduce the risk of overfitting and improve the model's generalization ability.

[0047] 2. Construct a neural network model for directed target recognition in remote sensing images.

[0048] To obtain better classification and recognition results, this invention constructs a remote sensing directed target recognition network with joint feature decoupling and refinement and confusion minimization constraints, including a basic backbone network, a directed region proposal generation network, a regression branch, a coarse classification branch, a fine recognition branch, confusion minimization loss, and an interactive verification strategy. Figure 1 The processing procedure for this network model is given.

[0049] The backbone network is used to extract features from the input image. It employs a ResNet50 residual network with a Feature Pyramid Network (FPN) structure and is initialized with weights pre-trained on the ImageNet dataset.

[0050] The Directed Region Proposal (RPP) network generates directed candidate boxes for the target. It employs a fully convolutional network based on the RPN network, consisting of one 3×3 convolutional layer and two parallel 1×1 convolutional layers. One 1×1 convolutional layer is used for foreground / background classification, and its output feature map has the same 3 channels as the RPN network. The other 1×1 convolutional layer is used for regression processing, and its output feature map has 18 channels instead of 12 in the RPN. After obtaining the directed candidate boxes, a directed region proposal alignment operation is performed to obtain the features corresponding to the target (feature size 7×7×256), which are used in subsequent regression, coarse classification, and fine recognition branches.

[0051] The regression branch performs position regression on the generated directed candidate boxes. It takes the target features obtained through the directed region proposal alignment operation as input and outputs the target's position information (x, y, w, h, θ), where (x, y) represents the target's center coordinates, (w, h) represents the target's width and height, and θ represents the target's rotation angle. The regression branch contains three fully-connected layers (FC).

[0052] The coarse classification branch primarily performs coarse-grained classification of the target. It takes the target features obtained through directed region proposal alignment as input and outputs the target's classification score P = (p0, p1, ..., p) across K coarse categories. K Here, coarse categories mainly refer to general categories such as background (BG), airplanes, and ships. The coarse classification branch contains three fully connected layers (FC), and it shares the first two fully connected layers (FC) with the regression branch.

[0053] Fine-grained branch identification, confusion-minimized loss, and cross-validation are used to accurately determine the subclass of the target, such as Boeing 737 (B737), Boeing 747 (B747), other aircraft (OA), passenger ships (PS), and speedboats (MB).

[0054] The fine-grained identification branch, composed of a lightweight network, works in parallel with the coarse classification and regression branches, outputting the target's classification score in its subclass. A confusion-minimizing loss function is used to train the fine-grained identification branch, improving its classification ability on targets prone to misidentification. An interactive verification strategy establishes a connection between the coarse classification and fine-grained identification branches, enabling joint inference and ensuring the robustness of the inference process. These three parts are the core of this invention, as detailed below:

[0055] (1) The fine-grained recognition branch aims to extract discriminative features of the target through feature decoupling and refinement, thereby achieving high-precision recognition. This branch is a compact network composed of multiple convolutional layers and fully connected layers, capable of running efficiently in parallel with other branches. The specific processing procedure is as follows:

[0056] Step a: For target features First, its function is decoupled to avoid interference with regression features, and then its spatial features are refined. This involves applying two independent 1×1 convolution operations (Convolution, Conv) to feature X, generating feature maps respectively. Spatial attention map The refinement process for this feature is as follows:

[0057]

[0058] Where Φ1 and Φ2 are the parameter matrices of the convolution operation, Conv(·) represents the convolution operation, and attention map A is mainly used to encode the regions of interest in the target features.

[0059] Step b: To facilitate subsequent calculations, the spatial attention map A is normalized to the range [0,1] using the sigmoid activation function σ(·). This process can be represented as:

[0060] A=σ(A) (6)

[0061] Step c: Perform element-wise multiplication of the feature map U and the normalized attention map A to generate spatially weighted features. V can be represented as:

[0062]

[0063] in, This represents an element-wise multiplication operation. During element-wise multiplication, A is broadcast along the channel dimension. Through spatial feature refinement operations, the spatial attention map A guides the enhancement of key features and suppresses irrelevant features, thereby helping to extract more discriminative information.

[0064] Step d: Based on this, further refinement of channel features is performed to enhance the discriminative power of the features. First, global average pooling (GAP) is performed on V to extract global channel descriptors. Each element represents the global response value for a specific channel. V gap It can be represented as:

[0065] V gap =GAP(V) (8)

[0066] In the formula, GAP(·) represents the global average pooling operation, and its calculation formula is:

[0067]

[0068] H and W represent the height and width of the feature map, respectively, and c is the channel index. The channel descriptor V... gap Input to fully connected layer, generate channel modulation weights Used to adjust the response intensity of different channels. The formula for calculating b is as follows:

[0069] b = FC(V) gap ,ψ1) (10)

[0070] Where FC(·) represents the fully connected operation, and Ψ1 is the parameter matrix of the fully connected layer. After obtaining the channel modulation weight b, the channel responses of feature V are recalibrated, and the response intensity of each channel is adjusted by element-wise multiplication to generate features refined by dual features. The formula for calculating Y is as follows:

[0071]

[0072] When multiplying the channel modulation weights element-wise with the feature V, b is broadcast along the spatial dimension to meet the dimensionality requirements of matrix element-wise multiplication. Through channel refinement, the network can dynamically adjust the feature representation based on the global response of each channel, further improving the discriminative ability of the features.

[0073] After obtaining the feature Y, it is reduced to a 1024-dimensional vector z using a fully connected layer to reduce parameters and adapt to subsequent fine-grained classification tasks. Then, using the low-dimensional feature vector as input, a fully connected layer generates scores for Q+1 subclasses. (For the FAIR1M-1.0 and FAIR1M-2.0 datasets, Q = 37). c0 represents the background score, K is the number of coarse classes (for the FAIR1M-1.0 and FAIR1M-2.0 datasets, K = 5), q k This represents the number of subclasses contained in the k-th major category (k = 1, ..., K), where Q = q1 + ... + q K , Represents the q-th class under the k-th coarse class i Subclass score (q) i =1,...,q k ).

[0074] Based on the scores of the target belonging to different subclasses output by the fine-grained identification branch, the scores are directly compared, the subclass corresponding to the highest score is selected as the target's subclass, and the result is output, thus completing the fine-grained classification.

[0075] (2) Confusion Minimized Loss (CML). By introducing an adaptive weight adjustment mechanism based on misclassified samples, the model can dynamically adjust its loss weights according to the separability of the samples, focusing on strengthening the learning of difficult-to-classify samples.

[0076] Specifically, given a candidate bounding box, its score set across Q+1 subclasses is C. Based on these confidence scores, the separability s∈[-1,1] of a sample is defined to measure the probability that the sample is correctly classified. The larger the value of s, the stronger the separability of the sample and the lower the probability of misclassification. Conversely, the smaller the value of s, the lower the discriminatory power of the sample between categories and the higher the probability of misidentification.

[0077] When calculating the separability *s* of a sample, we first need to obtain the confidence score α = C[g] of the sample in the true subclass, where g represents the index of the true subclass corresponding to the sample. α reflects the model's confidence that the sample belongs to the true class. However, considering α alone is insufficient to comprehensively measure the separability of a sample, because the distribution of confidence scores for other subclasses also affects the accuracy of classification. When α is the maximum value in C, it indicates that the model has the highest confidence in the true class of the sample, and the classification result is relatively reliable. In this case, the separability *s* is defined as the difference between α and the second-highest confidence score:

[0078]

[0079] Here, remove(C,α) represents the set of remaining confidence values ​​after removing α from the confidence vector C. By this definition, the s-value reflects the difference in confidence between the sample and other classes. The larger the difference, the stronger the model's confidence in classifying the sample, and the lower the risk of misclassification.

[0080] If α is not the maximum value in C, it indicates that the model has high confidence in other subclasses, and there is a significant risk of misclassification. In this case, s is defined as the difference between α and the maximum confidence value:

[0081]

[0082] By this definition, 's' not only measures the confidence of a sample in its true class but also reflects the degree of confusion the model presents with other subclasses. A larger 's' indicates greater confidence in the model's classification of the sample, and the sample is further from the classification decision boundary; conversely, a smaller 's', especially when 's' is negative, indicates higher classification uncertainty, and the model may misclassify the sample. Based on the separability 's' of the samples, the contribution of each sample to the loss can be dynamically adjusted. To achieve this, a weighting factor 'w' is introduced, which adjusts the importance of each sample during training based on its separability. The formula for calculating 'w' is as follows:

[0083]

[0084] Here, λ1 and λ2 are two hyperparameters used to control the magnitude and rate of change of w within different intervals. As can be seen from the formula, w is negatively correlated with s, meaning that as the separability s of a sample increases, the weight w decreases. This implies that for samples with high separability (larger s), i.e., easy samples, their corresponding weights are smaller. Combined with the weight factor w, the contribution of each sample in the loss function is dynamically adjusted, thus forming the confusion-minimizing loss function L. cm Its expression is:

[0085]

[0086] In the formula, N represents the total number of samples in a mini-batch. pos and N neg These represent the number of positive and negative samples, respectively. Confidence level C i This represents the confidence distribution of the i-th positive sample across the Q+1 subclasses. For the corresponding true label, it is a binary vector. For negative samples, C j This represents the prediction confidence of the j-th negative sample, where 0 represents the true label of the negative sample. BCE(·) in the loss function represents the cross-entropy loss.

[0087] By introducing separability (s) and a weight factor (w), the model can pay more attention to difficult samples with lower separability during training. Since these samples have confidence levels close to the classification boundary and are prone to misclassification, they are given higher weights in the loss function, prompting the model to learn their features more deeply and improving overall classification performance. During inference, the model can better handle these difficult-to-distinguish targets, thereby enhancing its generalization ability.

[0088] (3) Interactive Validation Strategy. The results of the coarse classification branch and the fine recognition branch are interactively validated to improve the robustness of the inference. This method mainly uses the high-confidence coarse classification results to assist the inference of the fine recognition, ensuring the reliability of the recognition results. The detailed process of interactive validation is shown below:

[0089] During interactive validation, the first step is to compare the maximum confidence scores of coarse and fine classification to determine the reliability of the coarse classification result. If the maximum confidence score of coarse classification, max(P), is greater than that of fine classification, max(C), it indicates that the model has higher confidence in coarse classification, while fine classification may fail to provide a clear judgment due to complexity of details or confusion between samples. In this case, using the coarse classification result to assist in correcting the fine classification result can improve the robustness of inference. Next, it is necessary to determine whether the coarse classification result belongs to the background class. If the coarse classification result corresponds to the background class (i.e., max(P) = p0, where p0 represents the confidence score of the background class), it means that the model is more inclined to classify the target as background. At this time, the confidence scores of all non-background subclasses in fine classification are set to 0, and only the confidence score of the background class is retained. The updated subclass confidence vector is M = [c0, 0, ..., 0]. This process ensures that when the target is unclear, it can effectively avoid classifying the target into other subclasses, thereby reducing the possibility of misclassification.

[0090] Conversely, if the maximum coarse classification confidence score is greater than the fine classification confidence score, and it corresponds to a specific coarse category rather than the background class (i.e., max(P) > max(C) and max(P) ≠ p0), then the coarse classification result is relatively reliable and can provide clear category information for the target. Based on this, the confidence score of the fine classification can be further confirmed. The specific steps are as follows: First, calculate the maximum value max(P) of the coarse classification confidence vector P, and find the corresponding coarse category index i = Index(P, max(P)). Then, find the maximum value among the fine classification confidence scores of all subclasses under this coarse category i. This involves selecting the most likely subclass from the coarse category. Based on the confidence level of this subclass, the fine-grained classification confidence vector M is updated, retaining only the subclass with the highest confidence level within the coarse category and setting the confidence levels of other subclasses to 0. The specific update rules are as follows:

[0091]

[0092] In the formula, j represents the subclass index. After the update, only the subclass with the highest confidence under the coarse classification category i is retained, while the confidence of other subclasses is set to 0. The reason for this approach is that coarse classification provides relatively reliable prior information for fine classification, thereby making the fine classification results more focused and accurate, and reducing uncertainty in classification.

[0093] If the maximum confidence score (max(C)) of the fine-grained classification is greater than or equal to the maximum confidence score (max(P)) of the coarse-grained classification, then the fine-grained classification is considered to provide sufficiently reliable information, and corrections are no longer needed based on the coarse-grained classification results. In this case, the results of the fine-grained classification are used directly for inference because they capture finer-grained classification features. This interactive verification strategy improves the local inference of the fine-grained classification by leveraging the global information from the coarse-grained classification, ensuring effective fusion of classification information at different levels. This allows the fine-grained classification to no longer rely solely on its own results, thereby enhancing the robustness of the model's inference.

[0094] 3. Network model training

[0095] The images in the image dataset constructed in step 1 are input into the network constructed in step 2 for training. The network is trained in an end-to-end manner to obtain a trained network.

[0096] The loss function L of the network is set as follows during training:

[0097] L = L orpn +L cm +L reg +L cls (17)

[0098] Among them, L orpn For the loss of the directed region suggestion generation network, L cm To train the confusion minimization loss for the fine recognition branch, L reg For the loss of the regression branch, L cls The loss term is for the coarse classification branch; these loss terms work together to optimize the model. Hyperparameters λ1 and λ2 are set to 0.5 and 0.10, respectively. The optimizer used for model training is AdamW, with an initial learning rate of 0.0001 and a batch size of 2. All other training hyperparameters, unless otherwise specified, use the default configurations in mmrotate.

[0099] 4. Directed target recognition from remote sensing images

[0100] The remote sensing image to be processed is input into the trained network model. The regression branch outputs the target's location information, and the class confidence output by the coarse classification branch and the fine recognition branch after interactive verification is the directional target recognition result of the image.

[0101] To verify the effectiveness of the method of the present invention, simulation experiments were conducted in the following operating environment: a 10-card-powered GPU server (CPU is Intel Xeon E5-2698, GPU is 12G RTX 2080Ti), the server operating system is Ubuntu 16.04.5LTS, and the experimental code is developed based on the open-source detection toolkit mmrotate.

[0102] Partial recognition results of the method of this invention on the remote sensing directed target recognition dataset FAIR1M-1.0 are shown below. Figure 2 The first row of images shows the identification results for different types of aircraft targets, the second row shows the identification results for different types of ships, and the third row shows the identification results for different types of vehicles. The experimental results demonstrate that the method of this invention can accurately locate and identify different types of targets even in challenging scenarios.

[0103] The effectiveness of the method of this invention is quantitatively evaluated using mAP (mean Average Precision), which is defined as follows:

[0104]

[0105] Where C represents the total number of categories in the training dataset, c represents the target category index, and AP c This represents the recognition accuracy for different categories.

[0106] The method of this invention was compared with the advanced single-stage directed target recognition method S2ANet and the two-stage directed target recognition algorithm based on Oriented R-CNN. The backbone network used in the comparison was ResNet50. The comparison results are shown in Table 1. It can be seen that the present invention can achieve higher directed target recognition accuracy.

[0107] Table 1

[0108] method mAP S2ANet 34.71％ Oriented R-CNN 38.85％ Method of the present invention 40.87％

Claims

1. A method for remote sensing directed object recognition combining feature decoupled refinement with obfuscated minimization constraints, characterized in that The steps are as follows: Step 1: Constructing a remote sensing image dataset; Step 2: Constructing a remote sensing oriented target recognition network with joint feature decoupling, refined and confusion minimization constraints, including a basic backbone network, an oriented region proposal generation network, a regression branch, a coarse classification branch, a fine recognition branch, a confusion minimization loss, and an interactive validation strategy; The backbone network is used to extract features of an input image, and a ResNet50 residual network with a feature pyramid structure is adopted; The oriented region proposal generation network is used to generate oriented candidate boxes of a target, and a full convolutional network based on an RPN network is adopted, with image features obtained by the backbone network as input, and features corresponding to the target being output after oriented region proposal alignment operation on the oriented candidate boxes of the target; The regression branch is used for position regression of the oriented candidate boxes, and contains three fully connected layers, with target features obtained by the oriented region proposal alignment operation as input, and position information of the target being output; The coarse classification branch is used for classification of the target at a coarse-grained level, and contains three fully connected layers, with the first two fully connected layers being shared with the regression branch, with target features obtained by the oriented region proposal alignment operation as input, and classification scores of the target in K coarse categories being output, wherein the value of K is determined by the dataset; The fine recognition branch is used to realize fine classification of the target, and is composed of a plurality of convolutional layers and fully connected layers, and works in parallel with the coarse classification branch and the regression branch, and outputs classification scores of the target in subcategories; The confusion minimization loss is used to train the fine recognition branch, and a calculation formula of the confusion minimization loss is as follows: where L cm denotes the confusion-minimized loss value, N denotes the total number of samples in one mini-batch, N pos and N neg denote the number of positive and negative samples, respectively, C i denotes the confidence distribution of the i-th positive sample over Q+1 sub-classes, is its corresponding true label, which is a binary vector; C j denotes the predicted confidence of the j-th negative sample, 0 denotes the true label of the negative sample; BCE(·) denotes the calculation of cross-entropy loss; w i denotes the weight factor of the i-th positive sample; Q is the total number of sub-classes contained in the data set, the value of Q is determined by the data set; The weight factor is calculated as follows: wherein w represents the weight factor of a sample, λ1 and λ2 are two hyperparameters, and the values of λ1 and λ2 are 0.5 and 0.1 respectively; s represents sample separability, and is determined as follows: wherein C represents a score set of the sample in Q+1 subcategories, remove(C, α) represents a remaining score set after α is removed from the score set, and α represents confidence of the sample in a real subcategory; The interactive validation strategy performs interactive validation on results output by the coarse classification branch and the fine recognition branch; Step 3: Training the remote sensing oriented target recognition network according to the image dataset in step 1; Step 4: Inputting a remote sensing image to be processed into the trained network model, wherein position information of a target is output by the regression branch, and a class confidence output by the coarse classification branch and the fine recognition branch after interactive validation is the oriented target recognition result of the image.

2. The method of claim 1, wherein the method is characterized by: In step 1, the public remote sensing oriented target recognition datasets FAIR1M-1.0 and FAIR1M-2.0 are adopted, original images in the FAIR1M-1.0 and FAIR1M-2.0 are cut to obtain image blocks with a size of 1024x1024, a sliding window size of 1024x1024 and a step size of 824 are used for cutting, and random vertical and horizontal flipping operations are performed on the image blocks to obtain the constructed remote sensing image dataset.

3. The method of claim 1, wherein the method is characterized by: The ResNet50 residual network with a feature pyramid structure described in step 2 is initialized by pre-training its weights on the ImageNet dataset.

4. The method of claim 1, wherein the method is characterized by: The RPN network-based fully convolutional network described in step 2 includes 1 3x3 convolutional layer and 2 parallel 1x1 convolutional layers, one of which is used for foreground / background classification, and the other is used for regression processing. The channel number of the feature map output by the former is consistent with that of the RPN network, both of which are 3. The channel number of the feature map output by the latter changes from 12 to 18.

5. The method of claim 1, wherein the method is characterized by: The specific processing process of the fine recognition branch described in step 2 is as follows: Step a: input the target feature X, and generate feature map U and spatial attention map A through two independent 1x1 convolution operations; Step b: normalize the spatial attention map A to the range [0, 1] through the sigmoid activation function σ(·); Step c: element-wise multiply the feature map U and the normalized attention map A to generate the spatial weighted feature V; Step d: a global average pooling operation is performed on the spatially weighted features to extract a global channel descriptor V gap The channel descriptor V gap An input full connection layer is used to generate a channel modulation weight b; the channel modulation weight b is multiplied element by element with the spatially weighted features V to generate double-feature-refined features Y; the features Y are reduced to a 1024-dimensional vector z by a full connection layer. Step e: Generate scores on Q+1 sub-classes by a fully connected layer with vector z as input where c0represents the score of the background, K is the number of coarse classes, q k represents the number of sub-classes included under the kth coarse class, represents the score of the q i th sub-class under the kth coarse class, k = 1,..., K, q i = 1,..., q k , Q = q1+ ··· + q K ; Select the sub-class corresponding to the highest score as the target sub-class and output.

6. The method of claim 1, wherein the method is a joint feature decoupled refinement and confusion minimization constrained remote sensing directed object recognition method. The interactive verification strategy described in step 2 is as follows: Compare the maximum confidence of coarse classification and fine classification. If the maximum confidence of coarse classification is greater than that of fine classification, and the result of coarse classification corresponds to the background class, set the fine classification confidence of all non-background sub-classes to 0, and only keep the confidence of the background class to obtain the updated sub-class confidence vector M. If the maximum confidence of coarse classification is greater than that of fine classification, and the result of coarse classification is not a background class, calculate the maximum value of the coarse classification confidence vector and find its corresponding coarse class index, denoted as i. Find the maximum value among the fine classification confidences of all sub-classes under the coarse class i, and only keep the sub-class corresponding to the maximum value. Set the confidences of other sub-classes to 0 to obtain the updated sub-class confidence vector M.

7. The method of claim 1, wherein the method is a joint feature decoupled refinement and confusion minimization constrained remote sensing directed object recognition method. In step 3, the loss function L of the network is set as follows during training: L = L orpn + L cm + L reg + L cls (4) wherein L orpn is the loss of the directional region proposal generation network, cm is the confusion minimization loss used to train the fine recognition branch, reg is the loss of the regression branch, cls is the loss of the coarse classification branch. The training is performed in an end-to-end manner using the AdamW optimizer with an initial learning rate of 0.0001, a batch size of 2, and other training hyperparameters using the default configuration in mmrotate.

Images