Feature decoupling refining and confusion minimization constraint combined remote sensing directed target identification method

Abstract

The invention provides a remote sensing directed target identification method combining feature decoupling refining and confusion minimization constraint. A remote sensing directed target recognition network combining feature decoupling refining and confusion minimization constraint is constructed, and the remote sensing directed target recognition network comprises a basic backbone network, a directed region suggestion box generation network, a regression branch, a coarse classification branch, a fine recognition branch, confusion minimization loss and an interactive verification strategy, target features can be decoupled and recoded by adopting a fine recognition branch, discriminative representation of a target can be extracted, and more accurate target recognition is realized; a fine recognition branch is trained by adopting confusion minimization loss, so that the feature characterization capability of the fine recognition branch can be further improved; and an interactive verification strategy is adopted, so that the robustness of remote sensing directed target recognition reasoning can be enhanced. The method can solve the problem that an existing remote sensing directed target recognition method is limited in feature representation capacity, and fine and accurate recognition of the remote sensing image directed target is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically relates to a remote sensing directional target recognition method combining feature decoupling refinement with confusion minimization constraints, which can simultaneously and precisely locate remote sensing targets in various directions and accurately identify the specific model or type of the target. Background Art

[0002] Remote sensing directed target recognition is an important part of computer vision. With the increasing complexity of tasks and the demand for refined interpretation in remote sensing applications, coarse-grained interpretation methods represented by remote sensing directed target detection have gradually shown their limitations. It is difficult to achieve more refined and accurate recognition of targets while locating targets in different directions. Therefore, remote sensing directed target recognition has received widespread attention.

[0003] At present, mainstream methods usually treat remote sensing directed target recognition as directed target detection. They achieve remote sensing directed target recognition by adjusting the output dimension of the classification branch of the existing advanced directed detector and retraining the improved detector on the relevant data set. Although such methods are direct and effective, they are difficult to achieve precise and accurate target recognition due to their limitations in feature representation capabilities, especially when distinguishing targets with similar appearances. These mainstream remote sensing directed target recognition usually adopt shared feature representation or independent feature representation. For shared feature representation, the recognition branch and the localization branch use the same features. Although shared features can reduce the number of parameters, since the recognition branch mainly focuses on the discriminative features of the target, while the regression branch focuses more on position-sensitive features, it is easy to cause feature conflicts and restrict discriminative feature learning if the same feature meets the needs of both fine recognition and regression. For independent feature representation, the recognition branch and the localization branch use independent features respectively, thereby effectively avoiding conflicts between features. However, since independent feature representation is usually implemented through a simple fully convolutional network, this representation method will face the problem of feature misalignment with the target, causing the target features to be easily interfered by the background or other targets, making it difficult to achieve high-precision target recognition. Therefore, how to extract discriminative feature representation is a key technology to break through remote sensing directional target recognition. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a remote sensing directed target recognition method that combines feature decoupling refinement with confusion minimization constraints. A neural network model for directed target recognition in remote sensing images is constructed, including a basic backbone network, a directed region proposal box generation network, a regression branch, a coarse classification branch, a fine recognition branch, a confusion minimization loss, and an interactive verification strategy. Among them, the directed target recognition head adopts a fine recognition branch to achieve more accurate target recognition; the fine recognition branch is trained using confusion minimization loss to improve feature characterization capabilities; and the interactive verification strategy is adopted to enhance the robustness of remote sensing directed target recognition reasoning. The present invention can solve the problem that the existing remote sensing directed target recognition methods are limited in feature characterization capabilities, and achieve fine and accurate recognition of directed targets in remote sensing images.

[0005] A remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraint, characterized by the following steps:

[0006] Step 1: Build a remote sensing image dataset;

[0007] Step 2: Construct a remote sensing directed target recognition network with joint feature decoupling refinement and confusion minimization constraints, including a basic backbone network, a directed region proposal box generation network, a regression branch, a coarse classification branch, a fine recognition branch, a confusion minimization loss, and an interactive verification strategy. The backbone network is used to extract features, the directed region proposal box generation network is used to generate directed candidate boxes of the target, the regression branch is used for position regression of the directed candidate box, the coarse classification branch is used to classify the target at a coarse-grained level, the fine recognition branch realizes fine classification of the target, the confusion minimization loss is used to train the fine recognition branch, and the interactive verification strategy interactively verifies the results output by the coarse classification branch and the fine recognition branch;

[0008] Step 3: Use the image data set in step 1 to train the network model in step 2;

[0009] Step 4: Input the remote sensing image to be processed into the trained network model, where the regression branch outputs the location information of the target, and the category confidence output by the coarse classification branch and the fine recognition branch after interactive verification is the directed target recognition result of the image.

[0010] Specifically, in step 1, the publicly available remote sensing directed target recognition datasets FAIR1M-1.0 and FAIR1M-2.0 are used, and the original images in FAIR1M-1.0 and FAIR1M-2.0 are cropped to obtain image blocks of size 1024×1024. The sliding window size used for cropping is 1024×1024, the step size is 824, and the image blocks are randomly flipped vertically and horizontally to obtain the constructed remote sensing image dataset.

[0011] Specifically, the backbone network described in step 2 adopts a ResNet50 residual network with a feature pyramid structure, and its weights are initialized by pre-training on the ImageNet dataset.

[0012] Specifically, the directed region proposal box generation network described in step 2 adopts a fully convolutional network based on the RPN network, including 1 3×3 convolutional layer and 2 parallel 1×1 convolutional layers, of which 1 1×1 convolutional layer is used for foreground / background classification, and the number of feature map channels of its output is consistent with that of the RPN network, both of which are 3. Another 1×1 convolutional layer is used for regression processing, and the number of feature map channels of its output is changed from 12 of RPN to 18; the directed region proposal box generation network takes the image features obtained by the backbone network as input, obtains the directed candidate box of the target, and then performs a directed region proposal alignment operation to output the features corresponding to the target.

[0013] Specifically, the regression branch described in step 2 contains 3 fully connected layers, which take the target features obtained by the directed region proposal alignment operation as input and output the location information of the target.

[0014] Specifically, the coarse classification branch described in step 2 contains 3 fully connected layers, and shares the first two fully connected layers with the regression branch. The coarse classification branch takes the target features obtained by the directed region proposal alignment operation as input and outputs the classification scores of the target on K coarse categories, where the value of K is determined by the dataset.

[0015] Specifically, the fine recognition branch described in step 2 is composed of several convolutional layers and fully connected layers, and its specific processing process is as follows:

[0016] Step a: Input the target feature X, and after two independent 1×1 convolution operations, generate the feature map U and the spatial attention map A respectively;

[0017] Step b: The spatial attention map A is normalized to the range [0, 1] using the sigmoid activation function σ(·);

[0018] Step c: Multiply the feature map U and the normalized attention map A element by element to generate the spatial weighted feature V;

[0019] Step d: Perform global average pooling on the spatial weighted features to extract the global channel descriptor V gap , the channel descriptor V gap Input the fully connected layer to generate the channel modulation weight b; multiply the channel modulation weight b and the spatial weighted feature V element by element to generate the feature Y after double feature refinement; the feature Y is reduced to a 1024-dimensional vector z through the fully connected layer;

[0020] Step e: Take vector z as input and generate scores for Q+1 subclasses through a fully connected layer Among them, c0 represents the background score, K is the number of coarse classes, and q k Indicates the number of subcategories contained in the kth category, Indicates the qth coarse class under the kth coarse class i The score of the subclass, k=1,...,K,q i =1,...,q k ,q1+···+q K =Q, the values ​​of Q and K are determined by the data set;

[0021] The subcategory corresponding to the highest score is selected as the subcategory of the target and output.

[0022] Specifically, the calculation formula for the confusion minimization loss described in step 2 is as follows:

[0023]

[0024] Among them, L cm represents the confusion minimization loss value, N represents the total number of samples in a small batch, N pos and N neg Represents the number of positive samples and negative samples, C i represents the confidence distribution of the i-th positive sample on Q+1 subclasses, The corresponding true label is a binary vector; C j represents the prediction confidence of the jth negative sample, 0 represents the true label of the negative sample; BCE(·) represents the calculation of cross entropy loss; w i Represents the weight factor of the i-th positive sample;

[0025] The weight factor is calculated as follows:

[0026]

[0027] Among them, w represents the weight factor of the sample, λ1 and λ2 are two hyperparameters with values ​​of 0.5 and 0.1; s represents the sample separability, which is determined by the following formula:

[0028]

[0029] Among them, C represents the score set of samples on Q+1 subclasses, and remove(C,α) represents the remaining score set after removing α from the score set.

[0030] Specifically, the interactive verification strategy described in step 2 is as follows:

[0031] Compare the maximum confidence of coarse classification and fine classification. If the maximum confidence of coarse classification is greater than the maximum confidence of fine classification, and the result of coarse classification corresponds to the background class, set the fine classification confidence of all non-background subclasses to 0, and only retain the confidence of the background class, and obtain the updated subclass confidence vector M; if the maximum confidence of coarse classification is greater than the maximum confidence of fine classification, and the result of coarse classification is not the background class, calculate the maximum value of the coarse classification confidence vector, and find its corresponding coarse category index, recorded as i, find the maximum value among all subclass fine classification confidences under the coarse category i, and only retain the subclass corresponding to the maximum value, set the confidence of other subclasses to 0, and obtain the updated subclass confidence vector M; the coarse classification confidence refers to the result obtained by the coarse classification branch; the fine classification confidence refers to the result obtained by the fine recognition branch.

[0032] Specifically, in step 3, the network loss function L is set as follows during training:

[0033] L=L orpn +L cm +L reg +L cls (4)

[0034] Among them, L orpn is the loss of the directed region proposal generation network, L cm is the confusion minimization loss for training the fine recognition branch, L reg is the loss of the regression branch, L cls is the loss of the coarse classification branch;

[0035] The training was performed end-to-end using the AdamW optimizer. The initial learning rate was set to 0.0001, the batch size was 2, and other training hyperparameters used the default configuration in mmrotate.

[0036] The beneficial effects of the present invention are as follows: since a fine recognition branch is designed to run in parallel with the positioning branch, the branch functionally decouples the fine classification features and the regression features, refines the target features in the spatial and channel dimensions, and re-encodes to generate feature representations with stronger discriminability, which can achieve more accurate target recognition; since a confusion minimization loss is designed for training the fine recognition branch, the loss dynamically adjusts the loss contribution of the sample in the training process according to the separability of each sample, which can further improve the discriminative ability of the feature representation and better handle targets that are easily misidentified; since an interactive verification strategy is introduced to combine the results of the fine recognition branch and the coarse-grained classification branch, the robustness of remote sensing directed target recognition reasoning can be enhanced by fusing the information of the two; the present invention can solve the problem that the existing remote sensing directed target recognition methods are limited in feature representation capabilities, and has significant advantages in remote sensing directed target recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the network structure of remote sensing directed target recognition with joint feature decoupling refinement and confusion minimization constraints of the present invention;

[0038] Figure 2 It is a result image of remote sensing directional target recognition processing using the method of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0040] The present invention provides a remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraints, and the specific implementation process is as follows:

[0041] 1. Build a remote sensing image dataset

[0042] The present invention constructs a remote sensing image dataset for network model training based on the public remote sensing directed target recognition datasets FAIR1M-1.0 and FAIR1M-2.0. The format of the annotation box is a directed rectangular box. The original images in FAIR1M-1.0 and FAIR1M-2.0 are cropped to obtain image blocks of size 1024×1024. The sliding window size used for cropping is 1024×1024, the step size is 824, and the image blocks are randomly flipped vertically and horizontally to reduce the risk of overfitting and improve the generalization ability of the model.

[0043] 2. Construct a neural network model for directed target recognition in remote sensing images

[0044] In order to obtain better classification and recognition results, the present invention constructs a remote sensing directed target recognition network with joint feature decoupling refinement and confusion minimization constraints, including a basic backbone network, a directed region proposal box generation network, a regression branch, a coarse classification branch, a fine recognition branch, a confusion minimization loss and an interactive verification strategy. Figure 1 The processing process of the network model is given.

[0045] Among them, the backbone network is used to extract the features of the output image. It adopts the ResNet50 residual network with a feature pyramid network (FPN) structure and is initialized with weights pre-trained on the ImageNet dataset.

[0046] The directed region proposal box generation network is used to generate directed candidate boxes of the target. It uses a fully convolutional network based on the RPN network, including 1 3×3 convolutional layer and 2 parallel 1×1 convolutional layers. One 1×1 convolutional layer is used for foreground / background classification, and the number of channels of its output feature map is consistent with that of the RPN network, both of which are 3. Another 1×1 convolutional layer is used for regression processing, and the number of channels of its output feature map is changed from 12 of RPN to 18. After obtaining the directed candidate box of the target, the directed region proposal alignment operation is performed to obtain the features corresponding to the target (the size of the features is 7×7×256), which are used for subsequent regression branches, coarse classification branches, and fine recognition branches.

[0047] The regression branch is used to regress the position of the generated oriented candidate boxes. It takes the target features obtained through the oriented region proposal alignment operation as input and outputs the target's position information (x, y, w, h, θ), where (x, y) represents the center point coordinates of the target, (w, h) represents the width and height of the target, and θ represents the rotation angle of the target. The regression branch contains 3 fully-connected layers (FC).

[0048] The coarse classification branch mainly realizes the coarse-grained classification of the target. It takes the target features obtained through the directed region proposal alignment operation as input and outputs the classification score P of the target on K coarse categories = (p0, p1, L, p K ), where the coarse categories mainly refer to general categories such as background (BG), airplanes, ships, etc. The coarse classification branch contains three fully connected layers FC, which shares the first two fully connected layers FC with the regression branch.

[0049] The refined identification branch, confusion minimization loss, and cross-validation are used to precisely determine the subclasses of the targets, such as Boeing 737 (B737), Boeing 747 (B747), other aircraft (OA), passenger ship (PS), and speedboat (MB), etc.

[0050] Among them, the fine recognition branch is composed of a lightweight network, which works in parallel with the coarse classification branch and the regression branch to output the classification score of the target on the subclass; the confusion minimization loss function is used to train the fine recognition branch to improve its classification ability on targets that are easily misidentified; the interactive verification strategy establishes a connection between the coarse classification branch and the fine recognition branch to perform joint reasoning to ensure the robustness of the reasoning process. These three parts are the core of the present invention, as follows:

[0051] (1) The fine recognition branch aims to extract the discriminative features of the target by means of feature decoupling and refinement to achieve high-precision recognition. This branch is a compact network consisting of multiple convolutional layers and fully connected layers, which can run efficiently in parallel with other branches. The specific processing process is as follows:

[0052] Step a: For the target feature First, decouple its function to avoid the impact on the regression features, and then refine its spatial features. That is, two independent 1×1 convolution operations (Convolution, Conv) are used for feature X to generate feature maps respectively. and spatial attention map The feature refinement process is as follows:

[0053]

[0054] Among them, Φ1 and Φ2 are the parameter matrices of the convolution operation, Conv(·) represents the convolution operation, and the attention map A is mainly used to encode the areas that need to be paid attention to in the target features.

[0055] Step b: To facilitate subsequent calculations, the spatial attention map A is normalized to the range [0, 1] using the sigmoid activation function σ(·). The process can be expressed as:

[0056] A=σ(A) (6)

[0057] Step c: Multiply the feature map U and the normalized attention map A element by element to generate spatial weighted features V can be expressed as:

[0058]

[0059] in, Represents an element-by-element multiplication operation. A is broadcast along the channel dimension during element-by-element multiplication. Through the spatial feature refinement operation, the spatial attention map A guides the enhancement of key features and suppresses irrelevant features, thereby helping to extract more discriminative information.

[0060] Step d: On this basis, further refine the channel features to enhance the discriminative ability of the features. First, perform a global average pooling (GAP) operation on V to extract the global channel descriptor Each element of V represents the global response value of a specific channel. gap It can be expressed as:

[0061] V gap =GAP(V) (8)

[0062] In the formula, GAP(·) represents the global average pooling operation, and its calculation formula is:

[0063]

[0064] H and W represent the height and width of the feature map, respectively, and c is the channel index. gapInput the fully connected layer to generate channel modulation weights Used to adjust the response strength of different channels. The calculation formula of b is as follows:

[0065] b=FC(V gap ,ψ1) (10)

[0066] Among them, FC(·) represents the fully connected operation, and Ψ1 is the parameter matrix of the fully connected layer. After obtaining the channel modulation weight b, the channel response of the feature V is recalibrated, and the response strength of each channel is adjusted by element-by-element multiplication to generate the feature after double feature refinement The calculation formula for Y is as follows:

[0067]

[0068] When the channel modulation weights are element-wise multiplied with the feature V, b is broadcasted along the spatial dimension to meet the dimensionality requirements of the matrix element-wise multiplication. Through channel refinement, the network can dynamically adjust the feature expression according to the global response of each channel, further improving the discriminative ability of the feature.

[0069] After obtaining the feature Y, it is reduced to a 1024-dimensional vector z through a fully connected layer to reduce the parameters and adapt to the subsequent fine classification tasks. Then, the low-dimensional feature vector is used as input to generate scores on Q+1 subclasses through a fully connected layer. (For the datasets FAIR1M-1.0 and FAIR1M-2.0, Q=37). c0 represents the score of the background, K is the number of coarse classes (for the datasets FAIR1M-1.0 and FAIR1M-2.0, K=5), q k represents the number of subcategories contained in the kth category (k=1,...,K), q1+···+q K =Q, Indicates the qth coarse class under the kth coarse class i The score of the subclass (q i =1,...,q k ).

[0070] According to the scores of the targets belonging to different subcategories output by the fine recognition branch, the scores are directly compared, the subcategory corresponding to the highest score is selected as the subcategory of the target, and the result is output to complete the fine classification.

[0071] (2) Confusion Minimized Loss (CML): By introducing an adaptive weight adjustment mechanism based on misclassified samples, the model can dynamically adjust its loss weight according to the separability of the samples, focusing on strengthening the learning of difficult-to-classify samples.

[0072] Specifically, given a candidate box, its score set on Q+1 subclasses is C. Based on these confidences, the separability of the sample s∈[-1,1] is defined to measure the possibility of the sample being correctly classified. The larger the value of s, the stronger the separability of the sample and the lower the possibility of being misclassified. On the contrary, the smaller the value of s, the lower the distinction between the samples and the greater the possibility of being misclassified.

[0073] When calculating the sample separability s, we first need to obtain the confidence α=C[g] of the sample on the true subclass, where g represents the true subclass index corresponding to the sample. α reflects the model's confidence that the sample belongs to the true category. However, considering α alone is not enough to fully measure the separability of the sample, because the confidence distribution of the sample on other subclasses also affects the accuracy of classification. When α is the maximum value in C, it indicates that the model has the highest confidence in the true category of the sample, and the classification result is more reliable. In this case, the separability s is defined as the difference between α and the second highest confidence:

[0074]

[0075] Among them, remove(C,α) represents the remaining confidence set after removing α from the confidence vector C. Through this definition, the s value can reflect the confidence difference between the sample and other categories. The larger the difference, the stronger the model's classification confidence in the sample, and the lower the risk of misclassification.

[0076] If α is not the maximum value in C, it means that the model has a high confidence in other subclasses and there is a high risk of misclassification. In this case, s is defined as the difference between α and the maximum confidence:

[0077]

[0078] With this definition, s not only measures the confidence of the sample in the true category, but also reflects the degree of confusion of the model on other subclasses. The larger s is, the more confident the model is in classifying the sample and the farther the sample is from the classification decision boundary; conversely, the smaller s is, especially when s is a negative value, it indicates that the classification uncertainty of the sample is high and the model may misclassify. According to the separability s of the sample, the contribution of the sample to the loss can be adjusted dynamically. To achieve this goal, a weight factor w is introduced, which adjusts the importance of each sample in the training process according to the separability of the sample. The calculation formula of w is as follows:

[0079]

[0080] Among them, λ1 and λ2 are two hyperparameters used to control the magnitude and speed of change of w in different intervals. As can be seen from the formula, w is negatively correlated with s, that is, when the separability s of the sample increases, the weight w will decrease. This means that for samples with higher separability (larger s), that is, easy samples, their corresponding weights are smaller. Combined with the weight factor w, the contribution of each sample in the loss function is dynamically adjusted, thus forming the confusion minimization loss function L cm , whose expression is:

[0081]

[0082] In the formula, N represents the total number of samples in a small batch, N pos and N neg Represents the number of positive samples and negative samples respectively. Confidence C i represents the confidence distribution of the i-th positive sample on Q+1 subclasses, is the corresponding true label, which is a binary vector. For negative samples, C j represents the prediction confidence of the jth negative sample, and 0 represents the true label of the negative sample. BCE(·) in the loss function represents the cross entropy loss.

[0083] By introducing separability s and weight factor w, the model can pay more attention to those difficult samples with low separability during training. Since the confidence of these samples is close to the classification boundary and they are easily misclassified, they are given higher weights in the loss function, prompting the model to learn their features more deeply and improve the overall classification performance. In the inference stage, the model can better handle these difficult-to-distinguish targets, thereby enhancing the generalization ability of the model.

[0084] (3) Interactive verification strategy. The results of the coarse classification branch and the fine recognition branch are interactively verified to improve the robustness of reasoning. This method mainly uses the high-confidence coarse classification results to assist the inference of fine recognition to ensure the reliability of the recognition results. The detailed process of interactive verification is as follows:

[0085] In the interactive verification process, we first need to compare the maximum confidence of the coarse classification and the fine classification to determine whether the coarse classification result is reliable. If the maximum confidence of the coarse classification max(P) is greater than the maximum confidence of the fine classification max(C), this indicates that the model has a higher confidence in the coarse classification, while the fine classification may fail to provide a clear judgment due to complex details or confusion between samples. In this case, using the coarse classification results to assist in correcting the fine classification results can improve the robustness of the inference. Then we need to determine whether the coarse classification result belongs to the background class. If the coarse classification result corresponds to the background class (i.e., max(P) = p0, where p0 represents the confidence of the background class), it means that the model is more inclined to classify the target as background. At this time, the confidence of the fine classification of all non-background subclasses is set to 0, and only the confidence of the background class is retained. The updated subclass confidence vector is M = [c0,0,...,0]. This process ensures that when the target is unclear, it can effectively avoid classifying the target into other subclasses, thereby reducing the possibility of misclassification.

[0086] On the contrary, if the maximum value of the coarse classification confidence is greater than the fine classification confidence, and it corresponds to a specific coarse category rather than the background category, that is, max(P)>max(C) and max(P)≠p0, then the coarse classification result is relatively reliable and can provide clear category information for the target. On this basis, the confidence of the fine classification can be further confirmed. The specific operation is as follows. First, calculate the maximum value max(P) of the coarse classification confidence vector P, and find the corresponding coarse category index i=Index(P,max(P)). Then, find the maximum value among the fine classification confidences of all subcategories under the coarse category i. That is, the most likely subclass is selected within the scope of the coarse category. According to the confidence of the subclass, the fine classification confidence vector M is updated, only the subclass with the highest confidence under the coarse category is retained, and the confidence of other subclasses is set to 0. The specific update rules are as follows:

[0087]

[0088] In the formula, j represents the subclass index. After the update, only the subclass with the highest confidence under the coarse classification category i is retained, and the confidence of other subclasses is set to 0. The reason for this is that the coarse classification provides relatively reliable prior information for the fine classification, making the fine classification results more concentrated and accurate, reducing the uncertainty in the classification.

[0089] If the maximum confidence max(C) of the fine classification is greater than or equal to the maximum confidence max(P) of the coarse classification, the fine classification is considered to provide sufficiently reliable information and no longer relies on the coarse classification results for correction. In this case, the results of the fine classification are directly used for inference because they can capture more fine-grained classification features. This interactive verification strategy uses the global information of the coarse classification to improve the local inference of the fine classification, ensuring effective fusion between classification information at different levels, so that the fine classification no longer relies solely on its own results, thereby enhancing the robustness of model reasoning.

[0090] 3. Network model training

[0091] The images in the image dataset constructed in step 1 are input into the network constructed in step 2 for training, and the network is trained in an end-to-end manner to obtain a trained network.

[0092] The loss function L of the network is set during training as follows:

[0093] L=L orpn +L cm +L reg +L cls (17)

[0094] Among them, L orpn is the loss of the directed region proposal generation network, L cm is the confusion minimization loss for training the fine recognition branch, L reg is the loss of the regression branch, L cls is the loss of the coarse classification branch; these loss terms complement each other and jointly optimize the model. The hyperparameters λ1 and λ2 are set to 0.5 and 0.10. The optimizer for model training is AdamW, the initial learning rate is set to 0.0001, and the batch size is 2. Other training hyperparameters not specifically stated use the default configuration in mmrotate.

[0095] 4. Directed target recognition in remote sensing images

[0096] The remote sensing image to be processed is input into the trained network model, where the regression branch outputs the location information of the target, and the category confidence output by the coarse classification branch and the fine recognition branch after interactive verification is the directed target recognition result of the image.

[0097] To verify the effectiveness of the method of the present invention, simulation experiments were carried out in the following operating environment: 10 Karsten Heli GPU servers (CPU is Intel Xeon E5-2698, GPU is 12G RTX 2080Ti), the server operating system is Ubuntu 16.04.5LTS, and the experimental code is developed based on the open source detection toolkit mmrotate.

[0098] The partial recognition results of the remote sensing directional target recognition dataset FAIR1M-1.0 using the method of the present invention are shown in Figure 2 The first row of images shows the recognition results of different types of aircraft targets, the second row of images shows the recognition results of different types of ships, and the third row of images shows the recognition results of different types of vehicles. The above experimental results show that the method of the present invention can still accurately locate and identify different types of targets in challenging scenarios.

[0099] mAP (mean Average Precision) is used to quantitatively evaluate the effectiveness of the method of the present invention, and its definition is as follows:

[0100]

[0101] Among them, C represents the total number of categories contained in the training data set, c represents the target category index, and AP c Represents the recognition accuracy of different categories.

[0102] The method of the present invention is compared with the advanced single-stage directed target recognition method S2ANet and the two-stage directed target recognition algorithm based on Oriented R-CNN, and the backbone network used in the comparison process is ResNet50. The comparison results are shown in Table 1. It can be seen that the present invention can achieve higher directed target recognition accuracy.

[0103] Table 1

[0104] method mAP S2ANet 34.71％ Oriented R-CNN 38.85％ Method of the present invention 40.87％

Claims

1. A remote sensing directional target recognition method combining feature decoupling refinement and confusion minimization constraints, characterized in that Here are the steps: Step 1: Build a remote sensing image dataset; Step 2: Construct a remote sensing directed target recognition network with joint feature decoupling refinement and confusion minimization constraints, including a basic backbone network, a directed region proposal box generation network, a regression branch, a coarse classification branch, a fine recognition branch, a confusion minimization loss, and an interactive verification strategy. The backbone network is used to extract features, the directed region proposal box generation network is used to generate directed candidate boxes of the target, the regression branch is used for position regression of the directed candidate box, the coarse classification branch is used to classify the target at a coarse-grained level, the fine recognition branch realizes fine classification of the target, the confusion minimization loss is used to train the fine recognition branch, and the interactive verification strategy interactively verifies the results output by the coarse classification branch and the fine recognition branch; Step 3: Use the image data set in step 1 to train the network model in step 2; Step 4: Input the remote sensing image to be processed into the trained network model, where the regression branch outputs the location information of the target, and the category confidence output by the coarse classification branch and the fine recognition branch after interactive verification is the directed target recognition result of the image.

2. A remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraint as claimed in claim 1, characterized in that: In step 1, the public remote sensing directed target recognition datasets FAIR1M-1.0 and FAIR1M-2.0 are used to crop the original images in FAIR1M-1.0 and FAIR1M-2.0 to obtain image blocks of size 1024×1024. The sliding window size used for cropping is 1024×1024, the step size is 824, and the image blocks are randomly flipped vertically and horizontally to obtain the constructed remote sensing image dataset.

3. A remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraint as claimed in claim 1, characterized in that: The backbone network described in step 2 adopts a ResNet50 residual network with a feature pyramid structure, and its weights are initialized by pre-training on the ImageNet dataset.

4. A remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraint as claimed in claim 1, characterized in that: The directed region proposal box generation network described in step 2 adopts a fully convolutional network based on the RPN network, including 1 3×3 convolutional layer and 2 parallel 1×1 convolutional layers, of which 1 1×1 convolutional layer is used for foreground / background classification, and the number of feature map channels of its output is consistent with that of the RPN network, both of which are 3. Another 1×1 convolutional layer is used for regression processing, and the number of feature map channels of its output is changed from 12 of RPN to 18; the directed region proposal box generation network takes the image features obtained by the backbone network as input, obtains the directed candidate box of the target, and then performs the directed region proposal alignment operation to output the features corresponding to the target.

5. The remote sensing directed target recognition method of joint feature decoupling refinement and confusion minimization constraint as claimed in claim 1, characterized in that: The regression branch described in step 2 contains 3 fully connected layers, which take the target features obtained by the directed region proposal alignment operation as input and output the location information of the target.

6. A remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraint as claimed in claim 1, characterized in that: The coarse classification branch described in step 2 contains 3 fully connected layers, and shares the first two fully connected layers with the regression branch. The coarse classification branch takes the target features obtained by the directed region proposal alignment operation as input and outputs the classification score of the target on K coarse categories. The value of K is determined by the dataset.

7. A remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraint as claimed in claim 1, characterized in that: The fine recognition branch described in step 2 consists of several convolutional layers and fully connected layers, and its specific processing process is as follows: Step a: Input the target feature X, and after two independent 1×1 convolution operations, generate the feature map U and the spatial attention map A respectively; Step b: The spatial attention map A is normalized to the range [0, 1] using the sigmoid activation function σ(·); Step c: Multiply the feature map U and the normalized attention map A element by element to generate the spatial weighted feature V; Step d: Perform global average pooling on the spatial weighted features to extract the global channel descriptor V gap , the channel descriptor V gap Input the fully connected layer to generate the channel modulation weight b; multiply the channel modulation weight b and the spatial weighted feature V element by element to generate the feature Y after double feature refinement; the feature Y is reduced to a 1024-dimensional vector z through the fully connected layer; Step e: Take vector z as input and generate scores for Q+1 subclasses through a fully connected layer Among them, c0 represents the background score, K is the number of coarse classes, and q k Indicates the number of subcategories contained in the kth category, Indicates the qth coarse class under the kth coarse class i The score of the subclass, k=1,...,K,q i =1,...,q k ,q1+···+q K =Q, the value of Q is determined by the data set; The subcategory corresponding to the highest score is selected as the subcategory of the target and output.

8. The remote sensing directed target recognition method of claim 1, characterized in that: The calculation formula for the confusion minimization loss described in step 2 is as follows: Among them, L cm represents the confusion minimization loss value, N represents the total number of samples in a small batch, N pos and N neg Represents the number of positive samples and negative samples, C i represents the confidence distribution of the i-th positive sample on Q+1 subclasses, The corresponding true label is a binary vector; C j represents the prediction confidence of the jth negative sample, 0 represents the true label of the negative sample; BCE(·) represents the calculation of cross entropy loss; w i Represents the weight factor of the i-th positive sample; The weight factor is calculated as follows: Among them, w represents the weight factor of the sample, λ1 and λ2 are two hyperparameters with values ​​of 0.5 and 0.1; s represents the sample separability, which is determined by the following formula: Among them, C represents the score set of samples on Q+1 subclasses, and remove(C,α) represents the remaining score set after removing α from the score set.

9. A remote sensing directed target recognition method combining feature decoupling refinement and confusion minimization constraint as claimed in claim 1, characterized in that: The interactive verification strategy described in step 2 is as follows: Compare the maximum confidence of the coarse classification and the fine classification. If the maximum confidence of the coarse classification is greater than the maximum confidence of the fine classification, and the result of the coarse classification corresponds to the background class, set the fine classification confidence of all non-background subclasses to 0, and only keep the confidence of the background class. The updated subclass confidence vector is M. If the maximum confidence value of the coarse classification is greater than the maximum confidence value of the fine classification, and the result of the coarse classification is not the background class, calculate the maximum value of the coarse classification confidence vector, and find its corresponding coarse category index, recorded as i, find the maximum value among all subclass fine classification confidence values ​​under the coarse category i, retain only the subclass corresponding to the maximum value, set the confidence values ​​of other subclasses to 0, and obtain the updated subclass confidence vector M; the coarse classification confidence refers to the result obtained by the coarse classification branch; the fine classification confidence refers to the result obtained by the fine recognition branch.

10. The remote sensing directed target recognition method of claim 1, characterized in that: In step 3, the network loss function L is set as follows during training: L=L orpn +L cm +L reg +L cls (4) Among them, L orpn is the loss of the directed region proposal generation network, L cm is the confusion minimization loss for training the fine recognition branch, L reg is the loss of the regression branch, L cls is the loss of the coarse classification branch; The training was performed end-to-end using the AdamW optimizer. The initial learning rate was set to 0.0001, the batch size was 2, and other training hyperparameters used the default configuration in mmrotate.

Images