A method to improve the accuracy of fine-grained remote sensing target detection using attribute description

Abstract

The present application relates to a method for improving the accuracy of fine-grained remote sensing target detection by using attribute descriptions. The method comprises: constructing a remote sensing target detection model, including a visual encoder, a pre-trained language encoder, a visual language deep enhancement encoder, a bidirectional multi-head attention layer and a detection head; extracting remote sensing image features of an input remote sensing image according to the visual encoder; extracting category embedding features and attribute embedding features of category text and attribute text using the language encoder; inputting the category embedding features and the corresponding attribute embedding features of each category into the bidirectional multi-head attention layer for enhancement to obtain enhanced category embedding features; inputting the remote sensing image features and the enhanced category embedding features into the visual language deep enhancement encoder, inputting the enhanced image features and the category embedding features after secondary enhancement into the detection head to obtain detection results. The present method can improve the accuracy of fine-grained remote sensing target detection by using attribute descriptions.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a method for improving the accuracy of fine-grained remote sensing target detection by utilizing attribute description. Background Art

[0002] Remote sensing object detection is a core task at the intersection of computer vision and remote sensing technologies, with important applications in urban planning, ecological protection, and other fields. Conventional remote sensing object detection only requires locating and identifying broad categories of targets, such as aircraft and ships. However, in practical applications, understanding fine-grained classifications, such as the type or specific model of aircraft and ships, is often required to enable more refined decision-making. Existing fine-grained remote sensing object detection methods typically improve the detection accuracy of fine-grained categories through multi-scale feature enhancement, contrastive learning, and data augmentation. However, these methods only achieve fine-grained detection by optimizing visual features and have never explored leveraging language knowledge to enhance the detector's understanding of fine-grained objects. When human experts learn to identify fine-grained aircraft models, they can often summarize the attributes of a particular aircraft model, such as the wing structure, tail shape, and number of engines. More precisely, they can also understand data such as wingspan and fuselage length. This attribute knowledge helps human experts accurately identify aircraft models. However, existing methods fail to leverage attribute knowledge for fine-grained object detection. Summary of the Invention

[0003] Based on this, it is necessary to provide a method that can improve the accuracy of fine-grained remote sensing target detection by using attribute description to address the above technical problems.

[0004] A method for improving fine-grained remote sensing target detection accuracy using attribute description, the method comprising:

[0005] Obtain input remote sensing images, category text, and attribute text; build a remote sensing target detection model; the remote sensing target detection model includes a visual encoder, a pre-trained language encoder, a vision-language deep enhancement encoder, a bidirectional multi-head attention layer, and a detection head;

[0006] Extracting remote sensing image features of the input remote sensing image based on the visual encoder; extracting category embedding features and attribute embedding features of category text and attribute text using the pre-trained language encoder;

[0007] The category embedding features and corresponding attribute embedding features of each category are input into the bidirectional multi-head attention layer for enhancement to obtain the enhanced category embedding features;

[0008] Input the remote sensing image features and the enhanced category embedding features into the visual language deep enhancement encoder to obtain the enhanced image features and the secondary enhanced category embedding features;

[0009] The enhanced image features and the secondary enhanced category embedding features are input into the detection head to obtain the detection results.

[0010] This method for improving fine-grained remote sensing target detection accuracy using attribute descriptions breaks through the limitations of traditional reliance on visual features. It utilizes a pre-trained language encoder to extract category and attribute embeddings, and then injects attribute knowledge into the category embeddings via a bidirectional multi-head attention layer. This enriches the network's cognitive dimension for fine-grained targets, enabling the model to more accurately identify targets based on attribute characteristics, similar to human experts. Secondly, a deep fusion mechanism for vision and language is constructed. The language encoder is used to extract attribute annotation embedding features. After bidirectional attention fusion with the category embedding features, the detection head calculates the similarity between visual region features and category embedding features, improving fine-grained detection accuracy. Furthermore, an innovative masked attribute reconstruction pre-training method enables the language encoder to understand attribute descriptions in conjunction with visual examples. The pre-trained parameters function stably in the detection network, ensuring the reliable utilization of attribute knowledge. Finally, by building a multi-component collaborative system, utilizing visual encoders, language encoders, and bidirectional multi-head attention layers, the attribute descriptions are deeply integrated into the entire detection process, comprehensively improving the model's detection capabilities for fine-grained targets. This approach has important application value in fields such as urban planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of a method for improving fine-grained remote sensing target detection accuracy by using attribute description in one embodiment is shown;

[0012] Figure 2 is a schematic diagram of a remote sensing target detection model in one embodiment;

[0013] Figure 3 Schematic diagram of the pre-training process of a language encoder in one embodiment. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0015] In one embodiment, Figure 1 As shown, a method for improving the accuracy of fine-grained remote sensing target detection using attribute description is provided, including the following steps:

[0016] Step 102, obtain input remote sensing images, category text and attribute text; build a remote sensing target detection model; the remote sensing target detection model includes a visual encoder, a pre-trained language encoder, a visual language deep enhancement encoder, a bidirectional multi-head attention layer and a detection head.

[0017] Get the input remote sensing image, category text and attribute text, where the category text is a string of all the category names to be detected concatenated by ".", and the attribute text is the attribute description of all categories, and each category attribute is a string. Figure 2 As shown, it includes a visual encoder, a pre-trained language encoder, a visual-language deep enhancement encoder, a bidirectional multi-head attention layer, and a detection head.

[0018] Step 104 , extracting remote sensing image features of the input remote sensing image using the visual encoder; and extracting category embedding features and attribute embedding features of the category text and attribute text using the pre-trained language encoder.

[0019] Vision encoder through the network Extracting remote sensing image features , a language encoder completed through pre-training Extracting category embedding features and attribute embedding features , Indicates the maximum number of words, represents the implicit dimension, Indicates the number of categories.

[0020] Furthermore, the present invention designs a mask attribute reconstruction pre-training method for a language encoder, comprising:

[0021] Input the remote sensing image and the calibration box and attribute text of each target into the network, and the attribute of each target is a string.

[0022] The attribute text is randomly masked. Specifically, 15% of the words are randomly selected; 10% of the words are randomly selected from these 15% and replaced with a random word; 80% of the words are randomly selected from these 15% and replaced with [MASK]; the remaining 10% of the words remain unchanged. This results in the masked attributes and the original attributes.

[0023] Vision encoder through the network Extracting remote sensing image features , through the language encoder Extracting mask attribute embedding features , Indicates the number of targets.

[0024] The instance feature extractor extracts features from the image by labeling the box Extract all target instance features and get , is the number of instance feature channels, is the instance feature size.

[0025] Embed attributes into features Obtained by mapping to the number of visual instance feature channels through a linear layer , adjust the dimension of the instance feature to obtain ; In the second dimension and The spliced features are input into the visual language joint encoder for fusion to obtain .

[0026] Will The input is fed into the prediction head to generate attribute predictions, where the prediction head consists of two linear layers.

[0027] The cross entropy loss between the predicted attributes and the original attributes is calculated to optimize the network. The language encoder parameters obtained after the above pre-training process are loaded into the language encoder in the remote sensing object detection model and the language encoder parameters are frozen.

[0028] During pre-training, the network, comprised of a visual encoder and a language encoder, predicts attributes using visual examples, resulting in a language encoder with a specific understanding of attribute descriptions. During object detection network training, the pre-trained language encoder parameters are loaded, and its gradient updates are disabled, allowing only other network parameters to be updated. This ensures that the understanding of attribute descriptions learned by the language encoder can be stably applied to the detection model, providing a reliable foundation for the model to leverage attribute knowledge during detection, thereby improving the accuracy of fine-grained remote sensing object detection.

[0029] In step 106, the category embedding features and the corresponding attribute embedding features of each category are input into the bidirectional multi-head attention layer for enhancement to obtain the enhanced category embedding features.

[0030] The embedding features of each category and its corresponding attribute embedding features are enhanced through a bidirectional multi-head attention layer to For example, the process is expressed as: ,in and denote bidirectional multi-head attention layer and layer normalization respectively, is a category The category embedding features of is a category Attribute embedding features, For category The number of words; and then enhance the category embedding features through residual connection: ,in are the network learnable parameters, Embedding features for enhanced categories.

[0031] First, the language encoder is used to extract attribute annotation embedding features, which are then bidirectionally fused with the category embedding features through attention. This fusion method can better combine attribute information with category information, so that the category concept is integrated with the details of the attribute description. Therefore, when calculating the similarity between the visual area features and the category embedding features in the detection head, judgments can be made based on richer and more accurate category representations, thereby improving the accuracy of fine-grained detection.

[0032] At the same time, a pre-trained language encoder is used to extract category embeddings and category attribute embeddings, and a bidirectional multi-head attention layer is used to infuse category attribute knowledge into the category embeddings. This enables the network to absorb and utilize attribute knowledge, overcoming the previous limitation of relying solely on visual features and enriching its understanding of fine-grained objects from a knowledge perspective. For example, human experts can more accurately identify aircraft models by summarizing attribute characteristics such as wing structure and tail shape. This method similarly allows the network to learn this attribute knowledge, laying the foundation for more accurate detection.

[0033] In step 108, the remote sensing image features and the enhanced category embedding features are input into a visual language deep enhancement encoder to obtain enhanced image features and secondary enhanced category embedding features.

[0034] The image features and category embedding features Input the visual language deep enhancement encoder to obtain enhanced image features and category embedding features The visual language deep enhancement encoder consists of 6 network blocks, each of which is composed of a cross-modal multi-head attention module, a DyHead module, and a BERT layer. The cross-modal multi-head attention module is used to realize cross-modal feature interaction, and the DyHead module is used to encode visual features and extract image region features. , the BERT layer is used to encode language features.

[0035] The remote sensing image features and enhanced category embedding features are input into the visual language deep enhancement encoder to further achieve interactive enhancement between visual and language features. Through this deep interaction, the image features can obtain supplementary information brought by attribute knowledge, while the category embedding features can also better match the target features in the image, making the model's understanding of fine-grained target features more accurate, thereby improving detection accuracy.

[0036] In step 110 , the enhanced image features and the secondary enhanced category embedding features are input into the detection head to obtain the detection results.

[0037] Image region features and category embedding features Input to the detection head, the detection head includes two branches: positioning and classification. The positioning branch consists of a layer The convolutional layer is composed of the classification branch, which involves the similarity calculation between the category embedding features and the image region features; the regional features The box prediction is obtained by the positioning branch, and the similarity between the category embedding features and the image region features is calculated in the classification branch: , and the similarity is the classification score of each category.

[0038] When the model begins training, the pre-trained parameters of the visual encoder and language encoder are loaded, and the language encoder parameters are frozen. During model training, the ground truth is combined with the output of the classification branch to calculate the Focal Loss, and the output of the localization branch to calculate the Intersection over Union Loss, to optimize the network parameters.

[0039] This method for improving fine-grained remote sensing target detection accuracy using attribute descriptions breaks through the limitations of traditional reliance on visual features. It utilizes a pre-trained language encoder to extract category and attribute embeddings, and then injects attribute knowledge into the category embeddings via a bidirectional multi-head attention layer. This enriches the network's cognitive dimension for fine-grained targets, enabling the model to more accurately identify targets based on attribute characteristics, similar to human experts. Secondly, a deep fusion mechanism for vision and language is constructed. The language encoder is used to extract attribute annotation embedding features. After bidirectional attention fusion with the category embedding features, the detection head calculates the similarity between visual region features and category embedding features, improving fine-grained detection accuracy. Furthermore, an innovative masked attribute reconstruction pre-training method enables the language encoder to understand attribute descriptions in conjunction with visual examples. The pre-trained parameters function stably in the detection network, ensuring the reliable utilization of attribute knowledge. Finally, by building a multi-component collaborative system, utilizing visual encoders, language encoders, and bidirectional multi-head attention layers, the attribute descriptions are deeply integrated into the entire detection process, comprehensively improving the model's detection capabilities for fine-grained targets. This approach has important application value in fields such as urban planning.

[0040] In one embodiment, Figure 3 As shown in Figure 2, the pre-training process of the language encoder includes:

[0041] Obtain the input remote sensing image and the calibrated bounding box and attribute text of each target therein; randomly mask the attribute text to obtain the masked attributes and original attributes; use the visual encoder to extract remote sensing image features, and use the language encoder to extract the masked attribute embedding features; use the instance feature extractor to extract the instance features of all targets from the remote sensing image features through the calibrated bounding box;

[0042] The mask attribute embedding feature is mapped to the number of visual instance feature channels through a linear layer to obtain the mapped feature. After adjusting the dimension of the instance feature, the mapped feature and the adjusted instance feature are spliced in the second dimension. The spliced feature is input into the visual language joint encoder for fusion to obtain the fused feature.

[0043] The fused features are input into the prediction head to generate attribute predictions, the cross entropy loss between the predicted attributes and the original attributes is calculated, the network is optimized, and a trained language encoder is obtained.

[0044] In one embodiment, the attribute text is randomly masked to obtain masked attributes and original attributes, including:

[0045] Randomly select 15% of the words in the attribute text, randomly select 10% of the words from the 15% of the words and replace them with a random word; randomly select 80% of the words from the 15% of the words and replace them with [MASK]; the remaining 10% of the words remain unchanged, and the masked attributes and original attributes are obtained.

[0046] In one embodiment, the mask attribute embedding feature is mapped to the number of visual instance feature channels through a linear layer to obtain a mapped feature, the dimension of the instance feature is adjusted, and then the mapped feature and the adjusted instance feature are concatenated in a second dimension, including:

[0047] Embedding mask attributes into features The mapped features are obtained by mapping to the number of visual instance feature channels through the linear layer , adjust the dimension of the instance feature to obtain the adjusted instance feature ; In the second dimension and Perform splicing to obtain the spliced features; among them, Indicates the number of targets, Indicates the maximum number of words, represents the implicit dimension, is the number of instance feature channels.

[0048] In one embodiment, the category text is a string obtained by concatenating the names of all categories to be detected by "."; the attribute text is a description of the attributes of all categories, and each category attribute is a string.

[0049] In one embodiment, the category embedding features and the corresponding attribute embedding features of each category are input into a bidirectional multi-head attention layer for enhancement, to obtain enhanced category embedding features and attribute embedding features, including:

[0050] The category embedding features and corresponding attribute embedding features of each category are input into the bidirectional multi-head attention layer for enhancement, and the enhanced category embedding features are obtained as follows:

[0051] ;

[0052] ;

[0053] in, and denote bidirectional multi-head attention layer and layer normalization respectively, is a category The category embedding features of is a category Attribute embedding features, For category The number of words, are the network learnable parameters, Embedding features for enhanced categories.

[0054] In one embodiment, the visual language deep enhancement encoder includes 6 network blocks, each of which is composed of a cross-modal multi-head attention module, a DyHead module and a BERT layer. The cross-modal multi-head attention module is used to realize cross-modal feature interaction, the DyHead module is used to encode visual features and extract image area features, and the BERT layer is used to encode language features.

[0055] In one embodiment, the detection head includes two branches: positioning and classification. The positioning branch consists of a layer The classification branch is composed of convolutional layers. It involves calculating the similarity between the category embedding features and the image region features. The enhanced image features and the secondary enhanced category embedding features are input into the detection head to obtain the detection results, including:

[0056] Enhanced remote sensing image features Get box prediction by positioning branch, and embed features of category after secondary enhancement Similarity calculation is performed with the enhanced remote sensing image features in the classification branch: , the similarity is the classification score of each category, and the category with the highest similarity is taken as the classification result.

[0057] In a specific embodiment, during experiments, a remote sensing aircraft detection dataset containing specific civil aircraft models was constructed using aircraft data from the remote sensing aircraft detection dataset MAR20 and the remote sensing fine-grained object detection dataset FAIR1M. This dataset contains 30 aircraft models, with instance locations annotated with rotating boxes. Each aircraft model is annotated with fine-grained attributes, including wing configuration, tail shape, and fuselage length. The object detection method used the mean average precision (mAP) with an Intersection over Union (IoU) threshold of 0.5; the masked attribute reconstruction method used the accuracy metric. Masked attribute reconstruction was pre-trained on the large-scale remote sensing aircraft detection dataset RarePlanes. The pre-training results are shown in Table 1, demonstrating that the pre-trained model can accurately predict attributes based on aircraft instances. Experimental results on the constructed dataset of 30 aircraft models are shown in Table 2, demonstrating that the integration of attribute knowledge improves the network's fine-grained detection accuracy. Detection accuracy is further improved after adding the pre-trained language encoder for masked attribute reconstruction.

[0058] Table 1

[0059]

[0060] Table 2

[0061]

[0062] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0063] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are intended to fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for improving the accuracy of fine-grained remote sensing target detection using attribute description, characterized in that: The method comprises: Obtaining input remote sensing images, category text, and attribute text; building a remote sensing target detection model; the remote sensing target detection model includes a visual encoder, a pre-trained language encoder, a vision-language deep enhancement encoder, a bidirectional multi-head attention layer, and a detection head; Extracting remote sensing image features of the input remote sensing image according to the visual encoder; extracting category embedding features and attribute embedding features of the category text and attribute text using the language encoder; Inputting the category embedding features and corresponding attribute embedding features of each category into the bidirectional multi-head attention layer for enhancement to obtain enhanced category embedding features; Inputting the remote sensing image features and the enhanced category embedding features into a visual language deep enhancement encoder to obtain enhanced image features and secondary enhanced category embedding features; Inputting the enhanced image features and the secondary enhanced category embedding features into a detection head to obtain a detection result; The pre-training process of the language encoder includes: Obtain an input remote sensing image and the calibrated bounding box and attribute text of each target therein; randomly mask the attribute text to obtain masked attributes and original attributes; extract remote sensing image features using a visual encoder and embed features of masked attributes using a language encoder; and extract instance features of all targets from the remote sensing image features using the calibrated bounding box using an instance feature extractor; The mask attribute embedding feature is mapped to the number of visual instance feature channels through a linear layer to obtain the mapped feature. After adjusting the dimension of the instance feature, the mapped feature and the adjusted instance feature are concatenated in the second dimension. The concatenated feature is input into the visual language joint encoder for fusion to obtain the fused feature. The fused features are input into the prediction head to generate attribute predictions, the cross entropy loss between the predicted attributes and the original attributes is calculated, the network is optimized, and a trained language encoder is obtained.

2. The method according to claim 1, characterized in that The attribute text is randomly masked to obtain masked attributes and original attributes, including: Randomly select 15% of the words in the attribute text, randomly select 10% of the words from the 15% of the words and replace them with a random word; randomly select 80% of the words from the 15% of the words and replace them with [MASK]; the remaining 10% of the words remain unchanged, and the masked attributes and the original attributes are obtained.

3. The method according to claim 1, characterized in that The mask attribute embedding feature is mapped to the number of visual instance feature channels through a linear layer to obtain the mapped feature. After adjusting the dimension of the instance feature, the mapped feature and the adjusted instance feature are concatenated in the second dimension, including: Embedding mask attributes into features The mapped features are obtained by mapping to the number of visual instance feature channels through the linear layer , adjust the dimension of the instance feature to obtain the adjusted instance feature ; In the second dimension and Perform splicing to obtain the spliced features; among them, Indicates the number of targets, Indicates the maximum number of words, represents the implicit dimension, is the number of instance feature channels.

4. The method according to claim 1, wherein The category text is a string obtained by concatenating the names of all categories to be detected by ". "; the attribute text is a description of the attributes of all categories, and each category attribute is a string.

5. The method according to claim 1, wherein The category embedding features and corresponding attribute embedding features of each category are input into the bidirectional multi-head attention layer for enhancement to obtain enhanced category embedding features and attribute embedding features, including: The category embedding features and corresponding attribute embedding features of each category are input into the bidirectional multi-head attention layer for enhancement, and the enhanced category embedding features are obtained as follows: in, and denote bidirectional multi-head attention layer and layer normalization respectively, is a category The category embedding features of is a category Attribute embedding features, For category The number of words, are the network learnable parameters, is the enhanced category embedding feature, Indicates the maximum number of words, Represents an implicit dimension.

6. The method according to claim 1, characterized in that The visual language deep enhancement encoder includes 6 network blocks, each of which is composed of a cross-modal multi-head attention module, a DyHead module and a BERT layer. The cross-modal multi-head attention module is used to realize cross-modal feature interaction, the DyHead module is used to encode visual features and extract image area features, and the BERT layer is used to encode language features.

7. According to the method of claim 1, the detection head includes two branches: positioning and classification. The positioning branch consists of a layer The convolutional layer is composed of a classification branch that involves calculating the similarity between the category embedding features and the image region features. The enhanced image features and the secondary enhanced category embedding features are input into the detection head to obtain the detection results, including: The enhanced image features Get box prediction by positioning branch, and embed features of category after secondary enhancement Similarity calculation is performed with the enhanced remote sensing image features in the classification branch: The similarity is the classification score of each category, and the category with the highest similarity is taken as the classification result.

Images