Aerial image-oriented real-time open vocabulary target detection model

Abstract

The invention provides a real-time open vocabulary target detection model AOV-RTDETR for an aerial image, and aims to solve the problems that a traditional detection model cannot recognize unseen categories, small targets are difficult to detect and the real-time performance is insufficient. The model is based on an RT-DETR framework, cross-modal feature alignment of images and texts is achieved in combination with a CLIP visual language model, region-text comparison loss is introduced to break through predefined category limitation, and detection of unseen categories is supported. In order to improve the detection capability of a small target and a high aspect ratio target, a large strip-shaped convolution structure is provided to enhance the feature modeling capability, and a multi-scale spatial pyramid attention encoder and a text-guided multi-stage fusion decoder are designed to improve the feature expression and semantic alignment capability of the small target. The method has high precision and real-time balance capability, and is suitable for application scenes with high requirements for target detection precision and speed, such as aerial remote sensing, emergency rescue, unmanned aerial vehicle patrol and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a real-time open vocabulary target detection model for aerial images. Background Technology

[0002] Aerial target detection aims to locate and classify targets of interest from aerial imagery, and is widely used in fields such as Earth monitoring and disaster search and rescue, thus attracting considerable attention. However, most existing aerial detectors can only identify predefined categories in the training set, focusing primarily on detection accuracy while neglecting detection efficiency. These methods typically rely on mapping image features to fixed category indices, i.e., fixed-vocabulary (closed-set) detection, thus limiting their ability to discover new categories. In recent years, Visual Language Models (VLMs), such as CLIP, have been able to map visual and linguistic features to a shared representation space, providing new possibilities for overcoming the category limitations of traditional target detection. In fact, recent research has explored the relationship between images and text, endowing models with Open Vocabulary Detection (OVD) capabilities, and has made significant progress. These methods can be broadly categorized into two types: (1) Two-stage detector-based methods: These methods typically use networks such as FPN to generate candidate target regions and rely on pre-trained VLMs for classification. For example, VILD enhances open vocabulary detection capabilities by extracting knowledge from pre-trained VLMs; VL-PLM relies on pre-trained VLMs to generate region-text pair labels and uses them to train the detector, enabling it to identify categories not present in the training set. However, these methods are computationally intensive and have low inference efficiency. (2) End-to-end learning-based methods: These methods either utilize pre-trained VLMs to generate region-text pairs or integrate object detection with phrase alignment (grounding) tasks to construct a VLM specifically for OVD tasks. For example, GLIP transforms the object detection task into a phrase alignment problem, thereby unifying object detection and text understanding; Grounding-DINO integrates phrase alignment strategies into the Transformer detection architecture, further enhancing the generalization ability of OVD. However, most of these methods are based on computationally intensive detectors such as ViT or DINO, and are mainly designed for natural images, making them difficult to apply directly to aerial images.

[0003] Furthermore, objects are primarily displayed at small scales in aerial imagery. Previous research on aerial target detection has mainly employed deep neural networks or coarse-to-fine architectures to enhance the detection performance for small targets. However, these methods are typically closed-set detectors. Only a few works have explored open-vocabulary detection in aerial images, and the limited training data makes it difficult to directly transfer OVD methods from natural images to aerial images. For example, DescReg enhances text description capabilities by introducing descriptive regularization methods, but it does not fully consider the importance of visual semantic interaction. CastDet employs a complex multi-teacher framework based on a two-stage detector, which, while possessing open-vocabulary detection capabilities, has high computational costs, making it difficult to meet the real-time requirements of aerial target detection.

[0004] Meanwhile, successfully detecting high aspect ratio targets, which are prevalent in aerial target detection, remains a challenging problem. Existing target detection methods often struggle with elongated objects, where detection performance degrades as the object's aspect ratio increases. This paper argues that the difficulty in detecting these objects stems from two main challenges: First, high aspect ratio targets contain rich feature information in one spatial dimension but exhibit relatively sparse features in another. Traditional detectors based on convolutional neural networks mostly extract input feature maps within a square window. This design severely limits their ability to effectively capture anisotropic environments, which are common in aerial images, resulting in the inclusion of irrelevant information from the surrounding region. Second, due to their unique geometric properties, high aspect ratio targets present considerable challenges in regression tasks. Aerial target detection, unlike general target detection, requires additional angle regression. For high aspect ratio targets, even small errors in angle estimation can lead to significant deviations from ground reality. To date, few works have considered how to handle challenging high aspect ratio objects. A common approach widely used in previous methods is to expand the receptive field of models with large kernel convolutions. A prime example is LSKNet, which introduced large kernel convolutions with spatial selection mechanisms to capture long-range contextual information. PKINet further extended LSKNet and proposed a multi-branch architecture to enhance the detection capability for objects with large scale variations. However, the parallel paradigm utilizing multiple large kernel convolutions exacerbates the computational burden, and the complex block design further limits model efficiency. How to effectively utilize large kernel convolutions for objects with high aspect ratio variations remains an unresolved issue.

[0005] Currently, the technical solutions similar to the "real-time open vocabulary target detection model for aerial images" of this invention mainly include the following:

[0006] OV-DETR, or "Open-Vocabulary DETR with Conditional Matching," proposes a Transformer-based approach for open vocabulary detection. It identifies new object categories through cross-modal interaction between a visual encoder and a text encoder. While this method performs reasonably well in detecting large targets, its ability to detect small targets is insufficient. For small targets in aerial images, the global self-attention mechanism struggles to accurately locate these local features in high-resolution images, resulting in unsatisfactory detection performance.

[0007] VLDet's approach, "Learning Object-Language Alignments for Open-Vocabulary Object Detection," allows the detector to directly learn the correspondence between objects and language on image-text pairing data. However, this technique falls short when dealing with small images in aerial photography, as targets occupying very few pixels are often not prominent in feature maps. This makes the model more prone to misjudgment or failure to capture, especially when encountering unseen images, where its generalization ability is not ideal.

[0008] RegionCLIP (Region-based Language-Image Pretraining) attempts to extend the visual-text pretraining approach derived from CLIP to object recognition, hoping to achieve the detection of unknown categories through cross-modal alignment. While large-scale image and text data gives it the potential to identify new categories, it encounters bottlenecks when dealing with very small targets. The fundamental reason is that this method relies more on the overall semantic relationships of the image, lacking sufficient capture of detailed-level features.

[0009] DetPro's "Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model" enhances the features of specific objects during training, aiming to expand the adaptability of open-vocabulary object detection. While this enhancement method is helpful for targets of average size, its effectiveness is limited for small targets; when the target belongs to a completely new category and is also very small, the model's recognition expansion capability is still insufficient.

[0010] In summary, the existing technical solutions for open-vocabulary small target detection suffer from the following main drawbacks: heavy computational burden preventing real-time performance; neglect of the high proportion of small targets and the abundance of high aspect ratio targets in aerial photography; insufficient matching of visual and textual features; poor generalization ability for new categories; and low sample utilization. Against this backdrop, this invention proposes AOV-RTDETR, a lightweight real-time open-vocabulary target detection method based on RT-DETR. RT-DETR, as an efficient detection framework, possesses excellent real-time performance. AOV-RTDETR further incorporates large strip convolutions, effectively combining the advantages of square and strip convolutions while reducing feature redundancy. Furthermore, it designs advanced cross-modal fusion strategies to enhance feature extraction capabilities for small targets, achieving high-performance detection while maintaining inference speed. Summary of the Invention

[0011] This patent proposes a real-time open vocabulary object detection model for aerial images (AOV-RTDETR), a highly efficient open vocabulary detector specifically designed for aerial photography scenarios. Specifically, AOV-RTDETR is built upon the standard RTDETR framework and is suitable for the efficiency requirements of aerial detection. Based on the concept of image-text alignment, AOV-RTDETR integrates a CLIP encoder and introduces a region-text contrast loss, replacing the traditional category regression loss in conventional detectors, thus breaking the limitations of predefined categories. Furthermore, considering small objects and complex backgrounds in aerial images, large strip convolutions with sequential orthogonality are used instead of ordinary convolutions in the network backbone to better capture the anisotropic features of targets in aerial images. This patent also proposes a bidirectional vision-language fusion method, a text-guided encoder-decoder structure, which effectively extracts class-related image features by utilizing cross-modal fusion. This structure includes a multi-scale spatial pyramid attention (MAFPN) encoder and a text-guided attention multi-level fusion (TGAMF) decoder. This method enhances the extraction of image features with class information by utilizing cross-modal fusion between image features and class embeddings. MAFPN enhances the feature extraction process in encoder components, while TGAMF aims to improve small object detection performance.

[0012] Specifically, AOV-RTDETR is a method for real-time detection of small targets with open vocabulary in aerial remote sensing images, including the following steps:

[0013] Step 1: Input the input image into the CLIP text encoder and ResNet image encoder, and introduce a large strip convolutional structure to extract image features. Input the class name into the CLIP text encoder to generate multi-level class embeddings to support subsequent cross-modal fusion.

[0014] Step 2: Align image features with text features using the comparison head module, and calculate the similarity score between each image region and the category word vector. Use the Hungarian matching algorithm to assign positive and negative samples, and introduce region-text contrast loss to replace the traditional classification loss.

[0015] Step 3: Image features are fed into the Multi-Scale Spatial Pyramid Attention Encoder (MAFPN) to extract fine-grained and global features; the decoder stage introduces a text-guided multi-level fusion mechanism (TGAMF) to guide the attention of the detection query to focus on semantically relevant regions using multi-level class embeddings, especially enhancing the perception ability of small targets.

[0016] Step 4: The decoder outputs the text similarity scores of the predicted bounding box and the corresponding region, and filters the detection results by setting a threshold to achieve real-time recognition and localization of open vocabulary targets.

[0017] As a preferred embodiment, a method for real-time detection of small targets with open vocabulary in aerial remote sensing images, wherein the image encoding method in step one is as follows:

[0018] The first step is to input an image feature tensor with C channels. First, apply a small kernel (such as a standard depthwise convolution of 3×3 or 5×5) to it to extract local contextual information and generate preliminary feature representations;

[0019] The second step involves introducing two large, orthogonal strip-shaped depth convolution kernels sequentially after the standard convolution: one strip convolution kernel uses a larger 1×k kernel (e.g., 1×15) to extract fine-grained features along the horizontal direction, and the other strip convolution kernel uses a larger k×1 kernel (e.g., 15×1) to extract the vertical structural information of the target along the vertical direction.

[0020] Third, the two orthogonal strip convolutions mentioned above are applied to different spatial dimensions of the feature map, which can effectively expand the receptive field and enhance the network's directional sensitivity and length and width structure modeling ability for high aspect ratio targets.

[0021] The fourth step is to fuse the outputs of the horizontal strip convolution and the vertical strip convolution, and then connect them to a pointwise 1×1 convolution to enhance the information interaction between channels and further integrate directional features.

[0022] Fifth, the final fused feature map not only retains fine-grained structure in the spatial dimension but also possesses high responsiveness to strip-shaped and elongated targets. This feature map is used as an attention weight to perform element-wise weighting on the original input feature X, thereby enhancing the representation effect of salient target regions.

[0023] Preferably, in step one, the class embeddings generated by the text encoder contain three levels: T low T mid T high , respectively, represent semantic representations applicable to low, medium, and high-scale image features, providing refined text guidance for subsequent multi-level fusion.

[0024] Preferably, the region-text contrast loss in step two includes:

[0025] The first step is to construct text feature embeddings for each input image and a given category vocabulary, and then output N object prediction results through a Transformer decoder. Each prediction result includes a bounding box location and a region feature representation;

[0026] The second step is to use the Hungarian matching algorithm to assign positive and negative samples between the predicted bounding box and the ground truth bounding box. In the matching process, the traditional category score is no longer used as the matching basis. Instead, the similarity between the predicted region features and the category text embedding is used as the matching score, thereby achieving image-text feature alignment.

[0027] The third step is to calculate the region-text similarity loss L for each pair of matching results. c The loss is based on the VarifocalLoss design and is used to measure the cross-modal matching strength between each region and its corresponding class, which can simultaneously take into account the enhancement of positive samples and the suppression of negative samples.

[0028] The fourth step, while completing category alignment, further optimizes the target bounding boxes using regression, employing two regression loss functions: L1 loss for location loss. L1 It measures the deviation between the predicted bounding box and the ground truth bounding box in the coordinate dimension. Size and shape matching uses IOU loss. IoU Improve the overlap of target boundaries;

[0029] Fifth, the three losses mentioned above are weighted and summed to form the final training loss function, which is expressed as follows:

[0030] L det =λL c +μL iou +vL L1

[0031] Wherein, λ, μ, and v represent the weighting coefficients of contrast loss, IOU loss, and L1 localization loss, respectively, which can be flexibly adjusted according to the task scenario to balance detection accuracy and training stability.

[0032] Preferably, the specific method of the multi-scale spatial pyramid attention encoder (MAFPN) in step three is as follows:

[0033] The first step is to input the feature map F output by the image encoder into the Hierarchical Residual Convolution (HRC) module, and divide it into s sub-blocks along the channel dimension, i.e., F = {F1, F2, ..., F...} s}, each sub-block F i Include One channel;

[0034] The second step is to process each sub-block F separately. i A standard 3×3 convolutional layer is applied, combined with a BatchNormalization layer (BN) for local feature extraction, and features are accumulated layer by layer through residual connections to form an information aggregation path from local to global, thereby achieving hierarchical expansion of the receptive field;

[0035] The third step is to fuse the results of all sub-blocks to form a spatial pyramid feature representation with multi-scale information. This structure avoids the problems of redundant calculation and feature information redundancy in traditional multi-scale structures through hierarchical residual connections.

[0036] The fourth step is to perform cross-modal alignment between the fused image features and the class embeddings from the text encoder, and use a cross-attention mechanism to guide the model to focus on the region in the image that is most relevant to the semantics of the current category.

[0037] In the fifth step, in cross-attention, image features are used as queries and text features are used as keys and values ​​to exchange information between channels, enabling the model to complete the deep fusion of image and text features during the encoding stage and improve the image semantic alignment capability.

[0038] The sixth step is to output the fused image and text features from the encoder and pass them to the subsequent decoder module for further target localization and open category recognition. This structure significantly enhances the model's feature discrimination ability in complex backgrounds and is especially suitable for fine-grained small target detection tasks.

[0039] Preferably, the specific method for the Text Guided Multilevel Fusion Decoder (TGAMF) in step three is as follows:

[0040] The first step is to construct a multi-layer Transformer decoder and embed a text guidance mechanism in each layer to introduce text feature representations at different levels, including a high-level text vector T. high Mid-level text vector T mid Low-level text vector T lowTo achieve cross-scale semantic guidance;

[0041] The second step is to fuse the multi-scale features output by the image encoder with the class embeddings output by the text encoder to generate multi-level class embeddings. Each layer of text vectors not only contains the original category semantics but also incorporates visual information of the corresponding scale, thereby constructing a cross-modal feature representation corresponding to the image scale.

[0042] The third step, based on the characteristic that small targets are easily diluted in high-level semantics, adopts a hierarchical fusion strategy in the TGAMF decoder: the first-layer decoder and T... high Fusion, focusing on semantic global information, the second and third layer decoders and T mid Fusion, taking into account both semantics and structure, the decoders of layers 4 to 6 and T low Multiple interactions are used to enhance the perception and focus on the detailed features of small targets;

[0043] The fourth step is to introduce a multi-head cross-attention (MHCA) mechanism in each decoder layer to fuse the query vector in the current decoder with the corresponding multi-level class embeddings, thereby achieving fine-grained alignment between the decoder query and semantic information.

[0044] Fifth step, by providing T low Design more interaction layers to enhance the model's attention to low-resolution image regions in the subsequent decoding stage, so that small targets have a higher probability of being detected and higher localization accuracy in the final prediction.

[0045] The sixth step involves using the fused decoder output to predict the target location and category-related similarity scores, ensuring that the detector has the ability to accurately respond to small-sized, low-sample, and semantically weak targets, significantly improving the small target detection performance under open vocabulary conditions.

[0046] Preferably, the reasoning output method in step four is as follows:

[0047] High-confidence target boxes are selected by region-text similarity scores, and the similarity threshold is dynamically adjusted according to the task scenario to balance recall and precision, ensuring that the system meets the requirements for real-time detection.

[0048] This invention has the following characteristics and beneficial effects:

[0049] First, the open-vocabulary detection capability is enhanced. By introducing an image-text alignment mechanism and a region-text contrast loss function, the model gains the ability to detect new categories not present in the training set, thus overcoming the inherent limitations of traditional detectors in terms of categories. This approach not only improves the model's generalization ability to unknown categories but also provides effective theoretical and practical basis for open-vocabulary detection.

[0050] Secondly, in terms of improving the accuracy (especially for small targets) and recall of target detection, by introducing large strip convolutions into the backbone network and designing a multi-scale spatial pyramid attention encoder and a multi-level text-guided fusion decoder, the model can extract fine-grained features of small targets more fully. By fusing text semantics and local image features layer by layer, the accuracy and recall of small targets in the detection task are significantly improved, demonstrating that it has a clear advantage in improving the performance of small target detection in complex scenes.

[0051] Furthermore, to achieve efficient real-time inference, this study constructed a detection model based on the RT-DETR framework. While ensuring high detection accuracy, the system achieved fast inference speed, meeting the stringent requirements of real-time aerial detection scenarios. It achieved a balance between accuracy and speed, providing strong support for real-time applications.

[0052] Finally, by improving data utilization, a CLIP pre-trained model was introduced, enabling the model to achieve efficient open-vocabulary detection even on limited aerial image data. This pre-trained model provides a good alignment foundation between visual features and textual semantics, thereby significantly improving data utilization efficiency and further enhancing the robustness of the detector in small-sample environments. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is the overall framework for AOV-RTDETR.

[0055] Figure 2 This is a schematic diagram of the MAFPN encoder.

[0056] Figure 3 This is a schematic diagram of the TGAMF decoder. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description, in conjunction with the accompanying drawings and embodiments, further illustrates the invention. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention.

[0058] The proposed AOV-RTDETR is a real-time Transformer architecture for open-vocabulary object detection, aiming to solve the problem that traditional aerial photography object detection models struggle to identify open-vocabulary categories. It also enhances small object detection capabilities, enabling efficient object recognition and detection in zero-shot and few-shot object detection tasks. AOV-RTDETR employs strategies such as feature-adaptive multi-scale encoding, multi-level text-guided decoding, dynamic matching query mechanisms, and open-vocabulary classification heads to improve the accuracy of object detection in open-vocabulary environments.

[0059] AOV-RTDETR is based on the RT-DETR framework and mainly consists of the following modules:

[0060] Image encoder: It adopts the ResNet-50 backbone network and introduces large strip convolutions.

[0061] Text encoder: CLIP text encoder is used.

[0062] Bidirectional Vision-Language Fusion Module (Bi-VLF): A multi-scale spatial pyramid attention (MAFPN) encoder and a text-guided attention multi-level fusion (TGAMF) decoder.

[0063] Comparison Head: Calculates region-text similarity by aligning visual and text feature dimensions through linear layers.

[0064] Detection head: Outputs target bounding box coordinates and region-text similarity score.

[0065] The real-time open-vocabulary object detection model for aerial images includes the following steps:

[0066] Step 1: The image and text encoder performs image encoding;

[0067] The first step involves encoding the input image. The image encoder is based on CLIP-ResNet50 and introduces large striped convolutions, utilizing sequential orthogonal large striped convolutions to capture spatial information. This enhances the ability to perceive small object features. This convolution can expand the receptive field in a specific direction (horizontal or vertical), helping to capture the length and width characteristics of fine-grained objects, enhancing detection capabilities, and extracting multi-scale feature maps. Given an input tensor X∈R with C channels... C×H×W First, apply the core. Z is obtained by using depthwise convolution to extract local contextual features, where H×W and k H ×k W These are the feature size and kernel size, respectively. In practical applications, k... H ×k W Set to 5x5. After the initial depthwise convolution, two consecutive depthwise convolutions with a large strip kernel are used to better capture high aspect ratio objects. The output is represented as... Unlike standard convolutions that extract features from square regions each time, large strip convolutions allow the network to focus more on features along the horizontal or vertical axis. The combined use of horizontal and vertical large strip convolutions enables the network to collect directional features across two spatial axes, thereby enhancing the representation of elongated or narrow structures in the spatial dimension. To further enhance the interaction of features in the channel dimension, a simple pointwise convolution is applied... This is transformed into Y. In this way, each location in the generated feature map Y encodes both horizontal and vertical features in a wide spatial region. Finally, treating the feature map Y as attention weights to reweight the input X, can be expressed as follows: The "·" symbol represents element-wise multiplication.

[0068] The second step is to encode the input text. Input the category name (e.g., "airplane" or "vehicle"), and input the category vocabulary C∈R. n Class embeddings T = TextEncoder(C) ∈ R are generated using the CLIP text encoder. n×d .

[0069] Step 2: Align the region with the text for features;

[0070] The first step is to align the image feature dimensions using linear projection and calculate the region-text similarity: α and β are learnable parameters.

[0071] The second step, Hungarian assignment, involves matching predicted bounding boxes with ground truth bounding boxes using region-text similarity (rather than classification scores) to minimize matching costs.

[0072] The third step is to calculate the loss function: the total loss is the weighted sum of the contrast loss Lc, the IOU loss Liou, and the L1 loss LL1.

[0073] L det =λL c +μL iou +vL L1

[0074] The values ​​are set as follows: λ = 1.0, μ = 2.0, and v = 1.0.

[0075] Step 3: Multi-scale feature fusion is performed through the bidirectional vision-language fusion module (Bi-VLF);

[0076] The first step involves using a multi-scale spatial pyramid attention (MAFPN) encoder (such as...). Figure 2 (As shown). Input multi-scale image features {F3, F4, F5} with class embeddings T. Image features F extracted from the backbone. image Send to HRC for multi-scale modeling: HRC employs hierarchical convolution (3×3 standard convolution) + local residual connections to enhance information about small targets, generating features at different scales and ensuring the coexistence of local details and global information. Finally, the cross-attention score between image features and class embeddings is calculated.

[0077]

[0078] The second step involves using a text-guided attention multilevel fusion (TGAMF) decoder (such as...). Figure 3 (As shown). Input: Multilevel text embeddings{T high T mid， T low} and decoder query Q.

[0079] Multi-level guidance strategy:

[0080] High-level decoder (Layer 1): Fusion T high Focus on semantic-level information.

[0081] Mid-layer decoder (layers 2-3): Fusion T mid Balancing semantics and details.

[0082] Low-level decoder (layers 4-6): Fusion T low Enhance the detailed features of small objects.

[0083] Update rules:

[0084] If i = 1:

[0085] Q′ i =Q i +MHCA(Q i T high-level T high-level )

[0086] If i = 2 or 3:

[0087] Q′ i =Q i +MHCA(Q i T mid-level Tmid-level )

[0088] If i = 4 or 5 or 6:

[0089] Q′ i =Q i +MHCA(Q i T low-level T low-level )

[0090] MHCA stands for Multi-Head Cross-Attention Module. The model uses a 6-layer decoder, with layers 4, 5, and 6 all connected to T. low MHCA is performed. That is, the latter three layers of the decoder focus on interacting with the low-level class embeddings, so that the model can make more use of the rich low-level features to identify small targets in the final output.

[0091] Step four: Perform model training and inference.

[0092] (1) Training phase:

[0093] The first step involves inputting images and a category vocabulary, and then extracting features using an image encoder and a text encoder.

[0094] The second step is to enhance foreground features by fusing multi-scale image features with class embeddings using MAFPN;

[0095] The third step involves TGAMF decoding layer by layer, combining multi-level text guidance to generate bounding boxes and similarity scores.

[0096] The fourth step is to calculate the region-text contrast loss and backpropagate to optimize the parameters.

[0097] (2) Reasoning stage:

[0098] The first step is to input an image and any category name, and generate bounding boxes and similarity scores in real time;

[0099] The second step is to filter out low-confidence detection results using a threshold and output the final detection box.

[0100] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A real-time open vocabulary object detection model AOV-RTDETR for aerial images, characterized in that, Includes the following steps: Step 1: ResNet-50 is used as the image encoder, and large strip convolutions are introduced to enhance the feature perception capability of high aspect ratio targets. Step two: The CLIP text encoder is used to convert category words into class embeddings, and these class embeddings are then fused with image features; Step 3: Design a region-text contrast loss head, use region-text similarity to replace the traditional category regression loss, and use Hungarian assignment to match predicted boxes with ground truth boxes; Step 4: Fuse multi-scale image features using a multi-scale spatial pyramid attention (MAFPN) encoder and perform cross-modal fusion with class embeddings; Step 5: The text-guided attention multi-level fusion (TGAMF) decoder is used to decode image features layer by layer, and class embeddings are combined to enhance the detection capability of small targets. Step 6: Optimize network parameters based on region-text contrast loss, IOU loss, and L1 loss to improve target detection accuracy and real-time inference capability.

2. The AOV-RTDETR aerial open vocabulary target detection model according to claim 1, characterized in that: The large strip convolution in step one includes sequentially orthogonal strip convolution structures that expand the receptive field in the horizontal or vertical direction in order to capture targets with a high aspect ratio.

3. The aerial open vocabulary target detection model AOV-RTDETR according to claim 1, characterized in that: In step two, the CLIP text encoder generates class embeddings in the form: T = TextEncoder(C) ∈ Rn × d, where n is the number of classes and d is the dimension of the class embeddings.

4. The aerial open vocabulary target detection model AOV-RTDETR according to claim 1, characterized in that: The region-text contrast loss in step three is calculated in the following way: The first step is to calculate the similarity between image features and class embeddings, and then calculate the matching cost based on the similarity. The second step is to use the Hungarian assignment algorithm to assign labels and select the minimum matching cost. The third step is to combine the Varifocal loss (VFLoss) to weight the matching results in order to enhance the alignment accuracy between regions and text.

5. The AOV-RTDETR aerial open vocabulary target detection model according to claim 1, characterized in that: The Multi-Scale Spatial Pyramid Attention (MAFPN) encoder in step four uses a hierarchical residual convolutional module (HRC) to extract multi-scale features layer by layer, and performs feature fusion of image and text through a cross-attention module. MAFPN improves detection capability through the following steps: The first step is to extract image features from different scales to ensure that multi-level image information is fully utilized; The second step is to fuse features of different scales with class embeddings through a cross-attention mechanism, thereby improving the detection capability of small targets; The third step is to optimize the feature extraction process through a hierarchical convolutional structure to enhance the model's ability to perceive fine-grained targets. The fourth step is to effectively enhance the expression of image features by utilizing a hierarchical convolutional structure, and further fuse low-level and high-level features.

6. The aerial open vocabulary target detection model AOV-RTDETR according to claim 1, characterized in that: The Text-Guided Attention Multilevel Fusion (TGAMF) decoder in step five enhances the preservation of details in low-level features through multi-level text guidance, enabling the model to more sensitively locate small targets. TGAMF achieves effective fusion of low-level features and class embeddings through multi-level text-guided cross-attention modules, improving the detection capability of small targets. The following steps optimize small target detection: The first step is to use a multi-level text-guided cross-attention module to enable better fusion of image features at each layer with class embeddings; The second step is low-level feature fusion: by utilizing the interaction between low-level image features and class embeddings, we can capture detailed information and improve the detection capability of small targets. The third step is mid-level feature fusion: balancing semantic and detail information in the mid-level decoder to ensure accurate target localization. The fourth step is high-level feature fusion: high-level decoders are combined with high-level class embeddings to improve overall detection performance, especially to enhance target recognition capabilities in complex backgrounds.

Images