Open aerial photography target detection automatic labeling method based on large model

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an open aerial photography target detection automatic labeling method based on a large model, which comprises three stages of data preprocessing, instance level sentence generation and data post-processing, sentence labels containing self-description, relative positions and absolute positions are generated in combination with multiple rounds of interaction, and accurate association of the labels and instances is achieved based on attribute similarity matching. According to the open aerial photography target detection automatic labeling method based on the large model, the problems of insufficient data scale, single semantic granularity and high labeling cost in aerial photography image labeling are solved, the generalization ability and robustness of an open set target detection model are remarkably improved, and the open aerial photography target detection automatic labeling method is suitable for the fields of disaster rescue, smart cities, environment monitoring and the like.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an automatic labeling method for open aerial photography target detection based on a large model. Background Art

[0002] Object detection is key to accurately identifying and locating targets in aerial imagery. It plays a vital role in a variety of applications, including environmental monitoring, urban planning, and rescue operations. Specific application scenarios for object detection include, but are not limited to, the following: Disaster response: Rapidly identifying affected areas (such as collapsed buildings) to guide the allocation of rescue resources; Urban planning: Dynamically monitoring traffic flow and infrastructure distribution to optimize urban spatial layout; Agricultural management: Accurately identifying crop types and pest and disease areas to improve farmland management efficiency; Environmental monitoring: Tracking ecological changes such as deforestation and water pollution to support sustainable development decisions.

[0003] In the field of artificial intelligence, data is the cornerstone of model capabilities. Especially for open-set object detection tasks, large-scale, high-quality annotated data directly determines the model's generalization and robustness. Taking the natural image field as an example, GroundingDINO v1.5 achieved accurate detection of arbitrary text input by training on 20 million annotated data, verifying the principle that "data size determines performance ceiling." However, in the field of aerial imagery, the scarcity of datasets and the high cost of annotation have severely constrained technological development, mainly reflecting the following two difficulties:

[0004] 1. Data size gap: The largest aerial visual localization dataset (OPT-RSVG) contains only 48,000 annotated pairs, less than one-sixth of the natural image dataset, making it difficult for models to learn the diverse features of complex scenes.

[0005] 2. Labeling costs are rising exponentially: Aerial images contain densely packed targets, small in size, and subject to strong background interference. Manual labeling is time-consuming and susceptible to subjective factors, resulting in an error rate of up to 5%-10%.

[0006] In addition, current mainstream methods rely on closed annotated datasets and manual templates, which have the following bottlenecks:

[0007] ① Single semantic granularity: Labeling is limited to the word level (e.g., "vehicle"), lacking phrase ("small white vehicle") and sentence-level descriptions ("white ambulance in the right-turn lane at the intersection"). This prevents the model from understanding fine-grained attributes (color, location, context). In disaster relief, recognizing only "vehicle" makes it impossible to distinguish between an "ambulance" and an "ordinary car," delaying the localization of critical targets.

[0008] Limited scene complexity: Existing datasets (such as DIOR-RSVG) impose a strict limit on the number of instances per image (less than five objects of the same type), failing to reflect the dense, multi-class distribution of objects in real-world scenarios. In urban traffic monitoring, complex intersections can contain dozens of vehicles, pedestrians, and traffic facilities, making existing models prone to missed or false positives.

[0009] ③ Insufficient labeled data size and quality: Low-altitude equipment (such as drones) requires real-time recognition of dynamic targets (such as birds and temporary obstacles) in complex environments. However, the largest existing aerial visual positioning datasets (such as OPT-RSVG) contain only approximately 50,000 labeled pairs, far fewer than those used in the natural image domain (e.g., Flickr30k, which has 300,000 labeled pairs). This results in weak model generalization. In agricultural management, small sample sizes make it difficult for models to adapt to the diverse crop morphologies across different regions, resulting in significant fluctuations in recognition accuracy.

[0010] Data shortage and high labeling costs form a "vicious cycle", leading to two core contradictions:

[0011] ① The disconnect between technical capabilities and application needs: Closed detection models can only identify a limited number of categories, while real-world scenarios require processing unknown categories and dynamic descriptions, such as new rescue equipment in sudden disasters.

[0012] ② Data quality and model performance constrain each other: small-scale, low-semantic-density data limits model learning, while low-performance models further hinder data labeling efficiency, forming a vicious cycle.

[0013] Therefore, the following problems exist in the technology now:

[0014] 1. Model generalization bottleneck caused by data loss: Existing aerial image datasets are small (for example, OPT-RSVG has only 50,000 annotation pairs) and have single semantics (only vocabulary-level labels). They cannot cover the diversity and dynamics of real-world scenes, resulting in insufficient model learning during training.

[0015] 2. Degradation of aerial photography detection capabilities in open environments: Data loss and semantic sparsity directly lead to decreased model performance in open environments; Failure of fine-grained detection: The inability to distinguish between a "white ambulance" and an "ordinary car" delays the positioning of key targets in disaster relief; Missed detection in dense scenes: When densely packed with vehicles and pedestrians intersect in urban transportation hubs, the false detection rate of traditional models surges.

[0016] To address the above problems, this paper proposes an automatic annotation method for open aerial target detection based on a large model, aiming to lay a data foundation for open set aerial target detection. Summary of the Invention

[0017] In order to break the bottleneck of model generalization caused by data missing and solve the technical problem of degradation of aerial photography detection capabilities in open environments, the present invention proposes an automatic labeling method for open aerial photography target detection based on a large model, which can help to quickly construct new data sets, covering three levels of semantic descriptions of words, phrases, and sentences, and significantly improve the model's adaptability to unknown categories and complex scenes; through multi-level semantic labeling and automated scene segmentation technology, the model's understanding of fine-grained attributes and complex spatial relationships is enhanced, significantly improving the robustness of open environment detection.

[0018] The technical solutions provided by the present invention are as follows:

[0019] The large-model-based automatic labeling method for open aerial photography target detection proposed in the present invention includes three stages: data preprocessing, instance-level sentence generation, and data post-processing.

[0020] S1. Data preprocessing: Data preprocessing aims to simplify complex aerial images so that VLMs can effectively focus on relevant areas.

[0021] Specifically, the present invention processes images to extract three key components: instance regions, foreground regions, and partial instance attributes.

[0022] ① Instance region: Based on the coordinates provided in the detection annotation, the instance region can be easily obtained by cropping the sub-image.

[0023] ② Foreground region: Considering that aerial images are densely populated with instances and have a significant background, the present invention uses Algorithm 1 to effectively separate the foreground region. Specifically, the present invention calculates the maximum enclosing rectangle of the object bounding box to isolate multiple object clusters in each image.

[0024] ③ Partial instance attributes: This invention utilizes instance attributes as components to generate different titles. It focuses on six basic attributes: category, size, color, geometry, relative position, and absolute position. While categories are predefined, size and absolute position attributes are determined based on manual rules due to their inherent subjectivity and spatial complexity.

[0025] The following is a detailed description of the instance region and Algorithm 1: Instance region: Use the lightweight detection model YOLO-World-Tiny to locate the target, and output the bounding box set B = {b1, b2, ..., b n The instance subimage is cropped based on the detection bounding box coordinates, and the output size is unified to 128×128. The lightweight detection model YOLO-World-Tiny mentioned above can be replaced by other lightweight detection models.

[0026]

[0027]

[0028] S2. Instance-level sentence generation: This step aims to generate three sentence titles with different levels of detail for each instance, as well as additional instance attributes, by interacting with VLMs. The interaction between each instance and VLMs is divided into four rounds: ① Introducing the entire annotation workflow to VLMs. ② Providing instance region images as well as instance category and size attributes prompts VLMs to extract color and geometric attributes and then generate self-describing titles. ③ Submitting a foreground region image containing the instance allows VLMs to determine the relative position attributes of the instance based on the surrounding context, expanding the previous title into a relative position title. ④ Providing absolute position attributes prompts VLMs to integrate this information into the previously generated title, thereby creating an absolute position title. Each interaction uses a JSON template to adjust the VLMs output. Therefore, each instance provides three different sentence titles with different description levels and a comprehensive set of six attributes. The VLMs model selected by the present invention is InternVL25-38B-AWQ, which can be replaced by other open source models.

[0029] S3. Data post-processing: Based on the attributes obtained from the previous steps, the present invention generates three phrase-level titles for each instance using a combination of category, color, and size attributes, thereby generating six unique titles for each instance. However, due to instance similarity, titles with fewer attributes usually correspond to multiple instances. By utilizing the attribute-based title and the attribute information recorded for each instance, the present invention effectively establishes title-instance pairs by comparing the attribute similarity between the title and the instance. The present invention uses Sentence-BERT to calculate attribute similarity, and other text feature similarity calculation methods can be used instead.

[0030] Compared with related technologies, the large-scale model-based open aerial photography target detection and automatic labeling method proposed in this invention has the following beneficial effects:

[0031] Data-driven model evolution: Provide the model with sufficient, high-quality training samples to break the limitations of closed-loop detection.

[0032] Open capabilities are actually implemented: supporting zero-sample migration and dynamic command response, enabling high-value scenarios such as disaster relief and smart cities.

[0033] Reduce costs and increase efficiency in the industry ecosystem: Accelerate the large-scale application of aerial image analysis technology with low-cost, high-efficiency labeling solutions.

[0034] In summary, this invention, through multi-level semantic annotation and automated scalability, not only addresses the core pain points of data scarcity and high annotation costs in the field of aerial imagery, but also directly promotes breakthroughs in the performance of open-set detection models. The proposed large-scale model-based automatic annotation method for open aerial object detection is applicable to fields such as remote sensing image analysis, drone monitoring, urban planning, and disaster response. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flowchart for automatic data generation in the large-model-based open aerial photography target detection and automatic labeling method proposed by the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] like Figure 1 As shown, Figure 1 A flowchart is automatically generated for the data in the automatic labeling pipeline. The labeling process consists of three main components: data preprocessing, instance-level sentence caption generation, and data post-processing. Each aerial image undergoes a comprehensive labeling process, including attribute extraction, caption generation at different levels of detail using VLMs, and accurate matching of caption-instance pairs based on attribute similarity.

[0038] The large-scale model-based open aerial photography target detection and automatic labeling method proposed in this invention has the following specific implementation process:

[0039] Input image: An aerial image (resolution 1024×1024) from the DIOR dataset, containing multiple object instances. The bounding box set B = {b1, b2, b3} corresponds to three object instances.

[0040] (1) Foreground area extraction (Algorithm 1):

[0041] Step 1: Dynamically scale the bounding box. For each detection box b i Calculate area A i And set the scaling factor s i :

[0042] b1: size 30×30 (area A1=900), scaling factor s1=1.2 → expanded to 36×36.

[0043] b2: size 25×25 (area A2=625), scaling factor s2=1.2 → expanded to 30×30.

[0044] b3: size 15×15 (area A3=225), scaling factor s3=1.55 → expanded to 22.5×22.5 (rounded to 23×23).

[0045] Step 2: Merge overlapping areas

[0046] Initial unmerged box set: U = {b1′, b2′, b3′} (expanded bounding box).

[0047] Merging process: Starting from b1′, detect overlap with b2′ and merge to generate region r1. b3′ has no overlap and is retained independently as region r2. Output foreground region set: R = {r1, r2}.

[0048] (2) Instance-level sentence generation:

[0049] Input instance region: the cropped target instance subimage (r1 and r2).

[0050] By prompting VLMs to interact, the results are obtained:

[0051] Basic description: r1: "color one", "shape one", "object one". r2: "color two", "shape two", "object two".

[0052] Position enhancement: Based on the absolute position grid code, r1 is located in the "lower right" grid → updated to "red circular object 1 in the lower right grid." r2 is located in the "center left" grid → updated to "small silver object 2 in the center left grid."

[0053] (3) Data post-processing and annotation matching:

[0054] Phrase compression:

[0055] r1: Extract key attributes to generate the phrase “medium-sized red object - lower right”.

[0056] r2: Generate the phrase "small silver object two-center".

[0057] Dynamic matching:

[0058] The present invention uses a combination of category, color and shape attributes to generate three phrase-level titles for each instance, thereby generating six unique titles for each instance. Using the Sentence-BERT model, the present invention effectively establishes title-instance pairs by comparing the attribute similarity between titles and instances.

[0059] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A large-scale model-based open aerial photography target detection and automatic labeling method, characterized by: The following steps are involved: S1. Data preprocessing: Extract target instance regions, foreground regions, and target attributes, where target attributes include category, size, color, geometric shape, relative position, and absolute position; S2. Instance-level sentence annotation generation: Utilizes the vision-language model to generate sentence annotations containing self-description, relative position, and absolute position information through multiple rounds of interaction; S3. Data post-processing: Generate phrase-level annotations based on instance attributes, and use attribute matching algorithms to match the generated annotations with the corresponding target instances.

2. The large model-based open aerial photography target detection and automatic labeling method according to claim 1 is characterized in that: In the data preprocessing step S1, the object size attribute is classified according to the ratio of the object bounding box area to the image area and according to a preset threshold.

3. The large model-based open aerial photography target detection and automatic labeling method according to claim 1 is characterized in that: In the data preprocessing step S1, the image is divided into several fixed areas to determine the absolute position attributes of the target.

4. The large model-based open aerial photography target detection and automatic labeling method according to claim 1 is characterized in that: In the instance-level sentence annotation generation step S2, self-descriptive sentences, relative position sentences and absolute position sentences are generated respectively through at least three rounds of interaction, and the output is standardized using a JSON template.

5. The large model-based open aerial photography target detection and automatic labeling method according to claim 1 is characterized in that: In the data post-processing step S3, automatic matching between annotations and instances is achieved by comparing the similarity between the attributes explicitly mentioned in the generated annotations and the attributes of the target instance.

Images