Visual language model adaptation method and device for aerial target detection

Abstract

The present disclosure relates to the field of computer vision and deep learning, and provides a visual language model adaptation method and device for aerial target detection, which comprises: acquiring a UAV aerial image dataset and target text; loading pre-training weights of a visual language model, freezing a visual backbone network and a text backbone network of the visual language model; generating a background vocabulary table, converting the background vocabulary through the text backbone network and a corresponding projection layer into an input of a background suppression gate module, replacing an original query selection module in the visual language model with the background suppression gate module to obtain a final visual language model; adding a contrast loss between target text features and background text features to the original training loss of the visual language model to determine a model training loss constraint; and training the final visual language model using the UAV aerial image dataset and the model training loss constraint. The present disclosure can reduce missed detection and false detection, and has high efficiency and good interpretability.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and deep learning technology, and in particular to a visual language model adaptation method and apparatus for aerial target detection. Background Technology

[0002] In recent years, traditional research on UAV image target detection has mainly focused on addressing specific challenges in this field, particularly the prevalence of small targets and extremely low foreground-to-background ratios. Some techniques attempt to improve the discrimination ability of small targets through strategies such as multi-scale feature refinement or contextual feature enhancement. Others focus on mitigating background by explicitly separating foreground regions. Despite progress on specific datasets, these existing techniques suffer from a fundamental flaw: isolated, closed-set design. They are tightly coupled with their training data distribution, not only in terms of predefined target categories but also in terms of specific scene features (such as the natural environment and weather conditions). Therefore, adapting them to new environments or targets requires a cumbersome and resource-intensive modification process. This lack of flexibility makes customizing UAV models for various specific task requirements extremely inefficient.

[0003] Visual language models (VLMs), with their powerful image-text alignment and generalization capabilities, offer a viable path to overcome the bottlenecks of traditional models. Without requiring the design of complex domain-specific modules, the powerful representational capabilities of VLMs can be guided for specific UAV tasks through extremely simple adaptation mechanisms. This approach points to an efficient and elegant paradigm: leveraging powerful base models combined with widely applicable adaptation strategies to solve specific UAV detection challenges. In existing technologies, VLM detectors such as GLIP, YOLO-World, and GroundingDINO identify targets through visual-language matching. To apply these powerful large-scale models to downstream tasks, the Parametric Efficient Fine-Tuning (PEFT) method has gained popularity among researchers and engineers. By fine-tuning only a small subset of parameters, PEFT has achieved significant success in adapting VLMs to routine ground vision tasks, effectively mitigating the high overhead and catastrophic forgetting problems associated with full-parameter fine-tuning. However, experiments on VisDrone and UAVDT demonstrate that standard methods for uniformly adjusting existing features, such as LoRA and Adapter, perform poorly on UAV images. Summary of the Invention

[0004] This disclosure aims to address at least one of the problems existing in the prior art by providing a visual language model adaptation method and apparatus for aerial target detection.

[0005] One aspect of this disclosure provides a visual language model adaptation method for aerial target detection, including: Acquire a dataset of drone aerial images, perform image preprocessing on the drone aerial image dataset, and determine the target text to be detected; Load the pre-trained weights of the visual language model, and freeze the model weights of the visual backbone network and the text backbone network of the visual language model. The visual language model further includes a query selection module, a projection layer, a multimodal encoder, a multimodal decoder, and a detection head. Generate a background vocabulary list, which includes background vocabulary for multiple background types; The background suppression gating module is initialized by converting the background words in the background vocabulary into background text features through the text backbone network and its corresponding projection layer, which are then used as input to the background suppression gating module. The original query selection module in the visual language model is replaced by the background suppression gating module to obtain the final visual language model. The background suppression gating module includes multiple feature gates for suppressing and regulating the background. Based on the original training loss of the visual language model, a contrast loss between target text features and background text features is added to determine the model training loss constraints; The final visual language model is trained using the drone aerial image dataset and the model training loss constraints.

[0006] Optionally, the generation of the background vocabulary includes: Define the range of background vocabulary and multiple background types; A large language model is used to generate multiple alternative background vocabulary types within the given background vocabulary range; The candidate background vocabulary was validated using a visual language model; The final background vocabulary is determined based on the candidate background vocabulary through manual verification, and the background vocabulary list is generated.

[0007] Optionally, the multiple feature gates include a protection gate, a suppression gate, and a candidate adjustment gate; The protection gate is used to protect the local features of the image when the target text features are highly similar to the local features of the input image. The suppression gate is used to suppress the local features of the image when the background text features of the background vocabulary and the local features of the input image have a high similarity. The candidate adjustment gate is used to adjust the scores corresponding to the protection gate and the suppression gate as a whole to ensure coordination.

[0008] Optionally, the background suppression gating module is used to perform semantic gating query selection according to the following formula: ; ; ; in, This represents the adjusted similarity matrix. This represents the adjusted maximum similarity score; express A function used to select the maximum value of each element in a matrix; This represents the selected candidate query index. This indicates the top score with the highest adjusted maximum similarity score. One index, This indicates the set number of candidate queries. This indicates a semantic gating module. Represents the original image-text similarity matrix and , For image feature representation, For the feature representation of the target text The maximum foreground similarity score for each local query in the image and , The maximum background similarity score for local queries of each image and , The background image-text similarity matrix and , For background text feature representation, superscript Indicates transpose; Semantic gating module Used for regulation according to the following formula: ; ; ; ; ; in, and These represent the output activation values ​​of the protection gate and the suppression gate, respectively. and The initial adjustment signal value and the final candidate adjustment vector of the candidate adjustment gate are respectively... It is the Sigmoid activation function. The scaling hyperparameter represents the score adjustment. The Hadamard product of a matrix. They are respectively Protective weights, suppression weights, and adjustment weights. They are respectively Protective weights, suppression weights, and adjustment weights. These are respectively the protection gate bias term, the suppression gate bias term, and the candidate adjustment gate bias term.

[0009] Optionally, the contrast loss between the target text features and the background text features is expressed as: ; in, This is a word-level contrastive loss function used to amplify the distance distribution of different semantic features. for The InfoNCE loss between each term in the word, This is the joint feature representation obtained by concatenating background text features and target text features. , For feature splicing operations, Background text features Features of the target text.

[0010] Optionally, the visual language model is any one of the GroundingDINO model and its subsequent derivative models.

[0011] Another aspect of this disclosure provides a visual language model adaptation device for aerial target detection, comprising: The acquisition module is used to acquire a dataset of drone aerial images, perform image preprocessing on the drone aerial image dataset, and determine the target text to be detected. The first initialization module is used to load the pre-trained weights of the visual language model and freeze the model weights of the visual backbone network and the text backbone network of the visual language model. The visual language model also includes a query selection module, a projection layer, a multimodal encoder, a multimodal decoder, and a detection head. A generation module is used to generate a background vocabulary list, which includes background vocabulary for multiple background types. The second initialization module is used to initialize the background suppression gating module. It converts the background words in the background vocabulary into background text features through the text backbone network and its corresponding projection layer, and uses these features as input to the background suppression gating module. It replaces the original query selection module in the visual language model with the background suppression gating module to obtain the final visual language model. The background suppression gating module includes multiple feature gates for suppressing and controlling the background. The loss determination module is used to add a contrast loss between the target text features and the background text features to the original training loss of the visual language model, and to determine the training loss constraints of the model. The training module is used to train the final visual language model using the UAV aerial image dataset and the model training loss constraints.

[0012] Another aspect of this disclosure provides an electronic device comprising: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the visual language model adaptation method for aerial target detection described above.

[0013] Another aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the visual language model adaptation method for aerial target detection described above.

[0014] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the visual language model adaptation method for aerial target detection described above.

[0015] Compared with the prior art, this disclosure has the following advantages: it changes the "passive ignoring of background" to "actively identifying and suppressing background", which can reduce false negatives and false positives and has initiative; the background word embedding can be pre-computed, and only matrix multiplication and simple gating operations are involved during inference, which hardly increases the latency. In addition, only a small number of trainable parameters are introduced, which has high parameter and inference efficiency and high efficiency; the suppression of noise is achieved through the intuitive concept of background vocabulary, which has good interpretability. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with the corresponding pictures in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0017] Figure 1 A flowchart illustrating a visual language model adaptation method for aerial target detection provided in one embodiment of this disclosure; Figure 2 A schematic diagram illustrating the process of generating a background vocabulary list for another embodiment of this disclosure; Figure 3A schematic diagram of the final visual language model provided for another embodiment of this disclosure. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the various embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this disclosure to facilitate a better understanding of the disclosure. However, the technical solutions claimed in this disclosure can be implemented even without these technical details and with various variations and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of this disclosure. The various embodiments can be combined with and referenced by each other without contradiction.

[0019] One embodiment of this disclosure relates to a visual language model adaptation method for aerial target detection. By actively focusing on background noise and making targeted adjustments, the model is guided to avoid focusing on the background, thereby achieving targeted adaptation and good adaptation performance.

[0020] like Figure 1 As shown, the visual language model adaptation method for aerial target detection provided in this embodiment includes steps S1 to S6.

[0021] Step S1: Obtain the drone aerial image dataset, perform image preprocessing on the drone aerial image dataset, and determine the target text to be detected.

[0022] Specifically, step S1 is used to prepare the drone aerial image dataset, complete the selection and image preprocessing of the drone aerial image dataset, and establish the target text to be detected, i.e., the vocabulary of the target to be detected.

[0023] Image preprocessing may include, but is not limited to, image enhancement and image normalization operations.

[0024] Step S2: Load the pre-trained weights of the visual language model, and freeze the model weights of the visual backbone network and text backbone network of the visual language model. The visual language model also includes a query selection module, a projection layer, a multimodal encoder, a multimodal decoder, and a detection head.

[0025] Specifically, step S2 is used to initialize the visual language model. The visual language model can be any type of visual language model with a query selection module. For example, the visual language model can be any of the GroundingDINO model and its subsequent derivative models.

[0026] Step S3: Generate a background vocabulary list, which includes background vocabulary for multiple background types.

[0027] Specifically, when generating a background vocabulary list, a general range of background vocabulary can be established first. Then, candidate background vocabulary can be generated through a large language model. Finally, the background vocabulary can be determined through visual language model verification and manual screening to generate the background vocabulary list.

[0028] For example, combined Figure 2 Generate a background vocabulary, including the following content.

[0029] Scope Definition: Define the scope of background vocabulary and multiple background types. Specifically, the scope of background vocabulary and background types can be determined through text analysis of image content descriptions or by manual specification. For example, multiple background types can involve various common backgrounds, including: street scenes, such as buildings; natural landscapes, such as grass; atmospheric conditions, such as fog.

[0030] Vocabulary Generation: A Large Language Model (LLM) is used to generate multiple candidate background vocabulary words for different background types within the existing vocabulary range. Specifically, when generating candidate background vocabulary words, a large language model, such as a mainstream commercial large language model, can be used to generate multiple candidate background vocabulary words for different background types based on the existing vocabulary range. The mainstream commercial large language model can be the web version of GPT or Gemini. For example, the prompt words provided to the large language model could be: generate a common background vocabulary list for a drone aerial photography dataset, centered around three categories: street scene, natural landscape, and atmospheric conditions. In this case, the candidate background vocabulary words generated by the large language model could include: buildings, sidewalks, walls, fences, stones, dust, sand, rivers, mountains, trees, grass, shrubs, sky, water, clouds, rain, shadows, night, ground surface, etc.

[0031] Model Validation: The candidate background words are validated using a visual language model. Specifically, when validating candidate background words, the drone aerial image dataset obtained in step S1 can be sampled. Based on the sampled images, a visual language model such as DINO-X is used to detect whether the target corresponding to the candidate background word can be identified. Candidate background words for which the visual language model cannot identify the corresponding target are eliminated. For example, when validating the candidate background words, prompt words corresponding to the input image can be provided to the visual language model. These prompt words could be: "Please find targets such as buildings and sidewalks in the image." In this case, the visual language model can detect targets corresponding to the candidate background words such as road, tree, fog, and river.

[0032] Manual Verification: Based on the candidate background words, manual verification determines the final background words and generates a background vocabulary list. Specifically, during manual verification, candidate background words with ambiguous meanings or unclear referents can be eliminated to determine the final background words and generate the background vocabulary list. For example, during manual verification, it can be verified whether the visual language model can detect the target corresponding to the candidate background word, whether the target detected by the visual language model is consistent with the actual word meaning of the corresponding candidate background word, and whether the semantics of the candidate background word are clear. If the visual language model can detect the target corresponding to the candidate background word, and the target detected by the visual language model is consistent with the actual word meaning of the corresponding candidate background word, and the semantics of the candidate background word are relatively clear, then the corresponding candidate background word is considered to have passed the model verification and manual verification and can be included as a background word in the background vocabulary list. For example, the final generated background vocabulary list can include background words such as building, road, pedestrian crossing, wall, fence, dust, stone, texture, and background.

[0033] This implementation generates a background vocabulary, allowing the model to suppress noise through the intuitive concept of background words in the vocabulary, thus providing good interpretability.

[0034] Step S4: Initialize the background suppression gating module. The background words in the background vocabulary are converted into background text features through the text backbone network and its corresponding projection layer, which are used as inputs to the background suppression gating module. The original query selection module in the visual language model is replaced by the background suppression gating module to obtain the final visual language model. The background suppression gating module includes multiple feature gates for suppressing and regulating the background.

[0035] Specifically, when initializing the background suppression gating module, a direct feature loading mechanism is established. The background words in the background vocabulary are transformed into background text features through the text backbone network and its corresponding projection layer, and then directly used as an input to the background suppression gating module. After that, the parameters of multiple feature gates are initialized, and the original query selection module in the visual language model is replaced by the background suppression gating module.

[0036] For example, combined Figure 3 Multiple feature gates include protection gates, suppression gates, and candidate adjustment gates.

[0037] The protection gate is used to protect local image features when the target text features are highly similar to the local features of the input image.

[0038] Suppression gates are used to suppress local image features when the background text features of the background vocabulary and the local features of the input image have high similarity.

[0039] Candidate adjustment gates are used to adjust the scores corresponding to the protection gate and the suppression gate as a whole to ensure coordination.

[0040] For example, the background suppression gating module is used to perform semantic gating query selection according to the following formula: ; ; ; in, This represents the adjusted similarity matrix. This represents the adjusted maximum similarity score; express This function is used to select the maximum value of each element in a matrix. This represents the selected candidate query index. This indicates the top score with the highest adjusted maximum similarity score. One index, This indicates the set number of candidate queries. This indicates a semantic gating module. Represents the original image-text similarity matrix and , For image feature representation, For the feature representation of the target text The maximum foreground similarity score for each local query in the image and , The maximum background similarity score for local queries of each image and , The background image-text similarity matrix and , For background text feature representation, superscript This indicates transpose.

[0041] Semantic gating module Used for regulation according to the following formula: ; ; ; ; ; in, and These represent the output activation values ​​of the protection gate and the suppression gate, respectively. and The initial adjustment signal values ​​and final candidate adjustment vectors of the candidate adjustment gates are respectively... It is the Sigmoid activation function. The scaling hyperparameter represents the score adjustment. The Hadamard product, representing a matrix, is an element-wise multiplication. They are respectively Protective weights, suppression weights, and adjustment weights. They are respectively Protective weights, suppression weights, and adjustment weights. These are respectively the protection gate bias term, the suppression gate bias term, and the candidate adjustment gate bias term.

[0042] For example, the structure of the final visual language model is as follows: Figure 3 As shown, the model includes the original text backbone network, visual backbone network, projection layer, multimodal encoder, multimodal decoder, detector head, and a background suppression gating module that replaces the original query selection module in the visual language model by adopting a background-guided query selection mechanism. The text backbone network extracts target text features based on target text prompts and background text features based on background text prompts. The visual backbone network extracts image features from the input image. The projection layer converts the target text features and image features into target text feature representations and image feature representations acceptable to the multimodal encoder through cross-modal alignment, and also converts background text features into a background text feature representation as input to the background suppression gating module through cross-modal alignment. The multimodal encoder fuses and enhances the target text feature representations and image feature representations output by the projection layer, outputting enhanced image features, key vector K, and value vector V. The background suppression gating module utilizes the background text feature representation output by the projection layer to perform semantically gated query selection using a background-guided query selection mechanism. The semantically gated query selection result is then used to reorder and select the original query vector formed from the enhanced image features output by the multimodal encoder according to the query score, generating the final effective query vector Q. The multimodal decoder outputs the prediction result based on the effective query vector Q output by the background suppression gating module and the key vector K and value vector V output by the multimodal encoder. The detection head characterizes the prediction result output by the multimodal decoder, such as outputting the target bounding box corresponding to the target text and its corresponding confidence score.

[0043] Step S5: Based on the original training loss of the visual language model, add a contrast loss between the target text features and the background text features to determine the model training loss constraints.

[0044] Specifically, step S5 is used to establish model training loss constraints. Since this implementation improves the visual language model by replacing the original query selection module with a background suppression gating module, and by directly using background words from the background vocabulary as input to the background suppression gating module after being converted into background text features through the text backbone network and its corresponding projection layer, a contrast loss between target text features and background text features is added to the original training loss of the visual language model to further enhance the understanding and expressive capabilities of the final visual language model.

[0045] For example, when the visual language model is any one of the GroundingDINO model and its subsequent derivative models, the original training loss of the visual language model is the target detection loss of the GroundingDINO series, including the L1 loss related to bounding box regression, the GIOU loss, and the Focal Loss related to classification.

[0046] For example, the contrast loss between the target text features and the background text features added in this embodiment is expressed as: ; in, This is a word-level contrastive loss function used to amplify the distance distribution of different semantic features. for The InfoNCE loss between each term in the word, This is the joint feature representation obtained by concatenating background text features and target text features. , For feature splicing operations, Background text features Features of the target text.

[0047] Step S6: Using the drone aerial image dataset and model training loss constraints, train the final visual language model.

[0048] Specifically, step S6 is used to adapt and train the final visual language model. The final visual language model is obtained by training the drone aerial image dataset obtained in step S1, replacing the original query selection module in the visual language model with a background suppression gating module. During the training process, the output of the original query selection module in the visual language model is modified using the background suppression gating module, and the model training loss constraints determined in step S5 are used to optimize the final visual language model.

[0049] To enable those skilled in the art to better understand the above embodiments, a specific example is provided below.

[0050] A visual language model adaptation method for aerial target detection includes the following steps.

[0051] 1. Dataset Preparation. Complete the selection of the dataset, data preprocessing, and dataset partitioning.

[0052] 1.1 The selected image dataset must contain real images and corresponding object annotation information. To verify the model's performance and robustness, this example selects the publicly available VisDrone and UAVDT datasets for drone aerial image target detection tasks, and unifies the image annotations into COCO JSON format to form a drone aerial image dataset.

[0053] 1.2 Image preprocessing is performed on the UAV aerial image dataset. Image preprocessing includes image enhancement and image normalization operations. Image enhancement operations include resizing and random flipping. On the VisDrone and UAVDT datasets, this example proportionally resizes the images to 1333×800 pixels. Subsequently, on the VisDrone and UAVDT datasets, this example performs probabilistic random horizontal and vertical flipping operations on the images, and finally performs image normalization.

[0054] 1.3 Target text preparation: Convert the categories officially labeled in the VisDrone dataset and UAVDT dataset into a text list to obtain the target text to be detected, such as car, truck, bus.

[0055] 2. Design the visual language model. In this example, the initial visual language model is mmGroundingDINO-T based on mmdet software library version 3.3.0, and the model weights of the visual backbone network SwinTransformer and the text backbone network BERT are frozen.

[0056] 3. Generate a background vocabulary list. From three aspects, street view, natural terrain, and atmospheric conditions, background vocabulary is generated through the GPT web interface. Then, it is checked by DINO-X and manually filtered to obtain 20 background vocabulary words, which are then generated into a background vocabulary list. These 20 background vocabulary words include: buildings, roads, sidewalks, walls, fences, dust, stones, sand, rivers, mountains, trees, grass, shrubs, sky, clouds, rain, shadows, night, background, and texture.

[0057] 4. Initialize the background suppression gating module. The background words in the background vocabulary are converted into background text features through a text backbone network and used as an input to the background suppression gating module. The language-guided query selection module in mmGroundingDINO-T is replaced with the background suppression gating module provided in the above implementation method to obtain the final visual language model, and the feature gates in the background suppression gating module are initialized.

[0058] 5. Establish model training loss constraints. Based on the existing visual language model training loss of mmGroundingDINO-T, add a contrast loss between target text features and background text features to determine the model training loss constraints. Specifically, the contrast loss between target text features and background text features uses the InfoNCE loss, and the specific implementation method described above is adopted. The loss hyperparameter is set to 1.0.

[0059] 6. Adaptive training was performed on the final visual language model. The training batch size was 8 images, trained on 8 Nvidia RTX 2080Ti images. The learning rate was set to 1e-4 on the VisDrone dataset images and 1e-5 on the UAVDT dataset images. The final visual language model underwent 50 rounds of adaptive training under the model training loss constraint. After adaptive training, compared to the original model before training (i.e., the original visual language model without replacing the query selection module with the background suppression gating module and without training), the visual language model adaptation method for aerial object detection provided in this disclosure achieved a 21.4% mAP improvement and a 33.9% mAP improvement on the VisDrone dataset. 50 Improvements were achieved, resulting in a 16.5% mAP improvement and a 23.5% mAP improvement on the UAVDT dataset. 50 Improvements. Compared to existing visual language models trained without using the visual language model adaptation method for aerial target detection provided in this disclosure (i.e., the original visual language model that has not replaced the query selection module with a background suppression gating module), the visual language model adaptation method for aerial target detection provided in this disclosure achieves a 0.8% mAP improvement and a 1.2% mAP improvement on the VisDrone dataset. 50 Improvements were achieved, resulting in a 1.3% mAP improvement and a 1.1% mAP improvement on the UAVDT dataset. 50 Improvements. Compared to traditional adapter methods, the visual-language model adaptation method for aerial target detection disclosed in this paper achieves a 0.5% mAP improvement and a 0.7% mAP improvement on the VisDrone dataset. 50 Improvements were achieved, resulting in a 3.5% mAP improvement and a 2.7% mAP improvement on the UAVDT dataset. 50 promote.

[0060] The visual language model adaptation method for aerial target detection provided in this disclosure has the following advantages over the prior art: it changes the approach from "passively ignoring the background" to "actively identifying and suppressing the background," which can reduce missed detections and false detections, and is proactive; the background word embedding can be pre-calculated, and only matrix multiplication and simple gating operations are involved during inference, which hardly increases the latency; in addition, only a small number of trainable parameters are introduced, which has high parameter and inference efficiency, and is highly efficient; it suppresses noise through the intuitive concept of a background vocabulary, which has good interpretability.

[0061] Another embodiment of this disclosure relates to a visual language model adaptation device for aerial target detection, including an acquisition module, a first initialization module, a generation module, a second initialization module, a loss determination module, and a training module.

[0062] The acquisition module is used to acquire drone aerial image datasets, perform image preprocessing on the drone aerial image datasets, and determine the target text to be detected.

[0063] The first initialization module is used to load the pre-trained weights of the visual language model and freeze the model weights of the visual backbone network and text backbone network of the visual language model. The visual language model also includes a query selection module, a projection layer, a multimodal encoder, a multimodal decoder, and a detection head.

[0064] The generation module is used to generate a background vocabulary list, which includes background vocabulary for multiple background types.

[0065] The second initialization module is used to initialize the background suppression gating module. It converts the background words in the background vocabulary into background text features through the text backbone network and its corresponding projection layer, which are then used as inputs to the background suppression gating module. The original query selection module in the visual language model is replaced by the background suppression gating module to obtain the final visual language model. The background suppression gating module includes multiple feature gates for suppressing and regulating the background.

[0066] The loss determination module is used to add a contrast loss between target text features and background text features to the original training loss of the visual language model, thereby determining the model training loss constraints.

[0067] The training module is used to train the final visual language model using a drone aerial image dataset and model training loss constraints.

[0068] The specific implementation method of the visual language model adaptation device for aerial target detection provided in this disclosure can be found in the visual language model adaptation method for aerial target detection provided in this disclosure, and will not be repeated here.

[0069] Another embodiment of this disclosure relates to an electronic device, including at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enables the at least one processor to perform the visual language model adaptation method for aerial target detection described in the above embodiments.

[0070] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0071] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0072] Another embodiment of this disclosure relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements the visual language model adaptation method for aerial target detection described in the above embodiments.

[0073] That is, those skilled in the art will understand that all or part of the steps in the methods described in the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0074] Another embodiment of this disclosure relates to a computer program product, including a computer program that, when executed by a processor, implements the visual language model adaptation method for aerial target detection described in the above embodiments.

[0075] Those skilled in the art will understand that the above embodiments are specific implementations of this disclosure, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this disclosure.

Claims

1. A visual language model adaptation method for aerial target detection, characterized in that, include: Acquire a dataset of drone aerial images, perform image preprocessing on the drone aerial image dataset, and determine the target text to be detected; Load the pre-trained weights of the visual language model, and freeze the model weights of the visual backbone network and the text backbone network. The visual language model further includes a query selection module, a projection layer, a multimodal encoder, a multimodal decoder, and a detection head. Generate a background vocabulary list, which includes background vocabulary for multiple background types; The background suppression gating module is initialized by converting the background words in the background vocabulary into background text features through the text backbone network and its corresponding projection layer, which are then used as input to the background suppression gating module. The original query selection module in the visual language model is replaced by the background suppression gating module to obtain the final visual language model. The background suppression gating module includes multiple feature gates for suppressing and regulating the background. Based on the original training loss of the visual language model, a contrast loss between target text features and background text features is added to determine the model training loss constraints; The final visual language model is trained using the drone aerial image dataset and the model training loss constraints.

2. The visual language model adaptation method for aerial target detection according to claim 1, characterized in that, The generated background vocabulary includes: Define the range of background vocabulary and multiple background types; A large language model is used to generate multiple alternative background vocabulary types within the given background vocabulary range; The candidate background vocabulary was validated using a visual language model; The final background vocabulary is determined based on the candidate background vocabulary through manual verification, and the background vocabulary list is generated.

3. The visual language model adaptation method for aerial target detection according to claim 1, characterized in that, Multiple feature gates include protection gates, suppression gates, and candidate adjustment gates; The protection gate is used to protect the local features of the image when the target text features are highly similar to the local features of the input image. The suppression gate is used to suppress the local features of the image when the background text features of the background vocabulary and the local features of the input image have a high similarity. The candidate adjustment gate is used to adjust the scores corresponding to the protection gate and the suppression gate as a whole to ensure coordination.

4. The visual language model adaptation method for aerial target detection according to claim 3, characterized in that, The background suppression gating module is used to perform semantic gating query selection according to the following formula: ； ； ； in, This represents the adjusted similarity matrix. This represents the adjusted maximum similarity score; express A function used to select the maximum value of each element in a matrix; This represents the selected candidate query index. This indicates the top score with the highest adjusted maximum similarity score. One index, This indicates the set number of candidate queries. This indicates a semantic gating module. Represents the original image-text similarity matrix and , For image feature representation, For the feature representation of the target text The maximum foreground similarity score for each local query in the image and , The maximum background similarity score for local queries of each image and , The background image-text similarity matrix and , For background text feature representation, superscript Indicates transpose; Semantic gating module Used for regulation according to the following formula: ； ； ； ； ； in, and These represent the output activation values ​​of the protection gate and the suppression gate, respectively. and The initial adjustment signal value and the final candidate adjustment vector of the candidate adjustment gate are respectively... It is the Sigmoid activation function. The scaling hyperparameter represents the score adjustment. The Hadamard product of a matrix. They are respectively Protective weights, suppression weights, and adjustment weights. They are respectively Protective weights, suppression weights, and adjustment weights. These are respectively the protection gate bias term, the suppression gate bias term, and the candidate adjustment gate bias term.

5. The visual language model adaptation method for aerial target detection according to claim 1, characterized in that, The contrast loss between target text features and background text features is expressed as: ； in, This is a word-level contrastive loss function used to amplify the distance distribution of different semantic features. for The InfoNCE loss between each term in the word, This is the joint feature representation obtained by concatenating background text features and target text features. , For feature splicing operations, Background text features Features of the target text.

6. The visual language model adaptation method for aerial target detection according to any one of claims 1 to 5, characterized in that, The visual language model is any one of the GroundingDINO model and its subsequent derivative models.

7. A visual language model adaptation device for aerial target detection, characterized in that, include: The acquisition module is used to acquire a dataset of drone aerial images, perform image preprocessing on the drone aerial image dataset, and determine the target text to be detected. The first initialization module is used to load the pre-trained weights of the visual language model and freeze the model weights of the visual backbone network and the text backbone network of the visual language model. The visual language model further includes a query selection module, a projection layer, a multimodal encoder, a multimodal decoder, and a detection head. A generation module is used to generate a background vocabulary list, which includes background vocabulary for multiple background types. The second initialization module is used to initialize the background suppression gating module. It converts the background words in the background vocabulary into background text features through the text backbone network and its corresponding projection layer, and uses these features as input to the background suppression gating module. It replaces the original query selection module in the visual language model with the background suppression gating module to obtain the final visual language model. The background suppression gating module includes multiple feature gates for suppressing and controlling the background. The loss determination module is used to add a contrast loss between the target text features and the background text features to the original training loss of the visual language model, and to determine the training loss constraints of the model. The training module is used to train the final visual language model using the UAV aerial image dataset and the model training loss constraints.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the visual language model adaptation method for aerial target detection as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the visual language model adaptation method for aerial target detection as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the visual language model adaptation method for aerial target detection as described in any one of claims 1 to 6.

Images