A method for identifying aerial time-sensitive targets based on large models

Abstract

The present invention discloses a method for identifying aerial time-sensitive targets based on a large model, which belongs to the field of image processing and recognition. Specifically, the flight trajectories of aircraft of the same model at different time points are collected to obtain image data and text data containing aerial time-sensitive targets. Then, the image is preprocessed and converted into a visual feature vector using a CLIP model to obtain encoded image features. Next, the text is preprocessed to obtain a natural language description. Definition rules are designed to divide the natural language description into multiple types of semantics, and then an organic fusion is implemented by designing a semantic hint merger, which is input into the CLIP model to obtain a global semantic hint feature. Finally, the global semantic hint feature of each frame of the image is used as a query vector, and the encoded image feature is used as a key and value vector. By enhancing the image features, the time-sensitive target is finally effectively identified. The present invention improves the acquisition of targets with strong descriptive and discriminative properties.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and recognition, and relates to realizing image recognition of military flight targets in combination with large model technology, and is specifically a method for recognizing aerial time-sensitive targets based on large models. Background Art

[0002] With the rapid development of zero-shot and few-shot classification techniques, pre-trained vision-language models (VLMs) such as CLIP have shown great potential. However, due to the lack of large-scale camera datasets for events, training models directly on event data remains infeasible. Therefore, cross-modal adaptation of existing VLMs to event vision has become a significant research challenge.

[0003] In this context, Wu et al. [1] proposed EventCLIP, which uses CLIP for event-based object recognition and supports zero-shot and few-shot learning. They generalize the image encoder of CLIP by converting the raw events into a 2D grid representation. To further improve performance, Wu et al. also proposed a feature adapter to aggregate temporal information on event frames and refine text embeddings to better align with visual input.

[0004] In multimodal object recognition, an important challenge facing machine learning is domain generalization: this ability is particularly important when dealing with tasks that have a different distribution from the training data. Current methods mainly focus on pure visual object recognition, while ignoring the integration of natural language. However, with the advancement of vision-language pre-training technology, it has become possible to use a large number of vision-language pairs for supervised learning. This not only allows learning across different domains, but also enhances recognition capabilities in multimodal scenarios. For example, Qiao et al. [2] proposed the CLIPood method, which demonstrated excellent transfer learning capabilities. However, CLIPood still has some limitations: the loss function used is inconsistent with its document description, it only evaluates a single model architecture and lacks universality, and it ignores category-aware visual fusion. To address these problems, Qiao et al. proposed a new method to derive the actual loss function based on implementation and expand the evaluation scope to a larger vision-language base model. In addition, Mixup-CLIPood is introduced to enhance category-aware visual fusion through a novel mixed loss.

[0005] Unlike traditional representation learning based on discrete labels, vision-language pre-training aligns images and text into a common feature space, allowing zero-shot transfer to downstream tasks through prompting, i.e., classification weights are generated through natural language describing the category of interest. However, one of the main challenges faced in the actual deployment of these models is prompt engineering, which requires domain-specific expertise and is extremely time-consuming - as small changes in wording can have a huge impact on performance. Inspired by recent progress in prompt learning in the field of natural language processing (NLP), Zhou et al. [3] proposed a method called Context Optimization (CoOp) to automatically adapt pre-trained vision-language models such as CLIP to downstream image recognition. By learning continuous vectors to represent prompt words without manual tuning, CoOp can significantly improve the performance of few-shot learning and is more efficient and robust than hand-crafted prompts and linear detection models.

[0006] Despite this, CoOp still has a key problem: the learned context cannot generalize well to a wider range of categories that have not been seen in the same dataset, which means that CoOp may overfit the basic categories observed during training. To overcome this problem, Zhou et al. [4] proposed a method called Conditional Context Optimization (CoCoOp), which aims to solve the problem of insufficient generalization ability of existing CoOp when facing unseen categories. CoCoOp further learns a lightweight neural network to generate input conditional tokens for each image, so that dynamic prompts can better adapt to each instance, thereby reducing sensitivity to category changes. Experiments show that CoCoOp not only outperforms CoOp in generalization of unseen categories, but also demonstrates good transfer ability across datasets and stronger domain generalization performance.

[0007] Sun et al. [5] proposed a fine-grained visual cue learning method (FG-VPL) to improve the image recognition performance of pre-trained visual-language models with a small number of samples. By introducing fine-grained visual cues into the image encoder, the model can focus on the target objects of the downstream task and generate more discriminative visual features. In addition, a dual-path adaptive recognition module is proposed to narrow the domain gap and utilize cross-modal matching knowledge and visual information in the small-sample training set for test image classification. Miao et al. [6] proposed the MuDPT (Multimodal Deep Fusion Prompt Tuning) method, which aims to improve prompt learning by pre-training visual-language models to achieve co-alignment of text and visual representations. To achieve this goal, the researchers designed a lightweight conversion block (Injection Model) to establish a cross-attention mechanism between different modalities, thereby promoting deep bidirectional prompt fusion. Chen et al. [7] proposed an energy-based multi-prompt learning method (EMPL), which aims to improve the rapid adaptation ability of visual-language pre-trained models (VLMs) to downstream tasks by generating multiple prompt embeddings. This approach utilizes instances drawn from the energy distribution implicitly defined by the VLM to generate prompts, which not only improves parameter efficiency but also balances the relationship between in-domain and out-of-domain open-vocabulary generalization.

[0008] Bahng et al. [8] explored the effectiveness of visual prompting in adjusting large-scale visual models. By learning a single image perturbation, the frozen model performs well on the new task. HiCLIP proposed by Geng et al. [9] introduces a hierarchical perceptual attention mechanism based on CLIP, aiming to gradually discover semantic hierarchies from images and text in an unsupervised manner. By applying this hierarchical perceptual attention on the visual and language branches, HiCLIP can better capture high-level and fine-grained semantic information, thereby improving cross-modal alignment and performing well in visual recognition and multimodal tasks. Shi et al.

[10] proposed a new method to enhance the performance of pre-trained models in downstream tasks by using synthetic text images as visual prompts. This method is called LoGoPrompt, which solves the "chicken and egg" problem of whether to add category-related visual prompts first or predict the category first, and redefines the classification goal as visual prompt selection. GOPRO proposed by Singha et al.

[11] is a model designed for image recognition tasks, which improves visual recognition performance by combining the advantages of CLIP (a large-scale base model that can embed images into a semantically rich space) and self-supervised learning (SSL). The model leverages learnable image and text projectors to ensure similarity between different augmented views of the input image in a shared image-text embedding space, thereby improving the model's invariance and generalization capabilities. During training, GOPRO employed a comprehensive loss function, including cross-domain contrast loss, visual contrast loss, and a novel cue consistency loss, to address issues with loss weighting and consistency across different viewpoints of an image that may be encountered in a multi-task framework.

[0009] Although the above methods have made significant progress in the field of vision-language models, there are still some shortcomings:

[0010] First, many methods rely heavily on prompt engineering, which requires a lot of time and domain expertise to manually design prompts, which is not efficient enough in practical applications.

[0011] Second, existing methods have limited generalization capabilities in the presence of very few or even zero samples, especially when dealing with unseen categories or out-of-domain data.

[0012] In addition, some methods still lack the deep fusion of visual features and text features, and cannot fully utilize cross-modal information to improve recognition performance.

[0013] References are as follows:

[0014] [1]Wu Z,Liu X,Gilitschenski I.Eventclip:Adapting clip for event-basedobject recognition[J].arXiv preprint arXiv:2306.06354,2023.

[0015] [2]Qiao Y,Li K,Lin J,et al.Robust Domain Generalization for Multi-modal Object Recognition[J].arXiv preprint arXiv:2408.05831,2024.

[0016] [3]Zhou K,Yang J,Loy C C,et al.Learning to prompt for vision-languagemodels[J].International Journal of Computer Vision,2022,130(9):2337-2348.

[0017] [4]Zhou K,Yang J,Loy C C,et al.Conditional prompt learning forvision-language models[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2022:16816-16825.

[0018] [5]Sun H,He X,Zhou J,et al.Fine-grained visual prompt learning ofvision-language models for image recognition[C] / / Proceedings of the 31st ACMInternational Conference on Multimedia.2023:5828-5836.

[0019] [6]Miao Y,Li S,Tang J,et al.MuDPT:Multi-modal Deep-symphysis PromptTuning for Large Pre-trained Vision-Language Models[C] / / 2023IEEEInternational Conference on Multimedia and Expo(ICME).IEEE,2023:25-30.

[0020] [7]Chen Z,Huang X,Guan Q,et al.A retrospect to multi-prompt learningacross vision and language[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.2023:22190-22201.

[0021] [8]Bahng H,Jahanian A,Sankaranarayanan S,et al.Exploring visualprompts for adapting large-scale models[J].arXiv preprint arXiv:2203.17274,2022.

[0022] [9]Geng S,Yuan J,Tian Y,et al.HiCLIP:Contrastive language-imagepretraining with hierarchy-aware attention[J].arXiv preprint arXiv:2303.02995,2023.

[0023]

[10] Shi C,Yang S.Logoprompt:Synthetic text images can be good visualprompts for vision-language models[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2023:2932-2941.

[0024]

[11] Singha M, Jha A, Banerjee B.GOPRO: Generate and Optimize Prompts inCLIP using Self-Supervised Learning[J].arXiv preprint arXiv:2308.11605,2023. Summary of the Invention

[0025] To address the above problems, the present invention proposes a large-model-based aerial time-sensitive target recognition method, which integrates image multi-text semantic modeling and visual-semantic association. It not only strengthens the correlation between semantic features and visual features, but also achieves the technical effect of improving the model's ability to understand and represent image semantics, and obtaining target sample feature descriptions with strong descriptive and discriminative features.

[0026] The large-model-based aerial time-sensitive target recognition method is divided into the following steps:

[0027] Step 1: In a scenario where multiple aircraft of different models are identifying time-sensitive targets in the air, the flight trajectories of the same aircraft at different time points are collected to obtain the image data and text data corresponding to each aircraft model;

[0028] All flight trajectory videos of the same aircraft are divided into frames to obtain an image data set P containing time-sensitive targets: P = {p1, p2, ..., p M}; M is the number of flight trajectory videos;

[0029] At the same time, the text list T of all flight trajectories of this type of aircraft is counted. M}, each text list corresponds to a flight trajectory video.

[0030] Step 2: Preprocess the image data corresponding to each flight trajectory video of the same model aircraft separately;

[0031] Step 3: Use the image encoder of the CLIP model to convert the preprocessed images of each video into a set of visual feature vectors to obtain the image features of each video after encoding.

[0032] The CLIP model uses a 12-layer Vision Transformer module as the image encoder and uses a Patch size of 16.

[0033] The specific conversion process is:

[0034] First, for each flight trajectory video, each preprocessed frame is decomposed into 16×16 patches using an image encoder. Each patch is then linearly mapped to a one-dimensional vector, called a token, and a class token is added.

[0035] Then, through the linear embedding layer, the image data corresponding to the video is embedded into the visual tokens sequence to obtain the video sequence X:

[0036]

[0037] Where f is the number of frames in the current video; 512 is the embedding dimension of each token, and 197 is the number of all tokens in each frame image.

[0038] Finally, the sequence X is averaged and pooled along the frame channel to obtain the corresponding encoded image feature tensor

[0039] Feature average pooling means that the token features at the same position of all frames of the video are averagely pooled into one feature, tensor X v It is the feature matrix of 197 tokens, which is also the encoded image modality feature.

[0040] Step 4: Preprocess the text data of the same aircraft model to obtain the natural language description of each text list;

[0041] Step 5: Design the definition rules for natural language descriptions and divide the natural language descriptions corresponding to all text lists into three semantic types:

[0042] 1) Category description semantics, directly using the text list extracted by the text encoder of the CLIP model to correspond to the semantic cues related to the category;

[0043] The CLIP model uses a 12-layer Transformer module as the text encoder and uses a patch size of 16.

[0044] 2) Visual perception semantics, using a discrete semantic cue encoder to extract discrete semantic cues of visual perception and a continuous semantic cue encoder to extract continuous semantic cues of visual perception;

[0045] 3) High-order knowledge semantics: a discrete semantic cue encoder is used to extract discrete semantic cues of high-order knowledge, and a continuous semantic cue encoder is used to extract continuous semantic cues of high-order knowledge.

[0046] First, the discrete semantic cue encoder is designed as follows: visual perception semantic text and high-order knowledge semantic text are input into the T5 model respectively, and a simplified summary of the text is output, thereby extracting the key information of the input text; the key information of the input text is summarized and extracted through the T5 model, and discrete semantic cues of visual perception semantics and high-order knowledge semantics are obtained respectively.

[0047] Then, a continuous semantic cue encoder is designed to input visual perception semantic text and high-level knowledge semantic text into the pre-trained KEPLER model respectively. Combined with a lightweight projection mapping network, it constructs continuous semantic cues of visual perception and continuous semantic cues of high-level knowledge respectively.

[0048] Step 6: Design a semantic cue merger to organically integrate the multi-type semantics corresponding to all text lists and input it into the text encoder of the CLIP model to obtain the final global semantic cue features.

[0049] Step 7: Use the global semantic cue features of each video as the query vector and the encoded image features as the key and value vectors. Obtain enhanced image features through the cross-modal attention mechanism, and aggregate all regional visual information to obtain a highly descriptive and discriminative feature description of the target sample, ultimately effectively identifying the time-sensitive target category in the video.

[0050] The specific process is:

[0051] First, for the i-th frame of the current video, the global semantic cue feature corresponding to its text list is w i , the encoded image feature is X i ; Enhanced image features of this frame for:

[0052]

[0053] Where CrossAttention(·) refers to cross attention, LN is layer normalization, x cls Represents the features of the [CLS] tag of the input image.

[0054] Then, the average value of the enhanced image features of all frames in the video is calculated to obtain the average image features as global image features.

[0055] Next, the normalized cosine similarity is calculated to calculate the image feature X after encoding the frame. i The probability of belonging to a global image feature:

[0056]

[0057] where τ is a learnable temperature parameter and sim(·) represents the cosine similarity.

[0058] Finally, the probability is used to optimize the cosine similarity so that the image feature X after encoding the frame is i The similarity between the correct text features is maximized, and the similarity between other categories is minimized;

[0059] The specific optimization is to minimize the cross entropy loss:

[0060]

[0061] where y i is the one-hot encoding of the i-th frame image.

[0062] The advantages of the present invention are:

[0063] 1. A large-scale model-based aerial time-sensitive target recognition method designs image multi-semantic definition rules and divides the multi-semantics corresponding to visible light images into three types: category description semantics, visual perception semantics, and high-order knowledge semantics. This method comprehensively defines and effectively models the semantics of target images, establishes a strong correspondence between images and paired semantics, and enriches the semantic information of image samples from multiple angles.

[0064] 2. A large-scale model-based method for identifying time-sensitive aerial targets. This method interactively learns the target's visual content and semantic information. Through a cross-modal attention mechanism, the image's visual feature vector interacts with global semantic cue features, strengthening the correspondence between the target's visual content and multi-dimensional semantic information. This mechanism highlights key visual areas within the image, fostering a deep interaction between visual and semantic features, ensuring that each visual information segment effectively corresponds to its semantic cue. This interactive mechanism not only enhances the descriptiveness and discriminative nature of image features but also significantly improves target recognition accuracy, providing technical support for the precise identification of time-sensitive aerial targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flow chart of a method for identifying aerial time-sensitive targets based on a large model according to the present invention;

[0066] Figure 2 This is a framework diagram of an algorithm model of a large-model-based aerial time-sensitive target recognition method of the present invention;

[0067] Figure 3 Schematic diagram of the network architecture of the CLIP image encoder ViT-B / 16 of the present invention;

[0068] Figure 4 Graphs of samples of various categories in the dataset according to the embodiment of the present invention;

[0069] Figure 5 This is a graph showing the average accuracy of various types of small-scale data sets of the present invention;

[0070] Figure 6 This is a graph of various indicators of a small-scale data set of the present invention;

[0071] Figure 7 This is a graph showing the average accuracy of various types of large-scale data sets of the present invention;

[0072] Figure 8 This is a graph of various indicators of a large-scale data set of the present invention;

[0073] Figure 9 This is the Attention Map visualization diagram of the small-scale dataset of the present invention;

[0074] Figure 10 This is the Attention Map visualization diagram of the large-scale dataset of the present invention. DETAILED DESCRIPTION

[0075] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0076] The present invention provides a method for identifying aerial time-sensitive targets based on a large model, which is based on the demand for accurate capture and identification of specific aerial targets in the scenarios of national defense construction and regional security control, and provides reliable technical support for the investigation, monitoring and early warning of key targets. The core of the present invention is to explore and develop a multimodal aerial target capture and recognition technology based on a CLIP pre-trained large model and text semantic prompts, and to achieve high-precision capture and recognition of aerial targets by making full use of the complementary information of visible light images and text prompts. Specifically, the present invention uses massive image training samples as data support and sample semantic prompts as an external knowledge source, and with the help of the large model's effective modeling of text content and the perception of image visual content, it realizes the effective interaction and learning of knowledge semantic information and visual content information, thereby improving the system's capture and recognition accuracy of specific aerial targets.

[0077] The present invention adopts a method of image multi-text semantic modeling and visual-semantic association fusion. By dividing the multi-text semantics corresponding to the visible light image into category description semantics, visual perception semantics and high-order knowledge semantics, the semantic information contained in the text is comprehensively mined and integrated into global semantic hints. Then, through the cross-modal attention mechanism, the important role of key visual areas in global feature representation is highlighted, making the semantic information of the image more structured and hierarchical. At the same time, the global semantic hint feature is used as the query vector, and the encoded image feature is used as the key and value vector to highlight the important role of key visual areas in global feature representation. Such a design not only strengthens the correlation between semantic features and visual features, but also achieves the technical effect of improving the model's ability to understand and represent image semantics, and obtaining a target sample feature description with strong descriptiveness and discrimination.

[0078] Compared with existing methods, the present invention has the advantages of enhancing the richness and distinctiveness of image features and improving the accuracy of target recognition, providing a more accurate semantic basis for subsequent target recognition tasks and providing new ideas for cross-modal feature fusion.

[0079] The large model-based aerial time-sensitive target recognition method is as follows: Figure 1 and Figure 2 As shown, the following steps are taken:

[0080] Step 1: In a scenario where multiple aircraft of different models are identifying time-sensitive targets in the air, the flight trajectories of the same aircraft at different time points are collected to obtain the image data and text data corresponding to each aircraft model;

[0081] All flight trajectory videos of the same aircraft are divided into frames to obtain an image data set P containing time-sensitive targets:

[0082] M is the number of flight trajectory videos; F is the total number of frames, C is the number of image channels, H and W are the height and width of the image respectively;

[0083] At the same time, a text list of all flight trajectories of this type of aircraft is compiled Each text list corresponds to a flight trajectory video.

[0084] Step 2: Preprocess the image data corresponding to each flight trajectory video of the same model aircraft separately;

[0085] Preprocessing means: for the current video, adjusting the resolution of each frame image in the video to 224×224 while maintaining the aspect ratio, and filling the insufficient pixels with zero pixels.

[0086] Step 3: Use the image encoder of the CLIP model to convert the preprocessed images of each video into a set of visual feature vectors to obtain the image features of each video after encoding.

[0087] The CLIP model uses a 12-layer Vision Transformer module as the image encoder and adopts a Patch size of 16.

[0088] The specific conversion process is:

[0089] First, for each flight trajectory video, the preprocessed image frame is decomposed into 16×16 patches using an image encoder (196 ÷ 16 = 14). Each row and column contains 14 patches, for a total of 14×14 = 196 patches. Each patch is linearly mapped to a one-dimensional vector, called a token, for a total of 196 tokens. A class token is also added, resulting in a total of 197 tokens per frame.

[0090] Then, through the linear embedding layer, the image data corresponding to the video is embedded into the visual tokens sequence to obtain the video sequence X:

[0091]

[0092] Where f is the number of frames in the current video; 512 is the embedding dimension of each token, and 197 is the number of all tokens in each frame image.

[0093] Finally, the sequence X is averaged and pooled along the frame channel to obtain the corresponding encoded image feature tensor

[0094] Feature average pooling means that the token features at the same position of all frames of the video are averagely pooled into one feature, tensor X v It is the feature matrix of 197 tokens, which is also the encoded image modality feature.

[0095] Step 4: Preprocess the text data of the same aircraft model to obtain the natural language description of each text list;

[0096] Each text list is expanded to obtain a natural language description; the image semantic definition includes aerodynamic layout, appearance description, paint color, paint material, thrust layout, deployment orientation, usage, intelligence and other information.

[0097] After text preprocessing, the semantics of each image is effectively standardized.

[0098] Step 5: Design the definition rules for natural language descriptions and divide the natural language descriptions corresponding to all text lists into three semantic types:

[0099] 1) Category description semantics, that is, a direct description of the category to which the image containing the target belongs. The text list extracted by the text encoder of the CLIP model directly corresponds to the category-related semantic hints; for example, "This is an image of [category label], where [category label] is the image category information such as reconnaissance aircraft, B2 stealth bomber, etc."

[0100] The CLIP model uses a 12-layer Transformer module as the text encoder and uses a patch size of 16.

[0101] 2) Visual perception semantics, i.e., text descriptions related to the visual characteristics of the target. A discrete semantic cue encoder is used to extract discrete semantic cues of visual perception, and a continuous semantic cue encoder is used to extract continuous semantic cues of visual perception; for example, descriptions of target shape, contour, and color.

[0102] 3) High-order knowledge semantics, which does not directly reflect the visual characteristics of the target but reflects the deep-level attributes of the target or specific text descriptions. A discrete semantic cue encoder is used to extract discrete semantic cues of high-order knowledge, and a continuous semantic cue encoder is used to extract continuous semantic cues of high-order knowledge; such as dynamic parameters, intelligence descriptions, etc.

[0103] For category description semantics, the CLIP model is directly used to pre-train the text feature encoder to extract category-related semantic cues;

[0104] For visual perception semantics and high-order knowledge semantics, a discrete semantic cue encoder and a continuous semantic cue encoder are designed respectively to extract discrete semantic cues and continuous semantic cues of visual perception and high-order knowledge, and combined with the image-semantic cue interaction method to achieve the purpose of fully utilizing semantic information.

[0105] First, the discrete semantic cue encoder process is designed as follows:

[0106] Visual perception semantics and high-level knowledge semantics can be used as external knowledge data to improve the accuracy of image object recognition. However, textual descriptions are often redundant and contain redundant (noise) information irrelevant to visual recognition. To remove this information from both types of semantic information, the visual perception semantic text and high-level knowledge semantic text are separately input into the T5 model. The T5 model is an end-to-end pre-trained model that takes a piece of text as input and outputs a simplified summary of that text, thereby extracting the key information of the input text. The T5 model summarizes and extracts the key information from the input text, obtaining discrete semantic cues for visual perception semantics and high-level knowledge semantics, respectively.

[0107] Next, we designed a continuous semantic cue encoder. During the discrete cue learning process described above, the simplified text description inevitably filters out some category-related information as redundant. To ensure that the generated cue carries a wider range of contextual information covering the current image category, we fed the visually perceived semantic text and the high-level knowledge semantic text into the pre-trained KEPLER model. This was then combined with a lightweight projection mapping network to construct continuous semantic cues based on visual perception and high-level knowledge, respectively.

[0108] Step 6: Design a semantic cue merger to organically integrate the multi-type semantics corresponding to all text lists and input it into the text encoder of the CLIP model to obtain the final global semantic cue features.

[0109] Organic fusion means: concatenating category description semantic cues, visual perception discrete semantic cues, visual perception continuous semantic cues, high-order knowledge discrete semantic cues and high-order knowledge continuous semantic cues to obtain global semantic cues, and inputting them into the text encoder of the CLIP model to obtain global semantic cues features.

[0110] Step 7: Use the global semantic cue feature as the query vector and the encoded image feature as the key and value vectors. Obtain enhanced image features through the cross-modal attention mechanism, aggregate all regional visual information, and obtain a highly descriptive and discriminative feature description of the target sample, ultimately effectively identifying the time-sensitive target category in the video.

[0111] The CLIP model is a typical dual-encoder architecture consisting of an image encoder and a text encoder. Each image and paired text are converted into a high-dimensional embedding using the image encoder and text encoder, respectively. The training goal is to align unimodal embeddings through contrastive learning, where the model matches paired images and text together while separating unpaired images and text in the latent space. Pre-training on large-scale image-text datasets ensures that the learned visual features are discriminative and transferable to zero- and few-shot downstream tasks.

[0112] To further adapt the visual representation to image content matching and target image recognition tasks to reduce the interference of task-irrelevant visual concepts, this method constructs a task-specific visual adaptation head that focuses on salient visual cues related to category information.

[0113] The specific process is:

[0114] First, for the i-th frame of the current video, the global semantic cue feature corresponding to its text list is w i , which is used as the query vector, and its encoded image feature is X i As key and value vectors; the enhanced image features of the frame for:

[0115]

[0116] Where CrossAttention(·) refers to cross attention, LN is layer normalization, x cls Represents the features of the [CLS] tag of the input image.

[0117] Then, in order to summarize all the information, the enhanced image features of all frames in the video are averaged to obtain the average image features as global image features.

[0118] Next, the normalized cosine similarity is calculated to calculate the image feature X after encoding the frame. i The probability of belonging to a global image feature:

[0119]

[0120] where τ is a learnable temperature parameter and sim(·) represents the cosine similarity.

[0121] Finally, the probability is used to optimize the cosine similarity between the image feature and the global semantic clue feature so that the image feature X after encoding the frame is i The similarity between the correct text features is maximized, and the similarity between other categories is minimized;

[0122] During training, the gradients of the knowledge-aware cue and adaptation head are updated while keeping the parameters of CLIP unchanged. The training goal is to minimize the cross-entropy loss:

[0123] where y i is the one-hot encoding of the i-th frame image.

[0124] In this way, the performance of the model in processing specific tasks can be fine-tuned without affecting the generalization ability of the CLIP model that has been pre-trained with large-scale data.

[0125] Example:

[0126] The dataset is divided into five aircraft models: B-2, B-52, E2D, F18, and F35. The samples of each category are as follows: Figure 4 During the experiment, the first batch of data sets was smaller, with only about 1,000 sample images per category, while the second batch of data sets had about 10,000 sample images per category.

[0127] Since the entire dataset is composed of frames of aircraft at different time points in the video, they have a high degree of similarity, which may affect the accuracy and efficiency of subsequent analysis. To solve this problem, a structured approach was adopted to organize and divide the original dataset. Specifically, the dataset was first classified by video track according to the aircraft model. The purpose of this is to group similar images, which facilitates subsequent processing and analysis. Next, the dataset of each aircraft model was subdivided: every 60 frames were selected as a grouping unit. The design was based on the following settings:

[0128] Temporal continuity: Every 60 frames roughly corresponds to a few seconds of aircraft video images during data collection, which provides a reasonable time window for the target recognition task to capture the movement and changes of the aircraft during this period.

[0129] The amount of data is just right: 60 frames is neither too much to overload the processing nor too little to capture enough information.

[0130] Information richness: Each 60-frame image sequence can provide enough visual information to identify specific features and behaviors of the aircraft.

[0131] Using the above sampling method, the dataset is divided into 60-frame images per aircraft model, ultimately generating tracks consisting of multiple 60 images. Each track represents the flight path of a specific aircraft model over a period of time. This data organization not only improves data processing efficiency but also facilitates subsequent analysis and model training, as these tracks can be used for tasks such as feature extraction, pattern recognition, and behavior analysis.

[0132] Finally, the original dataset is randomly divided into training and test sets according to 80% and 20% of the tracks. Specifically, about 4,000 / 40,000 images are used as training sets, and 1,000 / 10,000 images are used as test sets.

[0133] During the experimental implementation, the CLIP model based on ViT-B / 16 was used as the backbone of the main network architecture, considering its efficiency and accuracy. This CLIP model uses the Vision Transformer and Transformer with 12 layers of Transformer modules as the image encoder and text encoder, respectively, and adopts a patch size of 16.

[0134] The pre-processed image set P is passed through the CLIP image encoder. The image is first decomposed into 16×16 patches, so a 224×224 image will be decomposed into (224 / 16) 2= 196 patches, and then each patch is mapped to a one-dimensional vector through linear mapping. These one-dimensional vectors are called a token. The specific process is as follows Figure 3 As shown, the encoded image data is then embedded into the visual tokens sequence X through a linear embedding layer. Then, the feature is averaged into a tensor X along the time channel (i.e., the frame channel) v .

[0135] During training, the parameters of the CLIP encoder are fixed. This allows the model to leverage the rich features and knowledge already learned by the pre-trained CLIP model while preventing damage to the pre-trained CLIP model parameters and reducing the computational burden during training. The model's learning rate is 0.001, a relatively small value that helps the model converge stably during training and avoid overfitting. Weight decay, set to 1e-4, is also introduced as a regularization technique to prevent model parameters from becoming excessively large and further improve the model's generalization capabilities.

[0136] The target category text list needs to be processed into corresponding language descriptions to fully utilize the CLIP text encoder, which uses the same number of Transformer layers as the image encoder. Specifically, using a carefully designed prompt template, each category text is first expanded to obtain a natural language description. For example, "B-2" can be processed into "This is a photo of B-2, a type of airplane."

[0137] For two datasets of different sizes, the model was trained for 50 and 20 epochs, respectively, with a batch size of 32. Furthermore, the number of frames was set to 10, meaning that in each iteration the model processed 10 consecutive frames of image data. This helps the model capture dynamic changes and temporal information in the consecutive images. Adam was used as the optimizer. In the experiments, the model was implemented using Python and the PyTorch framework and trained on a server equipped with an A40.

[0138] During the experiment of this project, experiments were conducted on the datasets before and after expansion. Next, the experimental results are introduced using the test set evaluation of the two-scale datasets.

[0139] When evaluating the model effect, the widely used accuracy, precision, recall and F1 score were used as evaluation indicators. It should be noted that the results reported in the experiment were obtained by averaging multiple categories. The specific formula is as follows:

[0140]

[0141] Among them, TP, TN, FP, and FN are the number of correctly predicted positive examples, the number of correctly predicted negative examples, the number of negative examples incorrectly predicted as positive examples, and the number of positive examples incorrectly predicted as negative examples, respectively.

[0142] Accuracy is the most intuitive performance metric, measuring the proportion of samples that the model correctly predicts to the total number of samples. Accuracy is easy to calculate even in imbalanced datasets, but it is not always the best performance metric, because in the case of a majority of negative examples and a minority of positive examples, a model that always predicts negative examples can also have high accuracy.

[0143] Precision measures the proportion of examples that the model predicts as positive that are actually positive. It focuses on the accuracy of the predicted positive examples. When there are few positive examples in the dataset, precision is an important metric because it can reduce false positives for rare events.

[0144] Recall measures the proportion of actual positive examples that are correctly predicted by the model. It focuses on the model's ability to capture all positive examples. Recall is particularly important when minimizing false negatives.

[0145] The F1 score is the harmonic mean of precision and recall, attempting to find a balance between these two. The F1 score is high when both are high, so it's a metric that strikes a balance between precision and recall. The F1 score is particularly useful when the dataset is extremely imbalanced.

[0146] Figure 5 and Figure 7 It is a graph showing the average precision (AP) of various data sets of different sizes changing with epoch. Figure 6 and Figure 8 It is the curve of the average precision (average_mAP), accuracy (average_ACC), recall rate (average_RECALL), and F1 score (average_F1) of different scale data sets over epochs.

[0147] It can be seen from the four curves that no matter which indicator, it basically increases gradually with the increase of epochs, and then tends to be stable in the last few epochs.

[0148] Attention Map visualization is crucial for understanding the workings of deep learning models, particularly Transformer models and their variants. It provides an explanatory view of the model's internal decision-making process. By visualizing the distribution of attention, we can better understand the model's decision-making process and identify whether the model's focus on the data is consistent with expectations.

[0149] Therefore, samples from each of the five categories were input into the model. The multi-head attention weight matrix obtained in the 12-layer Transformer Encoder of the image encoder was extracted and averaged. After extracting the token vectors related to the classification, the attention weight matrix was reshaped, interpolated and scaled to the same size as the input image. This attention weight matrix was then overlaid on the input image for display, resulting in the Attention Map visualization. The first, third, fifth, seventh, ninth, and twelfth layers of the 12-layer Transformer Encoder were displayed for two datasets of different sizes.

[0150] according to Figure 9 and Figure 10 The Attention Map visualization results show that in the first few layers of the Transformer Encoder, the network's attention is more focused on the target, while in the later layers, the attention is dispersed to surrounding locations in the image. Let's analyze this phenomenon below.

[0151] In Transformer models, a common phenomenon is that attention distribution changes as layers deepen. In early attention layers, the model tends to focus on the most salient and relevant parts of the input sequence, such as the primary target or object in image processing tasks. This is because lower-level attention layers are closer to the original input and are therefore more likely to capture local features and direct visual information.

[0152] However, as the model layers increase, the output of each attention layer serves as the input to the next layer, allowing each layer to further abstract and combine features based on the previous layer. This hierarchical feature abstraction allows the model to capture more complex and abstract concepts at higher levels. Therefore, in deeper layers, attention is no longer limited to specific regions, but is spread across the entire input sequence to understand more global information and contextual relationships.

[0153] This phenomenon occurs in both natural language processing and computer vision tasks. For example, when processing natural language, early attention layers may focus on individual words or phrases, while later attention layers may focus on entire sentences or paragraphs to understand the overall meaning of the text. In vision tasks, early layers may focus on specific objects or textures in an image, while later layers may focus on the layout and structure of the entire scene.

[0154] This dispersion of attention is not random, but an effective strategy learned by the model to capture different levels of information at different levels. This also explains why the Transformer model can perform well in a variety of tasks because it can flexibly process feature representations ranging from simple to complex.

[0155] In addition, the distribution of attention also helps the model process a wider range of contextual information, which is especially important when processing long sequences or large-scale images. By distributing attention at different levels, the model is able to better understand the global structure and dependencies of the input data, thereby improving the accuracy and robustness of its predictions.

[0156] Therefore, as the number of Transformer layers increases, the dispersion of attention reflects the model's gradual abstraction process from local features to global context, which is one of the key advantages of the Transformer architecture design.

Claims

1. A method for identifying time-sensitive aerial targets based on a large model, characterized in that: The specific steps are as follows: Step 1: In a scenario where multiple aircraft of different models are identifying time-sensitive targets in the air, the flight trajectories of the same aircraft at different time points are collected to obtain the image data and text data containing the target corresponding to each aircraft model; Step 2: Preprocess the image data corresponding to each flight trajectory video of the same model aircraft; Step 3: Use the image encoder of the CLIP model to convert the preprocessed images of each video into a set of visual feature vectors to obtain the image features of each video after encoding; Step 4: Preprocess the text data of the same aircraft model to obtain the natural language description of each text list; Step 5: Design the definition rules for natural language descriptions and classify the natural language descriptions corresponding to all text lists into three semantic types: 1) Category description semantics, directly using the text list extracted by the text encoder of the CLIP model to correspond to the semantic cues related to the category; 2) Visual perception semantics, using a discrete semantic cue encoder to extract discrete semantic cues of visual perception and a continuous semantic cue encoder to extract continuous semantic cues of visual perception; 3) High-order knowledge semantics: a discrete semantic cue encoder is used to extract discrete semantic cues of high-order knowledge, and a continuous semantic cue encoder is used to extract continuous semantic cues of high-order knowledge; Step 6: Design a semantic hint merger to organically integrate the multi-type semantics corresponding to all text lists and input it into the text encoder of the CLIP model to obtain the final global semantic hint feature; Step 7: Using the global semantic cue features of each video as the query vector and the encoded image features as the key and value vectors, we use a cross-modal attention mechanism to obtain enhanced image features. We then aggregate all regional visual information to obtain highly descriptive and discriminative feature descriptions of target samples, ultimately effectively identifying time-sensitive target categories in the video. The specific process is: First, for the i-th frame of the current video, the global semantic cue feature corresponding to its text list is w i , the encoded image feature is X i ; Enhanced image features of this frame for: Where CrossAttention(·) refers to cross attention, LN is layer normalization, x cls Represents the features of the [CLS] tag of the input image; X is the video sequence obtained by embedding the image data corresponding to the video into the visual tokens sequence; Then, the average value of the enhanced image features of all frames in the video is calculated to obtain the average image features As a global image feature; Next, the normalized cosine similarity is calculated to calculate the image feature X after encoding the i-th frame. i The probability of belonging to a global image feature: Where τ is a learnable temperature parameter and sim(·) represents the cosine similarity; Finally, the probability is used to optimize the cosine similarity so that the image feature X after encoding the frame is i The similarity between the correct text features is maximized, and the similarity between other categories is minimized; The specific optimization is to minimize the cross entropy loss: where y i is the one-hot encoding of the i-th frame image.

2. A method for identifying time-sensitive aerial targets based on a large model as claimed in claim 1, characterized in that: In step 1, all flight trajectory videos of the same type of aircraft are divided by frame to obtain an image data set P containing time-sensitive targets: M is the number of flight trajectory videos; F is the total number of frames, C is the number of image channels, H and W are the height and width of the image respectively; At the same time, a text list of all flight trajectories of this type of aircraft is compiled Each text list corresponds to a flight trajectory video.

3. The method for identifying aerial time-sensitive targets based on a large model according to claim 1, wherein: In the step 2, the pre-processing means: for the current video, adjusting the resolution of each frame image in the video to 224×224 while maintaining the aspect ratio, and filling the insufficient pixels with zero pixels.

4. The method for identifying aerial time-sensitive targets based on a large model according to claim 1, wherein: In step 3, the CLIP model uses a 12-layer Vision Transformer module as the image encoder, and the patch size is 16; In step 5, the CLIP model uses a Transformer with 12 layers of Transformer modules as the text encoder, and the patch size is 16.

5. A method for identifying aerial time-sensitive targets based on a large model as claimed in claim 4, characterized in that: The specific conversion process of step 3 is as follows: First, for each flight trajectory video, the preprocessed image frame is decomposed into 16×16 patches using an image encoder. Each patch is then mapped to a one-dimensional vector, called a token, using linear mapping. A class token is also added. Then, through the linear embedding layer, the image data corresponding to the video is embedded into the visual tokens sequence to obtain the video sequence X: Where f is the number of frames in the current video; 512 is the embedding dimension of each token, 197 is the number of all tokens in each frame image; F is the total number of frames; Finally, the sequence X is averaged and pooled along the frame channel to obtain the corresponding encoded image feature tensor Feature average pooling means that the token features at the same position of all frames of the video are averagely pooled into one feature, tensor X v It is the feature matrix of 197 tokens, which is also the encoded image modality feature.

6. A method for identifying time-sensitive aerial targets based on a large model as claimed in claim 1, characterized in that: In the step 4, each text list is expanded to obtain a natural language description, including aerodynamic layout, appearance description, paint color, paint material, thrust layout, deployment direction, usage and intelligence information.

7. The method for identifying aerial time-sensitive targets based on a large model according to claim 1, wherein: In step 5, the discrete semantic hint encoder is designed as follows: the visual perception semantic text and the high-order knowledge semantic text are input into the T5 model respectively, and a simplified summary text is output, thereby extracting the key information of the input text; the key information of the input text is summarized and extracted by the T5 model, and discrete semantic hints of visual perception semantics and high-order knowledge semantics are obtained respectively; The process of designing a continuous semantic cue encoder is as follows: visual perception semantic text and high-order knowledge semantic text are input into the pre-trained KEPLER model respectively, and combined with a lightweight projection mapping network to construct continuous semantic cues of visual perception and continuous semantic cues of high-order knowledge respectively.

8. The method for identifying aerial time-sensitive targets based on a large model according to claim 1, wherein: In step six, organic fusion refers to: concatenating category description semantic cues, visual perception discrete semantic cues, visual perception continuous semantic cues, high-order knowledge discrete semantic cues, and high-order knowledge continuous semantic cues to obtain global semantic cues, and inputting them into the text encoder of the CLIP model to obtain global semantic cues features.

Images