Aerial time-sensitive target identification method based on large model

Abstract

The invention discloses an air time-sensitive target recognition method based on a large model, and belongs to the field of image processing and recognition. The method specifically comprises the following steps: collecting flight trajectories of aircrafts of the same model at different time points, and obtaining image data and text data containing an air time-sensitive target; then, the image is preprocessed, the image is converted into a visual feature vector by using a CLIP model, and encoded image features are obtained; secondly, preprocessing the text to obtain natural language description; and designing a definition rule, dividing the natural language description into multiple types of semantics, designing a semantic prompt combiner to implement organic fusion, and inputting the semantic prompt combiner into a CLIP model to obtain a global semantic prompt feature. And finally, taking the global semantic prompt feature of each frame of image as a query vector, taking the encoded image feature as a key and value vector, and finally effectively identifying the time-sensitive target by enhancing the image feature. According to the method, the goal of obtaining high descriptive and distinguishable performance is improved.

Description

Technical Field

[0001] The invention belongs to the field of image processing and recognition, and relates to realizing image recognition of military flight targets in combination with large model technology, and specifically is an aerial time-sensitive target recognition method based on a large model. Background Art

[0002] With the rapid development of zero-shot and few-shot classification techniques, pre-trained vision-language models (VLMs) such as CLIP have shown great potential. However, due to the lack of camera datasets in large-scale events, models trained directly on event data are still not feasible. Therefore, adapting existing VLMs cross-modally to event vision has become an important research challenge.

[0003] In this context, Wu et al. [1] proposed EventCLIP, which uses CLIP for event-based object recognition and supports zero-shot and few-shot learning. The image encoder of CLIP is generalized by converting the original event into a 2D grid representation. To further improve the performance, Wu et al. also proposed a feature adapter to aggregate the temporal information on the event frames and refine the text embedding to better align with the visual input.

[0004] In multimodal object recognition, an important challenge facing machine learning is domain generalization: this capability is particularly important when dealing with tasks with a different distribution from the training data. Current methods mainly focus on pure visual object recognition, while ignoring the integration of natural language. However, with the advancement of vision-language pre-training technology, it has become possible to use a large number of vision-language pairs for supervised learning. This not only allows learning across different domains, but also enhances recognition capabilities in multimodal scenarios. For example, Qiao et al. [2] proposed the CLIPood method, which demonstrated superior transfer learning capabilities. However, CLIPood still has some limitations: the loss function used is inconsistent with its document description, only evaluating a single model architecture lacks universality, and ignoring category-aware visual fusion. To address these problems, Qiao et al. proposed a new method to derive the actual loss function based on the implementation and expand the evaluation scope to a larger vision-language base model. In addition, Mixup-CLIPood is introduced to enhance category-aware visual fusion through a novel mixed loss.

[0005] Unlike traditional representation learning based on discrete labels, vision-language pre-training aligns images and text into a common feature space, allowing zero-shot transfer to downstream tasks through prompting, i.e., classification weights are generated through natural language describing the class of interest. However, one of the main challenges in deploying these models in practice is prompt engineering, which requires domain-specific expertise and is extremely time-consuming, as small changes in wording can have a huge impact on performance. Inspired by recent progress in prompt learning in the field of natural language processing (NLP), Zhou et al. [3] proposed a method called Context Optimization (CoOp) to automatically adapt pre-trained vision-language models such as CLIP to downstream image recognition. By learning continuous vectors to represent prompt words without manual tuning, CoOp can significantly improve the performance of few-shot learning and is more efficient and robust than hand-crafted prompts and linear detection models.

[0006] Despite this, CoOp still has a key problem: the learned context cannot generalize well to broader categories that have not been seen in the same dataset, which means that CoOp may overfit the underlying categories observed during training. To overcome this problem, Zhou et al. [4] proposed a method called Conditional Context Optimization (CoCoOp) to address the problem of insufficient generalization ability of existing CoOp when facing unseen categories. CoCoOp further learns a lightweight neural network to generate input conditional tokens for each image, so that dynamic prompts can better adapt to each instance, thereby reducing sensitivity to category changes. Experiments show that CoCoOp not only outperforms CoOp in generalization of unseen categories, but also exhibits good transfer ability across datasets and stronger domain generalization performance.

[0007] Sun et al. [5] proposed a fine-grained visual cue learning method (FG-VPL) to improve the image recognition performance of pre-trained visual-language models with a small number of samples. By introducing fine-grained visual cues into the image encoder, the model can focus on the target objects of the downstream task and generate more discernible visual features. In addition, a dual-path adaptive recognition module is proposed to narrow the domain gap and utilize cross-modal matching knowledge and visual information in the small-sample training set for test image classification. Miao et al. [6] proposed the MuDPT (Multimodal Deep Fusion Prompt Tuning) method to improve prompt learning by pre-training visual-language models to achieve co-alignment of text and visual representations. To achieve this goal, the researchers designed a lightweight transformation block (Injection Model) to establish a cross-attention mechanism between different modalities to promote deep bidirectional prompt fusion. Chen et al. [7] proposed an energy-based multi-prompt learning method (EMPL) to improve the rapid adaptation ability of visual-language pre-trained models (VLMs) to downstream tasks by generating multiple prompt embeddings. This approach utilizes examples drawn from the energy distribution implicitly defined by the VLM to generate prompts, which not only improves parameter efficiency but also balances the relationship between in-domain and out-of-domain open-vocabulary generalization.

[0008] Bahng et al. [8] explored the effectiveness of visual prompting in adjusting large-scale visual models. By learning a single image perturbation, the frozen model performs well on new tasks. HiCLIP proposed by Geng et al. [9] introduces a hierarchical perceptual attention mechanism based on CLIP, aiming to gradually discover semantic hierarchies from images and texts in an unsupervised manner. By applying this hierarchical perceptual attention mechanism on the visual and language branches, HiCLIP can better capture high-level and fine-grained semantic information, thereby improving cross-modal alignment and performing well in visual recognition and multimodal tasks. Shi et al.

[10] proposed a new method to enhance the performance of pre-trained models in downstream tasks by using synthetic text images as visual cues. This method, called LoGoPrompt, solves the "chicken and egg" problem of whether to add category-related visual cues first or predict the category first, and redefines the classification goal as visual cue selection. GOPRO proposed by Singha et al.

[11] is a model designed for image recognition tasks, which improves visual recognition performance by combining the advantages of CLIP (a large-scale basic model that can embed images into a semantically rich space) and self-supervised learning (SSL). The model uses learnable image and text projectors to ensure the similarity between different enhanced views of the input image in a shared image-text embedding space, thereby improving the model's invariance and generalization capabilities. GOPRO uses a comprehensive loss function during training, including cross-domain contrast loss, visual contrast loss, and a new prompt consistency loss, which aims to solve the loss weighting and consistency issues between different image perspectives that may be encountered in a multi-task framework.

[0009] Although the above methods have made significant progress in the field of visual-language models, there are still some shortcomings:

[0010] First, many methods rely heavily on prompt engineering, which requires a lot of time and domain expertise to manually design prompts, which is not efficient enough in practical applications.

[0011] Second, existing methods have limited generalization capabilities in the presence of very few or even zero samples, especially when dealing with unseen categories or out-of-domain data.

[0012] In addition, some methods still lack the deep fusion of visual features and text features, and cannot fully utilize cross-modal information to improve recognition performance.

[0013] References are as follows:

[0014] [1]Wu Z,Liu X,Gilitschenski I.Eventclip:Adapting clip for event-basedobject recognition[J].arXiv preprint arXiv:2306.06354,2023.

[0015] [2]Qiao Y,Li K,Lin J,et al.Robust Domain Generalization for Multi-modal Object Recognition[J].arXiv preprint arXiv:2408.05831,2024.

[0016] [3]Zhou K,Yang J,Loy C C,et al.Learning to prompt for vision-languagemodels[J].International Journal of Computer Vision,2022,130(9):2337-2348.

[0017] [4]Zhou K,Yang J,Loy C C,et al.Conditional prompt learning forvision-language models[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2022:16816-16825.

[0018] [5]Sun H,He X,Zhou J,et al.Fine-grained visual prompt learning ofvision-language models for image recognition[C] / / Proceedings of the 31st ACMInternational Conference on Multimedia.2023:5828-5836.

[0019] [6]Miao Y,Li S,Tang J,et al.MuDPT:Multi-modal Deep-symphysis PromptTuning for Large Pre-trained Vision-Language Models[C] / / 2023IEEEInternational Conference on Multimedia and Expo(ICME).IEEE,2023:25-30.

[0020] [7]Chen Z,Huang X,Guan Q,et al.A retrospect to multi-prompt learningacross vision and language[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.2023:22190-22201.

[0021] [8]Bahng H,Jahanian A,Sankaranarayanan S,et al.Exploring visualprompts for adapting large-scale models[J].arXiv preprint arXiv:2203.17274,2022.

[0022] [9]Geng S,Yuan J,Tian Y,et al.HiCLIP:Contrastive language-imagepretraining with hierarchy-aware attention[J].arXiv preprint arXiv:2303.02995,2023.

[0023]

[10] Shi C,Yang S.Logoprompt:Synthetic text images can be good visualprompts for vision-language models[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2023:2932-2941.

[0024]

[11] Singha M, Jha A, Banerjee B.GOPRO: Generate and Optimize Prompts inCLIP using Self-Supervised Learning[J].arXiv preprint arXiv:2308.11605,2023. Summary of the invention

[0025] In response to the above problems, the present invention proposes an aerial time-sensitive target recognition method based on a large model, which integrates image multi-text semantic modeling and visual-semantic association, not only strengthening the correlation between semantic features and visual features, but also achieving the technical effect of improving the model's ability to understand and represent image semantics, and obtaining target sample feature descriptions with strong descriptive and discriminative features.

[0026] The large model-based aerial time-sensitive target recognition method is divided into the following steps:

[0027] Step 1: In the scenario where multiple aircraft of different models are used to identify time-sensitive targets in the air, the flight trajectories of the same model of aircraft at different time points are collected to obtain the image data and text data corresponding to each model of aircraft;

[0028] All flight trajectory videos of the same type of aircraft are divided into frames to obtain an image data set P containing time-sensitive targets: P = {p1, p2, ..., p M}; M is the number of flight trajectory videos;

[0029] At the same time, a text list T = {t1, t2, ..., t M}, each text list corresponds to a flight trajectory video.

[0030] Step 2: Preprocess the image data corresponding to each flight trajectory video of the same model of aircraft respectively;

[0031] Step 3: Use the image encoder of the CLIP model to convert the preprocessed images of each video into a set of visual feature vectors to obtain the image features of each video after encoding.

[0032] The CLIP model uses the Vision Transformer with 12 layers of Transformer modules as the image encoder, and the Patch size is 16;

[0033] The specific conversion process is:

[0034] First, for each flight trajectory video, each preprocessed frame image is decomposed into 16×16 patches through an image encoder, and each patch is mapped to a one-dimensional vector called a token through linear mapping; and a class token is added.

[0035] Then, through the linear embedding layer, the image data corresponding to the video is embedded into the visual tokens sequence to obtain the video sequence X:

[0036]

[0037] Where f is the number of frames in the current video; 512 is the embedding dimension of each token, and 197 is the number of all tokens in each frame.

[0038] Finally, the sequence X is averaged and pooled along the frame channel to obtain the corresponding encoded image feature tensor of the group.

[0039] Feature average pooling means that the token features at the same position in all frames of the video are averagely pooled into one feature. The tensor X v It is the feature matrix of 197 tokens, which is also the encoded image modality feature.

[0040] Step 4: Preprocess the text data of the same model of aircraft to obtain a natural language description of each text list;

[0041] Step 5: Design the definition rules of natural language descriptions and divide the natural language descriptions corresponding to all text lists into three semantic types:

[0042] 1) Category description semantics, directly using the text list extracted by the text encoder of the CLIP model to correspond to the semantic cues related to the category;

[0043] The CLIP model uses a 12-layer Transformer module as the text encoder, and the Patch size is 16;

[0044] 2) Visual perception semantics, using a discrete semantic cue encoder to extract discrete semantic cues of visual perception and a continuous semantic cue encoder to extract continuous semantic cues of visual perception;

[0045] 3) For high-order knowledge semantics, a discrete semantic cue encoder is used to extract discrete semantic cues of high-order knowledge and a continuous semantic cue encoder is used to extract continuous semantic cues of high-order knowledge.

[0046] Firstly, the discrete semantic cue encoder is designed as follows: visual perception semantic text and high-order knowledge semantic text are input into the T5 model respectively, and a simplified summary of the text is output, thereby extracting the key information of the input text; the key information of the input text is summarized and extracted through the T5 model, and discrete semantic cues of visual perception semantics and high-order knowledge semantics are obtained respectively.

[0047] Then, a continuous semantic cue encoder is designed to input the visual perception semantic text and high-order knowledge semantic text into the pre-trained KEPLER model respectively, and combined with a lightweight projection mapping network to construct the continuous semantic cues of visual perception and the continuous semantic cues of high-order knowledge respectively;

[0048] Step 6: Design a semantic cue merger to organically integrate the multi-type semantics corresponding to all text lists and input it into the text encoder of the CLIP model to obtain the final global semantic cue features.

[0049] Step 7: Use the global semantic cue features of each video as the query vector and the encoded image features as the key and value vectors, obtain enhanced image features through the cross-modal attention mechanism, and summarize all regional visual information to obtain a target sample feature description with strong descriptive and discriminative features, and finally effectively identify the time-sensitive target category in the video.

[0050] The specific process is:

[0051] First, for the i-th frame of the current video, the global semantic cue feature corresponding to its text list is w i , the encoded image feature is X i ; Enhanced image features of this frame for:

[0052]

[0053] Among them, CrossAttention(·) refers to cross attention, LN is layer normalization, and x cls Represents the features of the [CLS] tag of the input image.

[0054] Then, the average enhanced image features of all frames in the video are calculated to obtain the average image features. as a global image feature.

[0055] Next, the normalized cosine similarity is calculated to calculate the image feature X after encoding the frame. i The probability of belonging to a global image feature:

[0056]

[0057] where τ is a learnable temperature parameter and sim(·) represents the cosine similarity.

[0058] Finally, the cosine similarity is optimized by probability so that the image feature X after encoding the frame i The similarity with the correct text features is maximized, and the similarity with other categories is minimized;

[0059] The specific optimization is to minimize the cross entropy loss:

[0060]

[0061] where y i is the one-hot encoding of the i-th frame image.

[0062] The advantages of the present invention are:

[0063] 1. A method for aerial time-sensitive target recognition based on a large model designs image multi-semantic definition rules and divides the multi-semantics corresponding to visible light images into three types: category description semantics, visual perception semantics, and high-order knowledge semantics. This allows for a comprehensive definition and effective modeling of the target image semantics, a strong correspondence between the image and the paired semantics, and the enrichment of the image sample semantic information from multiple angles.

[0064] 2. A method for identifying aerial time-sensitive targets based on a large model, interactive learning of target visual content information and semantic information, and interaction of the image's visual feature vector with the global semantic cue features through a cross-modal attention mechanism, strengthening the correspondence between the target's visual content and multi-dimensional semantic information. Under this mechanism, the key visual areas in the image are highlighted, forming a deep interaction between visual features and semantic features, so that each visual information fragment can effectively correspond to its semantic cue. This interactive mechanism not only strengthens the descriptiveness and resolution of image features, but also significantly improves target recognition accuracy, providing technical support for the accurate recognition of aerial time-sensitive targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of a method for identifying aerial time-sensitive targets based on a large model according to the present invention;

[0066] Figure 2 It is an algorithm model framework diagram of an aerial time-sensitive target recognition method based on a large model of the present invention;

[0067] Figure 3 This is a schematic diagram of the network architecture of the CLIP image encoder ViT-B / 16 of the present invention;

[0068] Figure 4 It is a sample diagram of each category of the data set in the embodiment of the present invention;

[0069] Figure 5 It is a graph of various average accuracy curves of small-scale data sets of the present invention;

[0070] Figure 6 It is a graph of various indicators of a small-scale data set of the present invention;

[0071] Figure 7 It is a graph of various average accuracy curves of large-scale data sets of the present invention;

[0072] Figure 8 It is a curve chart of various indicators of the large-scale data set of the present invention;

[0073] Fig. 9 This is the Attention Map visualization diagram of the small-scale dataset of the present invention;

[0074] Fig.10 This is a visualization diagram of the Attention Map of the large-scale dataset of the present invention. DETAILED DESCRIPTION

[0075] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0076] The present invention provides an aerial time-sensitive target recognition method based on a large model, which is based on the demand for accurate capture and recognition of specific aerial targets in the scenarios of national defense construction and regional security control, and provides reliable technical support for the detection, monitoring and early warning of key targets. The core of the present invention is to explore and develop a multimodal aerial target capture and recognition technology based on a CLIP pre-trained large model and text semantic prompts, and to achieve high-precision capture and recognition of aerial targets by making full use of the complementary information of visible light images and text prompts. Specifically, the present invention uses massive image training samples as data support and sample semantic prompts as an external knowledge source. With the help of the large model's effective modeling of text content and the perception of image visual content, the present invention realizes effective interaction and learning of knowledge semantic information and visual content information, and improves the system's capture and recognition accuracy of specific aerial targets.

[0077] The present invention adopts the method of image multi-text semantic modeling and visual-semantic association fusion. By dividing the multi-text semantics corresponding to the visible light image into category description semantics, visual perception semantics and high-order knowledge semantics, the semantic information contained in the text is fully mined and integrated into global semantic cues. Then, through the cross-modal attention mechanism, the important role of key visual areas in global feature representation is highlighted, making the semantic information of the image more structured and hierarchical. At the same time, the global semantic hint feature is used as the query vector, and the encoded image feature is used as the key and value vector to highlight the important role of key visual areas in global feature representation. Such a design not only strengthens the correlation between semantic features and visual features, but also achieves the technical effect of improving the model's ability to understand and represent image semantics, and obtaining a target sample feature description with strong descriptiveness and resolution.

[0078] Compared with existing methods, the present invention has the advantages of enhancing the richness and distinctiveness of image features and improving the accuracy of target recognition, providing a more accurate semantic basis for subsequent target recognition tasks and providing new ideas for cross-modal feature fusion.

[0079] The large model-based aerial time-sensitive target recognition method is as follows: Figure 1 and Figure 2 As shown, the following steps are taken:

[0080] Step 1: In the scenario where multiple aircraft of different models are used to identify time-sensitive targets in the air, the flight trajectories of the same model of aircraft at different time points are collected to obtain the image data and text data corresponding to each model of aircraft;

[0081] All flight trajectory videos of the same type of aircraft are divided by frames to obtain an image data set P containing time-sensitive targets:

[0082] M is the number of flight trajectory videos; F is the total number of frames, C is the number of image channels, H and W are the height and width of the image respectively;

[0083] At the same time, a text list of all flight tracks of this model of aircraft is compiled Each text list corresponds to a flight trajectory video.

[0084] Step 2: Preprocess the image data corresponding to each flight trajectory video of the same model of aircraft respectively;

[0085] Preprocessing means: for the current video, the resolution of each frame image in the video is adjusted to 224×224 while maintaining the aspect ratio, and insufficient pixels are filled with zero pixels.

[0086] Step 3: Use the image encoder of the CLIP model to convert the preprocessed images of each video into a set of visual feature vectors to obtain the image features of each video after encoding.

[0087] The CLIP model uses a Vision Transformer with 12 layers of Transformer modules as the image encoder, and the Patch size is 16.

[0088] The specific conversion process is:

[0089] First, for each flight trajectory video, the preprocessed frame image is decomposed into 16×16 patches through the image encoder, that is, 196÷16=14, each row and column has 14 patches, a total of 14×14=196 patches. Each patch is mapped to a one-dimensional vector through linear mapping, called a token, a total of 196 tokens; and a class token is added; therefore, each frame image contains a total of 197 tokens.

[0090] Then, through the linear embedding layer, the image data corresponding to the video is embedded into the visual tokens sequence to obtain the video sequence X:

[0091]

[0092] Where f is the number of frames in the current video; 512 is the embedding dimension of each token, and 197 is the number of all tokens in each frame image.

[0093] Finally, the sequence X is averaged and pooled along the frame channel to obtain the corresponding encoded image feature tensor of the group.

[0094] Feature average pooling means that the token features at the same position in all frames of the video are averagely pooled into one feature. The tensor X v It is the feature matrix of 197 tokens, which is also the encoded image modality feature.

[0095] Step 4: Preprocess the text data of the same model of aircraft to obtain a natural language description of each text list;

[0096] Each text list is expanded to obtain a natural language description; the image semantic definition includes aerodynamic layout, appearance description, paint color, paint material, thrust layout, deployment orientation, usage, intelligence and other information.

[0097] After text preprocessing, the semantics of each image is effectively standardized.

[0098] Step 5: Design the definition rules of natural language descriptions and divide the natural language descriptions corresponding to all text lists into three semantic types:

[0099] 1) Category description semantics, that is, a direct description of the category to which the image containing the target belongs. The text list extracted by the text encoder of the CLIP model directly corresponds to the category-related semantic prompts; such as "This is an image of [category label], where [category label] is the image category information such as reconnaissance aircraft, B2 stealth bomber, etc."

[0100] The CLIP model uses a 12-layer Transformer module as the text encoder, and the Patch size is 16;

[0101] 2) Visual perception semantics, i.e., text descriptions related to the visual characteristics of the target, using a discrete semantic cue encoder to extract discrete semantic cues of visual perception and a continuous semantic cue encoder to extract continuous semantic cues of visual perception; such as target shape description, contour description, color description, etc.

[0102] 3) High-order knowledge semantics, which does not directly reflect the visual characteristics of the target but reflects the deep-level attributes of the target or specific text description. A discrete semantic cue encoder is used to extract discrete semantic cues of high-order knowledge and a continuous semantic cue encoder is used to extract continuous semantic cues of high-order knowledge; such as power parameters, intelligence descriptions, etc.

[0103] For category description semantics, the CLIP model is directly used to pre-train the text feature encoder to extract category-related semantic cues;

[0104] For visual perception semantics and high-order knowledge semantics, discrete semantic cue encoders and continuous semantic cue encoders are designed respectively to extract discrete semantic cues and continuous semantic cues of visual perception and high-order knowledge, combined with the image-semantic cue interaction method to achieve the purpose of making full use of semantic information.

[0105] First, the discrete semantic cue encoder process is designed as follows:

[0106] Visual perception semantics and high-order knowledge semantics can be used as external knowledge data to improve the accuracy of image object recognition, but text descriptions are usually redundant and contain redundant (noise) information that is irrelevant to visual recognition. In order to remove this information from the two types of semantic information, the visual perception semantic text and high-order knowledge semantic text are respectively input into the T5 model. The T5 model is an end-to-end pre-trained model that takes a piece of text as input and outputs a simplified summary of the text, thereby extracting the key information of the input text; the T5 model is used to summarize and extract the key information of the input text, and obtain discrete semantic cues of visual perception semantics and high-order knowledge semantics respectively.

[0107] Then, a continuous semantic cue encoder is designed. In the above discrete cue learning process, the simplified text description will inevitably filter out some category-related information as redundant information. In order to allow the generated cue to carry background context information that may cover a wider range of current image categories, the visual perception semantic text and high-order knowledge semantic text are respectively input into the pre-trained KEPLER model, combined with a lightweight projection mapping network, to construct visual perception continuous semantic cues and high-order knowledge continuous semantic cues respectively;

[0108] Step 6: Design a semantic cue merger to organically integrate the multi-type semantics corresponding to all text lists and input it into the text encoder of the CLIP model to obtain the final global semantic cue features.

[0109] Organic fusion means: concatenating category description semantic cues, visual perception discrete semantic cues, visual perception continuous semantic cues, high-order knowledge discrete semantic cues and high-order knowledge continuous semantic cues to obtain global semantic cues, and inputting them into the text encoder of the CLIP model to obtain global semantic cues features.

[0110] Step 7: Use the global semantic cue feature as the query vector and the encoded image feature as the key and value vectors, obtain enhanced image features through the cross-modal attention mechanism, and summarize all regional visual information to obtain a target sample feature description with strong descriptive and discriminative features, and finally effectively identify the time-sensitive target category in the video.

[0111] The CLIP model is a typical dual encoder architecture, consisting of an image encoder and a text encoder. Each image and paired text are converted into high-dimensional embeddings by the image encoder and the text encoder respectively. The training goal is to align unimodal embeddings through contrastive learning, where the model matches paired images and texts together and pulls unpaired images and texts farther apart in the latent space. By pre-training on large-scale image and text datasets, the learned image visual features are discriminative and can be transferred to zero / few-shot downstream tasks.

[0112] To further adapt the visual representation to image content matching and target image recognition tasks to reduce the interference of task-irrelevant visual concepts, our method constructs a task-specific visual adaptation head that focuses on salient visual cues related to category information.

[0113] The specific process is:

[0114] First, for the i-th frame of the current video, the global semantic cue feature corresponding to its text list is w i , which is used as the query vector, and its encoded image feature is X i As key and value vectors; the enhanced image features of the frame for:

[0115]

[0116] Among them, CrossAttention(·) refers to cross attention, LN is layer normalization, and x cls Represents the features of the [CLS] tag of the input image.

[0117] Then, in order to summarize all the information, the enhanced image features of all frames in the video are averaged to obtain the average image feature as a global image feature.

[0118] Next, the normalized cosine similarity is calculated to calculate the image feature X after encoding the frame. i The probability of belonging to a global image feature:

[0119]

[0120] where τ is a learnable temperature parameter and sim(·) represents the cosine similarity.

[0121] Finally, the probability is used to optimize the cosine similarity between the image feature and the global semantic hint feature, so that the image feature X i The similarity with the correct text features is maximized, and the similarity with other categories is minimized;

[0122] During training, the gradients of the knowledge-aware cue and adaptation head are updated while keeping the parameters of CLIP unchanged. The training objective is to minimize the cross entropy loss:

[0123] where y i is the one-hot encoding of the i-th frame image.

[0124] In this way, the performance of the model can be fine-tuned when processing specific tasks without affecting the generalization ability of the CLIP model that has been pre-trained with large-scale data.

[0125] Example:

[0126] The dataset is divided into five aircraft models, namely B-2, B-52, E2D, F18, and F35. The samples of each category are as follows: Figure 4 During the experiment, the first batch of data sets had a small number of sample images, with only about 1,000 sample images for each category, while the second batch of data sets had about 10,000 sample images for each category.

[0127] Since the entire dataset is composed of frames of the aircraft at different time points in the video, they have high similarity, which may affect the accuracy and efficiency of subsequent analysis. In order to solve this problem, a structured approach was adopted to organize and divide the original dataset. Specifically, the dataset was first classified by video track according to the aircraft model. The purpose of this is to group similar images, which is convenient for subsequent processing and analysis. Next, the dataset of each aircraft model is subdivided: every 60 frames are selected as a grouping unit. The design is based on the following settings:

[0128] Temporal continuity: Every 60 frames roughly corresponds to a few seconds of aircraft video images during data collection, which provides a reasonable time window for the target recognition task to capture the movement and changes of the aircraft during this period.

[0129] The right amount of data: 60 frames is neither too much to overload processing nor too little to capture enough information.

[0130] Information Richness: Each 60-frame image sequence can provide enough visual information to identify specific features and behaviors of the aircraft.

[0131] Through the above sampling method, the data set is divided into 60 frames of pictures according to different aircraft models, and finally a track consisting of multiple 60 pictures is obtained. Each track represents the flight trajectory of a specific aircraft model over a period of time. This data organization method not only helps to improve the efficiency of data processing, but also helps with subsequent analysis and model training, because tasks such as feature extraction, pattern recognition, and behavior analysis can be performed based on these tracks.

[0132] Finally, the original dataset is randomly divided into training and test sets according to 80% and 20% of the tracks. Specifically, about 4000 / 40000 images are used as training sets, and 1000 / 10000 images are used as test sets.

[0133] During the experimental implementation, considering the efficiency and accuracy of the CLIP model based on ViT-B / 16, this model was used as the backbone network of the main network architecture during the project implementation. The CLIP model uses the Vision Transformer and Transformer with 12 layers of Transformer modules as image encoder and text encoder, and the Patch size is 16.

[0134] The preprocessed image set P is passed through the CLIP image encoder. The image is first decomposed into 16×16 patches, so a 224×224 image will be decomposed into (224 / 16) 2= 196 patches, and then each patch is mapped to a one-dimensional vector through linear mapping. These one-dimensional vectors are called a token. The specific process is as follows Figure 3 As shown in Figure 1, the encoded image data is then embedded into the visual tokens sequence X through a linear embedding layer. Then, the feature is averaged into a tensor X along the time channel (i.e., frame channel) v .

[0135] During the training phase, the parameters of the CLIP encoder are fixed. This is done to take advantage of the rich features and knowledge that the pre-trained CLIP model has learned, while preventing the destruction of the pre-trained CLIP model parameters and reducing the computational burden during training. The learning rate of the model is 0.001, which is a relatively small value that helps the model converge stably during training and avoid overfitting. At the same time, the weight decay technology is introduced and set to 1e-4. This is a regularization method to prevent the model parameters from being too large and further improve the generalization ability of the model.

[0136] For the target category text list, it needs to be processed into the corresponding language description to make full use of the CLIP text encoder, which uses the same number of Transformer layers as the image encoder. Specifically, using a carefully designed prompt template, each category text is first expanded to obtain a natural language description. For example, "B-2" is processed into "This is a photo of B-2, a type of airplane".

[0137] For two datasets of different sizes, the model was trained for 50 and 20 epochs respectively, with a batch size of 32. In addition, the number of frames was set to 10, which means that in each iteration, the model processes 10 frames of continuous image data, which helps the model capture the dynamic changes and timing information in continuous images. Adam is used as the optimizer. In the experiment, the model was implemented using Python and PyTorch frameworks and trained on a server equipped with A40.

[0138] During the experiment of this project, experiments were conducted on the datasets before and after expansion. Next, the experimental results are introduced by evaluating the test sets of the two-scale datasets.

[0139] When evaluating the model effect, the widely used accuracy, precision, recall and F1 score are used as evaluation indicators. It should be noted that the results reported in the experiment are obtained by averaging multiple categories. The specific formula is as follows:

[0140]

[0141] Among them, TP, TN, FP, and FN are the number of correctly predicted positive examples, the number of correctly predicted negative examples, the number of negative examples incorrectly predicted as positive examples, and the number of positive examples incorrectly predicted as negative examples, respectively.

[0142] Accuracy is the most intuitive performance indicator, which measures the proportion of samples that the model predicts correctly to the total number of samples. Accuracy is easy to calculate even in an unbalanced dataset, but it is not always the best performance indicator, because in the case of most negative examples and a few positive examples, a model that always predicts negative examples can also have a high accuracy.

[0143] Precision: Precision measures the proportion of samples predicted by the model as positive that are actually positive. It focuses on the accuracy of the prediction of positive examples. When there are few positive examples in the dataset, precision is an important indicator because it can reduce false positives for rare events.

[0144] Recall measures the proportion of all actual positive examples that are correctly predicted as positive by the model. It focuses on the model's ability to capture all positive examples. Recall is particularly important when it is necessary to minimize missed positives.

[0145] The F1 score is the harmonic mean of precision and recall, which tries to find a balance between precision and recall. The F1 score is high when both are high, so it is an indicator that strikes a balance between precision and recall. The F1 score is particularly useful when the dataset is extremely unbalanced.

[0146] Figure 5 and Figure 7 It is a curve chart showing the average precision (AP) of different scale data sets as the epoch changes. Figure 6 and Figure 8 It is the change curve of average precision (average_mAP), accuracy (average_ACC), recall rate (average_RECALL), and F1 score (average_F1) of different scale data sets over epochs.

[0147] It can be seen from the four curves that no matter which indicator it is, it basically increases gradually with the increase of epochs, and then tends to stabilize in the last few epochs.

[0148] Attention Map visualization is of great significance for understanding the working mechanism of deep learning models, especially Transformer models and their variants, and provides an explanatory view of the decision-making process within the model. By visualizing the attention distribution, we can better understand the decision-making process of the model and identify whether the key areas that the model focuses on in the data are in line with expectations.

[0149] Therefore, samples of five categories are input into the model respectively, and then the multi-head attention weight matrix obtained in the 12-layer Transformer Encoder of the image encoder is extracted for averaging. After extracting the token vector related to the classification, the attention weight matrix is ​​reshaped, interpolated and scaled to the same size as the input image, and then the attention weight matrix is ​​superimposed on the input image for display to obtain the visualization result of the Attention Map. According to two data sets of different sizes, the first, third, fifth, seventh, ninth, and twelfth layers of the 12-layer Transformer Encoder are displayed respectively.

[0150] according to Fig. 9 and Fig.10 The Attention Map visualization result shows that in the first few layers of Transformer Encoder, the network's attention is more focused on the target, while in the later layers, the attention is dispersed to the surrounding positions in the image. Let's analyze this phenomenon.

[0151] In the Transformer model, it is common to see changes in attention distribution as the layers go deeper. In the early attention layers, the model tends to focus on the most salient and relevant parts of the input sequence, which may be the main target or object in image processing tasks. This is because the lower-level attention layers are closer to the original input, and therefore more likely to capture local features and direct visual information.

[0152] However, as the model layers increase, the output of each attention layer serves as the input of the next layer, so that each layer can further abstract and combine features based on the previous layer. This hierarchical feature abstraction allows the model to capture more complex and abstract concepts at higher levels. Therefore, in deeper layers, attention is no longer limited to a specific area, but is spread over the entire input sequence to understand more global information and contextual relationships.

[0153] This phenomenon is found in both natural language processing and computer vision tasks. For example, when processing natural language, early attention layers may focus on individual words or phrases, while later attention layers may focus on entire sentences or paragraphs to understand the overall meaning of the text. In vision tasks, early layers may focus on specific objects or textures in an image, while later layers may focus on the layout and structure of the entire scene.

[0154] This dispersion of attention is not random, but an effective strategy learned by the model to capture different levels of information at different levels. This also explains why Transformer models can perform well in a variety of tasks because they can flexibly handle feature representations ranging from simple to complex.

[0155] In addition, spreading attention also helps the model process a wider range of contextual information, which is especially important when processing long sequences or large-scale images. By spreading attention across different levels, the model is able to better understand the global structure and dependencies of the input data, thereby improving the accuracy and robustness of its predictions.

[0156] Therefore, as the number of Transformer layers increases, the dispersion of attention reflects the model's gradual abstraction process from local features to global context, which is one of the key advantages of the Transformer architecture design.

Claims

1. A method for identifying aerial time-sensitive targets based on a large model, characterized in that: The specific steps are as follows: Step 1: In the scenario where multiple aircraft of different models are used to identify time-sensitive targets in the air, the flight trajectories of the same model of aircraft at different time points are collected to obtain the image data and text data containing the target corresponding to each model of aircraft; Step 2: Preprocess the image data corresponding to each flight trajectory video of the same model of aircraft respectively; Step 3: Using the image encoder of the CLIP model, convert the preprocessed images of each video into a set of visual feature vectors to obtain the image features of each video after encoding; Step 4: Preprocess the text data of the same model of aircraft to obtain the natural language description of each text list; Step 5: Design the definition rules of natural language descriptions and divide the natural language descriptions corresponding to all text lists into three semantic types: 1) Category description semantics, directly using the text list extracted by the text encoder of the CLIP model to correspond to the semantic cues related to the category; 2) Visual perception semantics, using a discrete semantic cue encoder to extract discrete semantic cues of visual perception and a continuous semantic cue encoder to extract continuous semantic cues of visual perception; 3) High-order knowledge semantics, using discrete semantic cue encoder to extract discrete semantic cues of high-order knowledge and continuous semantic cue encoder to extract continuous semantic cues of high-order knowledge; Step 6: Design a semantic hint merger to organically merge the multi-type semantics corresponding to all text lists and input it into the text encoder of the CLIP model to obtain the final global semantic hint feature; Step 7: Take the global semantic cue features of each video as the query vector and the encoded image features as the key and value vectors, obtain enhanced image features through the cross-modal attention mechanism, and summarize all regional visual information to obtain a highly descriptive and discriminative feature description of the target sample, and finally effectively identify the time-sensitive target category in the video; The specific process is: First, for the i-th frame of the current video, the global semantic cue feature corresponding to its text list is w i , the encoded image feature is X i ; Enhanced image features of this frame for: Among them, CrossAttention(·) refers to cross attention, LN is layer normalization, and x cls The features of the [CLS] tag representing the input image; Then, the average enhanced image features of all frames in the video are calculated to obtain the average image features. As a global image feature; Next, the normalized cosine similarity is calculated to calculate the image feature X after encoding the frame. i The probability of belonging to a global image feature: Where τ is a learnable temperature parameter and sim(·) represents the cosine similarity; Finally, the cosine similarity is optimized by probability so that the image feature X after encoding the frame i The similarity with the correct text features is maximized, and the similarity with other categories is minimized; The specific optimization is to minimize the cross entropy loss: where y i is the one-hot encoding of the i-th frame image.

2. A method for identifying aerial time-sensitive targets based on a large model as claimed in claim 1, characterized in that: In the step 1, all flight trajectory videos of the same type of aircraft are divided by frames to obtain an image data set P containing time-sensitive targets: M is the number of flight trajectory videos; F is the total number of frames, C is the number of image channels, H and W are the height and width of the image respectively; At the same time, a text list of all flight tracks of this model of aircraft is compiled Each text list corresponds to a flight trajectory video.

3. A method for identifying aerial time-sensitive targets based on a large model as claimed in claim 1, characterized in that: In the step 2, the preprocessing means: for the current video, the resolution of each frame image in the video is adjusted to 224×224 while maintaining the aspect ratio, and the insufficient pixels are filled with zero pixels.

4. A method for identifying aerial time-sensitive targets based on a large model as claimed in claim 1, characterized in that: In step 3, the CLIP model uses a Vision Transformer with 12 layers of Transformer modules as the image encoder, and the Patch size is 16; In step 5, the CLIP model uses a Transformer with 12 layers of Transformer modules as the text encoder, and the Patch size is 16.

5. A method for identifying aerial time-sensitive targets based on a large model as claimed in claim 4, characterized in that: The specific conversion process of step 3 is as follows: First, for each flight trajectory video, each preprocessed frame image is decomposed into 16×16 patches through an image encoder, and each patch is mapped to a one-dimensional vector called a token through linear mapping; and a class token is added; Then, through the linear embedding layer, the image data corresponding to the video is embedded into the visual tokens sequence to obtain the video sequence X: Where f is the number of frames in the current video; 512 is the embedding dimension of each token, and 197 is the number of all tokens in each frame image; Finally, the sequence X is averaged and pooled along the frame channel to obtain the corresponding encoded image feature tensor of the group. Feature average pooling means that the token features at the same position in all frames of the video are averagely pooled into one feature. The tensor X v It is the feature matrix of 197 tokens, which is also the encoded image modality feature.

6. A method for identifying aerial time-sensitive targets based on a large model as claimed in claim 1, characterized in that: In the step 4, each text list is expanded to obtain a natural language description, including aerodynamic layout, appearance description, paint color, paint material, thrust layout, deployment position, usage and intelligence information.

7. A method for identifying aerial time-sensitive targets based on a large model as claimed in claim 1, characterized in that: In the step 5, the discrete semantic hint encoder is designed as follows: the visual perception semantic text and the high-order knowledge semantic text are respectively input into the T5 model, and the simplified summary of the text is output, so as to extract the key information of the input text; the key information of the input text is summarized and extracted by the T5 model, and the discrete semantic hints of the visual perception semantics and the high-order knowledge semantics are respectively obtained; The process of designing a continuous semantic cue encoder is as follows: visually perceived semantic text and high-order knowledge semantic text are input into the pre-trained KEPLER model respectively, and a lightweight projection mapping network is combined to construct continuous semantic cues of visual perception and continuous semantic cues of high-order knowledge respectively.

8. A method for identifying aerial time-sensitive targets based on a large model as claimed in claim 1, characterized in that: In the step six, organic fusion means: concatenating category description semantic cues, visual perception discrete semantic cues, visual perception continuous semantic cues, high-order knowledge discrete semantic cues and high-order knowledge continuous semantic cues to obtain global semantic cues, and inputting them into the text encoder of the CLIP model to obtain global semantic cues features.

Images