Image-level automatic prompt generation target detection method based on thinking chain

Abstract

The invention provides an image-level prompt automatic generation target detection method based on a thinking chain, and belongs to the field of multi-modal target detection. The method provided by the invention comprises the following steps: 1) inputting an image into a concept extraction module, and screening and identifying a foreground object region; 2) establishing a common sense knowledge graph GC, and deleting non-coexisting objects in the preliminary prompt Praw; 3) performing scene association and concept reasoning supplementation on the Ppreune by using a large language model; 4) designing a self-adaptive threshold algorithm, and deleting a concept with low confidence in the associated and expanded prompt Psearch to obtain an image-level prompt Pimage; and 5) inputting the prompt Pimage and the corresponding image I into the visual language model to complete detection. Based on the thinking chain idea, the image level prompt automatic generation method is designed by utilizing the generalization ability of the multi-modal large model and the text generation ability of the large language model, and the accuracy of visual language model target detection is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal natural image detection, and in particular, relates to an image-level automatic prompt generation target detection method based on a thought chain. Background Art

[0002] Visual recognition tasks such as image classification, object detection, and semantic segmentation have been long-standing problems in computer vision research and are the basis for many computer vision applications such as autonomous driving, remote sensing, and robotics. With the emergence of deep learning, visual recognition research has achieved great success by utilizing end-to-end trainable deep neural networks (DNNs). However, the transition from traditional machine learning to deep learning faces two major difficulties: the slow convergence of training DNN networks from scratch and the difficulty of collecting large-scale, task-specific, and manually labeled training data.

[0003] In recent years, a new "pre-training-fine-tuning-prediction" learning paradigm has gradually emerged and has shown strong effectiveness in many visual recognition tasks. Under this new paradigm, the DNN model is first pre-trained using some existing large-scale data. Some of this data is labeled and some is not. After that, the model is fine-tuned using labeled training data for a specific task. Because the model has learned more comprehensive knowledge during the pre-training stage, this paradigm can accelerate network convergence and make it easier to train models with good performance for various downstream tasks.

[0004] Nevertheless, this "pre-training-fine-tuning-prediction" paradigm still requires an additional fine-tuning stage, using annotated data from downstream tasks, to achieve the desired effect. Later, with the rapid development of large language models in the field of natural language processing, a new deep learning paradigm - visual language model pre-training-zero-shot prediction has attracted a lot of attention recently. In this paradigm, large-scale image-text pairs of visual language models (VLMs) are pre-trained. These image-text pairs are almost all from the Internet, so the pre-trained model can be directly applied to downstream visual recognition tasks without further fine-tuning.

[0005] First, one of the core technologies of visual language models is visual feature extraction. By using advanced deep learning architectures such as convolutional neural networks (CNN) and transformers, VLMs are able to extract high-dimensional visual features from images, which can capture objects, scenes, and details in the image. For example, Residual Network (ResNet) and Vision Transformer (ViT) are common visual feature extraction models that perform well in various visual tasks.

[0006] Pre-training VLMs is usually guided by a specific visual language objective, which allows the model to learn the connection between images and text through a large number of image-text pairs. For example, CLIP applies an image-text contrast objective, which is achieved in the latent space by reducing the distance between matching images and texts and increasing the distance between mismatched pairs. In this way, the pre-trained visual language model can capture a large amount of knowledge about the association between vision and language, and can match it with the given image-text embedding vectors to give zero-shot prediction results. This new learning paradigm can effectively utilize network data and can directly perform zero-shot prediction without the need for fine-tuning of specific tasks. It is not only easy to deploy, but also has ideal results. For example, pre-trained CLIP has achieved the best zero-shot prediction results in 36 visual recognition tasks, ranging from classic image classification to human action recognition to visual symbol recognition.

[0007] Secondly, the processing of language features usually relies on pre-trained language models, such as BERT, GPT, etc. These models are trained on large-scale corpora and have strong language understanding and generation capabilities. In visual language models, language features and visual features are usually fused through a cross-modal attention mechanism, allowing the model to generate descriptions that are closely related to the image content or perform tasks such as visual question answering.

[0008] Large-scale language models (LLMs) are large-scale, pre-trained statistical language models based on neural networks. The recent success of large-scale language models is the accumulation of decades of research and development of language models. According to different starting points and development speeds, they can be roughly divided into four stages. They are statistical language models, neural language models, pre-trained language models, and large-scale language models.

[0009] Statistical language models (SLMs) view text as a sequence of words and estimate the probability of a text as the product of its word probabilities. The main form of SLM is a Markov chain model called the n-gram model, which computes the probability of a word based on the next (n-1) words to be processed. Since word probabilities are estimated using word and n-gram counts collected from a text corpus, the model needs to handle data sparsity by using smoothing (i.e., assigning zero probability to unseen words or n-grams), where the model retains probabilistic redundancy for some unseen n-gram words. N-gram models are widely used in many NLP systems. However, these models are incomplete due to data sparsity and they cannot fully capture the diversity and variability of natural language.

[0010] Early neural language models (NLMs) dealt with data sparsity by mapping words to low-dimensional continuous vectors, also called embedding vectors, and predicted the next word based on an aggregation of words, embedding the vectors of its subsequent words using a neural network. The embedding vectors learned by NLMs define a latent space where the semantic similarity between vectors can be easily computed as their distance. This opens the door to computing the semantic similarity of any two inputs, regardless of their form (e.g., queries vs. documents in web search, sentences in different languages ​​in machine translation) or modality (e.g., images and text in image captioning). Early NLMs were task-specific models in that they were trained on task-specific data and the latent space they learned was task-specific.

[0011] Unlike early NLMs, pre-trained language models (PLMs) are task-agnostic. This generalization also extends to the learned hidden embedding space. Training and inference of PLMs follow the “pretrain-finetune” paradigm, where a language model with a recurrent neural network or transformer is pre-trained on a network-scale collection of unlabeled text for a general task (e.g., word prediction) and then fine-tuned using a small amount of labeled, task-specific data.

[0012] Large language models mainly refer to transformer-based neural language models that contain hundreds of millions to hundreds of millions of parameters and have been pre-trained on massive text data, such as PaLM, LLaMA, and GPT-4. Compared with pre-trained language models, large language models are not only much larger in model size, but also show stronger language understanding and generation capabilities. More importantly, large language models have emerging capabilities that have never appeared in small-scale language models, including (1) contextual learning, in which LLM learns new tasks through a small number of examples in the prompt during the reasoning stage; (2) instruction following, in which LLM can follow new task instructions without explicit examples after instruction fine-tuning; (3) multi-step reasoning, in which LLM decomposes a complex task into multiple intermediate reasoning steps based on thought chain prompts. Large language models can also be enhanced with external knowledge and tools to achieve effective interaction with users and the environment, and use feedback data collected during the interaction process to continuously optimize themselves.

[0013] Through advanced usage and enhancement techniques, large language models can be deployed as so-called AI agents, artificial entities that can perceive the environment, make decisions, and take actions. Previous research has focused on developing agents for specific tasks and domains. The emerging capabilities demonstrated by large language models make it possible to build general AI agents based on LLMs. While large language models are trained to produce responses in static environments, AI agents need to take actions to interact with dynamic environments. Therefore, LLM-based agents usually require enhancements to the LLM, for example, to obtain updated information from external knowledge bases, to verify that system operations produce expected results, and to handle when things do not go as expected.

[0014] As large models in the fields of computer vision and natural language have developed rapidly, large models in multimodal fields have also gradually emerged, among which visual language models are the most typical.

[0015] The visual language model is a model that analyzes spatially related information of image blocks. An image can be divided into many regions, each of which can be expressed by a visual word based on its visual features. These visual words not only have spatial dependencies, but also co-occurrence frequencies. By using these characteristics between visual words, the semantics of the image can be easily mined.

[0016] The smallest semantic unit of natural language processing is a word. Correspondingly, the smallest semantic unit of an image is a visual word. Assuming that the visual words of an image are arranged according to a certain pattern, if this arrangement is completely random, it cannot reflect the meaningful image semantics. For convenience, it is assumed that visual words are not independent of each other, and that adjacent visual words obey a certain arrangement pattern. The semantics of an image can be mined by modeling the arrangement pattern of visual words.

[0017] In this process, text is needed as a prompt to guide the model to mine the visual features of the image, so prompt learning came into being. The prompt learning method was first proposed in the field of natural language processing, aiming to achieve efficient application of large models in multiple downstream tasks. Specifically, it embeds task-related information in the input layer or intermediate layer of the model to guide the large model to solve multiple tasks. By inheriting the pre-training form, the application of maintaining consistency between downstream tasks and pre-training is achieved. In this section, we first briefly introduce the pre-training model in the field of natural language processing, and then introduce the prompt learning method based on the pre-training model, including basic concepts, downstream application forms, and prompt template types.

[0018] A pre-trained model is a model that is trained on large-scale data through specific proxy tasks. It usually has strong feature extraction and data understanding capabilities and can be widely used in various downstream tasks. In the field of natural language processing, there are currently four typical pre-trained models, namely:

[0019] (1) Autoregressive language model: In the pre-training stage, language representation learning is performed by predicting words one by one from left to right. Specifically, given a text sequence, the model needs to predict the word at the current position based on the previously visible text sequence on the left. Representative models of this type of work include GPT-3, RETRO, GPT-Neo, etc.

[0020] (2) Masked language model: The autoregressive language model can only learn to represent text from left to right, which limits the model's ability to build bidirectional text representations. To this end, the masked language model randomly masks words or fragments in the text during the pre-training phase, predicts the words at the masked position based on the context, and achieves bidirectional understanding of the text. Representative models of this type include BERT and ERNIE.

[0021] (3) Prefix sequence language model: For tasks such as machine translation or text summarization based on given conditional text, the given text is used as a prefix sequence for bidirectional text modeling, and then the subsequent target text is predicted using an autoregressive approach. Representative models of this type of work include UniLM, UniLMv2, and ERNIEM.

[0022] (4) Encoder-decoder language model: Similar to the prefix sequence language model, this type of model uses an encoder to perform bidirectional text modeling on the given conditional text, and then uses an independent decoder to predict the subsequent text through autoregression. Representative models of this type of work include MASS, T5, BART, etc.

[0023] In order to use large-scale language models for downstream tasks in a parameter-efficient, data-efficient way and to reduce the gap between pre-training and downstream tasks, TemplateNER and KPT have been proposed. These methods use prompt templates to reconstruct various downstream tasks into the form of pre-training tasks, and then solve downstream tasks through prediction in the pre-training stage. For tasks such as text classification and named body recognition, the process of prompt learning to reconstruct tasks and make predictions mainly includes three parts:

[0024] (1) Adding a prompt template: Given a text input for a task, we need to add a corresponding prompt template based on the task form. For example, for a classification task, we can set the template as: "The classification result of this text is". Then, the prompt template is concatenated with the text to be classified as the input of the masked language model.

[0025] (2) Answer search: Based on the text input with the hint template added, the large-scale language model can predict the words in the blank in a way consistent with the prediction task in the pre-training stage. For example, for the first step of the text classification task and the masked language model, we can define the answer space as the entire vocabulary library in the pre-training stage, or a more refined small-scale vocabulary library guided by prior knowledge. Through the mask prediction task, the word with the highest probability of mask prediction is searched in the vocabulary library as the candidate answer.

[0026] (3) Answer mapping: The results predicted by large-scale language models may not completely correspond to the labels of downstream tasks, so prompt learning requires the design of the answer mapping step. For example, for text classification tasks, its classification labels are only "positive emotion" and "negative emotion". We need to map the results predicted in the blank space of the template to these two labels. For example, words such as "good", "happy", and "excited" can be mapped to the "positive emotion" label; words such as "bad", "negative", and "sad" can be mapped to the "negative emotion" label.

[0027] For different natural language processing downstream tasks and different types of pre-trained models, researchers usually choose different forms of prompt templates to reconstruct the tasks to ensure the consistency between downstream tasks and pre-trained tasks. There are two typical forms of prompt templates:

[0028] (1) Cloze template: Set a blank part in a text for answer prediction. Cloze templates are usually adapted to masked language models. For example, large-scale language models such as LAMA and Template use this cloze template as a prompt template, and its prediction method is exactly the same as that of the pre-training task. From the perspective of downstream tasks, tasks such as text classification, natural language reasoning, named entity recognition, common sense reasoning, and question-answering can all generate relevant words corresponding to the answer in the blank by inputting the text concatenated with the cloze template in a masked prediction manner, and then obtain the final result through answer mapping to complete the downstream tasks.

[0029] (2) Prefix sequence template: A text is concatenated to the original input to guide the subsequent text generation. The prefix sequence template is suitable for pre-trained models such as T5. For example, this prefix sequence template is used in methods such as Prefix-Tuning and Prompt-Tuning. Its prediction form is highly consistent with the pre-training tasks of the autoregressive language model, prefix sequence language model, and encoding-decoding language model. As a prompt template that can generate new text based on the previous text, the prefix sequence template is suitable for tasks such as text generation, information extraction, and text generation evaluation.

[0030] In addition to the fact that prompt templates can be divided into cloze and prefix sequence templates in terms of form, we can also classify the types of prompt templates from the perspective of whether the parameters of the prompt template itself will be optimized:

[0031] (1) Each word in the discrete template has actual semantics and exists in the pre-trained vocabulary. The model parameters of the word representations corresponding to these words are fixed after optimization in the pre-training stage and will not be further optimized in downstream applications. Discrete templates are suitable for tasks in zero-shot scenarios.

[0032] (2) Each word in the continuous template has no actual semantics and does not exist in the pre-trained vocabulary. The model parameters of the word representations corresponding to these words can be optimized for specific tasks and specific data in downstream applications. This optimization process is called prompt tuning. In scenarios where there is corresponding downstream data that can be used to assist in prompt tuning, continuous templates can demonstrate their advantages in targeting specific tasks and specific data. Continuous templates have been widely used in subsequent visual and multimodal prompt learning methods.

[0033] Visual-language multimodal cue learning methods are task-specifically designed for different downstream applications, including pure text cue learning, single text cue learning guided by visual information, text cue learning guided by text or external knowledge, joint text and visual cue learning, and combined cue learning for specific components.

[0034] Pure text prompt learning technology was first applied to multimodal pre-training models with a dual-tower structure such as CLIP. The initial design was an artificial prompt template such as "a photo of a[__]." By adding category words in the blank and inputting the prompt sentence into the text encoder to extract text features, and inputting the image into the visual encoder to extract features, the similarity between the prompt text and the image can be calculated, which can be applied to zero-shot visual classification tasks. Since artificial templates usually require a large number of attempts and cannot be optimized specifically for downstream datasets, pure text continuous prompt learning methods were proposed, inspired by continuous prompt learning methods such as Prefix-Tuning in natural language processing. This type of method sets the prompt template as a series of prompt vectors that can be optimized in a continuous space, and implements prompt tuning for specific tasks on downstream datasets based on optimization losses.

[0035] In multimodal scenarios, only learning task-specific prompts for text can easily lead to problems such as poor generalization and misalignment of image and text features. To solve these problems, visual information can be introduced into the text space as a guide for text prompt learning. CoCoOp, DPL, and StyLIP design networks to learn representations specific to image samples, and integrate them into the continuous prompt vector of pure text to achieve flexible and highly generalizable prompt learning. MAPL, PL-UIC, and LVP-M3 transfer image features to the text space through a mapping network to assist the language model in understanding vision. Img2Prompt uses the existing image description model to input the description generated for the image sample into the language model to enhance the understanding between modalities.

[0036] In the process of continuous prompt tuning of pure text, prompt vectors specific to downstream datasets are prone to overfitting. Inspired by the strong generalization characteristics of artificial prompt text, LAS and KgCoOp, in addition to the optimization loss of image-text matching, also designed the corresponding similarity loss between text and text or external knowledge, so that the prompt vector maintains a certain similarity with a series of artificial prompt texts or external knowledge, thereby retaining the generalization ability of prompt tuning.

[0037] The above multimodal prompt learning is limited to design on text. Due to the differences between text features and visual information, this unimodal approach limits the flexible adaptation of the two modalities on downstream tasks and is prone to suboptimal solutions. To this end, many methods propose to learn prompts in both text and vision. UPT designed a unified prompt vector for vision and text. MaPLe, CAVPT, MetaPrompt, and P3OVD designed their own prompt vectors in text and vision respectively, and MaPLe transferred visual information to the text space through a coupling function to further strengthen the interaction between modalities. Yang et al. designed a modality-consistent prompt vector based on the OFA model and concatenated it to the input sequence. These bimodal prompt vectors are optimized for prompts on downstream datasets based on task-related loss functions, realizing joint modality prompt learning.

[0038] Designing only one set of prompts for a task may result in insufficient representation of data with a large number of categories, high visual diversity, wide domain sources, and rich feature attributes. To solve this problem, PTP sets the same prompts for images with similar visual features and different prompts for images with different visual features. Lee et al. set corresponding prompts for three types of inputs: complete modalities, only images, and only text, facing real scenarios where modalities may be missing. When used, the corresponding prompt vector is selected and added to the model according to the missing modality. This includes prompt vectors that can be concatenated to the input sequence in the input layer and prompt vectors that are concatenated to the key and value in the transformer attention layer, which keep the length of the output sequence unchanged by controlling the query length. In the downstream application process, prompt tuning is achieved by optimizing the prompt vector parameters and the parameters of the external classification head on the comprehensive classification representation CLS. DualCoOp models the multi-classification problem as a positive and negative binary classification problem, and sets corresponding prompts for positive and negative features respectively. TaI sets global and local prompts to learn the information of the whole image and the region respectively. R-Tunin groups large-scale data sets by category and sets different prompt vectors for each group. Wang et al. set different prompts for data in different fields. In order to achieve prompt learning for multiple tasks, CPL sets task-general and task-specific prompts. For domain adaptation tasks, DAPL sets domain-general, domain-specific, and category-corresponding prompts. CSP and PromptCompVL set different prompts for attribute and category information for combined zero-shot learning tasks.

[0039] In addition to clustering data with similar features and setting different prompts for different classes, there is another method that models attributes and other features to satisfy a certain distribution to achieve prompt learning. ProDA inputs images and learnable prompt texts into the visual and text encoders to extract corresponding features, and achieves visual classification by matching the features corresponding to the two modalities into Gaussian distributions. Derakhshani et al. modeled image features into Gaussian distributions, and randomly sampled a vector from the distribution as a generalizable representation of visual information, and added it to the pure text prompt vector to ensure that the prompt vector has the diversity guided by visual information. PLOT and ZegOT regard visual features and text features corresponding to prompts as two discrete distributions, and achieve cross-modal matching through optimal transportation. Liu et al. proposed PBPrompt, which models each label as a variational distribution to introduce uncertainty into the label space, and then randomly samples the corresponding prompts from the distribution. Similarly, PBPrompt finally achieves matching of visual and text distributions through optimal transportation.

[0040] The above methods all design different prompts for different tasks, ignoring the correlation between tasks and limiting information sharing. In order to better utilize the correlation information between different tasks, Shen et al. proposed the MLVPT method to merge multiple source tasks and jointly optimize a set of prompts, thereby achieving multi-task sharing, and then use the shared prompts as the initialization of the target task prompts. Soft-CPT proposed a meta-network for multiple tasks, in which the name of each task is concatenated with the task meta-prompt, and the corresponding data label is concatenated with the label meta-prompt and input into the text encoder for feature fusion to obtain the final text prompt, and then the prompt is tuned on the downstream task.

[0041] Since tuning in continuous space will cause the prompt to overfit to the downstream data, in order to solve this problem, analyzing the gradient changes during the prompt tuning process and implementing gradient-guided prompt learning has become a solution. Pro-Grad orthogonally decomposes the gradient direction of each step in the tuning process into a direction representing general knowledge and its perpendicular direction. If the angle between the gradient direction and the general knowledge direction is acute, the prompt parameters are updated at this step, otherwise they are not updated. Ma et al. found that the initial stage of prompt tuning can retain generalization, while the later stages will gradually lose generalization. To this end, they proposed SubPT, defining the gradient direction in the early stage as the main feature direction with strong generalization, and orthogonally projecting the gradient onto the main feature direction in the later stages for prompt tuning.

[0042] Existing prompt learning methods all rely on downstream labeled data for prompt tuning. In order to achieve prompt learning in the scenario of unlabeled data, Huang et al. proposed an unsupervised prompt learning method UPL. It uses artificial prompt templates to perform zero-sample predictions on downstream data, thereby giving pseudo labels to unlabeled data, and then refers to CoOp to perform prompt tuning in continuous space based on pseudo labels.

[0043] In the field of natural language processing, most models are pre-trained through masked language modeling. In order to achieve cross-modal cue learning based on this form and predict the category of the target in the visual area from the mask part, CPT covers the targets in the picture with different color blocks according to their categories and establishes a mapping between target categories and colors. In cue learning, the color covered by the corresponding target is predicted in the blank space set by the cue statement, and then the final target category is predicted based on the mapping relationship.

[0044] In order to enable large-scale language pre-training models to understand visual information, Tsimpoukelli et al. proposed the FROZEN model. This model maps the features extracted from the image through the visual encoder to the language space to form visual information cues. During the optimization process on downstream data, the parameters of the language model remain unchanged, and only the parameters of the visual encoder need to be trained from scratch.

[0045] In the field of natural language processing, the prompt learning method achieves multi-task unified downstream applications by reconstructing various downstream tasks into pre-training tasks, and has achieved excellent results. This is due to two characteristics of the field of natural language processing: (1) The pre-training tasks of various large-scale language models are highly similar, that is, through language modeling, invisible target text is generated based on visible text sequence prediction. These pre-training tasks effectively realize the model's comprehensive understanding and representation capabilities of the corpus. (2) The solutions to downstream tasks can all be expressed as language outputs, so it is natural to reconstruct different language downstream tasks into language modeling tasks in the pre-training stage, and the downstream solutions can be generatively predicted through pre-training.

[0046] We compare the visual and multimodal cue learning methods with the cue learning methods in the field of natural language processing and analyze them from the following aspects:

[0047] First, current visual and multimodal cue learning methods are all specially designed for specific tasks, and there are few cue learning methods that can achieve multi-task unification and keep consistency with pre-training tasks. We believe that there are two main reasons: (1) The pre-training tasks of visual and multimodal models are numerous and complex, such as supervised classification pre-training tasks, contrastive self-supervised pre-training tasks, image restoration-style generative pre-training tasks, etc., and these pre-training tasks have different forms; (2) The forms of solutions to downstream tasks are also different. For example, image classification tasks require the output of classification labels, object detection tasks require the output of specific object coordinate positions and corresponding categories, and semantic segmentation tasks require the output of segmented region maps. Due to the huge differences between pre-training tasks, the huge differences between solutions to downstream tasks, and the huge differences between the forms of pre-training and downstream tasks, it is very difficult to achieve a cue learning method that can achieve multi-task unification and keep consistency between pre-training and downstream tasks based on existing visual or multimodal pre-training models.

[0048] Secondly, although hint learning can achieve efficient parameter fine-tuning in natural language processing, vision, and multimodal fields, the gradient still needs to be back-propagated and optimized during the entire process of downstream data tuning, which does not bring time efficiency.

[0049] In addition, the parameter scale of most models in the current vision and multimodal fields is much smaller than the parameter scale of models in the natural language processing field, which limits the amount of knowledge that the model can learn and its ultimate ability to understand and represent the data.

[0050] At present, there are still many feasible directions for future research in the field of prompt learning. Using the idea of ​​thinking chain to establish automatic generation of prompts is one of them. Summary of the invention

[0051] In view of the fact that the visual language model uses a unified default input for the dataset and uses all category labels as prompts for target detection, which leads to serious hallucinations or misclassification problems, an image-level automatic prompt generation target detection method based on thought chain is proposed to give full play to the image understanding and text generation capabilities of large models, thereby reducing the subjectivity and heavy workload of manually designed prompts and improving the accuracy of target detection. This method is flexible, not restricted by scenes, and can be migrated to other fields to complete visual cognition tasks in other scenes. It is suitable for the current popular multi-task status.

[0052] The present invention proposes an image-level automatic prompt generation method, which is characterized by combining the powerful image understanding ability of a large-scale visual model with the common sense reasoning ability of a large language model, using the idea of ​​a thinking chain to decompose a target detection task in one stage into multiple steps, and automatically generating image-level prompts for a specific data set, thereby improving the target detection accuracy, including the following steps:

[0053] 1) Input the image to be detected and the candidate labels into the concept extraction module to preliminarily identify the objects contained in the image and obtain the preliminary prompt P raw .

[0054] 2) Establish common sense knowledge graph G C , for P raw Further screening is performed, and unreasonable concepts are removed using common sense coexistence relations to obtain the processed prompt P prune .

[0055] 3) Use the large language model for reasoning, based on the processed prompt P prune Reasonable association, supplement the concepts that may be missed, and get the supplemented hints P search .

[0056] 4) Design an adaptive threshold algorithm to filter the concepts extracted through image understanding and common sense reasoning, delete the concepts with low prediction confidence, and obtain the final image-level prompt P image .

[0057] 5) The automatically generated P image It is paired with the corresponding image I and input into the visual language model to complete the final object detection task.

[0058] The above-mentioned concept extraction module contains two main modules, a foreground object extractor and a pre-trained text-image matching pair. The foreground object extractor uses a visual transformer as the basic network, uses a non-overlapping window of size 14*14 to extract feature maps, and uses the region of interest (RoI) head to filter the pre-selected box to obtain the area where the foreground object is located. The pre-trained text-image matching pair inherits from the results of contrastive learning language image pre-training (CLIP). This method uses 400 million text-image pairs for pre-training and has strong generalization and domain transfer capabilities. Common sense knowledge graph G C Based on the common sense knowledge base ConceptNet, it counts the number of related terms between two words to indicate the relevance of two concepts. The multi-step prompt generation process generates prompts for each module according to the thought chain process, guiding the generation of the final target detection text prompt P image .

[0059] The above-mentioned method for automatically generating image-level prompts based on thought chains is characterized by being applicable to zero-shot multimodal target detection tasks. Before completing detection reasoning, no additional pre-training and model fine-tuning processing for specific data sets is required. In addition, no manual text input is required for guidance. The model can completely autonomously generate specific prompts for different images to assist the visual language model in completing detection.

[0060] Features of the present invention:

[0061] It can be seen from the above scheme that the present invention is applied to multimodal target scenes, but there is no need for manual input of text as prompts to guide the visual language model for reasoning. The text prompts are automatically generated from the input image, and the operation is simple. This method can complete the detection under zero-sample conditions, does not require pre-training and fine-tuning of the model, occupies less video memory, and has low hardware requirements. This method has been tested on public data sets commonly used in multiple target detection fields, and the accuracy exceeds the default prompts and other prompt generation methods, which is concise and effective. In addition, the present invention can not only be used to improve the accuracy of target detection, but also can be expanded to other fields of computer vision, such as semantic segmentation, object tracking and other tasks, and is less restricted by the model structure, can meet the needs of actual production and life, and has practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a thought chain flowchart automatically generated by image-level prompts.

[0063] Figure 2 It is a comparison chart of the visualization results of target detection under zero-sample conditions using default prompts, other concept extraction results, and prompts automatically generated by the present invention as input. DETAILED DESCRIPTION

[0064] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0065] The present invention mainly solves the problem that when using default prompts for target detection, concepts unrelated to the image cause serious hallucinations and false detections in the model. Previously, the visual language model in the multimodal field performed target detection, using a unified fixed prompt for all images in a data set, and encoding and partitioning the entire image to establish the association between text and image. These two operations will cause hallucination problems and reduce the detection accuracy. The visual language framework used in the present invention is GLIP, and the pre-trained model is a text-visual multimodal model, which unifies the two tasks of target detection and visual positioning. The model finds objects based on the given text and constructs a unified training framework. Therefore, the model can be trained using data sets of both image and text modalities at the same time, and then cooperate with pseudo-label technology to amplify the data, so that the amount of training data reaches a considerable scale, so that it can still show good generalization performance under zero sample conditions.

[0066] For visual localization tasks, the difference from domain object detection tasks is that the label in the localization task is a sentence or a group of words or phrases, rather than a classification head. Therefore, the GLIP model obtains text features through a text encoder, calculates the matching degree between the text features and the region box features, and obtains a matching score to determine whether the words in the label match the image region. The formula is shown in 1.1.

[0067] O=Enc I (Img), P = Enc L (Prompt), S ground =OP T (1.1)

[0068] Among them, O is the encoded image feature, P is the encoded text feature, S ground is the similarity score. GLIP uses the above principle and shows good detection effect in the field of natural images, but the framework can only input a single fixed text as a prompt. Therefore, based on GLIP, the present invention realizes automatic prompt generation and autonomously generates image-level dedicated prompts for different images.

[0069] In the process of realizing image-level prompt generation, the present invention starts from the image level and only extracts concepts from the image. The specific scheme is:

[0070] 1. Input the image into the image encoder to obtain the embedded vector E after the entire image is encoded.

[0071] 2. Input the encoding vector E of the entire image into the foreground object extractor Ext f , through matching calculation, we get the region vector E containing the foreground object f .

[0072] 3. Input the default category label of the dataset into the text encoder to obtain the embedded vector T after encoding each label.

[0073] 4. Using the pre-trained text-image matching pairs, calculate the foreground area E f The matching degree with the category label T, thus obtaining the preliminary hint P obtained through image understanding raw .

[0074] In this scheme, the process of extracting foreground objects is shown in formula 1.2.

[0075]

[0076] Among them, λ iou and is a hyperparameter, b i Indicates the pre-selected area, L box Used to measure the pre-selected box b i The degree to which foreground objects are contained in box Minimum, get the most accurate pre-selected area b containing the foreground object i , and find the corresponding embedding vector

[0077] Although the idea of ​​extracting concepts through image understanding is simple and effective, this solution is greatly affected by the performance of the image understanding model used. In addition, due to the different complexity of different images, the completeness of the extracted concepts varies greatly between images, and the performance is unstable.

[0078] In addition, during the experiment, it was found that the image contained some small objects. If the size of the objects to be detected in the image varies greatly, the model will focus on the large objects, making it more difficult to identify the small objects, or even ignore them directly. The information that can be obtained in the image dimension is limited, but there is still room for mining in the text dimension. In natural images, there is a certain causal relationship between the objects contained in a specific scene in the image, such as inevitable non-coexistence relationship and potential correlation relationship.

[0079] Therefore, based on Solution 1, the present invention utilizes common sense knowledge at the text level to design a more complete prompt generation method.

[0080] First of all, the concept network ConceptNet is an open source semantic network, also known as a common sense knowledge base, which is used for large-scale storage of structured or unstructured data in information systems. The data structure in ConceptNet is now closer to multi-level data with semantics and organization. ConceptNet stores data in the form of knowledge graphs, centered on the relationships between entities. The graph is variable and semantic, which means that the meaning of the data is encoded with the data. The knowledge in the ConceptNet knowledge base comes in many different forms, such as ontology knowledge, relevance knowledge, rule base, case knowledge, etc., which focuses more on the construction of relevance knowledge and emphasizes the connection between data or things. The present invention uses ConceptNet data as the underlying foundation, and establishes a knowledge graph subgraph exclusive to different data sets for different data sets, which is used to measure the initial generated prompt P raw The rationality of each concept.

[0081] In the process of image understanding, in order to ensure that the model can identify all possible objects as much as possible, the present invention relaxes the prediction threshold limit in the design process to ensure the completeness of the extracted concepts. This approach is bound to lead to hallucinations. raw Some irrelevant concepts are introduced and cannot be removed. Most of these irrelevant concepts are illusions caused by similar visual features. In fact, these objects cannot appear in the scene. If the visual model determines that the object exists and cannot be excluded, the association knowledge in the knowledge base can be used to remove the unreasonable irrelevant concepts. The specific solution is:

[0082] 1. Establish entities according to the category labels of the objects to be detected in the data set, reconstruct ConceptNet, establish a knowledge graph for the data set, and count the number of associated terms between two entities in ConceptNet as the weight of the edge to obtain the data set knowledge graph G0;

[0083] 2. With P raw The concepts contained in it are nodes, and subgraph G is extracted from G0 sub ;

[0084] 3. Compute subgraph G sub The degree and connectivity of each node, remove the disconnected or unreasonable nodes, and the concepts represented by the remaining nodes are used as the processed hints P prune ;

[0085] Among them, the category labels of the objects to be detected are used as the names of the nodes in the knowledge graph G0, for example, laptop, television, keyboard, remote control, cell phone and microwave. Because the association relationship in ConceptNet is directional, the knowledge graph G0 is required to be a directed graph. In the process of screening nodes, if the degree of a node is 0, it means that it is not connected to other nodes, that is, there is no association relationship, and it should be deleted directly. If the degree of a node is 1, it means that the concept has only a unidirectional association relationship with another concept. The Dijkstra algorithm is used to calculate the shortest path from the node to other nodes. If the length of the path is greater than the subgraph G sub If the length of any side of other nodes in the , it means that the node is far away from other nodes and has a low degree of association, so it is also deleted. The specific calculation process is shown in formula 1.3.

[0086] Let G sub =(V, E), V represents the node set, E represents the edge set. For each node v∈V,

[0087] d(v)=min[d(u)+w(u,v)], u∈S (1.3)

[0088] Among them, d(v) represents the shortest path length from source point s to node v, S is the set of nodes for which the shortest path has been determined, and w(u, v) represents the weight of the edge from node u to node v.

[0089] At this time, the prompt P after the knowledge graph screening prune It can effectively alleviate the hallucination phenomenon of the model, but there are still some concept missing problems caused by the performance shortcomings of the image understanding model. Considering that the large language model has a strong knowledge reserve and text generation and association ability, this paper uses the large language model GPT-3.5 according to P prune The concepts contained in the image are reasonably associated with the scene, and other objects that may exist are inferred to obtain the expanded hint P search The specific plan is as follows:

[0090] 1. Set up the association template and guide the model to prune Think of the corresponding scene,

[0091] Template 1: This image contains {P prune} and other objects, please imagine the corresponding general scene;

[0092] 2. Set up reasoning templates to guide the model to infer possible objects based on the associated scenes. prune Supplement and get the expanded prompt Psearch ,

[0093] Template 2: Based on the above scenario, associate other objects that may exist.

[0094] Finally, the present invention counts the predicted probabilities of all concepts in the generated prompts, designs an adaptive threshold screening algorithm, removes concepts with low confidence, and thus obtains the final image-level prompt P image The present invention is based on the idea of ​​thinking chain and designs different prompt generation and screening rules. No manual adjustment is required throughout the process, ensuring that the final result is completely objective and avoiding subjective human interference.

[0095] The present invention has been tested on five public datasets, namely MSCOCO, PASCAL VOC, ODinW-35, ANIMALS and FOOD, with GLIP-T as the basic object detection model, and the evaluation index is the average precision (AP) of object detection. The detection results are better than those of the zero-shot object detection model and other prompt generation methods, as shown in Table 1. Under the condition of zero-shot detection, the algorithm of the present invention further improves the detection performance of GLIP, and surpasses the methods based on image-based concept extraction only, such as GRiT concept extraction, CLIP image understanding and other methods.

[0096] Table 1 Comparison of the detection effect of the present invention with the zero-sample target detection method and other prompt generation methods

[0097]

[0098] At the same time, in order to verify that the present invention has certain practical value in actual production and life, tests were conducted on more than 30 data sets applied to specific scenarios. In these data sets, there are fewer categories of objects to be detected, but the scenarios are more special. The specific experimental results are shown in Table 2 and Figure 2 The purpose is to illustrate that in addition to general common object detection tasks, this method is also applicable to prompt generation and detection in specific scenarios and has stronger generalization ability.

[0099] Table 2 Comparison of detection results of the present invention and other detection and prompt generation methods in different special scenarios

[0100]

[0101] In summary, based on the idea of ​​thought chain, this paper utilizes the generalization ability of the large image understanding model and the text generation ability of the large language model to design an image-level prompt automatic generation method, and verifies it on multiple public datasets to ensure the effectiveness of the method.

Claims

1. An image-level automatic prompt generation method based on thought chain, characterized in that: The idea of ​​thinking chain is used to decompose the target detection task of one stage into multiple steps, and automatically generate image-level prompts for specific data sets, thereby improving the accuracy of target detection. The steps include: 1) Input the image to be detected and the candidate labels into the concept extraction module to preliminarily identify the objects contained in the image and obtain the preliminary prompt P raw ; 2) Establish common sense knowledge graph G C , for P raw Further screening is performed, and unreasonable concepts are removed using common sense coexistence relations to obtain the processed prompt P prune ; 3) Use the large language model for reasoning, based on the processed prompt P prune Reasonable association, supplement the concepts that may be missed, and get the supplemented hints P search ; 4) Design an adaptive threshold algorithm to filter the concepts extracted through image understanding and common sense reasoning, delete the concepts with low prediction confidence, and obtain the final image-level prompt P image ; 5) The automatically generated P image The corresponding images are input into the visual language model in pairs to complete the final object detection task.

2. The method for automatically generating image-level prompts based on thought chains according to claim 1 is characterized in that: The concept extraction module consists of a foreground object extractor Ext f and pre-trained text-image matching pairs.

3. The method for automatically generating image-level prompts based on thought chains according to claim 1 is characterized in that: The common sense knowledge graph G C Based on the common sense knowledge base concept network, the category labels to be detected are entities, the association relationships are edges, the degree of association is weight, and the mathematical model is a weighted directed graph.

4. The method for automatically generating image-level prompts based on thought chains according to claim 1 is characterized in that: The large language model is a generative pre-trained transformer model that uses two-step question answering for scene association and concept supplementation reasoning.

5. The foreground object extractor Ext according to claim 2 f , characterized in that, The visual transformer is used as the basic network, a non-overlapping window of size 14*14 is used to extract the feature map, and the region of interest head is used to filter the pre-selected box to obtain the area where the foreground object is located.

Images