Open domain image recognition method based on combinable text prompt framework

By adopting a combinable text prompt framework in open domain image recognition, combining visual features and task-related prompts, the problem of difficult generalization of text prompt learning in the existing technology is solved, and high-precision recognition of new categories of images is achieved and visually irrelevant. Effective learning of words.

CN119046722APending Publication Date: 2024-11-29BEIJING NORMAL UNIVERSITY

Patent Information

Application Number
CN202411015053.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the prior art, simple text prompt learning is difficult to generalize to new categories, and integrating all visual content into text prompts will hinder the effective learning of visually irrelevant words, resulting in insufficient accuracy of open-domain image recognition.

Method used

Using a method based on a comboable text prompt framework, visual information is fused into text prompts through visual feature injectors and prompt integrators, and the combination of visual related and irrelevant prompts is learned by using cross-modal attention and self-attention mechanisms to optimize Prompt to combine modules.

Benefits of technology

The recognition accuracy of deep learning models for images is improved, especially in the open domain, and the recognition ability of unseen image categories is significantly improved, maintaining sensitivity to task requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046722A_ABST
    Figure CN119046722A_ABST
Patent Text Reader

Abstract

The invention discloses an open domain image recognition method based on a combinable text prompt framework, and the method comprises the steps: 1, carrying out the preprocessing and tokenization of an image x, and obtaining a tokenization feature Fv through a clip visual model; 2, initializing a task related text prompt ptt and a visual related text prompt pvt; step 3, pvt learns a prompt # imgabs0 # after visual fusion by using a cross-modal attention mechanism, step 4, ptt and # imgabs1 # adopt a self-attention mechanism to fuse a learning combination prompt # imgabs2 #, step 5, a clip text model is adopted to obtain # imgabs3 # features, a loss optimization prompt and a prompt combination module are calculated, the method learns visual related prompts and visual unrelated prompts respectively, and the visual related prompts, the visual unrelated prompts, the visual unrelated prompts, the visual unrelated prompts and the visual unrelated prompts are subjected to visual fusion. The visual related prompt captures visual information by using cross-modal attention, and the visual irrelevant prompt realizes mutual optimization through combination of self-attention and the visual related prompt.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to an open domain image recognition method based on a combinable text prompt framework. Background Art

[0002] Image recognition is a basic task of computer vision technology and a fundamental problem that needs to be solved in the field of artificial intelligence. Open domain image recognition technology is a method that realizes image recognition of any category through training with data of limited categories. It solves the problem of recognizing new types of images in the absence of prior information. Open domain image recognition methods have the ability of humans to recognize unseen objects based on descriptions alone. For example, humans have never seen a zebra but can recognize it through descriptive language such as "a horse with stripes". Open domain image recognition methods can recognize image categories that have never appeared in the model training set, reducing the image recognition method's dependence on data collection and labeling.

[0003] However, in the prior art, pure text prompt learning is difficult to generalize to new categories due to the lack of visual content. In addition, integrating all visual content into text prompts will hinder the effective learning of visually irrelevant words.

[0004] Based on the above technical problems existing in the prior art, the present invention proposes an open domain image recognition method based on a combinable text prompt framework. Summary of the invention

[0005] The purpose of the present invention is to provide an open domain image recognition method based on a combinable text prompt framework in view of the deficiencies of the prior art.

[0006] The present invention adopts the following technical solution:

[0007] An open domain image recognition method based on a composable textual hint framework, comprising:

[0008] Step 1: Preprocess and tokenize the image x, and use the clip visual model to obtain the tokenized features F v ;

[0009] Step 2: Initialize the task-related text prompt p tt and visually relevant textual cues vt ;

[0010] Step 3, p vt Learning to fuse visual cues using cross-modal attention

[0011] Step 4, p tt and Using self-attention mechanism to fuse learning combination hints

[0012] Step 5: Use the clip text model to obtain features, and calculate the loss optimization hint and hint combination modules.

[0013] Further, step 1 comprises:

[0014] The image is adjusted to the uniform size (224*224) required by the model, and the image is normalized and divided into 16*16 tokens. The ViT model is used to extract the feature F of the token. v .

[0015] Further, step 2 includes:

[0016] Task related prompt textp tt Initialized as “a photo of a”, visually relevant text hint p vt Initialized to "XX".

[0017] Furthermore, in step 3, the visual features are injected into the image-related textual hints through the cross self-attention mechanism. vt As a query, the image feature vector F v Generate visually relevant dynamic text prompts as keys and values

[0018]

[0019] Among them, norm(·) is the hierarchical normalization.

[0020] Furthermore, in step 4, a prompt integrator is used to fuse the task-related prompt text with the dynamic text prompt to generate p t :

[0021]

[0022] Using self-attention mechanism and feedforward neural network to optimize p t Generate standard text prompt content

[0023]

[0024] Furthermore, in step 5, the loss function consists of two parts: cross entropy loss and distillation losses

[0025]

[0026] Among them, λ is the weight parameter used to balance the two losses.

[0027] Furthermore, in step 5, the cross entropy loss as follows:

[0028]

[0029] in:

[0030]

[0031] Where K is the number of categories, f(x) is the feature of image x, is the combined hint for category y, and τ is a hyperparameter.

[0032] Further, in step 5, the distillation loss as follows:

[0033]

[0034] in, for "aphoto ofa"c i Input clip text model representation, c i is the class name of the i-th class.

[0035] The beneficial effects of the present invention are:

[0036] 1. The open domain image recognition method based on the combinable text prompt framework described in the present invention provides visually relevant semantic information for text prompts by effectively integrating visual feature injectors and prompt integrators, thereby improving the recognition accuracy of the deep learning model for images while maintaining sensitivity to task requirements. By learning visually relevant prompts and visually irrelevant prompts separately, the visually relevant prompts use cross-modal attention to capture visual information, and the visually irrelevant prompts are combined with visually relevant prompts through self-attention to achieve mutual optimization;

[0037] 2. The open domain image recognition method based on the combinable text prompt framework described in the present invention performs excellently in open domain image recognition and can effectively recognize classifications that do not appear in the training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the process of open domain image recognition based on the combinable text prompt framework in an embodiment of the present invention;

[0039] Figure 2 Schematic diagram for comparing the average recognition accuracy of the method described in the embodiment of the present invention with other methods. DETAILED DESCRIPTION

[0040] In order to more clearly understand the above-mentioned purposes, features and advantages of the present invention, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0041] Example

[0042] The feature calculation of the large model is extremely sensitive to the form and composition of the prompt word. Different prompt word compositions will result in different feature calculation results. In open domain image recognition based on prompt words, an effective prompt should contain two parts: one part of the prompt is related to the task, and the other part is related to the visual information of the image. Therefore, the embodiment of the present application proposes an innovative prompt framework, called a "combinable text prompt framework", in which the prompt text is divided into two parts: task-related prompt text and image-related prompt text, wherein the image-related part will be dynamically updated according to the image features, while the task-related part focuses on the task itself.

[0043] like Figure 1 As shown, the open domain image recognition method based on the composable text prompt framework includes two stages: model training and image recognition;

[0044] Phase 1: model training;

[0045] Step 1-1, model input preprocessing, where the model input consists of an image and a prompt text;

[0046] Step 1-1-1, image preprocessing, adjust the image to the uniform size of 224*224 required by the model; perform image normalization operation, that is, scale the image pixel value interval to the standard [0,1] value, so as to ensure the stability of the image value, facilitate information extraction, enhance the generalization ability of the deep model, and accelerate model convergence. Image normalization does not affect the information storage and visual features of the image;

[0047] Step 1-1-2, image feature calculation, the image pre-processed in step 1-1-1 is processed by an image encoder to extract visual features. For example, the image encoder selects a Vision Transformer (ViT-B / 16) deep learning network. This image encoder is pre-trained by ImageNet. In the embodiment of the present application, no secondary training and parameter update are performed. After the image is calculated by the image encoder, the image is output as a feature vector F of the image. v ;

[0048] Step 1-2, task related prompt text p tt Initialized as “a photo of a”, visually relevant text hint p vt Initialized to "XX";

[0049] Step 1-3, use the visual feature injector to inject the visual features obtained in step 1-1 into the p vt Through the cross self-attention mechanism, visual features are injected into the image-related text prompts. Specifically, p vt As a query, the image feature vector F v Generate visually relevant dynamic text prompts as keys and values

[0050]

[0051] Among them, norm(·) is hierarchical normalization;

[0052] Step 1-4: Fusion of task-related prompts and visual dynamic prompts. Using the prompt integrator, the task-related prompt text of step 1-2 and the dynamic text prompt output of step 1-3 are fused to generate p t :

[0053]

[0054] It uses self-attention mechanism and feedforward neural network to integrate task-related text and vision-related text to generate standard text prompt content.

[0055]

[0056] Step 1-5, text feature encoding. First, a text encoder is used to feature encode the standard text obtained in step 1-4 to generate a text feature vector. The encoder uses the CLIP pre-trained model with fixed parameters and no need to update. Second, the cross entropy loss function is used to implement back propagation and update the parameters of the visual feature injector and the prompt fuser.

[0057] Stage 2, image recognition;

[0058] Step 2-1: Input preprocessing, adjust the image to the uniform size of 224*224 required by the model; perform image normalization, that is, scale the image pixel value interval to the standard [0,1] value, and use a random initialization strategy for the image-related prompt text in the initialization stage;

[0059] Step 2-2, processing the preprocessed image through an image encoder to extract visual features;

[0060] Step 2-3, using the visual feature injector, the obtained visual features are combined with the initialized image-related text, and the visual features are injected into the image-related text prompts through the cross self-attention mechanism;

[0061] Step 2-4, task-related prompts are integrated with visual dynamic prompts. The prompt integrator is used to integrate the task-related prompt text with the output dynamic text prompt.

[0062] Step 2-5, text feature encoding, using a text encoder to perform feature encoding on the obtained standard text to generate a text feature vector;

[0063] Step 2-6, image recognition, calculate the cosine similarity between the image feature vector and the text feature vector, obtain the predicted probability of the different image-related prompt text categories based on the similarity, and select the category with the highest probability as the predicted category of the image:

[0064]

[0065] Where K is the number of categories, f(x) is the feature of image x, is the combined hint for category y, and τ is a hyperparameter.

[0066] In order to verify the superior technical effect of the method described in the embodiment of the present application, the method was tested on 11 public data sets, such as Figure 2 As shown, the test data sets are ImageNet, Caltech01, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101, respectively. CoP+CPT is the method described in the embodiment of the present application, Base is the test result of the known class, New is the test result of the unknown class, and HM is the average result. After testing, the method described in the embodiment of the present application has an average recognition accuracy of 83.98% for the existing classes, and an average recognition accuracy of 77.32% for the unknown classes, which are 1.36% and 2.74% higher than the currently best performing CoPrompt method, respectively.

[0067] The present invention is not limited by the above embodiments. The above embodiments and descriptions are only for explaining the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, which are within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the appended claims.

Claims

1. An open domain image recognition method based on a composable textual hint framework, It is characterized in that include: Step 1: Preprocess and tokenize the image x, and use the clip visual model to obtain the tokenized features F v ; Step 2: Initialize the task-related text prompt p tt and visually relevant textual cues vt ; Step 3, p vt Learning to fuse visual cues using cross-modal attention Step 4, p tt and Using self-attention mechanism to fuse learning combination hints Step 5: Use the clip text model to obtain features, and calculate the loss optimization hint and hint combination modules.

2. The open domain image recognition method based on the combinable text hint framework according to claim 1, It is characterized in that Step 1 includes: Adjust the image to the uniform size required by the model, perform image normalization, divide the tokens, and extract the token features F v .

3. The open domain image recognition method based on the combinable text hint framework according to claim 1, It is characterized in that Step 2 includes: Task related prompt textp tt Initialized to "aphoto of a", visually relevant text hint p vt Initialized to "XX".

4. The open domain image recognition method based on the combinable text hint framework according to claim 1, It is characterized in that In step 3, the visual features are injected into the image-related textual hints through the cross self-attention mechanism. vt As a query, the image feature vector F v Generate visually relevant dynamic text prompts as keys and values Among them, norm(·) is the hierarchical normalization.

5. The open domain image recognition method based on the combinable text hint framework according to claim 1, It is characterized in that In step 4, the prompt integrator is used to merge the task-related prompt text with the dynamic text prompt to generate p t : Using self-attention mechanism and feedforward neural network to optimize p t Generate standard text prompt content 6. The open domain image recognition method based on the combinable text hint framework according to claim 1, It is characterized in that In step 5, the loss function consists of two parts: cross entropy loss and distillation losses Among them, λ is the weight parameter used to balance the two losses.

7. The open domain image recognition method based on the combinable text hint framework according to claim 6, It is characterized in that In step 5, the cross entropy loss as follows: in: Where K is the number of categories, f(x) is the feature of image x, is the combined hint for category y, and τ is a hyperparameter.

8. The open domain image recognition method based on the combinable text hint framework according to claim 6, It is characterized in that In step 5, distillation loss as follows: in, for "a photo of a"c i Input clip text model representation, c i is the class name of the i-th class.

Citation Information

Patent Citations

  • Zero sample image recognition method based on CLIP

    CN117197568A

  • Transform-based cross-modal fine-grained retrieval method

    CN117874264A

  • Self-evolution zero sample target identification method based on sketch and text double prompt

    CN118072332A

  • Multi-target synchronous detection method and system

    CN118097266A

  • Method and apparatus for vision-language understanding

    US20240144651A1

Cited By

  • Mine car track foreign matter detection method and system based on deep neural network

    CN120472426A

  • Training method and device of open vocabulary detection model and electronic equipment

    CN120726615A