Image classification method based on visual feedback of large language model

Abstract

The invention discloses an image classification method based on visual feedback of a large language model, and aims to solve the problems of model performance limitation and classification accuracy reduction caused by inaccurate category descriptors. The method comprises a descriptor iterative optimization stage, a network training stage and an image classification stage. The descriptor iterative optimization stage comprises three steps of initializing category descriptors, constructing visual feedback and iteratively optimizing the descriptors; by combining the advantages of a large language model and a visual language model, accurate optimization and iterative improvement of class descriptors are realized. Firstly, initial class descriptors are generated through a large language model, then images are classified through a visual language model, and the class descriptors are gradually optimized and refined according to classification result feedback. According to the method, the classification accuracy is improved, and the distinguishing capability of the model in processing highly similar categories is enhanced, so that the overall performance and the application flexibility of the model are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to an image classification method based on large language model visual feedback. Background Art

[0002] In recent years, a variety of visual language models have demonstrated remarkable learning capabilities on a variety of visual tasks. These models are able to effectively align images and text in a shared embedding space by pre-training on large datasets. Different from traditional models, visual language models such as CLIP classify images by evaluating the similarity between input images and text descriptions. In particular, in zero-shot learning tasks, CLIP achieves excellent classification results by combining class names and preset templates as input prompts and comparing images with the most matching prompts. However, this approach is highly dependent on the quality of category prompts. In traditional methods, prompt templates are used to generate category texts, but these texts often lack accuracy and richness, which may cause the model to fail to fully utilize all available information during training and inference, thereby affecting overall performance. Summary of the invention

[0003] In response to the above-mentioned technical problems of model performance limitations and reduced classification accuracy caused by inaccurate category descriptors, the present technical solution provides an image classification method based on visual feedback from a large language model, which enables the large language model to discover the globally optimal category description, thereby significantly improving the overall classification performance of the model; it can effectively solve the above-mentioned problems.

[0004] The present invention is achieved through the following technical solutions:

[0005] An image classification method based on large language model visual feedback comprises the steps of:

[0006] Step 1: It includes the operations of the descriptor iteration optimization stage, the network training stage and the image classification stage; the descriptor iteration optimization stage includes the three steps of class descriptor initialization, visual feedback construction and iterative optimization of descriptors;

[0007] Step 2: Given a set of iteratively optimized category descriptions, visual feedback is constructed using task-related CLIP evaluation as an indicator to provide an overall evaluation of the current category description on the model;

[0008] Step 3: Train the CLIP network with the image and the iterated category descriptors so that they are aligned in a common embedding space through the image encoder ImageEncoder(·) and the text encoder TextEncoder(·); the expression is as follows:

[0009] F x =ImageEncoder(x)

[0010] F d =TextEncoder(d)

[0011] Score = F x F d

[0012] Where x is the input image, F x is the expression of image features, converting the input image x into a feature vector F x ; F d is the expression of text features, converting the input category descriptor d into a feature vector F d ; d represents the input category descriptor.

[0013] Furthermore, the operation mode of class descriptor initialization is as follows: first, all classes are divided into groups according to the similarity of class names; specifically, the text embeddings of the class names are extracted, and then the K-means algorithm is used to cluster them into groups, where each group represents a semantically similar class; in order to ensure that the large language model always pays attention to the easily confused classes, the clustering step is performed not only at initialization, but also dynamically at the beginning of each iteration, and the average embedding of all descriptors of each class is calculated; after clustering, the large language model is allowed to generate an initial class descriptor D 0 .

[0014] Furthermore, the step of constructing visual feedback is: combining the large language model with the visual knowledge in the visual language model, given a current set of class descriptors D, and constructing visual feedback V(D) through the task-related evaluation indicators of CLIP.

[0015] Furthermore, the iterative optimization descriptor step is as follows: in the i-th iteration, the previous category descriptor set D is provided. i-1 , V(D i-1 ) and memory bank M, asking the large language model to pick out the most useless n in the current descriptor set i descriptors, and use n i They are replaced by new descriptors to emphasize their unique visual features; the expression is:

[0016] D 0i ,...,D Ki =LLM(D i-1 ,M,V)

[0017] Among them, D 0i represents the first class descriptor generated in the ith iteration; D ki represents the k-th class descriptor generated in the i-th iteration; V represents the visual feedback used to evaluate and guide the generation of class descriptors;

[0018] In each iteration, k independent candidate descriptors {D 0i ,D 1i ,...,D ki} to ensure sufficient genetic diversity during the optimization process;

[0019] In addition, the generated K candidate samples and their corresponding visual feedback are used as input, and then the large language model is prompted to mix and match between different samples, and finally output a new sample D (K+1)i ; Thus, different useful descriptors of different samples are combined together to produce offspring with better overall performance; the expression is:

[0020] D (K+1)i =LLM(D 0i ,...,D Ki ,V).

[0021] Furthermore, the operation steps of the network training phase include:

[0022] S1: Collect and prepare a large number of annotated image datasets and their corresponding category descriptors, and then perform necessary preprocessing on the image data, such as image scaling, normalization, and data augmentation;

[0023] S2: Initialize the parameters of the CLIP network, which consists of an image encoder and a text encoder to map images and text into a common embedding space;

[0024] S3: CLIP network is trained with prepared images and category descriptors to align images and texts by minimizing their distance in a common embedding space, thus achieving mutual understanding between images and texts.

[0025] Furthermore, the operation steps of the image classification stage include:

[0026] Step A: prefabricate a prompt template for classification, and combine multiple class names with the prompt template to obtain a category prompt for classification;

[0027] Step B: Input the image and category prompt into the image encoder and text encoder respectively to extract their image features and text features respectively;

[0028] Step C: Calculate the credibility score of each text feature and image feature by aligning the image and text, and use the category with the highest credibility score as the predicted category of the image.

[0029] Furthermore, the specific operation method of step 2 is:

[0030] Step 2.1: Select the first m most confusing classes in the confusion matrix for CLIP calculation evaluation; the visual feedback method using the confusion matrix is ​​as follows:

[0031]

[0032] Where x represents the input image; X d An image representing the class descriptor d, used to compute the classification matrix in the visual feedback V(D); represents the confusion matrix; λ is a threshold hyperparameter used in formula calculation;

[0033] Step 2.2: Evaluation is achieved by computing the confidence scores of the category descriptions in an image-text aligned manner using CLIP.

[0034] Beneficial Effects

[0035] The present invention proposes an image classification method based on large language model visual feedback, which has the following beneficial effects:

[0036] (1) The present invention uses the feedback information of visual classification to gradually refine and optimize the category description, ensuring that the model can more accurately identify and distinguish different categories during the reasoning process. First, the initial category description is generated by the large language model, and then the visual language model is used to classify the image, and the category description is gradually optimized and refined based on the feedback information of the classification results. Through repeated iterations, this method can automatically generate high-quality category prompts, thereby significantly improving the accuracy of image classification. The image classification method based on visual feedback of the large language model provides an innovative method to improve the accuracy of image classification by integrating the open-world knowledge of the large language model and the classification ability of the visual language model.

[0037] (2) The present invention can finely optimize the category prompts to better reflect the characteristics and category information of the image. The optimized category prompts can more effectively guide the model to learn during the training process, thereby significantly improving the overall performance of the model. In practical applications, the precisely optimized category prompts can help the model understand and classify the image content more accurately, thereby improving the accuracy and reliability of classification.

[0038] (3) The iterative optimization method of the present invention has strong interpretability and generalization performance. Without the need for a complex training process, the model optimizes the prompts through multiple iterations, gradually adjusting and improving the expression of the prompts in each iteration. This process not only makes the decision-making process of the model more transparent, but also maintains good performance and stability in a variety of different data sets and application scenarios. The interpretability of the model helps users understand the inner workings of the model, thereby increasing their trust in the model's decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0040] Figure 2 Schematic diagram of the iterative optimization framework in the present invention.

[0041] Figure 3 Schematic diagram of the training framework in the present invention.

[0042] Figure 4 It is a schematic diagram of the classification framework in the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Under the premise of not departing from the design concept of the present invention, various modifications and improvements made by ordinary persons in the art to the technical solutions of the present invention should all fall within the protection scope of the present invention.

[0044] Embodiment 1:

[0045] like Figure 1 As shown, an image classification method based on large language model visual feedback includes the steps of:

[0046] Step 1: It includes three stages of operations: descriptor iterative optimization stage, network training stage and image classification stage.

[0047] Step 1.1: The descriptor iterative optimization phase includes three steps: initializing the category descriptor, building visual feedback, and iteratively optimizing the descriptor. The specific operation method is as follows:

[0048] Step 1.11: The operation mode of class descriptor initialization is as follows: first, all classes are divided into groups according to the similarity of class names; specifically, the text embeddings of class names are extracted and then clustered into groups using the K-means algorithm, where each group represents semantically similar classes; to ensure that the large language model always focuses on easily confused classes, the clustering step is performed not only at initialization, but also dynamically at the beginning of each iteration, and the average embedding of all descriptors for each class is calculated; after clustering, let the large language model generate the initial class descriptor D 0 .

[0049] Step 1.12: Construct visual feedback. The step is to combine the large language model with the visual knowledge in the visual language model. Given the current set of class descriptors D, construct visual feedback V(D) through CLIP's task-related evaluation indicators.

[0050] Step 1.13: Iteratively optimize the descriptor step: In the i-th iteration, provide the previous category descriptor set D i-1 , V(D i-1 ) and memory bank M, asking the large language model to pick out the most useless n in the current descriptor set i descriptors, and use n i They are replaced by new descriptors to emphasize their unique visual features; the expression is:

[0051] D 0i ,...,D Ki =LLM(D i-1 ,M,V)

[0052] Among them, D 0i represents the first class descriptor generated in the ith iteration; D ki represents the k-th class descriptor generated in the i-th iteration; V represents the visual feedback used to evaluate and guide the generation of class descriptors;

[0053] In each iteration, k independent candidate descriptors {D 0i ,D 1i ,...,D ki} to ensure sufficient genetic diversity during the optimization process.

[0054] In addition, the generated k candidate samples and their corresponding visual feedback are taken as input, and then the large language model is prompted to mix and match between different samples, and finally output a new sample D (k+1)i ; Thus, different useful descriptors of different samples are combined together to produce offspring with better overall performance; the expression is:

[0055] D (K+1)i =LLM(D 0i ,...,D Ki ,V).

[0056] Step 1.2: The network training phase includes:

[0057] S1: Collect and prepare a large number of annotated image datasets and their corresponding category descriptors, and then perform necessary preprocessing on the image data, such as image scaling, normalization, and data augmentation.

[0058] S2: Initialize the parameters of the CLIP network, which consists of an image encoder and a text encoder to map images and text into a common embedding space.

[0059] S3: CLIP network is trained with prepared images and category descriptors to align images and texts by minimizing their distance in a common embedding space, thus achieving mutual understanding between images and texts.

[0060] Step 1.3: The steps in the image classification phase include:

[0061] Step A: pre-prepare a prompt template for classification, and combine multiple class names with the prompt template to obtain a category prompt for classification.

[0062] Step B: Input the image and category prompt into the image encoder and text encoder respectively to extract their image features and text features respectively.

[0063] Step C: Calculate the credibility score of each text feature and image feature by aligning the image and text, and use the category with the highest credibility score as the predicted category of the image.

[0064] Step 2: Given a set of iteratively optimized category descriptions, use the task-related CLIP evaluation as an indicator to build visual feedback and provide an overall evaluation of the current category description on the model; the specific operation method is:

[0065] Step 2.1: Select the first m most confusing classes in the confusion matrix for CLIP calculation evaluation; the visual feedback method using the confusion matrix is ​​as follows:

[0066]

[0067] Where x represents the input image; X d An image representing the class descriptor d, used to compute the classification matrix in the visual feedback V(D); represents the confusion matrix; λ is a threshold hyperparameter used in formula calculation;

[0068] Step 2.2: Evaluation is achieved by computing the confidence scores of the category descriptions in an image-text aligned manner using CLIP.

[0069] Step 3: Train the CLIP network with the image and the iterated category descriptors so that they are aligned in a common embedding space through the image encoder ImageEncoder(·) and the text encoder TextEncoder(·); the expression is as follows:

[0070] F x =ImageEncoder(x)

[0071] F d =TextEncoder(d)

[0072] Score = Fx F d

[0073] Where x is the input image, F x is the expression of image features, converting the input image x into a feature vector F x ; F d is the expression of text features, converting the input category descriptor d into a feature vector F d ; d represents the input category descriptor.

[0074] The following is an explanation with examples:

[0075] In this case, in order to retrieve an image of "dog" in the CIFAR-100 database, the method of this embodiment is applied to perform classification operations; the operation steps include a class descriptor iterative optimization stage, a network training stage, and an image classification stage.

[0076] like Figure 1 As shown, the class descriptor iterative optimization phase includes the following steps:

[0077] Class descriptor initialization: First, all classes are divided into groups based on the similarity of their class names. Specifically, the text embeddings of the class names are extracted and then clustered into groups using the K-means algorithm, where each group represents a semantically similar class. To ensure that the large language model always focuses on easily confused classes, the clustering step is performed not only at initialization but also dynamically at the beginning of each iteration, and the average embedding of all descriptors for each class is calculated. After clustering, let the large language model generate the initial class descriptor D 0 .

[0078] After the above process, the most initial class descriptor and its set are obtained.

[0079] Constructing visual feedback: Given the current set of class descriptors D, construct visual feedback V(D) through CLIP’s task-related evaluation metrics.

[0080] The descriptors are input into the CLIP model for evaluation, and the credibility scores of the category descriptions are obtained using a visual feedback method.

[0081] Iterative optimization descriptor: In the i-th iteration, provide the previous class descriptor D i-1 、Visual feedback V(D i-1 ) and the memory bank M, and ask the large language model LLM to emphasize the unique visual features of the class descriptor. In each iteration, k independent candidate descriptors {D 0i ,D 1i ,...,D ki} to ensure sufficient genetic diversity during the optimization process. In addition, the generated k candidate class descriptors and their corresponding visual feedback are used as input, and then the large language model is prompted to mix and match between different samples, and finally a new class descriptor D is output. (k+1)i . Thus, different useful descriptors of different samples are combined together to produce offspring with better overall performance.

[0082] After the above process, an iterative optimization of the category description is achieved. The above steps are repeated to achieve multiple rounds of iterative optimization, and finally the class descriptor with the best performance is obtained.

[0083] like Figure 2 As shown, the network training phase includes the steps of:

[0084] S1: Collect and prepare a large number of annotated image datasets and their corresponding class descriptors, and then perform necessary preprocessing on the image data, such as image scaling, normalization, and data augmentation.

[0085] S2: Initialize the parameters of the CLIP network, which consists of an image encoder and a text encoder to map images and text into a common embedding space.

[0086] S3: CLIP network is trained with prepared images and class descriptors to align images and texts by minimizing their distance in a common embedding space, thus achieving mutual understanding between images and texts.

[0087] After the above process, the image and the iterated class descriptor are input into the CLIP network, and the image encoder and text encoder are trained by image-text alignment to obtain a CLIP model with stronger alignment ability in the common embedding space.

[0088] like Figure 3 As shown, the image classification stage includes the steps of:

[0089] Step A: pre-prepare a prompt template for classification, and combine multiple class names with the prompt template to obtain a category prompt for classification.

[0090] Pre-make "an image of [CLASS]." as a category prompt template, and fill in multiple class names into [CLASS] to obtain the category prompt for classification.

[0091] Step B: The image and category prompt of “dog” are input into the image encoder and text encoder to extract their image features and text features respectively.

[0092] Input an image of "dog" into the image encoder to obtain the visual features of the image. Input all the category prompts to be retrieved into the text encoder to obtain their semantic features.

[0093] Step C: Calculate the credibility score of each text feature and image feature by aligning the image and text, and use the category with the highest credibility score as the predicted category of the image.

[0094] Calculate the visual features of an image of "dog" and the semantic features of the text to be retrieved, and obtain the credibility score of the image and each text to be retrieved. Select the text to be retrieved with the highest credibility score as the predicted category of an image of "dog".

[0095] After the above process, the final category of an image of "dog" is dog.

[0096] In this embodiment, CIFAR-100 contains 100 categories and a total of 60,000 images, including 50,000 training set images and 10,000 test set images; each category corresponds to a set of class descriptors, and there are a total of 100 sets of class descriptors. In order to reflect the effectiveness of this embodiment, the following cases are compared:

[0097] 1. If the image classification is not performed in stage 1 and stage 2, the classification accuracy is:

[0098] CIFAR-100: 83.46%.

[0099] 2. If the image classification is performed completely according to the method of the present invention, the classification accuracy is:

[0100] CIFAR-100: 91.65%, an improvement of 8.19% over Case 1.

[0101] It can be concluded that the image classification method proposed in this embodiment can effectively obtain accurate class descriptors and improve the classification accuracy of images.

[0102] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the present invention.

Claims

1. An image classification method based on visual feedback of a large language model, characterized by: Includes steps: Step 1: It includes the operations of three stages: descriptor iterative optimization stage, network training stage and image classification stage; The descriptor iterative optimization stage includes three steps: class descriptor initialization, visual feedback construction, and iterative optimization of descriptors; Step 2: Given a set of iteratively optimized category descriptions, visual feedback is constructed using task-related CLIP evaluation as an indicator to provide an overall evaluation of the current category description on the model; Step 3: Train the CLIP network with the image and the iterated category descriptors so that they are aligned in a common embedding space through the image encoder ImageEncoder(·) and the text encoder TextEncoder(·); the expression is as follows: F x =ImageEncoder(x) F d =TextEncoder(d) Score=F x F d Where x is the input image, F x is the expression of image features, converting the input image x into a feature vector F x ; F d is the expression of text features, converting the input category descriptor d into a feature vector F d ; d represents the input category descriptor.

2. The image classification method based on large language model visual feedback according to claim 1, characterized in that: The operation mode of class descriptor initialization is as follows: first, all classes are divided into groups according to the similarity of class names; specifically, the text embeddings of the class names are extracted, and then the K-means algorithm is used to cluster them into groups, where each group represents a semantically similar class; in order to ensure that the large language model always pays attention to easily confused categories, the clustering step is performed not only at initialization, but also dynamically at the beginning of each iteration, and the average embedding of all descriptors of each category is calculated; after clustering, the large language model is allowed to generate an initial class descriptor D0.

3. The image classification method based on large language model visual feedback according to claim 1, characterized in that: The step of constructing visual feedback is: combining the large language model with the visual knowledge in the visual language model, given a current set of class descriptors D, and constructing visual feedback V(D) through the task-related evaluation indicators of CLIP.

4. The image classification method based on large language model visual feedback according to claim 1, characterized in that: The iterative optimization descriptor step is: in the i-th iteration, provide the previous category descriptor set D i-1 , V(D i-1 ) and memory bank M, asking the large language model to pick out the most useless n in the current descriptor set i descriptors, and use n i Replace them with new descriptors to emphasize their unique visual features; expression for: D 0i ,...,D Ki =LLM(D i-1 ,M,V) Among them, D 0i represents the first class descriptor generated in the ith iteration; D ki represents the k-th class descriptor generated in the i-th iteration; V represents the visual feedback used to evaluate and guide the generation of class descriptors; In each iteration, independent candidate descriptors {D 0i ,D 1i ,...,D ki } to ensure sufficient genetic diversity during the optimization process; In addition, the generated k candidate samples and their corresponding visual feedback are used as input, and then the large language model is prompted to mix and match between different samples, and finally output a new sample D (k+1)i ; Thus, different useful descriptors of different samples are combined together to produce offspring with better overall performance; the expression is: D (K+1)i =LLM(D 0i ,...,D Ki ,V)。 5. The image classification method based on large language model visual feedback according to claim 1, characterized in that: The operation steps of the network training stage include: S1: Collect and prepare a large number of annotated image datasets and their corresponding category descriptors, and then perform necessary preprocessing on the image data, such as image scaling, normalization, and data augmentation; S2: Initialize the parameters of the CLIP network, which consists of an image encoder and a text encoder to map images and text into a common embedding space; S3: CLIP network is trained with prepared images and category descriptors to align images and texts by minimizing their distance in a common embedding space, thus achieving mutual understanding between images and texts.

6. The image classification method based on large language model visual feedback according to claim 1, characterized in that: The operation steps of the image classification stage include: Step A: prefabricate a prompt template for classification, and combine multiple class names with the prompt template to obtain a category prompt for classification; Step B: Input the image and category prompt into the image encoder and text encoder respectively to extract their image features and text features respectively; Step C: Calculate the credibility score of each text feature and image feature by aligning the image and text, and use the category with the highest credibility score as the predicted category of the image.

7. The image classification method based on large language model visual feedback according to claim 1, characterized in that: The specific operation method of step 2 is: Step 2.1: Select the first m most confusing classes in the confusion matrix for CLIP calculation evaluation; the visual feedback method using the confusion matrix is ​​as follows: Where x represents the input image; X d An image representing the class descriptor d, used to compute the classification matrix in the visual feedback V(D); represents the confusion matrix; λ is a threshold hyperparameter used in formula calculation; Step 2.2: Evaluation is achieved by computing the confidence scores of the category descriptions in an image-text aligned manner using CLIP.

Images