Image classification prediction method fusing positive and negative text subgraphs

Abstract

The invention relates to the technical field of machine learning, and provides an image classification prediction method fusing positive and negative text subgraphs, and the method is characterized in that the method comprises the steps: obtaining a data set for small sample image classification, carrying out the processing, cleaning and packaging of the data set, and constructing a training set, a verification set and a test set; a GraphPromptAdapter model based on a positive and negative text graph prompt is constructed; the training set is input to a GraphPromptAdapter model, and training fitting is carried out on the GraphPromptAdapter model; the verification set is input to a GraphPromptAdapter model to be evaluated; and inputting the test set to the GraphPromptAdapter model to carry out prediction on image classification. According to the method, by fusing the negative text subgraph and triple loss optimization and combining graph prompt to enhance cross-modal structure modeling, the classification accuracy and discrimination ability of the vision-language model in a few-sample scene are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and in particular relates to an image classification prediction method integrating positive and negative text subgraphs. Background Art

[0002] In recent years, efficient transfer learning (ETL) methods for vision-language models (VLMs) have made significant progress in low-data environments. Adapter-style ETL methods introduce only a small number of additional parameters to mine task-specific knowledge while maintaining the generalization capabilities of pre-trained models. However, existing adapter methods suffer from the following two main problems: (1) Most methods only model task-specific knowledge from a single modality and fail to fully utilize cross-modal structural knowledge; (2) they ignore the structural relationships between categories in downstream tasks, resulting in suboptimal transfer effects.

[0003] Research represented by GraphAdapter attempts to introduce knowledge graphs to enhance the adaptability of visual-language models. GraphAdapter constructs text knowledge subgraphs and visual knowledge subgraphs to enable text features to utilize the structural information of both textual and visual modalities. However, this method still has the following shortcomings: (1) Failure to fully consider negative sample information. Existing methods mainly extract features through positive text subgraphs and visual subgraphs. However, the role of negative text samples (i.e., "images that do not belong to a certain category") in actual tasks is ignored, which may mislead the model when distinguishing different categories; (2) Model interaction methods are limited. Existing methods only consider the structural interaction between text and vision, but fail to fully utilize the structural knowledge of negative text, resulting in a lack of adversarial information during model training, which affects the model's discriminative ability; (3) The utilization of graph prompt features (GPF) is still relatively limited. Although GraphAdapter introduces dual knowledge graphs, its prompt method fails to optimize for different subgraphs, resulting in limitations in the utilization of structural information. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the existing technology, the present invention proposes an image classification prediction method that integrates positive and negative text subgraphs to improve the feature representation ability and classification accuracy of the vision-language model. The technical solution designed by the present invention includes the following steps: S1: Obtain a dataset for small sample image classification, process, clean, and package the dataset, and construct a training set, validation set, and test set; S2: Constructing a GraphPromptAdapter model based on positive and negative text graph prompts; S3: Input the training set to the GraphPromptAdapter model for training and fitting; S4: Input the validation set to the GraphPromptAdapter model for evaluation; S5: Input the test set to the GraphPromptAdapter model to predict the image classification.

[0005] Preferably, the S2 includes: S21: Building a deep learning environment; S22: Input the training set and the positive / negative text prompts of the corresponding category. The training set is used to extract visual features and generate visual vectors through the pre-trained visual encoder. The positive / negative text prompts are used to extract positive / negative text vectors through the pre-trained text encoder. S23: Construct three modal knowledge subgraphs respectively, including positive text subgraph, negative text subgraph and visual knowledge subgraph; S24: Introduce a learnable graph hint feature vector for each subgraph and update the hint feature through residual connection; S25: Based on the graph convolutional network, the text features of the text encoder interact with each subgraph, and the fusion output is adapted to the text features of the downstream tasks.

[0006] Preferably, the step S23 includes: The positive text subgraph uses the positive text vector as the node representation and constructs the edge weight matrix based on the cosine similarity between nodes; Negative text subgraph, which uses negative text vectors as node representations and constructs edge weight matrices based on the cosine similarity between nodes; The visual knowledge subgraph uses the mean of the visual features of each category in the training samples as the node representation, and generates edge weights based on feature similarity.

[0007] Preferably, the S24 includes: Define a learnable hint vector for each subgraph , Consistent with the node feature dimension, the initial features of all nodes in each subgraph are Update to prompt feature

[0008] Preferably, the step S25 includes: S251: Given text features from a text encoder ,Will Projecting to each subgraph to extract text modality and cross-modal structural knowledge; S252: Through the graph convolutional network, the graph interacts with each subgraph in the same graph space to obtain three feature vectors that respectively integrate three different structural knowledge; S253: Setting a hyperparameter The convolution vectors from the text subgraph and the visual knowledge subgraph are fused, and the updated vector weight of the text subgraph is set to , the updated vector weight of the visual knowledge subgraph is set to , get a fusion feature; S254: Setting a hyperparameter , using residual connection to fuse text features The fused features with the graph convolution output are used to obtain the final text features that are suitable for downstream tasks.

[0009] Preferably, the S3 includes: S31: Input the training set and positive / negative text prompts into the GraphPromptAdapter model; S32: Classification optimization using cross entropy loss function; S33: Design a triplet loss function to reduce the distance between visual features and positive text cues and expand the distance between visual features and negative text cues.

[0010] Preferably, the S4 includes: The GraphPromptAdapter model is evaluated using the built-in function of pytorch. After each epoch of training, the validation set is used to verify the current training model.

[0011] Preferably, the S5 includes: Load the optimal model parameter information saved during the training phase to initialize the GraphPromptAdapter model; Load the test set images and corresponding labels used to test the GraphPromptAdapter model; Statistics of the test rate and standard deviation of the GraphPromptAdapter model.

[0012] Beneficial effects: This application provides an image classification prediction method that integrates positive and negative text subgraphs. By integrating negative text subgraphs with triplet loss optimization and combining image prompts to enhance cross-modal structure modeling, the classification accuracy and discrimination ability of the vision-language model in few-sample scenarios are significantly improved, effectively solving the problem of confusion between similar categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of a GraphPromptAdapter model according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0014] The embodiments of the present invention are described in detail below. The following embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

[0015] This paper designs an image classification prediction method that integrates positive and negative text sub-graphs. Figure 1-2 As shown, the technical solution includes the following steps, specifically including: S1: Obtain a dataset for small sample image classification, process, clean, and package the dataset, and construct a training set, validation set, and test set; S2: Constructing a GraphPromptAdapter model based on positive and negative text graph prompts; S3: Input the training set to the GraphPromptAdapter model for training and fitting; S4: Input the validation set to the GraphPromptAdapter model for evaluation; S5: Input the test set to the GraphPromptAdapter model to predict the image classification.

[0016] Specifically, for S1, we download 11 datasets for few-shot classification tasks from https: / / github.com / KaiyangZhou / CoOp / blob / main / DATASETS.md, including ImageNet, StandfordCars, UCF101, Caltech101, Flowers102, SUN397, DTD, EuroSAT, FGVCAircraft, OxfordPets, and Food101. Among them, OxfordPets, Food101, StanfordCars, Flowers102 and FGVCAircraft belong to fine-grained classification tasks, EuroSAT is used for remote sensing classification, and DTD is a texture classification dataset; the original data in the dataset is cleaned, including removing the first line in the dataset, and removing spaces and punctuation marks from the text information in the dataset; for the cleaned dataset, since this experiment is a small sample classification task, the 1, 2, 4, 8, and 16 shots set in the configuration file respectively represent setting the training set samples to one category with one labeled picture for training, and so on. When 16 shots is used, one category has 16 labeled pictures for training, and the remaining pictures are used for the test set.

[0017] Preferably, S2 includes: S21: Building a deep learning environment; S22: Input the training set and the positive / negative text prompts of the corresponding category. The training set is used to extract visual features and generate visual vectors through the pre-trained visual encoder. The positive / negative text prompts are used to extract positive / negative text vectors through the pre-trained text encoder. S23: Construct three modal knowledge subgraphs respectively, including positive text subgraph, negative text subgraph and visual knowledge subgraph; S24: Introduce a learnable graph hint feature vector for each subgraph and update the hint feature through residual connection; S25: Based on the graph convolutional network, the text features of the text encoder interact with each subgraph, and the fusion output is adapted to the text features of the downstream tasks.

[0018] Preferably, S23 includes: The positive text subgraph uses the positive text vector as the node representation and constructs the edge weight matrix based on the cosine similarity between nodes; Negative text subgraph, which uses negative text vectors as node representations and constructs edge weight matrices based on the cosine similarity between nodes; The visual knowledge subgraph uses the mean of the visual features of each category in the training samples as the node representation, and generates edge weights based on feature similarity.

[0019] Preferably, S24 includes: Define a learnable hint vector for each subgraph , Consistent with the node feature dimension, the initial features of all nodes in each subgraph are Update to prompt feature .

[0020] Preferably, S25 includes: S251: Given text features from a text encoder ,Will Projecting to each subgraph to extract text modality and cross-modal structural knowledge; S252: Through the graph convolutional network, the graph interacts with each subgraph in the same graph space to obtain three feature vectors that respectively integrate three different structural knowledge; S253: Setting a hyperparameter The convolution vectors from the text subgraph and the visual knowledge subgraph are fused, and the updated vector weight of the text subgraph is set to , the updated vector weight of the visual knowledge subgraph is set to , get a fusion feature; S254: Setting a hyperparameter , using residual connection to fuse text features The fused features with the graph convolution output are used to obtain the final text features that are suitable for downstream tasks.

[0021] Specifically, for S2, a deep learning environment is built, including installing Anaconda on the server and creating a virtual environment. In the virtual environment, a Pytorch-GPU operating environment is built, and PyTorch related libraries and necessary dependency packages such as scipy, numpy, networkx, and torch-geometric are installed to support graph neural network (GNN) operations and calculations. For a given dataset, two types of template text prompts are generated for each category: positive text prompts (such as "a photo of a [class]") and negative text prompts (such as "Not a photo of a [class]"). The positive text prompts and negative text prompts are respectively input into the pre-trained text encoder to obtain the corresponding positive text vectors and negative text vectors, both of size . ,in represents the number of categories, is the feature dimension.

[0022] Preferably, S3 includes: S31: Input the training set and positive / negative text prompts into the GraphPromptAdapter model; S32: Classification optimization using cross entropy loss function; S33: Design a triplet loss function to reduce the distance between visual features and positive text cues and expand the distance between visual features and negative text cues.

[0023] Specifically, for S3, training and fitting were performed using the constructed GraphPromptAdapter model. The number of iterations was initialized with epoch=0, and the epoch value was fixed at 200 for the 1-, 2-, 4-, 8-, and 16-shot experiments. The batch size was fixed at 32 for each dataset. The Adam optimizer was used with an initial learning rate of 1e-3, which was decreased with a cosine learning rate decay schedule. To ensure stable training, a warm-up strategy was used, with a small learning rate of 1e-5 applied in the first epoch. The GraphPromptAdapter model was constructed using the PyTorch framework, and the loss functions were set to cross-entropy and triplet loss. The visual subgraph was constructed using the few-shot training samples. Before training, the pre-trained CLIP visual encoder was used to extract visual features from these few-shot samples of the same class and average them into nodes for that class. The number of images per class was determined by the number of shots. For example, for the 2-shot task, two images were used per node. The data augmentation strategies only included random resizing and cropping and random flipping.

[0024] Preferably, S4 includes: The GraphPromptAdapter model is evaluated using the built-in function of pytorch. After each epoch of training, the validation set is used to verify the current training model.

[0025] Preferably, S5 includes: Load the optimal model parameter information saved during the training phase to initialize the GraphPromptAdapter model; Load the test set images and corresponding labels used to test the GraphPromptAdapter model; Statistics of the test rate and standard deviation of the GraphPromptAdapter model.

[0026] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention can be made by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art based on the concepts of the present invention through logical analysis, reasoning, or limited experimentation based on the existing technology should be within the scope of protection defined by the claims.

Claims

1. An image classification prediction method integrating positive and negative text subgraphs, characterized in that: include: S1: Obtain a dataset for small sample image classification, process, clean, and package the dataset, and construct a training set, validation set, and test set; S2: Constructing a GraphPromptAdapter model based on positive and negative text graph prompts; S3: Input the training set to the GraphPromptAdapter model for training and fitting; S4: Input the validation set to the GraphPromptAdapter model for evaluation; S5: Input the test set to the GraphPromptAdapter model to predict the image classification.

2. The image classification prediction method for fusing positive and negative text subgraphs according to claim 1, characterized in that: Said S2 comprises: S21: Building a deep learning environment; S22: Input the training set and the positive / negative text prompts of the corresponding category. The training set is used to extract visual features and generate visual vectors through the pre-trained visual encoder. The positive / negative text prompts are used to extract positive / negative text vectors through the pre-trained text encoder. S23: Construct three modal knowledge subgraphs respectively, including positive text subgraph, negative text subgraph and visual knowledge subgraph; S24: Introduce a learnable graph hint feature vector for each subgraph and update the hint feature through residual connection; S25: Based on the graph convolutional network, the text features of the text encoder interact with each subgraph, and the fusion output is adapted to the text features of the downstream tasks.

3. The image classification prediction method for fusing positive and negative text subgraphs according to claim 2, characterized in that: Said S23 comprises: The positive text subgraph uses the positive text vector as the node representation and constructs the edge weight matrix based on the cosine similarity between nodes; Negative text subgraph, which uses negative text vectors as node representations and constructs edge weight matrices based on the cosine similarity between nodes; The visual knowledge subgraph uses the mean of the visual features of each category in the training samples as the node representation, and generates edge weights based on feature similarity.

4. The image classification prediction method for fusing positive and negative text subgraphs according to claim 2, characterized in that: The S24 includes: Define a learnable hint vector for each subgraph , Consistent with the node feature dimension, the initial features of all nodes in each subgraph are Update to prompt feature .

5. The image classification prediction method for fusing positive and negative text subgraphs according to claim 2, characterized in that: The S25 includes: S251: Given text features from a text encoder ,Will Projecting to each subgraph to extract text modality and cross-modal structural knowledge; S252: Through the graph convolutional network, the graph interacts with each subgraph in the same graph space to obtain three feature vectors that respectively integrate three different structural knowledge; S253: Setting a hyperparameter The convolution vectors from the text subgraph and the visual knowledge subgraph are fused, and the updated vector weight of the text subgraph is set to , the updated vector weight of the visual knowledge subgraph is set to , get a fusion feature; S254: Setting a hyperparameter , using residual connection to fuse text features The fused features with the graph convolution output are used to obtain the final text features that are suitable for downstream tasks.

6. The image classification prediction method for fusing positive and negative text subgraphs according to claim 1, characterized in that: The S3 includes: S31: Input the training set and positive / negative text prompts into the GraphPromptAdapter model; S32: Classification optimization using cross entropy loss function; S33: Design a triplet loss function to reduce the distance between visual features and positive text cues and expand the distance between visual features and negative text cues.

7. The image classification prediction method for fusing positive and negative text subgraphs according to claim 1, characterized in that: Said S4 comprises: The GraphPromptAdapter model is evaluated using the built-in function of pytorch. After each epoch of training, the validation set is used to verify the current training model.

8. The image classification prediction method for fusing positive and negative text subgraphs according to claim 1, characterized in that: Said S5 comprises: Load the optimal model parameter information saved during the training phase to initialize the GraphPromptAdapter model; Load the test set images and corresponding labels used to test the GraphPromptAdapter model; Statistics of the test rate and standard deviation of the GraphPromptAdapter model.

Images