Class increment image classification method based on image-text double-guide classifier extension

Abstract

The invention discloses a class increment image classification method based on image-text double-guide classifier extension, and aims to improve the image classification accuracy. Constructing a class increment image classification system composed of a text feature extraction module, an image feature extraction module, a contrast learning classification module, a contrast text generation module, an image-text dual-guide classifier, a classification result merging module, a feature mixing module and an entropy-guided loss weighting module; training the text feature extraction module and the image feature extraction module, performing preliminary and enhanced training on the image-text dual-guide classifier, and performing loss weighting in enhanced training according to an enhanced training feature entropy value calculated by the entropy-guided loss weighting module; and the enhanced training feature set is generated through cooperation of a classification result merging module, a comparison text generation module and a feature mixing module. And the trained class increment image classification system classifies the images to obtain image classes. The method can effectively reduce the prediction entropy and improve the image classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of incremental image classification, and specifically to an incremental image classification method based on an extension of a dual-guided image and text classifier. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, machine learning algorithms have been widely applied in many fields such as image recognition, intelligent security, autonomous driving, and medical image analysis. However, traditional image classification models typically require centralized training on labeled data containing all categories, and retraining the entire model whenever the data distribution changes or new categories appear. This is impractical in real-world scenarios where data is constantly growing and categories are dynamically expanding. For example, in scenarios such as new product launches on e-commerce platforms, new targets added to monitoring systems, and personalized photo album categorization in mobile applications, traditional methods require periodically collecting and retraining the entire dataset. This not only incurs high computational and time costs but also fails to meet the demands for real-time response and efficient updates. To address this issue, class-incremental image classification has emerged. Class-incremental image classification aims to enable models to learn only from data of newly added categories without retraining existing categories, while maintaining the ability to recognize already learned categories as much as possible. This approach significantly reduces computational overhead and storage requirements, enhances the model's continuous learning and adaptability, and makes it more suitable for dynamic environments in the real world where data arrives in streams and categories gradually increase.

[0003] However, incremental image classification faces a core challenge: during the learning of new categories, model parameter updates often overwrite or weaken the representations of already learned categories, leading to a significant performance drop—the catastrophic forgetting phenomenon. This raises a fundamental contradiction between stability and plasticity in incremental learning: stability refers to the model's ability to retain old knowledge, while plasticity refers to its ability to adapt to new knowledge. Extensive research has been dedicated to mitigating the forgetting problem in pre-trained models.

[0004] The paper "Zhou DW, Cai ZW, Ye HJ, et al. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptability are all you need[J]. International Journal of Computer Vision, 2025, 133(3): 1012-1032." by Zhou et al., titled "Revisiting class-incremental learning with pre-trained models: Generalizability and adaptability are all you need (SimpleCIL)," essentially freezes the pre-trained model and uses only "class prototype / mean features" to analytically set the classifier, achieving a strong baseline effect in the class-incremental image classification scenario using a minimalist mechanism; the paper "Zhou K, Yang J, Loy CC, et al. Learning to prompt for vision-language models[J]. International Journal of Computer Vision, 2022, 130(9):" by Zhou et al., titled "Revisiting class-incremental learning with pre-trained models: Generalizability and adaptability are all you need[J]. International Journal of Computer Vision, 2022, 130(9):" by Zhou et al., titled "Revisiting class-incremental learning with pre-trained models: Generalizability and adaptability are all you need[J]. International Journal of Computer Vision, 2025, 133(3): 1012-1032.", ...2, 130(9):" by Zhou et al., titled "Revisiting class-incremental 2337-2348. Zhou et al.'s paper: Learning how to build prompts for visual-language models (L2P) maintains a learnable "prompt pool" and retrieves and combines the most relevant prompts based on the input to guide the pre-trained model to complete continuous tasks, thereby slowing down forgetting without replaying the data; the paper "Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C] / / European conference on computervision. Cham: Springer Nature Switzerland, 2022: 631-648." Wang et al.'s paper: DualPromot: A complementary prompting method for rehearsal-free continual learning decomposes prompts into two types of complementary prompts, "general" and "expert", which simultaneously take into account stability and plasticity in rehearsal-free continual learning; the paper "Smith JS, Karlinsky L, Gutta V, et al."Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2023: 11909-11919. Smith et al.'s paper, (CODA-Promot), uses "decomposed attention" to generate / select prompts, improving rehearsal-free continual learning with higher capacity and end-to-end prompting.

[0005] In recent years, the emergence of large-scale vision-language pre-trained models has provided a powerful new foundation for incremental image classification. CLIP (see the paper "Alec Radford, et al. Learning transferable visual models from natural language supervision. International conference on machine learning. PMLR, 2021." by Alec Radford et al.) is a classic large-scale vision-language model. It constructs an aligned cross-modal shared representation space through contrastive learning on massive image-text pairs, possessing excellent zero-shot generalization ability and rich open-world knowledge. Incremental image classification methods based on CLIP have attracted widespread attention because they can utilize this prior knowledge.

[0006] The paper "Thengane V, Khan S, Hayat M, et al. Clip model is an efficient continual learner[J]. arXiv preprint arXiv:2210.03114, 2022." by Thengane V et al., "Continual CLIP," pioneered the application of CLIP models to incremental learning scenarios. This method does not introduce additional trainable parameters and directly utilizes pre-trained CLIP models, initially demonstrating its potential in continuous learning tasks. However, this method does not fully utilize the cross-modal semantic space constructed by CLIP. On the CIFAR100 dataset (see the paper "Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images[J]. 2009." by Krizhevsky et al.), the average classification accuracy only reached 81.81% in a 10-stage incremental benchmark test, indicating significant room for improvement. Furthermore, the model still faces a severe catastrophic forgetting problem.The paper "Zhou K, Yang J, Loy CC, et al. Learning to prompt for vision-language models[J]. International Journal of Computer Vision, 2022, 130(9): 2337-2348." by Zhou et al.: Learning to prompt for vision-language models (CoOp) replaces contextual words in text prompts with learnable vectors, and optimizes these contextual vectors to adapt vision-language models such as CLIP to downstream recognition tasks; the paper "Huang L, Cao X, Lu H, et al. Class-incremental learning with clip: Adaptive representation adjustment and parameter fusion[C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: The paper by Huang et al., “Category Incremental Learning of CLIP: Adaptive Representation Adjustment and Parameter Fusion (RAPF),” evaluates the impact of new classes on old classes through “adaptive representation adjustment” and corrects the representations with text features. Then, it uses “decomposed parameter fusion” to alleviate the forgetting caused by adapter fine-tuning. However, these methods do not make sufficient use of the text modalities present in CLIP. Simply using general category cues cannot further distinguish images at the text level. The paper "Zhou DW, Li KW, Ning J, et al. External Knowledge Injection for CLIP-Based Class-Incremental Learning[J]. arXiv preprint arXiv:2503.08510, 2025." by Zhou et al., entitled "External Knowledge Injection for CLIP-Based Class-Incremental Learning (ENGINE)," fixes the CLIP image encoder and text encoder to extract features from images and text. It adapts to the classification task at each stage of incremental learning by designing task-specific projection layers and enriches the extracted features from the image and text bimodal learning using the obtained external knowledge. As a result, the average classification accuracy of the model was further improved to 85.50% on the CIFAR100 benchmark, achieving the current state-of-the-art performance.

[0007] These incremental image classification methods based on the CLIP (Visual-Language Large Model) leverage CLIP's powerful feature extraction and generalization capabilities. However, they still utilize CLIP's inherent cosine similarity-based classifier during classification. Since CLIP itself is trained through contrastive learning to construct a cross-modal representation space rather than optimizing low-entropy predictions with high discriminative power, CLIP's original cosine similarity-based classifier often produces predictions with low confidence and high entropy, making CLIP a high-entropy classifier. This high-entropy state implies ambiguity in the model's attribution of inputs, making predictions highly unstable during subsequent incremental parameter updates, prone to confusion, thus weakening model stability and exacerbating catastrophic forgetting. Therefore, there is still room for improvement in the accuracy of incremental image classification.

[0008] Improving the accuracy of incremental image classification remains a technical issue of great concern to those skilled in the art. Summary of the Invention

[0009] The technical problem this invention aims to solve is the low prediction confidence, poor model stability, and severe catastrophic forgetting inherent in existing incremental image classification methods based on large vision-language models, which suffer from low prediction confidence, poor model stability, and severe catastrophic forgetting due to the direct use of high-entropy classifiers like CLIP. This invention proposes an incremental image classification method based on an extension of a dual-guided image-text classifier. This method introduces a trainable dual-guided image-text classifier and employs a two-stage optimization and output fusion strategy to effectively reduce prediction entropy and enhance the discriminative power of decision boundaries, thereby improving image classification accuracy.

[0010] To solve the above-mentioned technical problems, the technical solution of this invention is: to construct an incremental image classification system based on an extended image-text dual-guided classifier, referred to as an incremental image classification system. This system consists of a text feature extraction module, an image feature extraction module, a contrastive learning classification module, a contrastive text generation module, an image-text dual-guided classifier, a classification result merging module, a feature mixing module, and an entropy-guided loss weighting module; the system prepares and constructs the dataset required for training the incremental image classification system, dividing the dataset into a training set and a test set; the training set is used to initially train the text feature extraction module and the image feature extraction module; the contrastive learning classification module receives general text features sent by the text feature extraction module and image features sent by the image feature extraction module, calculates their similarity, and obtains the text classification prediction probability; the image-text dual-guided classifier performs text... After training, the feature extraction module and the image feature extraction module undergo training, which involves two stages: preliminary training and augmentation training. Based on the image features received from the trained image feature extraction module, predictions are made to obtain the image classification prediction probability. The classification result merging module merges the text classification prediction probability sent by the comparative learning classification module and the image classification prediction probability sent by the image-text dual-guided classifier to obtain the merged classification probability as the classification result of the class incremental image classification system. The comparative text generation module, feature mixing module, and entropy-guided loss weighting module only participate in the augmentation training of the image-text dual-guided classifier and do not participate in the classification process of the user input image. During the augmentation training of the image-text dual-guided classifier, the contrast text generation module generates corresponding contrast text features based on the merged classification probabilities received from the classification result merging module; the feature mixing module mixes the image features received from the image feature extraction module, the general text features received from the text feature extraction module, and the contrast text features received from the contrast text generation module, and combines them with the ground truth labels received from the contrast text generation module to obtain augmented training features; the entropy-guided loss weighting module calculates the entropy value of the augmented training features and applies loss weighting to the augmented feature data based on the entropy value during the augmentation training of the image-text dual-guided classifier; finally, the trained incremental image classification system is used to classify the user-input image to obtain the image category.

[0011] This invention includes the following steps: The first step is to construct an incremental image classification system based on an extension of a dual-guided image-text classifier. For example... Figure 1 As shown, the incremental image classification system consists of a text feature extraction module, an image feature extraction module, a contrastive learning classification module, an image-text dual-guided classifier module, a classification result merging module, a contrastive text generation module, a feature mixing module, and an entropy-guided loss weighting module.

[0012] The text feature extraction module is connected to the contrastive learning classification module, the feature mixing module, and the contrastive text generation module. The text feature extraction module constructs general text prompts using labels from the training set (e.g., when the label is "cat", the general text prompt is "A photo of cat") and extracts general text features from these prompts. The text feature extraction module consists of a text encoder, T text projection layers, and a first adder, where T is the number of training stages in the incremental image classification training process, and T is a positive integer, typically 10. The text encoder parameters are pre-trained using CLIP on the Transformer architecture (see the paper "Alec Radford, et al. Learning transferable visual models from natural language supervision. International conference on machine learning. PMLR, 2021." by Alec Radford et al.). These parameters are frozen during training and do not participate in the training process. The text encoder extracts features from the general text prompts, obtaining preliminary text features which are then sent to the T text projection layers. All T text projection layers are fully connected layers, projecting the initial text features received from the text encoder in parallel to obtain T projected text features containing corresponding task information. A first adder sums these T projected text features to obtain the final text feature. This method effectively incorporates feature information from all previous tasks, making it better suited for incremental image classification tasks. During training of the text feature extraction module, it extracts the first general text features from the general text prompts corresponding to all labels in the training set and sends them to the contrastive learning classification module. During training of the image-text dual-guided classifier module, the trained text feature extraction module re-extracts the second general text features from the general text prompts and sends them to the contrastive learning classification module and the feature mixing module. It also receives a list of contrastive text prompts from the contrastive text generation module, extracts the contrastive text features from all the prompts in the list, and sends the ground truth labels and contrastive text features corresponding to the images in the training set to the contrastive text generation module.

[0013] The image feature extraction module is connected to the contrastive learning classification module, the image-text dual-guided classifier, and the feature mixing module. The image feature extraction module extracts features from the images in the training set to obtain image features. The image feature module consists of an image encoder, T image projection layers, and a second adder. The image encoder parameters are pre-trained using CLIP on a Transformer architecture; these parameters are frozen during training and do not participate in the training process. The image encoder extracts features from the images in the training set, obtaining preliminary image features which are then sent to the T image projection layers. All T image projection layers are fully connected layers, projecting the preliminary image features obtained from the image encoder in parallel to obtain T projected image features containing corresponding task information. The second adder adds the T projected image features to obtain the final image features. During training, the image feature extraction module sends the first image features extracted from the training set images to the contrastive learning classification module. During training of the image-text dual-guided classifier module, the trained image feature extraction module re-extracts image features from the training set images and sends the resulting second image features to the contrastive learning classification module, the image-text dual-guided classifier, and the feature mixing module.

[0014] The contrastive learning classification module is connected to the text feature extraction module, the image feature extraction module, and the classification result merging module. The contrastive learning classification module uses the general text features received from the text feature extraction module and the image features received from the image feature extraction module to calculate the text classification prediction probability, and sends the text classification prediction probability to the classification result merging module.

[0015] The image-text dual-guided classifier is connected to the image feature extraction module, the classification result merging module, and the entropy-guided loss weighting module. The image-text dual-guided classifier consists of a single fully connected layer that ignores the bias term. It predicts the image features received from the image feature extraction module to obtain the image classification prediction probability, and sends the image classification prediction probability to the classification result merging module. At the same time, the image-text dual-guided classifier predicts the enhanced training features sent by the entropy-guided loss weighting module, and sends the image classification prediction probability corresponding to the enhanced training features to the entropy-guided loss weighting module.

[0016] The classification result merging module is connected to the contrastive learning classification module, the image-text dual-guided classifier, the contrastive text generation module, and the entropy-guided loss weighting module. It merges the text classification prediction probabilities received from the contrastive learning classification module and the image classification prediction probabilities received from the image-text dual-guided classifier to obtain the merged classification prediction probabilities, and sends the merged classification prediction probabilities to the contrastive text generation module and the entropy-guided loss weighting module.

[0017] The contrast text generation module is connected to the classification result merging module, feature mixing module, and text feature extraction module. The contrast text generation module prepares for the enhanced training of the image-text dual-guided classifier. This module receives the merged classification prediction probabilities from the classification result merging module, takes the Top-N results from the merged probability, and obtains the Top-N predicted labels for the training set images. Simultaneously, it receives the ground truth labels corresponding to the images in the training set from the text feature extraction module. It utilizes large language models (requiring strong language skills to accurately describe the contrast features of two classes in detail without images) (such as GPT-4 (see the paper "Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023." Achiam et al.'s paper: Gpt-4 Technical Report) and ChatGLM (see the paper "Zeng, Aohan, et al. The Eleventh International Conference on Learning Representations. 2022." Zeng, The paper GLM-130B by Aohan et al. (an open-ended bilingual pre-trained model) compares Top-N predicted labels with real labels to obtain a list of contrasting text prompts. This list is then sent to a text feature extraction module, which in turn receives the contrasting text features extracted from the text feature extraction module. Finally, the contrasting text features and the real labels corresponding to the images in the training set are sent to a feature mixing module.

[0018] The feature fusion module is connected to the text feature extraction module, image feature extraction module, contrast text generation module, and entropy-guided loss weighting module. The feature fusion module receives image features from the image feature extraction module, receives the real labels corresponding to the images in the training set and the contrast text features from the contrast text generation module, and receives general text features from the text feature extraction module. It combines the image features, general text features, and contrast text features to form an enhanced training feature set, and sends the enhanced training feature set and the real labels corresponding to the images in the training set to the entropy-guided loss weighting module.

[0019] The entropy-guided loss weighting module is connected to the feature mixing module, the classification result merging module, and the image-text dual-guided classifier. The entropy-guided loss weighting module receives the merged classification prediction probabilities from the classification result merging module and the enhanced training feature set from the feature mixing module. It calculates the entropy value based on the merged classification prediction probabilities to determine if the sample image is a high-entropy sample. Then, it forwards the enhanced training features from the enhanced training feature set to the image-text dual-guided classifier. The image-text dual-guided classifier calculates the image classification prediction probability corresponding to the enhanced training features and sends it to the entropy-guided loss weighting module. The entropy-guided loss weighting module calculates the loss for the corresponding sample, increasing the loss weight for the enhanced features of high-entropy samples during the calculation, thus obtaining the loss of the training set. This training set loss is then sent to the image-text dual-guided classifier. In this way, the incremental image classification system enhances its ability to distinguish high-entropy samples (high entropy values ​​indicate that the model is not confident in the results for that sample, making it a difficult sample to classify) during training, increasing the confidence of the incremental image classification system in the classification results, thereby further improving the accuracy of the incremental image classification task.

[0020] All the above modules participate in the training process, but the contrastive text generation module, feature mixing module, and entropy-guided loss weighting module only participate in training and do not participate in the user input image classification process. When classifying the user input image, the text feature extraction module only needs to extract the general text prompts corresponding to all image labels used in the training set during training (the incremental image classification domain training process is divided into T training stages, and each stage only uses a portion of the image labels in the training set) to obtain the corresponding general text features. The image feature extraction module extracts the image features of the user input image. The contrastive learning classification module obtains the text classification prediction probability of the user input image. The image-text dual-guided classifier obtains the image classification prediction probability of the user input image. The classification result merging module merges the text classification prediction probability and the image classification prediction probability of the user input image to obtain the merged classification probability of the user input image, which is taken as the final classification probability of the user input image. The label corresponding to the highest probability is taken as the final classification result of the user input image.

[0021] The second step involves collecting incremental learning scene images as a dataset for incremental image classification, and then dividing the incremental learning sample image classification dataset into a training set. Test set ,right The training set list is obtained by further dividing the training phase. ,Will Based on the training phase division, a test set list is obtained. The method is: Step 2.1 Collect incremental learning scene images as a class incremental image classification scene dataset. The method is as follows: We used the small image dataset CIFAR100 (see the paper "Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images[J]. 2009." by Krizhevsky et al.), the general scene dataset ImageNet-R (see the paper "Hendrycks D, Basart S, Mu N, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 8340-8349." by Hendrycks et al.), and the bird dataset CUB200 (see the paper "Wah C, Branson S, Welinder P, et al. The caltech-ucsd birds-200-2011 dataset[J]. 2011. "Wah et al.'s paper: Caltech-UC San Diego Birds 200-2011 Dataset), Food101 Dataset (see "BOSSARD, Lukas, et al. Food-101—mining discriminative components with random forests. In: Computer Vision–ECCV 2014:6-12, 2014.p. 446-461." BOSSARD et al.'s paper: Food101: Mining Discriminative Components with Random Forests), StanfordCars Dataset (see "Jonathan Krause, et al. 3D Object Representations for Fine-Grained Categorization[A]. 2013 IEEE International Conference on Computer Vision Workshops[C],2013" Jonathan Krause et al.'s paper: 3D Object Representations for Fine-Grained Categorization), and UCF101 Action Dataset (see "Khurram Soomro, et al.").UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild [J]. 2012. Khurram Soomro et al.'s paper: UCF101: A Dataset of 101 Human Actions Classes From Videos in the Wild [J]. 2012. Scene classification dataset SUN397 (see Xiao Jianxiong, et al. SUN database: Large-scale scene recognition from abbey to zoo [A]. 2010). Xiao Jianxiong et al.'s paper: SUN dataset: Large-scale scene recognition from abbey to zoo. Aircraft classification dataset Aircraft (see Subhransu Maji, et al. Fine-Grained Visual Classification of Aircraft [J]. 2013). Subhransu Maji et al.'s paper: Fine-grained visual classification of aircraft. Object recognition dataset Objectnet (see Barbu A, Mayo D, Alverio J, et al. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models [J]. Advances in neural information processing systems). The 2019 paper by Barbu et al. (32) presents a large-scale bias-controlled dataset for pushing the limits of object recognition models. Nine datasets (approximately 400,000 images in total) are used as incremental learning sample image classification datasets. Each image in these nine datasets is manually labeled, meaning each image is labeled with the true label of the object's category.

[0022] Step 2.2 Based on the original partitioning criteria in the dataset, divide the incremental learning sample image classification dataset into a training set. test set Let the total number of categories in the sample image classification dataset be C, and each image have a true label for its corresponding category (usually [1,2,…,c,…,C]). C is a positive integer.

[0023] Step 2.3 In the field of incremental image classification, the training process is divided into T training stages (T is a positive integer, typically 10) to simulate the continuous increase of data. During training, only the training data from the current stage is usually used, while a small amount of training data from previous stages is allowed to be saved. Therefore, a set partitioning method is used to partition the training set. The training set list is obtained by further dividing the training phase. , , And t is an integer, the method is: Step 2.3.1 Let the training phase t = 1; Step 2.3.2 Let the number of classes included in the t-th training phase be... (Assuming) There are 10 categories, and the incremental learning stage T is 10. The number of categories in each stage is 10. The set of class labels included in the t-th training phase. = (For example: for training phase 1, the first training phase contains the set of class labels) For training phase 2, the second training phase includes the set of category labels. ); Step 2.3.3 Find the training set China belongs to Images (i.e., finding the true labels in the training set) (same image), China belongs to Adding images to middle; Step 2.3.4 If ,make Proceed to step 2.3.2; if This indicates that the training set list has been obtained. Proceed to step 2.4; Step 2.4 In the field of incremental image classification, during testing, the test set for each stage should contain the categories of the current and previous stages. Therefore, the test set... Based on the training phase division, a test set list is obtained. , The method is: Step 2.4.1 Let the training phase t = 1, Step 2.4.2 Find the test set China belongs to Images (i.e., based on the test set) Find the tags included. (same image), add these images to middle; Step 2.4.3 If ,make Proceed to step 2.4.2; if This yields the final test set list. Proceed to step three; The third step is to initialize the training phase t=1; Fourth step, let the class label sequence of the t-th training phase be... , This is the number of categories included in the t-th training phase, and the total number of categories in the t-th phase is... n is the category label; the text feature extraction module uses a general text prompt template for... The class labels in the code correspond to the real categories, which are used to construct a general text hint list for the t-th training phase. The method is: Step 4.1 Let =1, initialize the general text prompt list for the t-th training phase. =[]; Step 4.2 Use a generic text prompt template (such as a photo of [CLASS]) to... The category label n in the text corresponds to the actual category, which is used to construct a general text hint. (If the actual category is cat, the text prompt would be "a photo of cat") Add to ; Step 4.3 If ,make Proceed to step 4.2; if This describes the general text hint list for all categories in the t-th training phase. Once the construction is complete, the general text prompt list for the t-th training phase is obtained. Proceed to step five.

[0024] Fifth step, using the training set The image feature extraction module and the text feature extraction module are trained. The method is as follows: Step 5.1 Initialize the model parameters of the image feature extraction module and the text feature extraction module. Initialize and freeze the parameters of the Transformer framework in the text feature extraction module using the pre-trained CLIP model's text encoder; initialize and freeze the parameters of the Transformer framework in the image feature extraction module using the pre-trained CLIP model's image encoder. Step 5.2 Randomly initialize the image projection layer for the t-th training phase. and text projection layer ,Will The projection layer sequence added to the image feature extraction module will Add a projection layer sequence to the text feature extraction module so that there is an image projection layer sequence in the t-th training stage. and text projection layer sequence The image projection layer sequence of the t-th training stage is concatenated with the frozen image encoder, and the text projection layer sequence of the t-th training stage is concatenated with the frozen text encoder. This ensures that the features extracted by the image encoder and text encoder contain stage task information, and that only the image projection layer sequence in the t-th training stage contains the stage task information. Trainable, only in the text projection layer sequence Trainable, the remaining projection layers remain frozen; Step 5.3 Set training parameters. Set the initial learning rate. The learning rate is set to 0.05, and the learning rate adjustment function is set to cosine annealing dynamic learning rate adjustment function. Stochastic gradient descent (SGD) is selected as the model training optimizer. The batch size for network training is set to 64. The maximum training step size is set to 10. Step 5.4 The text feature extraction module uses text feature extraction methods to extract the general text prompt list for the t-th stage. The general text features are obtained from the general text features. (It is a two-dimensional matrix with dimension 1) It is the feature dimension, usually 512), Send it to the contrastive learning classification module, specifically by: the text encoder... Text features are extracted from all common text prompts, and the resulting preliminary text features are sent to the image projection layer sequence. . The initial text features are projected in parallel to obtain t projected text features containing the corresponding task information. A second adder adds these t projected text features to obtain the final text features, which are the general text features for the current training phase. (It is a two-dimensional matrix with dimension 1) ),Will Send to the comparative learning classification module.

[0025] Step 5.5 Set the training epoch number to 1; Step 5.6 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , , (like no When the image is an integer multiple of the last batch, the last batch of images is... Remove from middle - All images remaining after one batch); Step 5.7 Image feature extraction module from Extract the training set for stage t. ,from Read the first batch The image will be the first one. batch The images are denoted in matrix form. The image is denoted by H, where H represents the height of the input image, W represents the width of the input image, and "3" represents the three RGB channels of the image. Step 5.8 The image feature extraction module uses image feature extraction methods to extract... The image features are used to obtain the first image features. (It is a two-dimensional matrix with dimension 1) ),Will Sending it to the contrastive learning classification module, specifically by: the image encoder... Feature extraction is performed on the image to obtain preliminary image features, which are then sent to the image projection layer sequence. . The initial image features are projected in parallel to obtain t projected image features containing corresponding task information. A second adder adds these t projected image features to obtain the final image features. (It is a two-dimensional matrix with dimension 1) ),Will Send to the comparative learning classification module.

[0026] Step 5.9 The contrastive learning classification module receives data from the image feature extraction module. Receive from the text feature extraction module ; Step 5.10 Comparative learning classification module calculates the first classification loss according to formula (1). : , formula (1).

[0027] Here i represents No. batch The image number, express No. batch The true label of the i-th image in a set of images. express No. batch Image features of the i-th image in a set of images. The label of the i-th image Textual features, Indicates calculation and cosine similarity, Represents the temperature coefficient (typically taken as 0.1). Step 5.11 The contrastive learning classification module is based on The trainable parameters in the image feature extraction module and the text feature extraction module are optimized using the gradient backpropagation method. Step 5.12: Order ,like Proceed to step 5.7; if This indicates the end of this round of training; proceed to step 5.13. Step 5.13: Order If epoch ≤ maxepoch, proceed to step 5.6; if epoch > maxepoch, it means that the training of the image feature extraction module and the text feature extraction module has ended, and the trained image feature extraction module and text feature extraction module are obtained. Freeze the image feature extraction module and text feature extraction module (i.e. the parameters will not be updated in subsequent training), and proceed to step 6. Step 6: Perform preliminary training on the image-text dual-guided classifier: the trained image feature extraction module extracts... Based on the image features, the prototype sequences of all categories in the t-th training stage are obtained. The weights of the image-text dual-guided classifier in the t-th training phase are obtained using the prototype sequence. The trained image feature extraction module uses image feature extraction methods to extract... The Middle batch One image The image features in the image are used to obtain the second image features. Image and text dual-guided classifier utilizes Calculate the second classification loss and based on The trainable parameters in the image-text dual-guided classifier are optimized using the gradient backpropagation method to obtain the initially trained image-text dual-guided classifier. The specific method is as follows: Step 6.1 The trained image feature extraction module extracts features from the training set list. Retrieve the training set for the t-th training phase ; Step 6.2 The trained image feature extraction module starts from... The prototype sequence of all classes in the t-th training phase is obtained from the following: , It is the prototype of the k-th category. And store the prototype sequences of all categories in the t-th training phase into the total sequence P in the t-th phase. The method is as follows: Step 6.2.1 Let the index of the first category in the t-th training phase be... ; Step 6.2.2 From Read the image of the k-th category and represent all images of the k-th category in matrix form. ; Step 6.2.3 Image Feature Extraction Module Extraction Image features ; Step 6.2.4 Calculate the average to obtain the prototype of the k-th category. ,Will Place it into the prototype sequence of all categories in the t-th training phase.

[0028] Step 6.2.5 ,like Proceed to step 6.2.2, if This indicates that the prototype sequences of all categories in the t-th training phase have been obtained. Proceed to step 6.3; Step 6.3 Utilizing prototype sequences The weights of the image-text dual-guided classifier in the t-th training phase :like , As in formula (2): , formula (2).

[0029] like , use The final image-text dual-guided classifier weights in the (t-1)th training phase The result is obtained by connecting the two formulas, as shown in formula (3): , formula (3).

[0030] Step 6.4 Set training parameters. Set the initial learning rate. The learning rate is set to 0.0005, and the learning rate adjustment function is set to the cosine annealing dynamic learning rate adjustment function. Adaptive moment estimation (Adam) is selected as the model training optimizer. The batch size for network training is set to 64. The maximum training step size (maxepoch) is set to 3. Step 6.5 Set the number of training rounds. ; Step 6.6 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , , (like no When the image is an integer multiple of the last batch, the last batch of images is... Remove from middle - All images remaining after one batch); Step 6.7 From Read the first batch The image will be the first one. batch The images are denoted in matrix form. ; Step 6.8 The trained image feature extraction module extracts features using the image feature extraction method described in step 5.8. The image features in the image are used to obtain the second image features. (It is a two-dimensional matrix with dimension 1) Since the image feature extraction module has already been trained at this point, it is more efficient than in step 5.8. The image features here encode the relevant image information of the t-th training phase. Send the data to the contrastive learning classification module, the image-text dual-guided classifier, and the feature fusion module; Step 6.9 Calculate the second classification loss using the image-text dual-guided classifier. As shown in formula (4): , formula (4); express Image features of the i-th image. This indicates that the i-th image belongs to the label. The predicted probability; Step 6.10 Image-text dual-guided classifier based on The gradient backpropagation method is used to optimize the trainable parameters in the image-text dual-guided classifier; Step 6.11 ,like Proceed to step 6.7; if This indicates the end of this round of training; proceed to step 6.12. Step 6.12 If epoch ≤ maxepoch, proceed to step 6.6; if epoch > maxepoch, it means that the initial training of the image-text dual-guided classifier is complete, and the initially trained image-text dual-guided classifier is obtained. Proceed to step 7. Step 7: The trained text feature extraction module uses text feature extraction methods to extract... The second general text feature is obtained from the general text features in the middle. The comparative learning classification module is based on and Calculate the text classification prediction probability at the t-th training stage. The image-text dual-guided classifier is based on... Calculate image classification prediction probability The classification result merging module will... and The probabilities are merged to obtain the merged classification prediction probabilities. ; Compare the text generation module to The classification prediction probabilities of each image are sorted, and the top-N predicted labels corresponding to the highest probabilities are obtained. A large language model is used to generate corresponding comparative text prompts, which are then passed to the text extraction module. The text feature extraction module extracts... Chinese text features are used to obtain contrast text features, which will then be used to... Send it to the comparison text generation module by: Step 7.1 Set training parameters. Set the initial learning rate. for The learning rate adjustment function is set to cosine annealing dynamic learning rate adjustment function. Adaptive moment estimation (Adam) is selected as the model training optimizer. The batch size for network training is set to 64. The maximum training step size (maxepoch) is set to 3. Step 7.2 Set the number of training rounds ; Step 7.3 From the training set list Retrieve the training set for the t-th training phase ; Step 7.4 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , ; Step 7.5 Let the first Loss per batch of images ; Step 7.6 The text feature extraction module extracts text features using the text feature extraction method described in step 5.4. The second general text feature is obtained from the general text features in the middle. (The text feature extraction module is trained; the general text features here encode the relevant text information of the current training phase.) The training set images are sent to the contrastive learning classification module and the feature mixing module, while the real labels corresponding to the training set images are sent to the contrastive text generation module and the feature mixing module. Step 7.7 The contrastive learning classification module receives the second image features. (It is a two-dimensional matrix with dimension 1) ),Will Send to the classification result merging module: , formula (5); Indicates to Perform normalization. This is a temperature parameter, typically set to 0.1. Step 7.8 The image-text dual-guided classifier receives the second image features from the image feature extraction module. The image classification prediction probability is calculated according to formula (6). (It is a two-dimensional matrix with dimension 1) ),Will Send to the classification result merging module: , formula (6); It represents the weights of the image-text dual-guided classifier in the t-th training phase; Step 7.9 The classification result merging module... and Add them together to get the combined classification probabilities. (It is a two-dimensional matrix with dimension 1) Logit is sent to the comparison text generation module and the entropy-guided loss weighting module; Step 7.10: Order the b batch... Image number in the image ; Step 7.11 The comparison text generation module receives the classification result merging module's output. ,right The predicted probabilities of each image are sorted, and the top-N highest predicted probabilities are selected to obtain the b-th batch. Top-N predicted labels for each image (It is a two-dimensional matrix with dimension 1) Each row represents the predicted probability of the top-N predicted labels for the corresponding image, where N is a positive integer, typically 5. , Let be the predicted label ranked 'a' in terms of predicted probability, where the value of 'a' ranges from [1, 2, ..., N]. Step 7.12: The text generation module receives the ground truth label corresponding to the m-th image in the training set from the text feature extraction module. ,from Obtain the top-N predicted labels for the m-th image. ; Step 7.13 Design the corresponding question template: "What are the unique visual features of [CLASS]?" i Compared to [CLASS] j in a photo? Focus on their key visual differences."; Step 7.14 Use Replace [CLASS] in the question template. i ,use The elements in [CLASS] are replaced sequentially. j N corresponding questions were obtained. ,Will The list of questions to be combined into the m-th image ,in This refers to the a-th corresponding question; Step 7.15 Compare the large language model in the text generation module with the question list. Answer the question to obtain a list of comparison text prompts for the m-th image. The list of comparison text prompts will be used. Send to the text feature extraction module; Step 7.16 The text feature extraction module receives the list of comparison text prompts. ,make ,make This serves as a list of stored prompt features for comparison with the list of text prompts.

[0031] Step 7.17 The text feature extraction module extracts from... Get the h-th comparison text prompt of the m-th image .

[0032] Step 7.18 The text feature extraction module extracts text features using the text feature extraction method described in step 5.4. Chinese text features ,Will join in .

[0033] Step 7.19 Order ,like Proceed to step 7.19; if ,right Perform vector averaging on all feature vectors to obtain the contrastive text features of the m-th image. (It is a one-dimensional vector with dimension D), Send to the comparison text generation module; Step 7.20: The contrast text generation module receives the contrast text features of the m-th image from the text feature extraction module. ,Will and the true label of the m-th image Forward to the feature mixing module.

[0034] Step 8: The feature fusion module performs a feature blending operation on the m-th image. First, perform a fusion operation on the contrast text features to obtain the fusion features of the m-th image. Then, perform a vector stack operation on the fusion features of the m-th image, the general text features, and the contrast text features to obtain the enhanced training features, which prepares for the enhanced training of the image-text dual-guided classifier. Step 8.1 The feature fusion module receives the ground truth label of the m-th image from the contrastive text generation module. and Received from the text feature extraction module Receive from image feature extraction module ,from Extract the general text features corresponding to the true label of the m-th image. Extract the image features of the m-th image .

[0035] Step 8.2 The feature blending module will and Mixing is performed to obtain mixed characteristics. As shown in formula (7): , formula (7); in To compare the mixing ratio of text features and image features, it is generally set to 0.75; Step 8.3 The feature blending module will Perform vector stack operations to form the first... Enhanced training features of the m-th image in the batch (It is a two-dimensional matrix with dimension 1) The CCP's three enhanced features) will With the true label of the m-th image Send it to the entropy-guided loss weighting module.

[0036] Step 9: The entropy-guided loss weighting module receives data from the feature mixing module. and It receives the merged classification probabilities Logit from the classification result merging module and uses them... Enhanced training is performed on the image-text dual-guided classifier, and high-entropy samples in the current class incremental image classification system are identified. The loss of high-entropy samples is weighted based on the entropy value of the samples. The method is as follows: Step 9.1 The entropy-guided loss weighting module receives Logit from the classification result merging module; Step 9.2 Normalize the Logit using the softmax function to obtain the normalized probability distribution Prob (a two-dimensional matrix with dimension 1). ); Step 9.3 Obtain the predicted probability distribution of the m-th image from the probability distribution Prob. (is a one-dimensional vector with dimension ) Find the entropy value of the m-th image. The calculation formula is as shown in formula (8): , formula (8); in The predicted probability distribution from the m-th image The predicted probability of the j-th label is extracted; Step 9.4 The entropy-guided loss weighting module receives from the feature mixing module ; Step 9.5 If ( This is the threshold value for the normalized entropy (usually set to 0.7), then let... Weights corresponding to the loss ;like ; Step 9.6 Let the enhancement loss value of the m-th image be... ; Step 9.7 Enhanced numbering ; Step 9.8 From Extract the s-th enhanced training feature. (It is a one-dimensional vector with dimension D), which is sent to the image-text dual-guided classifier; Step 9.9 Image and text dual-guided classifier reception And calculate the enhanced training features. Corresponding image classification prediction probability ,Will Send it to the entropy-guided loss weighting module.

[0037] Step 9.10 Entropy-guided loss weighting module receives ,from The enhanced features of image m obtained from the label belong to the label. Predicted probability Thus, the corresponding loss value can be obtained. As in formula (9): , formula (9); Step 9.11 Order ; Step 9.12 Order ,like Proceed to step 9.8, if This indicates that the loss calculation for all enhancement features of image m is complete, and the final enhancement loss value of the m-th image is obtained. Proceed to step 9.13.

[0038] Step 9.13 Set the final loss value ; Step 9.14 Order ,like Proceed to step 7.11, if Explanation of the first batch Once the image loss calculation is complete, proceed to step 9.15.

[0039] Step 9.15 The entropy-guided loss weighting module calculates the final loss value. Send to the image-text dual-guided classifier; Step 9.16 Image-text dual-guided classifier based on The gradient backpropagation method is used to optimize the trainable parameters in the image-text dual-guided classifier; Step 9.17 ,like Proceed to step 7.5; if This indicates the end of this round of training; proceed to step 9.18. Step 9.18 Order If epoch ≤ maxepoch, proceed to step 7.3; if epoch > maxepoch, it means that the image-text dual-guided classifier enhancement training has ended, and the final image-text dual-guided classifier parameters for the t-th training stage are obtained, then proceed to step 10. Step 10, Order ,like Proceed to step four; if This indicates that all T training phases have been completed, and the trained incremental image classification system has been obtained. Proceed to step eleven. Step 11: The trained incremental image classification system is used to classify the user-input image to obtain the predicted image result. The method is as follows: Step 11.1 The image feature extraction module receives the image to be classified input by the user. ; Step 11.2 The image feature extraction module extracts features using the image feature extraction method described in step 5.8. Image features in the image are obtained Image features (It is a two-dimensional matrix with dimension 1) ),Will Send to the image-text dual-guided classifier and the contrastive learning classification module; Step 11.3 Image and text dual-guided classifier receiving ,according to Calculate the image to be classified Image classification prediction probability ,Will Send to the classification result merging module: Step 11.4 The text feature extraction module uses the text feature extraction method described in step 5.4 to extract the general text features corresponding to all the general text prompts that have been classified and labeled during training. ,Will Send to the contrastive learning classification module; generate a general text prompt for each label, where the general text for all classified labels refers to the general text prompts generated for all classes encountered during training.

[0040] Step 11.5 Receive data from the contrastive learning classification module. and ,according to Calculate the image to be classified Text classification prediction probability ,Will Send to the classification result merging module: Step 11.6 Classification result merging module receives and Calculate according to formula (10) Predicted value , , formula (10); in The value is typically set to 0.6, which controls the output ratio of the image-text dual-guided classifier and the contrastive learning classification module. The category corresponding to the maximum value is the predicted result of the image to be classified, and the image classification is completed.

[0041] The following beneficial effects can be achieved by using the present invention: 1. This invention utilizes a pre-trained CLIP model to construct a cross-modal semantic space and strong generalization capabilities. The fifth step introduces a projection layer for each training stage, effectively learning new knowledge while retaining as much knowledge as possible from previous tasks, thus preventing catastrophic forgetting. Steps six through ten utilize image features to perform preliminary and augmented training on the image-text dual-guided classifier, obtaining low-entropy classifier prediction probabilities. Finally, the text classification prediction probability of the contrastive learning classification module is combined with the image classification prediction probability of the image-text dual-guided classifier. This effectively reduces the entropy value of the original contrastive learning classification module's prediction probability while preserving its generalization ability, thereby increasing the classification confidence and improving the accuracy of incremental image classification. 2. This invention constructs an incremental image classification system comprising a text feature extraction module, an image feature extraction module, a contrastive learning classification module, a contrastive text generation module, an image-text dual-guided classifier module, a feature mixing module, and an entropy-guided loss weighting module. During training, the system utilizes the contrastive text generation module to provide contrastive text discrimination information between difficult-to-classify labels, and strengthens the loss weights of high-entropy images through an entropy-guided loss weighting mechanism, thereby enhancing the learning of high-entropy samples and improving the system's ability to discriminate difficult samples (high-entropy samples). Experiments on multiple datasets such as CIFAR100, ImageNet-R, and CUB200 demonstrate that the classification accuracy of this invention in incremental image classification scenarios is significantly superior to existing CLIP-based incremental learning methods, effectively improving the classification accuracy of this invention in incremental image classification scenarios. Attached Figure Description

[0042] Figure 1 This is a logical structure diagram of the incremental image classification system constructed in the first step of this invention; Figure 2 This is the overall flowchart of the present invention. Detailed Implementation

[0043] Specific examples of the present invention will be described below with reference to the accompanying drawings. Figure 2 As shown, the present invention includes the following steps: The first step is to construct an incremental image classification system based on an extension of a dual-guided image-text classifier. For example... Figure 1 As shown, the incremental image classification system consists of a text feature extraction module, an image feature extraction module, a contrastive learning classification module, an image-text dual-guided classifier module, a classification result merging module, a contrastive text generation module, a feature mixing module, and an entropy-guided loss weighting module.

[0044] The text feature extraction module is connected to the contrastive learning classification module, the feature mixing module, and the contrastive text generation module. The text feature extraction module constructs general text prompts using labels from the training set and extracts general text features from these prompts. The text feature extraction module consists of a text encoder, T text projection layers, and a first adder, where T is the number of training stages in the incremental image classification training process, and T is a positive integer, typically 10. The text encoder parameters are pre-trained using CLIP on a Transformer architecture; these parameters are frozen during training and do not participate in the training process. The text encoder extracts features from the general text prompts, obtaining preliminary text features which are then sent to the T text projection layers. All T text projection layers are fully connected layers that project the preliminary text features received from the text encoder in parallel, obtaining T projected text features containing corresponding task information. The first adder adds the T projected text features to obtain the final text feature. This method effectively includes feature information from all previous tasks, making it better suited for incremental image classification tasks. During the training of the text feature extraction module, the text feature extraction module extracts the first general text features from the general text prompts corresponding to all labels in the training set and sends them to the contrast learning classification module. During the training of the image-text dual-guided classifier module, the trained text feature extraction module re-extracts the second general text features from the general text prompts and sends them to the contrast learning classification module and the feature mixing module. It also receives the list of contrast text prompts sent by the contrast text generation module, extracts the contrast text features of all contrast text prompts in the list, and sends the real labels and contrast text features corresponding to the images in the training set to the contrast text generation module.

[0045] The image feature extraction module is connected to the contrastive learning classification module, the image-text dual-guided classifier, and the feature mixing module. The image feature extraction module extracts features from the images in the training set to obtain image features. The image feature module consists of an image encoder, T image projection layers, and a second adder. The image encoder parameters are pre-trained using CLIP on a Transformer architecture; these parameters are frozen during training and do not participate in the training process. The image encoder extracts features from the images in the training set, obtaining preliminary image features which are then sent to the T image projection layers. All T image projection layers are fully connected layers, projecting the preliminary image features obtained from the image encoder in parallel to obtain T projected image features containing corresponding task information. The second adder adds the T projected image features to obtain the final image features. During training, the image feature extraction module sends the first image features extracted from the training set images to the contrastive learning classification module. During training of the image-text dual-guided classifier module, the trained image feature extraction module re-extracts image features from the training set images and sends the resulting second image features to the contrastive learning classification module, the image-text dual-guided classifier, and the feature mixing module.

[0046] The contrastive learning classification module is connected to the text feature extraction module, the image feature extraction module, and the classification result merging module. The contrastive learning classification module uses the general text features received from the text feature extraction module and the image features received from the image feature extraction module to calculate the text classification prediction probability, and sends the text classification prediction probability to the classification result merging module.

[0047] The image-text dual-guided classifier is connected to the image feature extraction module, the classification result merging module, and the entropy-guided loss weighting module. The image-text dual-guided classifier consists of a single fully connected layer that ignores the bias term. It predicts the image features received from the image feature extraction module to obtain the image classification prediction probability, and sends the image classification prediction probability to the classification result merging module. At the same time, the image-text dual-guided classifier predicts the enhanced training features sent by the entropy-guided loss weighting module, and sends the image classification prediction probability corresponding to the enhanced training features to the entropy-guided loss weighting module.

[0048] The classification result merging module is connected to the contrastive learning classification module, the image-text dual-guided classifier, the contrastive text generation module, and the entropy-guided loss weighting module. It merges the text classification prediction probabilities received from the contrastive learning classification module and the image classification prediction probabilities received from the image-text dual-guided classifier to obtain the merged classification prediction probabilities, and sends the merged classification prediction probabilities to the contrastive text generation module and the entropy-guided loss weighting module.

[0049] The contrast text generation module is connected to the classification result merging module, feature mixing module, and text feature extraction module. The contrast text generation module prepares for the augmented training of the image-text dual-guided classifier. This module receives the merged classification prediction probabilities from the classification result merging module, selects the Top-N results from the merged probabilities to obtain the Top-N predicted labels for the training set images, and simultaneously receives the ground truth labels corresponding to the images in the training set from the text feature extraction module. It then uses a large language model (such as GPT-4 or ChatGLM) to perform a comparative description of the Top-N predicted labels and ground truth labels, obtaining a contrast text prompt list. This list is sent to the text feature extraction module, which in turn receives the contrast text features extracted from the contrast text prompt list. Finally, the contrast text features and the ground truth labels corresponding to the images in the training set are sent to the feature mixing module.

[0050] The feature fusion module is connected to the text feature extraction module, image feature extraction module, contrast text generation module, and entropy-guided loss weighting module. The feature fusion module receives image features from the image feature extraction module, receives the real labels corresponding to the images in the training set and the contrast text features from the contrast text generation module, and receives general text features from the text feature extraction module. It combines the image features, general text features, and contrast text features to form an enhanced training feature set, and sends the enhanced training feature set and the real labels corresponding to the images in the training set to the entropy-guided loss weighting module.

[0051] The entropy-guided loss weighting module is connected to the feature mixing module, the classification result merging module, and the image-text dual-guided classifier. The entropy-guided loss weighting module receives the merged classification prediction probabilities from the classification result merging module and the enhanced training feature set from the feature mixing module. It calculates the entropy value based on the merged classification prediction probabilities to determine if the sample image is a high-entropy sample. Then, it forwards the enhanced training features from the enhanced training feature set to the image-text dual-guided classifier. The image-text dual-guided classifier calculates the image classification prediction probability corresponding to the enhanced training features and sends it to the entropy-guided loss weighting module. The entropy-guided loss weighting module calculates the loss for the corresponding sample, increasing the loss weight for the enhanced features of high-entropy samples during the calculation, thus obtaining the loss of the training set. This training set loss is then sent to the image-text dual-guided classifier. In this way, the incremental image classification system enhances its ability to distinguish high-entropy samples during training, increases the confidence of the classification results, and further improves the accuracy of the incremental image classification task.

[0052] All the above modules participate in the training process, but the contrastive text generation module, feature mixing module, and entropy-guided loss weighting module only participate in training and do not participate in the user input image classification process. When classifying the user input image, the text feature extraction module only needs to extract the general text prompts corresponding to all image labels used in the training set during training (the incremental image classification domain training process is divided into T training stages, and each stage only uses a portion of the image labels in the training set) to obtain the corresponding general text features. The image feature extraction module extracts the image features of the user input image. The contrastive learning classification module obtains the text classification prediction probability of the user input image. The image-text dual-guided classifier obtains the image classification prediction probability of the user input image. The classification result merging module merges the text classification prediction probability and the image classification prediction probability of the user input image to obtain the merged classification probability of the user input image, which is taken as the final classification probability of the user input image. The label corresponding to the highest probability is taken as the final classification result of the user input image.

[0053] The second step involves collecting incremental learning scene images as a dataset for incremental image classification, and then dividing the incremental learning sample image classification dataset into a training set. Test set ,right The training set list is obtained by further dividing the training phase. ,Will Based on the training phase division, a test set list is obtained. The method is: Step 2.1 Collect incremental learning scene images as a class incremental image classification scene dataset. The method is as follows: Nine datasets (totaling approximately 400,000 images) were used as incremental learning sample image classification datasets: the small image dataset CIFAR100, the general scene dataset ImageNet-R, the bird dataset CUB200, the food dataset Food101, the car dataset StanfordCars, the action dataset UCF101, the scene classification dataset SUN397, the aircraft classification dataset Aircraft, and the object recognition dataset ObjectNet. Each image in these nine datasets was manually labeled, meaning each image was marked with the true label of the object's category.

[0054] Step 2.2 Based on the original partitioning criteria in the dataset, divide the incremental learning sample image classification dataset into a training set. test set Let the total number of categories in the sample image classification dataset be C, and each image have a true label for its corresponding category (usually [1,2,…,c,…,C]). C is a positive integer.

[0055] Step 2.3 In the field of incremental image classification, the training process is divided into T training stages (T is a positive integer, typically 10) to simulate the continuous increase of data. During training, only the training data from the current stage is usually used, while a small amount of training data from previous stages is allowed to be saved. Therefore, a set partitioning method is used to partition the training set. The training set list is obtained by further dividing the training phase. , , And t is an integer, the method is: Step 2.3.1 Let the training phase t = 1; Step 2.3.2 Let the number of classes included in the t-th training phase be... (Assuming) There are 10 categories, and the incremental learning stage T is 10. The number of categories in each stage is 10. The set of class labels included in the t-th training phase. = (For example: for training phase 1, the first training phase contains the set of class labels) For training phase 2, the second training phase includes the set of category labels. ); Step 2.3.3 Find the training set China belongs to Images (i.e., finding the true labels in the training set) (same image), China belongs to Adding images to middle; Step 2.3.4 If ,make Proceed to step 2.3.2; if This indicates that the training set list has been obtained. Proceed to step 2.4; Step 2.4 In the field of incremental image classification, during testing, the test set for each stage should contain the categories of the current and previous stages. Therefore, the test set... Based on the training phase division, a test set list is obtained. , The method is: Step 2.4.1 Let the training phase t = 1, Step 2.4.2 Find the test set China belongs to Images (i.e., based on the test set) Find the tags included. (same image), add these images to middle; Step 2.4.3 If ,make Proceed to step 2.4.2; if This yields the final test set list. Proceed to step three; The third step is to initialize the training phase t=1; Fourth step, let the class label sequence of the t-th training phase be... , This is the number of categories included in the t-th training phase, and the total number of categories in the t-th phase is... n is the category label; the text feature extraction module uses a general text prompt template for... The class labels in the code correspond to the real categories, which are used to construct a general text hint list for the t-th training phase. The method is: Step 4.1 Let =1, initialize the general text prompt list for the t-th training phase. =[]; Step 4.2 Use a generic text prompt template (such as a photo of [CLASS]) to... The category label n in the text corresponds to the actual category, which is used to construct a general text hint. (If the actual category is cat, the text prompt would be "a photo of cat") Add to ; Step 4.3 If ,make Proceed to step 4.2; if This describes the general text hint list for all categories in the t-th training phase. Once the construction is complete, the general text prompt list for the t-th training phase is obtained. Proceed to step five.

[0056] Fifth step, using the training set The image feature extraction module and the text feature extraction module are trained. The method is as follows: Step 5.1 Initialize the model parameters of the image feature extraction module and the text feature extraction module. Initialize and freeze the parameters of the Transformer framework in the text feature extraction module using the pre-trained CLIP model's text encoder; initialize and freeze the parameters of the Transformer framework in the image feature extraction module using the pre-trained CLIP model's image encoder. Step 5.2 Randomly initialize the image projection layer for the t-th training phase. and text projection layer ,Will The projection layer sequence added to the image feature extraction module will Add a projection layer sequence to the text feature extraction module so that there is an image projection layer sequence in the t-th training stage. and text projection layer sequence The image projection layer sequence of the t-th training stage is concatenated with the frozen image encoder, and the text projection layer sequence of the t-th training stage is concatenated with the frozen text encoder. This ensures that the features extracted by the image encoder and text encoder contain stage task information, and that only the image projection layer sequence in the t-th training stage contains the stage task information. Trainable, only in the text projection layer sequence Trainable, the remaining projection layers remain frozen; Step 5.3 Set training parameters. Set the initial learning rate. The learning rate is set to 0.05, and the learning rate adjustment function is set to cosine annealing dynamic learning rate adjustment function. Stochastic gradient descent (SGD) is selected as the model training optimizer. The batch size for network training is set to 64. The maximum training step size is set to 10. Step 5.4 The text feature extraction module uses text feature extraction methods to extract the general text prompt list for the t-th stage. The general text features are obtained from the general text features. (It is a two-dimensional matrix with dimension 1) It is the feature dimension, usually 512), Send it to the contrastive learning classification module, specifically by: the text encoder... Text features are extracted from all common text prompts, and the resulting preliminary text features are sent to the image projection layer sequence. . The initial text features are projected in parallel to obtain t projected text features containing the corresponding task information. A second adder adds these t projected text features to obtain the final text features, which are the general text features for the current training phase. (It is a two-dimensional matrix with dimension 1) ),Will Send to the comparative learning classification module.

[0057] Step 5.5 Set the training epoch number to 1; Step 5.6 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , , (like no When the image is an integer multiple of the last batch, the last batch of images is... Remove from middle - All images remaining after one batch); Step 5.7 Image feature extraction module from Extract the training set for stage t. ,from Read the first batch The image will be the first one. batch The images are denoted in matrix form. The image is denoted by H, where H represents the height of the input image, W represents the width of the input image, and "3" represents the three RGB channels of the image. Step 5.8 The image feature extraction module uses image feature extraction methods to extract... The image features are used to obtain the first image features. (It is a two-dimensional matrix with dimension 1) ),Will Sending it to the contrastive learning classification module, specifically by: the image encoder... Feature extraction is performed on the image to obtain preliminary image features, which are then sent to the image projection layer sequence. . The initial image features are projected in parallel to obtain t projected image features containing corresponding task information. A second adder adds these t projected image features to obtain the final image features. (It is a two-dimensional matrix with dimension 1) ),Will Send to the comparative learning classification module.

[0058] Step 5.9 The contrastive learning classification module receives data from the image feature extraction module. Receive from the text feature extraction module ; Step 5.10 Comparative learning classification module calculates the first classification loss according to formula (1). : , formula (1); Here i represents No. batch The image number, express No. batch The true label of the i-th image in a set of images. express No. batch Image features of the i-th image in a set of images. The label of the i-th image Textual features, Indicates calculation and cosine similarity, Represents the temperature coefficient (typically taken as 0.1). Step 5.11 The contrastive learning classification module is based on The trainable parameters in the image feature extraction module and the text feature extraction module are optimized using the gradient backpropagation method. Step 5.12: Order ,like Proceed to step 5.7; if This indicates the end of this round of training; proceed to step 5.13. Step 5.13: Order If epoch ≤ maxepoch, proceed to step 5.6; if epoch > maxepoch, it means that the training of the image feature extraction module and the text feature extraction module has ended, and the trained image feature extraction module and text feature extraction module are obtained. Freeze the image feature extraction module and text feature extraction module (i.e. the parameters will not be updated in subsequent training), and proceed to step 6. Step 6: Perform preliminary training on the image-text dual-guided classifier: the trained image feature extraction module extracts... Based on the image features, the prototype sequences of all categories in the t-th training stage are obtained. The weights of the image-text dual-guided classifier in the t-th training phase are obtained using the prototype sequence. The trained image feature extraction module uses image feature extraction methods to extract... The Middle batch One image The image features in the image are used to obtain the second image features. Image and text dual-guided classifier utilizes Calculate the second classification loss and based on The trainable parameters in the image-text dual-guided classifier are optimized using the gradient backpropagation method to obtain the initially trained image-text dual-guided classifier. The specific method is as follows: Step 6.1 The trained image feature extraction module extracts features from the training set list. Retrieve the training set for the t-th training phase ; Step 6.2 The trained image feature extraction module starts from... The prototype sequence of all classes in the t-th training phase is obtained from the following: , It is the prototype of the k-th category. And store the prototype sequences of all categories in the t-th training phase into the total sequence P in the t-th phase. The method is as follows: Step 6.2.1 Let the index of the first category in the t-th training phase be... ; Step 6.2.2 From Read the image of the k-th category and represent all images of the k-th category in matrix form. ; Step 6.2.3 Image Feature Extraction Module Extraction Image features ; Step 6.2.4 Calculate the average to obtain the prototype of the k-th category. ,Will Place it into the prototype sequence of all categories in the t-th training phase.

[0059] Step 6.2.5 ,like Proceed to step 6.2.2, if This indicates that the prototype sequences of all categories in the t-th training phase have been obtained. Proceed to step 6.3; Step 6.3 Utilizing prototype sequences The weights of the image-text dual-guided classifier in the t-th training phase , As in formula (2): , formula (2); like , use The final image-text dual-guided classifier weights in the (t-1)th training phase The result is obtained by connecting the two formulas, as shown in formula (3): , formula (3).

[0060] Step 6.4 Set training parameters. Set the initial learning rate. The learning rate is set to 0.0005, and the learning rate adjustment function is set to the cosine annealing dynamic learning rate adjustment function. Adaptive moment estimation (Adam) is selected as the model training optimizer. The batch size for network training is set to 64. The maximum training step size (maxepoch) is set to 3. Step 6.5 Set the number of training rounds. ; Step 6.6 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , , (like no When the image is an integer multiple of the last batch, the last batch of images is... Remove from middle - All images remaining after one batch); Step 6.7 From Read the first batch The image will be the first one. batch The images are denoted in matrix form. ; Step 6.8 The trained image feature extraction module extracts features using the image feature extraction method described in step 5.8. The image features in the image are used to obtain the second image features. (It is a two-dimensional matrix with dimension 1) Since the image feature extraction module has already been trained at this point, it is more efficient than in step 5.8. The image features here encode the relevant image information of the t-th training phase. Send the data to the contrastive learning classification module, the image-text dual-guided classifier, and the feature fusion module; Step 6.9 Calculate the second classification loss using the image-text dual-guided classifier. As shown in formula (4): , formula (4); express Image features of the i-th image. This indicates that the i-th image belongs to the label. The predicted probability; Step 6.10 Image-text dual-guided classifier based on The gradient backpropagation method is used to optimize the trainable parameters in the image-text dual-guided classifier; Step 6.11 ,like Proceed to step 6.7; if This indicates the end of this round of training; proceed to step 6.12. Step 6.12 If epoch ≤ maxepoch, proceed to step 6.6; if epoch > maxepoch, it means that the initial training of the image-text dual-guided classifier is complete, and the initially trained image-text dual-guided classifier is obtained. Proceed to step 7. Step 7: The trained text feature extraction module uses text feature extraction methods to extract... The second general text feature is obtained from the general text features in the middle. The comparative learning classification module is based on and Calculate the text classification prediction probability at the t-th training stage. The image-text dual-guided classifier is based on... Calculate image classification prediction probability The classification result merging module will... and The probabilities are merged to obtain the merged classification prediction probabilities. ; Compare the text generation module to The classification prediction probabilities of each image are sorted, and the top-N predicted labels corresponding to the highest probabilities are obtained. A large language model is used to generate corresponding comparative text prompts, which are then passed to the text extraction module. The text feature extraction module extracts... Chinese text features are used to obtain contrast text features, which will then be used to... Send it to the comparison text generation module by: Step 7.1 Set training parameters. Set the initial learning rate. for The learning rate adjustment function is set to a cosine annealing dynamic learning rate adjustment function. Adaptive Moment Estimation (Adam) is selected as the model training optimizer. The batch size for network training is set to 64. The maximum training step size (maxepoch) is set to 3. Step 7.2 Set the number of training rounds ; Step 7.3 From the training set list Retrieve the training set for the t-th training phase ; Step 7.4 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , ; Step 7.5 Let the first Loss per batch of images ; Step 7.6 The text feature extraction module extracts text features using the text feature extraction method described in step 5.4. The second general text feature is obtained from the general text features in the middle. (The text feature extraction module is trained; the general text features here encode the relevant text information of the current training phase.) The training set images are sent to the contrastive learning classification module and the feature mixing module, while the real labels corresponding to the training set images are sent to the contrastive text generation module and the feature mixing module. Step 7.7 The contrastive learning classification module receives the second image features. (It is a two-dimensional matrix with dimension 1) ),Will Send to the classification result merging module: , formula (5); Indicates to Perform normalization. This is a temperature parameter, typically set to 0.1. Step 7.8 The image-text dual-guided classifier receives the second image features from the image feature extraction module. The image classification prediction probability is calculated according to formula (6). (It is a two-dimensional matrix with dimension 1) ),Will Send to the classification result merging module: , formula (6); It represents the weights of the image-text dual-guided classifier in the t-th training phase; Step 7.9 The classification result merging module... and Add them together to get the combined classification probabilities. (It is a two-dimensional matrix with dimension 1) Logit is sent to the comparison text generation module and the entropy-guided loss weighting module; Step 7.10: Order the b batch... Image number in the image ; Step 7.11 The comparison text generation module receives the classification result merging module's output. ,right The predicted probabilities of each image are sorted, and the top-N highest predicted probabilities are selected to obtain the b-th batch. Top-N predicted labels for each image (It is a two-dimensional matrix with dimension 1) Each row represents the predicted probability of the top-N predicted labels for the corresponding image, where N is a positive integer, typically 5. , Let be the predicted label ranked 'a' in terms of predicted probability, where the value of 'a' ranges from [1, 2, ..., N]. Step 7.12: The text generation module receives the ground truth label corresponding to the m-th image in the training set from the text feature extraction module. ,from Obtain the top-N predicted labels for the m-th image. ; Step 7.13 Design the corresponding question template: "What are the unique visual features of [CLASS]?" i Compared to [CLASS] j in a photo? Focus on their key visual differences."; Step 7.14 Use Replace [CLASS] in the question template. i ,use The elements in [CLASS] are replaced sequentially. j N corresponding questions were obtained. ,Will The list of questions to be combined into the m-th image ,in This refers to the a-th corresponding question; Step 7.15 Compare the large language model in the text generation module with the question list. Answer the question to obtain a list of comparison text prompts for the m-th image. The list of comparison text prompts will be used. Send to the text feature extraction module; Step 7.16 The text feature extraction module receives the list of comparison text prompts. ,make ,make This serves as a list of stored prompt features for comparison with the list of text prompts.

[0061] Step 7.17 The text feature extraction module extracts from... Get the h-th comparison text prompt of the m-th image .

[0062] Step 7.18 The text feature extraction module extracts text features using the text feature extraction method described in step 5.4. Chinese text features ,Will join in .

[0063] Step 7.19 Order ,like Proceed to step 7.19; if ,right Perform vector averaging on all feature vectors to obtain the contrastive text features of the m-th image. (It is a one-dimensional vector with dimension D), Send to the comparison text generation module; Step 7.20: The contrast text generation module receives the contrast text features of the m-th image from the text feature extraction module. ,Will and the true label of the m-th image Forward to the feature mixing module.

[0064] Step 8: The feature fusion module performs a feature blending operation on the m-th image. First, perform a fusion operation on the contrast text features to obtain the fusion features of the m-th image. Then, perform a vector stack operation on the fusion features of the m-th image, the general text features, and the contrast text features to obtain the enhanced training features, which prepares for the enhanced training of the image-text dual-guided classifier. Step 8.1 The feature fusion module receives the ground truth label of the m-th image from the contrastive text generation module. and Received from the text feature extraction module Receive from image feature extraction module ,from Extract the general text features corresponding to the true label of the m-th image. Extract the image features of the m-th image .

[0065] Step 8.2 The feature blending module will and Mixing is performed to obtain mixed characteristics. As shown in formula (7): , formula (7); in To compare the mixing ratio of text features and image features, it is generally set to 0.75; Step 8.3 The feature blending module will Perform vector stack operations to form the first... Enhanced training features of the m-th image in the batch (It is a two-dimensional matrix with dimension 1) The CCP's three enhanced features) will With the true label of the m-th image Send it to the entropy-guided loss weighting module.

[0066] Step 9: The entropy-guided loss weighting module receives data from the feature mixing module. and It receives the merged classification probabilities Logit from the classification result merging module and uses them... Enhanced training is performed on the image-text dual-guided classifier, and high-entropy samples in the current class incremental image classification system are identified. The loss of high-entropy samples is weighted based on the entropy value of the samples. The method is as follows: Step 9.1 The entropy-guided loss weighting module receives Logit from the classification result merging module. ; Step 9.2 Normalize the Logit using the softmax function to obtain the normalized probability distribution Prob (a two-dimensional matrix with dimension 1). ); Step 9.3 Obtain the predicted probability distribution of the m-th image from the probability distribution Prob. (is a one-dimensional vector with dimension ) Find the entropy value of the m-th image. The calculation formula is as shown in formula (8): , formula (8); in The predicted probability distribution from the m-th image The predicted probability of the j-th label is extracted; Step 9.4 The entropy-guided loss weighting module receives from the feature mixing module ; Step 9.5 If ( This is the threshold value for the normalized entropy (usually set to 0.7), then let... Weights corresponding to the loss ;like ; Step 9.6 Let the enhancement loss value of the m-th image be... ; Step 9.7 Enhanced numbering ; Step 9.8 From Extract the s-th enhanced training feature. (It is a one-dimensional vector with dimension D), which is sent to the image-text dual-guided classifier; Step 9.9 Image and text dual-guided classifier reception And calculate the enhanced training features. Corresponding image classification prediction probability ,Will Send it to the entropy-guided loss weighting module.

[0067] Step 9.10 Entropy-guided loss weighting module receives ,from The enhanced features of image m obtained from the label belong to the label. Predicted probability Thus, the corresponding loss value can be obtained. As in formula (9): , formula (9); Step 9.11 Order ; Step 9.12 Order ,like Proceed to step 9.8, if This indicates that the loss calculation for all enhancement features of image m is complete, and the final enhancement loss value of the m-th image is obtained. Proceed to step 9.13.

[0068] Step 9.13 Set the final loss value ; Step 9.14 Order ,like Proceed to step 7.11, if Explanation of the first batch Once the image loss calculation is complete, proceed to step 9.15.

[0069] Step 9.15 The entropy-guided loss weighting module calculates the final loss value. Send to the image-text dual-guided classifier; Step 9.16 Image-text dual-guided classifier based on The gradient backpropagation method is used to optimize the trainable parameters in the image-text dual-guided classifier; Step 9.17 ,like Proceed to step 7.5; if This indicates the end of this round of training; proceed to step 9.18. Step 9.18 Order If epoch ≤ maxepoch, proceed to step 7.3; if epoch > maxepoch, it means that the image-text dual-guided classifier enhancement training has ended, and the final image-text dual-guided classifier parameters for the t-th training stage are obtained, then proceed to step 10. Step 10, Order ,like Proceed to step four; if This indicates that all T training phases have been completed, and the trained incremental image classification system has been obtained. Proceed to step eleven. Step 11: The trained incremental image classification system is used to classify the user-input image to obtain the predicted image result. The method is as follows: Step 11.1 The image feature extraction module receives the image to be classified input by the user. ; Step 11.2 The image feature extraction module extracts features using the image feature extraction method described in step 5.8. Image features in the image are obtained Image features (It is a two-dimensional matrix with dimension 1) ),Will Send to the image-text dual-guided classifier and the contrastive learning classification module; Step 11.3 Image and text dual-guided classifier receiving ,according to Calculate the image to be classified Image classification prediction probability ,Will Send to the classification result merging module: Step 11.4 The text feature extraction module uses the text feature extraction method described in step 5.4 to extract the general text features corresponding to all the general text prompts that have been classified and labeled during training. ,Will Send to the contrastive learning classification module; generate a general text prompt for each label, where the general text for all classified labels refers to the general text prompts generated for all classes encountered during training.

[0070] Step 11.5 Receive data from the contrastive learning classification module. and ,according to Calculate the image to be classified Text classification prediction probability ,Will Send to the classification result merging module: Step 11.6 Classification result merging module receives and Calculate according to formula (10) Predicted value , , formula (10); in The value is typically set to 0.6, which controls the output ratio of the image-text dual-guided classifier and the contrastive learning classification module. The category corresponding to the maximum value is the predicted result of the image to be classified, and the image classification is completed.

[0071] To verify the effectiveness of this invention, test set data from nine datasets—CIFAR100, CUB, ObjectNet, ImageNet-R, Aircraft, Cars, Food, SUN, and UCF—were selected to replace the user's input images. The classification accuracy was then evaluated for each incremental training stage of this invention. The numerical tests were conducted in an experimental environment of Ubuntu 20.04 (a version of Linux), equipped with an Intel i9-10900K series CPU with a processing frequency of 3.70GHz, and an NVIDIA L40 GPU. The large language model in the comparative text generation module used GPT-4, and the temperature coefficient was T=10. The value is 0.1. The value is set to 0.6, and other parameter settings are consistent with those described in the steps. The test set of the above 9 datasets is input into the incremental image classification system of this invention to classify the test set data, and the average classification accuracy at all stages is calculated. And the classification accuracy in the final stage It is compared with other methods to assess the effectiveness of the invention.

[0072] First, we define the performance evaluation metrics for object detection algorithms. This experiment uses a standard accuracy evaluation method, with two specific metrics: and . This represents the average accuracy, i.e., the average classification accuracy across all stages, and is used to represent the overall classification performance of the incremental image classification system. It represents the final task accuracy and is used to indicate the performance of incremental image classification systems after training. It can reflect the degree of forgetting to some extent.

[0073] Secondly, the dataset partitioning is defined. This invention uses 'Bc Inc-d' to partition the categories in the incremental learning dataset, where c represents the number of categories in the first stage and d represents the number of categories in each subsequent stage. For example, 'B0Inc10' means that 0 categories are trained in the first stage (i.e., no first stage), and 10 categories are added in each subsequent stage; 'B50 Inc10' means that 50 categories are trained in the first stage, and 10 categories are added in each subsequent stage.

[0074] Based on the experimental results of this invention, the performance of this invention is compared with that of eight methods: the current state-of-the-art incremental image classification method ENGINE, the classic replay-based incremental image classification methods SimpleCIL, L2P, DualPromot, CODA-Promot, and CLIP-based incremental image classification methods Continual CLIP, CoOp, and RAPF.

[0075] Table 1: Table 1 shows the performance comparison of the present invention on three datasets: Aircraft, CIFAR100, and Cars. It can be seen that the present invention achieves the best results on all benchmark tests, and the classification accuracy of the final task is also the best. Both have seen significant improvements.

[0076] Table 2: Table 2 shows the performance comparison of the present invention on three datasets: ImageNet-R, CUB, and UCF. It can be seen that even on these datasets where the prediction accuracy of existing methods is already high, the present invention can still achieve a certain improvement.

[0077] Table 3: Table 3 shows the performance comparison of the present invention on three datasets: SUN, Food, and ObjectNet. It can be seen that the present invention has significant improvements in benchmark tests on these three datasets.

[0078] In summary, compared to the current state-of-the-art method ENGINE, this invention achieves a certain performance improvement on all datasets. Especially under the 'B0 Inc10' setting of the CIFAR100 dataset, It has been improved by 1.20. It improved by 1.98, with the ObjectNet dataset setting 'B0 Inc10'. It improved by 1.31. It improved by 3.65. And in some... Even on smaller datasets, this invention can improve the accuracy of the final task to some extent. For example, in the Aircraft dataset with the setting 'B0 Inc10', Only an increase of 0.01, while An improvement of 1.14 was achieved. And this can be observed across all benchmark tests. As the final classification performance of the incremental classification system, this invention can significantly improve the classification performance. In summary, this invention achieves state-of-the-art performance on multiple datasets, demonstrating high classification accuracy in incremental image classification tasks.

Claims

1. A class incremental image classification method based on an extended image-text dual-guided classifier, characterized in that... Includes the following steps: The first step is to construct an incremental image classification system based on an extension of the text-image dual-guided classifier. The incremental image classification system consists of a text feature extraction module, an image feature extraction module, a contrastive learning classification module, a text-image dual-guided classifier module, a classification result merging module, a contrastive text generation module, a feature mixing module, and an entropy-guided loss weighting module. The text-image dual-guided classifier undergoes initial training and enhancement training after training in the text feature extraction module and the image feature extraction module. The second step involves collecting incremental learning scene images as a dataset for incremental image classification, and then dividing the incremental learning sample image classification dataset into a training set. Test set ,right The training set list is obtained by further dividing the training phase. , , t is an integer, T is the number of training stages in the incremental image classification training process, and T is a positive integer; Based on the training phase division, a test set list is obtained. , ; The third step is to initialize the training phase t=1; Fourth step, let the class label sequence of the t-th training phase be... , This is the number of categories included in the t-th training phase, and the total number of categories in the t-th phase is... n is the category label; the text feature extraction module uses a general text prompt template for... The class labels in the code correspond to the real categories, which are used to construct a general text hint list for the t-th training phase. ; Fifth step, using the training set The image feature extraction module and the text feature extraction module are trained. The text feature extraction module uses a text feature extraction method to extract the general text prompt list at the t-th stage. The general text features are obtained from the general text features. The image feature extraction module uses image feature extraction methods to extract... The image features are used to obtain the first image features. ; The contrastive learning classification module calculates the first classification loss. ,based on The trainable parameters in the image feature extraction module and the text feature extraction module are optimized using the gradient backpropagation method to obtain the trained image feature extraction module and the text feature extraction module, and then the image feature extraction module and the text feature extraction module are frozen. Step 6: Perform preliminary training on the image-text dual-guided classifier: the trained image feature extraction module extracts... Based on the image features, the prototype sequences of all categories in the t-th training stage are obtained. The weights of the image-text dual-guided classifier in the t-th training phase are obtained using the prototype sequence. The trained image feature extraction module uses image feature extraction methods to extract... The Middle batch One image The image features in the image are used to obtain the second image features. Image and text dual-guided classifier utilizes Calculate the second classification loss and based on The trainable parameters in the image-text dual-guided classifier are optimized using the gradient backpropagation method to obtain the pre-trained image-text dual-guided classifier. Step 7: The trained text feature extraction module uses text feature extraction methods to extract... The second general text feature is obtained from the general text features in the middle. ; The comparative learning classification module is based on and Calculate the text classification prediction probability at the t-th training stage. The image-text dual-guided classifier is based on... Calculate image classification prediction probability ; The classification result merging module will and The probabilities are merged to obtain the merged classification prediction probabilities. ; Compare the text generation module to The classification prediction probabilities of each image are sorted, and the top-N predicted labels corresponding to the highest probabilities are obtained. A large language model is used to generate corresponding comparative text prompts, which are then passed to the text extraction module. The text feature extraction module extracts... Chinese text features, to obtain contrast text features ,Will Send it to the comparison text generation module; the method is: Step 7.1 Set training parameters, including the initial learning rate. The learning rate adjustment function is set, and Adam is selected as the model training optimizer; the batch size and maximum training step size (maxepoch) are set for network training. Step 7.2 Set the number of training rounds ; Step 7.3 From the training set list Retrieve the training set for the t-th training phase ; Step 7.4 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , ; Step 7.5 Let the first Loss per batch of images ; Step 7.6 The text feature extraction module uses text feature extraction methods to extract... The second general text feature is obtained from the general text features in the middle. ,Will The training set images are sent to the contrastive learning classification module and the feature mixing module, while the real labels corresponding to the training set images are sent to the contrastive text generation module and the feature mixing module. Step 7.7 The contrastive learning classification module receives the second image features. Second general text features Calculate the text classification prediction probability at the t-th training stage. Will Send to the classification results merging module; Step 7.8 The image-text dual-guided classifier receives the second image features from the image feature extraction module. Calculate the image classification prediction probability ,Will Send to the classification results merging module; Step 7.9 The classification result merging module... and Add them together to get the combined classification probabilities. Each of these actions Logit is sent to the comparison text generation module and the entropy-guided loss weighting module; Step 7.10: Order the b batch... Image number in the image ; Step 7.11 The comparison text generation module receives the classification result merging module's output. ,right The predicted probabilities of each image are sorted, and the top-N highest predicted probabilities are selected to obtain the b-th batch. Top-N predicted labels for each image , , Let be the predicted label ranked 'a' in terms of predicted probability, where the value of 'a' ranges from [1, 2, ..., N], and N is a positive integer. Step 7.12: The text generation module receives the ground truth label corresponding to the m-th image in the training set from the text feature extraction module. ,from Obtain the top-N predicted labels for the m-th image. ; Step 7.13 Design the corresponding question template; Step 7.14 Construct N corresponding questions ,Will The list of questions to be combined into the m-th image ,in This refers to the a-th corresponding question; Step 7.15 Compare the large language model in the text generation module with the question list. Answer the question to obtain a list of comparison text prompts for the m-th image. The list of comparison text prompts will be used. Send to the text feature extraction module; Step 7.16 The text feature extraction module receives the list of comparison text prompts. ,make ,make , which serves as a list of prompt features stored as a comparison text prompt list; Step 7.17 The text feature extraction module extracts from... Get the h-th comparison text prompt of the m-th image ; Step 7.18 The text feature extraction module uses text feature extraction methods to extract... Chinese text features ,Will join in ; Step 7.19 Order ,like Proceed to step 7.19; if ,right Perform vector averaging on all feature vectors to obtain the contrastive text features of the m-th image. It is a one-dimensional vector with dimension D. If it is a positive integer, then Send to the comparison text generation module; Step 7.20: The contrast text generation module receives the contrast text features of the m-th image from the text feature extraction module. ,Will and the true label of the m-th image Forwarded to the feature mixing module; Step 8: The feature fusion module performs a feature blending operation on the m-th image. First, perform a blending operation on the contrastive text features to obtain the blended features of the m-th image. Then, perform a vector stacking operation on the blended features of the m-th image, the general text features, and the contrastive text features to obtain the enhanced training features. ; Step 9: The entropy-guided loss weighting module receives data from the feature mixing module. and It receives the merged classification probabilities Logit from the classification result merging module and uses them... Enhanced training is performed on the image-text dual-guided classifier, and high-entropy samples in the current class incremental image classification system are identified. The loss of high-entropy samples is weighted based on the entropy value of the samples. The method is as follows: Step 9.1 The entropy-guided loss weighting module receives Logit from the classification result merging module; Step 9.2 Normalize Logit using the softmax function to obtain the normalized probability distribution Prob; Step 9.3 Obtain the predicted probability distribution of the m-th image from the probability distribution Prob. Find the entropy value of the m-th image. ; Step 9.4 The entropy-guided loss weighting module receives from the feature mixing module ; Step 9.5 If , It is the threshold of the normalized entropy value, let Weights corresponding to the loss ;like ; Step 9.6 Let the enhancement loss value of the m-th image be... ; Step 9.7 Enhanced numbering ; Step 9.8 From Extract the s-th enhanced training feature. Send it to the image and text dual-guided classifier; Step 9.9 Image and text dual-guided classifier reception And calculate the enhanced training features. Corresponding image classification prediction probability ,Will Send to the entropy-guided loss-weighting module; Step 9.10 Entropy-guided loss weighting module receives ,from The enhanced features of image m obtained from the label belong to the label. Predicted probability Thus, the corresponding loss value can be obtained. ; Step 9.11 Order ; Step 9.12 Order ,like Proceed to step 9.8, if This indicates that the loss calculation for all enhancement features of image m is complete, and the final enhancement loss value of the m-th image is obtained. Proceed to step 9.13; Step 9.13 Set the final loss value ; Step 9.14 Order ,like Proceed to step 7.11, if Explanation of the first batch Once the image loss calculation is complete, proceed to step 9.15; Step 9.15 The entropy-guided loss weighting module calculates the final loss value. Send to the image-text dual-guided classifier; Step 9.16 Image-text dual-guided classifier based on The gradient backpropagation method is used to optimize the trainable parameters in the image-text dual-guided classifier; Step 9.17 ,like Proceed to step 7.5; if This indicates the end of this round of training; proceed to step 9.

18. Step 9.18 Order If epoch ≤ maxepoch, proceed to step 7.3; if epoch > maxepoch, it means that the image-text dual-guided classifier enhancement training has ended, and the final image-text dual-guided classifier parameters for the t-th training stage are obtained, then proceed to step 10. Step 10, Order ,like Proceed to step four; if This indicates that all T training phases have been completed, and the trained incremental image classification system has been obtained. Proceed to step eleven. Step 11: The trained incremental image classification system is used to classify the user-input image to obtain the predicted image result. The method is as follows: Step 11.1 The image feature extraction module receives the image to be classified input by the user. ; Step 11.2 The image feature extraction module uses image feature extraction methods to extract... Image features in the image are obtained Image features ,Will Send to the image-text dual-guided classifier and the contrastive learning classification module; Step 11.3 Image and text dual-guided classifier receiving Calculate the image to be classified Image classification prediction probability ,Will Send to the classification result merging module: Step 11.4 The text feature extraction module uses text feature extraction methods to extract the general text features corresponding to all the general text prompts that have been classified and labeled during training. ,Will Send to the comparative learning classification module; Step 11.5 Receive data from the contrastive learning classification module. and Calculate the image to be classified Text classification prediction probability ,Will Send to the classification result merging module: Step 11.6 Classification result merging module receives and ,calculate Predicted value , The category corresponding to the maximum value is the predicted result of the image to be classified, and the image classification is completed.

2. The incremental image classification method based on image-text dual-guided classifier extension as described in claim 1, characterized in that... The text feature extraction module is connected to the contrastive learning classification module, the feature mixing module, and the contrastive text generation module. The text feature extraction module constructs general text prompts using labels from the training set and extracts general text features from these prompts. The text feature extraction module consists of a text encoder, T text projection layers, and a first adder. The text encoder parameters are pre-trained using CLIP on a Transformer architecture; these parameters are frozen during training and do not participate in the training process. The text encoder extracts features from the general text prompts, obtaining preliminary text features which are then sent to the T text projection layers. All T text projection layers are fully connected layers that project the preliminary text features received from the text encoder in parallel, obtaining features corresponding to the task. The information has T projected text features. The first adder adds the T projected text features to obtain the text features. When training the text feature extraction module, the text feature extraction module extracts the first general text features from the general text prompts corresponding to all labels in the training set and sends them to the contrast learning classification module. When training the image-text dual-guided classifier module, the trained text feature extraction module re-extracts the second general text features from the general text prompts and sends them to the contrast learning classification module and the feature mixing module. It also receives the contrast text prompt list sent by the contrast text generation module, extracts the contrast text features of all contrast text prompts in the list, and sends the real labels and contrast text features corresponding to the images in the training set to the contrast text generation module. The image feature extraction module is connected to the contrastive learning classification module, the image-text dual-guided classifier, and the feature mixing module. The image feature extraction module extracts features from the images in the training set to obtain image features. The image feature module consists of an image encoder, T image projection layers, and a second adder. The image encoder parameters are pre-trained using CLIP on a Transformer architecture; these parameters are frozen during training and do not participate in the training process. The image encoder extracts features from the images in the training set, obtaining preliminary image features which are then sent to the T image projection layers. All T image projection layers are fully connected layers, projecting the preliminary image features obtained from the image encoder in parallel to obtain T projected image features containing corresponding task information. The second adder adds the T projected image features to obtain the final image features. During training, the image feature extraction module sends the first image features extracted from the training set images to the contrastive learning classification module. During training of the image-text dual-guided classifier module, the trained image feature extraction module re-extracts image features from the training set images and sends the resulting second image features to the contrastive learning classification module, the image-text dual-guided classifier, and the feature mixing module. The contrastive learning classification module is connected to the text feature extraction module, the image feature extraction module, and the classification result merging module. The contrastive learning classification module uses the general text features received from the text feature extraction module and the image features received from the image feature extraction module to calculate the text classification prediction probability, and sends the text classification prediction probability to the classification result merging module. The image-text dual-guided classifier is connected to the image feature extraction module, the classification result merging module, and the entropy-guided loss weighting module. The image-text dual-guided classifier consists of a single fully connected layer that ignores the bias term. It predicts the image features received from the image feature extraction module to obtain the image classification prediction probability, and sends the image classification prediction probability to the classification result merging module. At the same time, the image-text dual-guided classifier predicts the enhanced training features sent by the entropy-guided loss weighting module, and sends the image classification prediction probability corresponding to the enhanced training features to the entropy-guided loss weighting module. The classification result merging module is connected to the contrastive learning classification module, the image-text dual-guided classifier, the contrastive text generation module, and the entropy-guided loss weighting module. It merges the text classification prediction probability received from the contrastive learning classification module and the image classification prediction probability received from the image-text dual-guided classifier to obtain the merged classification prediction probability, and sends the merged classification prediction probability to the contrastive text generation module and the entropy-guided loss weighting module. The comparison text generation module is connected to the classification result merging module, feature mixing module, and text feature extraction module. The comparison text generation module prepares for the enhanced training of the image-text dual-guided classifier. This module receives the merged classification prediction probabilities from the classification result merging module, takes the Top-N results from the merged classification prediction probabilities to obtain the Top-N predicted labels of the training set images, and simultaneously receives the real labels corresponding to the images in the training set sent by the text feature extraction module. It uses a large language model to perform comparative descriptions on the Top-N predicted labels and real labels to obtain a comparison text prompt list. The comparison text prompt list is sent to the text feature extraction module, and the module receives the comparison text features corresponding to the comparison text prompt list extracted by the text feature extraction module. The comparison text features and the real labels corresponding to the images in the training set are then sent to the feature mixing module. The feature fusion module is connected to the text feature extraction module, image feature extraction module, contrast text generation module, and entropy-guided loss weighting module. The feature fusion module receives image features from the image feature extraction module, receives the real labels corresponding to the images in the training set and the contrast text features from the contrast text generation module, and receives general text features from the text feature extraction module. It combines the image features, general text features, and contrast text features to form an enhanced training feature set, and sends the enhanced training feature set and the real labels corresponding to the images in the training set to the entropy-guided loss weighting module. The entropy-guided loss weighting module is connected to the feature mixing module, the classification result merging module, and the image-text dual-guided classifier. The entropy-guided loss weighting module receives the merged classification prediction probabilities from the classification result merging module and the enhanced training feature set from the feature mixing module. It calculates the entropy value based on the merged classification prediction probabilities to determine if the sample image is a high-entropy sample. Then, it forwards the enhanced training features from the enhanced training feature set to the image-text dual-guided classifier. The image-text dual-guided classifier calculates the image classification prediction probability corresponding to the enhanced training features and sends it to the entropy-guided loss weighting module. The entropy-guided loss weighting module calculates the loss for the corresponding sample, increasing the loss weight for the enhanced features of high-entropy samples during the calculation to obtain the training set loss. This training set loss is then sent to the image-text dual-guided classifier. In this way, the incremental image classification system enhances its ability to distinguish high-entropy samples during training, increases the confidence of the classification results, and further improves the accuracy of the incremental image classification task. All modules participate in training, except for the text generation module, feature mixing module, and entropy-guided loss weighting module, which only participate in training and do not participate in the user input image classification process.

3. The incremental image classification method based on a dual-guided image-text classifier as described in claim 1, characterized in that... The value of T is 10; the large language model in the comparison text generation module is required to have strong language capabilities and be able to accurately describe the comparison features of the two classes in detail without the presence of images.

4. The incremental image classification method based on a dual-guided image-text classifier as described in claim 3, characterized in that... The large language model is required to be GPT-4 or ChatGLM.

5. The incremental image classification method based on image-text dual-guided classifier extension as described in claim 1, characterized in that... The second step involves dividing the incremental learning sample image classification dataset into a training set. Test set ,right The training set list is obtained by further dividing the training phase. ,Will Based on the training phase division, a test set list is obtained. The method is: Step 2.1 Collect incremental learning scene images as a class incremental image classification scene dataset. The method is as follows: Nine datasets were used as incremental learning sample image classification datasets: the small image dataset CIFAR100, the general scene dataset ImageNet-R, the bird dataset CUB200, the food dataset Food101, the car dataset StanfordCars, the action dataset UCF101, the scene classification dataset SUN397, the aircraft classification dataset Aircraft, and the object recognition dataset ObjectNet. Each image in these nine datasets was labeled with the true label of the object category. Step 2.2 Based on the original partitioning criteria in the dataset, divide the incremental learning sample image classification dataset into a training set. test set Let the total number of categories in the sample image classification dataset be C, and each image have a true label [1,2,…,c,…,C] corresponding to its category. C is a positive integer; Step 2.3: Partition the training set using a set partitioning method. The training set list is obtained by further dividing the training phase. , , And t is an integer, the method is: Step 2.3.1 Let the training phase t = 1; Step 2.3.2 Let the number of classes included in the t-th training phase be... The set of class labels included in the t-th training phase = ; Step 2.3.3 Find the training set China belongs to The image, i.e., based on the training set Find the real labels included The same image, will China belongs to Adding images to middle; Step 2.3.4 If ,make Proceed to step 2.3.2; if This indicates that the training set list has been obtained. Proceed to step 2.4; Step 2.4 Test set Based on the training phase division, a test set list is obtained. , The method is: Step 2.4.1 Let the training phase t = 1, Step 2.4.2 Find the test set China belongs to The image, i.e., based on the test set Find the tags included. The same image, add these images to middle; Step 2.4.3 If ,make Proceed to step 2.4.2; if This yields the final test set list. Proceed to step three.

6. The incremental image classification method based on image-text dual-guided classifier extension as described in claim 1, characterized in that... The text feature extraction module described in step four uses a general text prompt template to... The class labels in the code correspond to the real categories, which are used to construct a general text hint list for the t-th training phase. The method is: Step 4.1 Let =1, initialize the general text prompt list for the t-th training phase. =[]; Step 4.2 Use a general text prompt template for... The category label n in the text corresponds to the actual category, which is used to construct a general text hint. ,Will Add to ; Step 4.3 If ,make Proceed to step 4.2; if This describes the general text hint list for all categories in the t-th training phase. Once the construction is complete, the general text prompt list for the t-th training phase is obtained. Proceed to step five.

7. The incremental image classification method based on image-text dual-guided classifier extension as described in claim 1, characterized in that... The fifth step involves using the training set. The method for training the image feature extraction module and the text feature extraction module is as follows: Step 5.1 Initialize the model parameters of the image feature extraction module and the text feature extraction module; initialize and freeze the parameters of the Transformer framework in the text feature extraction module using the pre-trained CLIP text encoder; initialize and freeze the parameters of the Transformer framework in the image feature extraction module using the pre-trained CLIP image encoder. Step 5.2 Randomly initialize the image projection layer for the t-th training phase. and text projection layer ,Will The projection layer sequence added to the image feature extraction module will Add a projection layer sequence to the text feature extraction module so that there is an image projection layer sequence in the t-th training stage. and text projection layer sequence The image projection layer sequence of the t-th training stage is concatenated with the frozen image encoder, and the text projection layer sequence of the t-th training stage is concatenated with the frozen text encoder. This ensures that the features extracted by the image encoder and text encoder contain stage task information, and that only the image projection layer sequence in the t-th training stage contains the stage task information. Trainable, only in the text projection layer sequence Trainable, the remaining projection layers remain frozen; Step 5.3 Set training parameters; set the initial learning rate. The learning rate is set to 0.05, and the learning rate adjustment function is set to the cosine annealing dynamic learning rate adjustment function; stochastic gradient descent is selected as the model training optimizer; the batch size of the network training is set to 64; and the maximum training step size is set to 10. Step 5.4 The text feature extraction module uses text feature extraction methods to extract the general text prompt list for the t-th stage. The general text features are obtained from the general text features. , It is a two-dimensional matrix with dimension 1. It is the feature dimension, which is 512. Send it to the contrastive learning classification module, specifically by: the text encoder... Text features are extracted from all common text prompts, and the resulting preliminary text features are sent to the image projection layer sequence. ; The initial text features are projected in parallel to obtain t projected text features containing the corresponding task information; the second adder adds the t projected text features to obtain the final text features, which are the general text features for the current training stage. ,Will Send to the comparative learning classification module; Step 5.5 Set the training epoch number to 1; Step 5.6 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , , ,like no When the image is an integer multiple of the last batch, the last batch of images is... Remove from middle -All images remaining after one batch; Step 5.7 Image feature extraction module from Extract the training set for stage t. ,from Read the first batch The image will be the first one. batch The images are denoted in matrix form. The image; where H represents the height of the input image, W represents the width of the input image, and "3" represents the three RGB channels of the image; Step 5.8 The image feature extraction module uses image feature extraction methods to extract... The image features are used to obtain the first image features. , It is a two-dimensional matrix with dimension 1. ,Will Sending it to the contrastive learning classification module, specifically by: the image encoder... Feature extraction is performed on the image to obtain preliminary image features, which are then sent to the image projection layer sequence. ; The initial image features are projected in parallel to obtain t projected image features containing corresponding task information; the second adder adds the t projected image features to obtain the final image features. ,Will Send to the comparative learning classification module; Step 5.9 The contrastive learning classification module receives data from the image feature extraction module. Receive from the text feature extraction module ; Step 5.10 Comparative learning classification module calculates the first classification loss according to formula (1). : Formula (1); i represents No. batch The image number, express No. batch The true label of the i-th image in a set of images. express No. batch Image features of the i-th image in a set of images. The label of the i-th image Textual features, Indicates calculation and cosine similarity, This represents the temperature coefficient, with a value of 0.

1. Step 5.11 The contrastive learning classification module is based on The trainable parameters in the image feature extraction module and the text feature extraction module are optimized using the gradient backpropagation method. Step 5.12: Order ,like Proceed to step 5.7; if This indicates the end of this round of training; proceed to step 5.

13. Step 5.13: Order If epoch ≤ maxepoch, proceed to step 5.6; if epoch > maxepoch, it means that the training of the image feature extraction module and the text feature extraction module has ended, and the trained image feature extraction module and text feature extraction module are obtained. Freeze the image feature extraction module and text feature extraction module, and proceed to step 6.

8. The incremental image classification method based on image-text dual-guided classifier extension as described in claim 1, characterized in that... The method described in step six for initially training the image-text dual-guided classifier using the training set image features extracted by the trained image feature extraction module is as follows: Step 6.1 The trained image feature extraction module extracts features from the training set list. Retrieve the training set for the t-th training phase ; Step 6.2 The trained image feature extraction module starts from... The prototype sequence of all classes in the t-th training phase is obtained from the following: , It is the prototype of the k-th category. And store the prototype sequences of all categories in the t-th training phase into the total sequence P in the t-th phase. The method is as follows: Step 6.2.1 Let the index of the first category in the t-th training phase be... ; Step 6.2.2 From Read the image of the k-th category and represent all images of the k-th category in matrix form. ; Step 6.2.3 Image Feature Extraction Module Extraction Image features ; Step 6.2.4 Calculate the average to obtain the prototype of the k-th category. ,Will Place it into the prototype sequence of all categories in the t-th training phase; Step 6.2.5 ,like Proceed to step 6.2.2, if This indicates that the prototype sequences of all categories in the t-th training phase have been obtained. Proceed to step 6.3; Step 6.3 Utilizing prototype sequences The weights of the image-text dual-guided classifier in the t-th training phase :like , As in formula (2): , formula (2); like , use The final image-text dual-guided classifier weights in the (t-1)th training phase The result is obtained by connecting the two formulas, as shown in formula (3): , Formula (3); Step 6.4 Set training parameters; set the initial learning rate. The learning rate is set to 0.0005, and the learning rate adjustment function is set to the cosine annealing dynamic learning rate adjustment function; the adaptive moment estimation Adam is selected as the model training optimizer; the batch size of the network training is set to 64; and the maximum training step size is set to 3. Step 6.5 Set the number of training rounds. ; Step 6.6 Set the batch number , indicating the first Image batches, each batch containing a total of Images, total batch size , , ; Step 6.7 From Read the first batch The image will be the first one. batch The images are denoted in matrix form. ; Step 6.8 The trained image feature extraction module uses image feature extraction methods to extract... The image features in the image are used to obtain the second image features. It is a two-dimensional matrix with dimension 1. ,Will Send the data to the contrastive learning classification module, the image-text dual-guided classifier, and the feature fusion module; Step 6.9 Calculate the second classification loss using the image-text dual-guided classifier. As shown in formula (4): , Formula (4); express Image features of the i-th image. This indicates that the i-th image belongs to the label. The predicted probability; Step 6.10 Image-text dual-guided classifier based on The gradient backpropagation method is used to optimize the trainable parameters in the image-text dual-guided classifier; Step 6.11 ,like Proceed to step 6.7; if This indicates the end of this round of training; proceed to step 6.

12. Step 6.12 If epoch ≤ maxepoch, proceed to step 6.6; if epoch > maxepoch, it means that the initial training of the image-text dual-guided classifier is complete, and the initially trained image-text dual-guided classifier is obtained. Proceed to step 7.

9. The incremental image classification method based on a dual-guided image-text classifier as described in claim 1, characterized in that... The method for setting training parameters in step 7.1 is as follows: set the initial learning rate... for Set the learning rate adjustment function to the cosine annealing dynamic learning rate adjustment function; set the batch size for network training to 64; set the maximum training stride to 3; in step 7.4... The text classification prediction probability in step 7.7 during the t-th training phase. Calculate according to formula (5) It is a two-dimensional matrix with dimension 1. : , Formula (5); Indicates to Perform normalization. The temperature parameter is set to 0.1; the image classification prediction probability in step 7.8 Calculate according to formula (6), It is a two-dimensional matrix with dimension 1. : , Formula (6); In step 7.11 It is a two-dimensional matrix with dimension 1. Each row ; The question template mentioned in step 7.13 is "What are the unique visual features of [CLASS]?" i Compared to [CLASS] j In a photo? Focus on their key visual differences.”; Step 7.14 involves constructing N corresponding questions. The method is: Use Replace [CLASS] in the question template. i ,use The elements in [CLASS] are replaced sequentially. j .

10. The incremental image classification method based on image-text dual-guided classifier extension as described in claim 1, characterized in that... The feature mixing module described in step eight applies to the m-th image. The method involves performing a blending operation on the contrastive text features to obtain the blended features of the m-th image, and then performing a vector stacking operation on the blended features of the m-th image, the general text features, and the contrastive text features to obtain the enhanced training features. Step 8.1 The feature fusion module receives the ground truth label of the m-th image from the contrastive text generation module. and Received from the text feature extraction module Receive from image feature extraction module ,from Extract the general text features corresponding to the true label of the m-th image. Extract the image features of the m-th image ; Step 8.2 The feature blending module will and Mixing is performed to obtain mixed characteristics. As shown in formula (7): Official (7) in To compare the mixing ratio of text features and image features, it is set to 0.75; Step 8.3 The feature blending module will Perform vector stack operations to form the first... Enhanced training features of the m-th image in the batch , It is a two-dimensional matrix with dimension 1. The CCP's three enhanced features will With the true label of the m-th image Send it to the entropy-guided loss weighting module.

11. The incremental image classification method based on image-text dual-guided classifier extension as described in claim 1, characterized in that... The Prob mentioned in step 9.2 is a two-dimensional matrix with dimension . ; In step 9.3 It is a one-dimensional vector with dimension . The entropy value corresponding to the m-th image The calculation formula is as shown in formula (8): Official (8) in The predicted probability distribution from the m-th image The predicted probability of the j-th label is extracted; The steps described in step 9.5 Set to 0.7; in step 9.8 It is a one-dimensional vector with dimension D; the loss value in step 9.10 Calculate according to formula (9): , Official (9).

12. The incremental image classification method based on image-text dual-guided classifier extension as described in claim 1, characterized in that... The steps described in step 11.2 It is a two-dimensional matrix with dimension 1. ; The steps described in step 11.3 The calculation formula is: The steps described in step 11.5 The calculation formula is: Step 11.6 Predicted value Calculate according to formula (10), , formula (10); in The value is 0.6, which controls the output ratio of the image-text dual-guided classifier and the contrastive learning classification module.

Images