A class-incremental learning method based on prompt guidance and multi-modal fusion

Abstract

This invention provides a class-based incremental learning method based on prompt guidance and multimodal fusion, relating to the fields of artificial intelligence and computer vision. First, the category labels are semantically expanded by constructing a semantically enhanced text representation using a text encoder. Then, a pre-trained visual encoder is used to perform block embedding and hierarchical feature extraction on the input image, constructing a cross-modal unified embedding space. A bimodal prompt gating fusion module is introduced within this unified embedding space, adaptively weighting text and image prompts according to gating weights to generate fused prompts. A bimodal prompt collaborative filtering module selects the set of prompts most relevant to the current task based on the similarity of image and text semantic features. In the incremental stage, the pre-trained backbone network is frozen, and only the prompt parameters and fusion layer weights are optimized. A joint loss function is used for parameter updates. Finally, in the inference stage, image and text data are input, cross-modal similarity is calculated, and classification prediction results are output.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to an incremental learning method based on prompting and multimodal fusion. Background Technology

[0002] In the fields of artificial intelligence and computer vision, deep learning models have made significant progress in various visual recognition tasks. In particular, models based on convolutional neural networks and visual Transformers, supported by large-scale supervised data, have achieved end-to-end training from low-level feature extraction to high-level semantic representation. However, real-world learning scenarios are often dynamic and uncertain. Models not only need to recognize known categories but also must learn and update under continuously emerging new categories or tasks, which leads to the problem of class-incremental learning (CIL).

[0003] Incremental learning requires models to learn new knowledge without forgetting old knowledge when faced with new categories of data. Traditional incremental learning methods typically rely on fixed-structure neural networks, maintaining model stability through techniques such as knowledge distillation, feature replay, or weight regularization. However, most of these methods assume a sufficient number of samples for new categories. In real-world applications, such as medical image recognition, remote sensing classification, or industrial inspection, the number of samples for new categories is often extremely small, sometimes only a few. In such cases, models are highly susceptible to overfitting or catastrophic forgetting due to a lack of sufficient data, severely limiting their generalization ability under small sample conditions.

[0004] To address the problem of scarce samples, researchers have proposed Few-Shot Learning (FSL) and Few-Shot Class-Incremental Learning (FSCIL) methods. FSCIL combines the goals of few-shot learning and incremental learning, requiring the model to learn new knowledge while retaining the discriminative ability of previous tasks when given very few samples of new categories. However, most existing FSCIL methods are limited to a single visual modality, relying solely on image features for category differentiation. In this case, the model lacks semantic constraints and linguistic priors, leading to blurred boundaries between different categories, severe feature overlap, and poor stability when new tasks arrive.

[0005] In recent years, the development of multimodal learning techniques has provided new ideas for solving the above problems. Typical visual language models, such as CLIP, achieve cross-modal representation learning by jointly training large-scale image and text data. These models can align visual and linguistic information in a unified feature space, thus possessing strong semantic understanding and zero-shot generalization capabilities. However, directly applying visual language models such as CLIP to incremental learning still has several shortcomings: First, the information interaction between image and text modalities is relatively coarse, the fusion mechanism lacks adaptability, and it is difficult to fully utilize the guiding role of text semantics on visual features; second, the prompt design process relies on human experience, and prompt parameters are difficult to share and dynamically update across different tasks, resulting in insufficient scalability of the model during task transitions; third, there is a lack of constraints on the stability of the cross-task semantic space, making it prone to feature drift and knowledge forgetting in the incremental stage.

[0006] Furthermore, existing multimodal fusion strategies, such as simple feature concatenation, linear projection, or attention weighting, while achieving some success in static tasks, often fail to adjust the modal fusion strength according to task changes in dynamic incremental scenarios, and cannot automatically filter redundant or irrelevant prompts. This leads to increased interference in the representation space of old tasks when the model learns new tasks, exhibiting a significant catastrophic forgetting effect.

[0007] Therefore, designing an innovative method that can achieve efficient information fusion between visual and linguistic modalities and maintain model stability and generalization ability during incremental learning with few samples has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0008] To address the shortcomings of existing incremental learning methods with limited sample sizes, such as insufficient modal fusion, rigid prompting mechanisms, and susceptibility to feature drift and forgetting during the incremental phase, this invention proposes a novel incremental learning method based on prompting guidance and multimodal fusion. This method comprehensively utilizes visual and textual modal information, employing modules such as semantic enhancement, gated fusion, prompting collaboration, and orthogonal constraints to achieve efficient alignment and dynamic adaptation of cross-modal features. This significantly improves the incremental learning performance and stability of the model under conditions of scarce samples.

[0009] On the one hand, this invention provides a method for incremental learning based on prompting and multimodal fusion, comprising the following steps:

[0010] Step 1: Construct semantically enhanced text representations; expand category labels based on language models to generate text sequences containing visual attributes and semantic features; input the expanded text sequences into a text encoder, and perform multi-layer semantic modeling through learnable general text task prompts and text task-specific prompts to obtain discriminative text feature vectors;

[0011] Specifically, the text encoder enriches the text input by adding visual feature descriptions to the category labels. A general hint for a learnable text task and A learnable text task-specific cue ,in , It is the embedding dimension of the text encoder; task-general cues are shared across different tasks and are placed before the text encoder. Layers are used, while task-specific cues are filtered from the task-specific cue pool and applied in subsequent layers after the (k+1)th layer of the text encoder.

[0012] The specific process of the text encoder is as follows: First, the GPT-2 model is used to generate enhanced visual feature descriptions for the category labels, and these descriptions are passed as input to the text encoder. Then, the tokenizer of the CLIP model is used to process the visual feature descriptions, converting them into text tokens. The tokens are mapped to corresponding embedding vectors through the embedding layer. Position encoding is added to the generated embedding vectors, resulting in embedding vectors with positional information. General hints for text tasks By splicing, the resulting form is: The sequence, where This represents the set of general text task hints, with a total of L general text hints. a indivual, Indicates the Lth a A general text prompt, the sequence is fed into the Transformer encoding layer of the text encoder for processing, defining... For the text encoder's first The forward propagation process of the first layer is shown in formula (1):

[0013] (1);

[0014] For up to the The subsequent layers of the layer are used as general knowledge extraction modules for the task. The forward propagation process of the general knowledge extraction module for the task is shown in formula (2):

[0015] (2);

[0016] in, This indicates a vertical concatenation operation; "_" indicates that the output at that position is discarded and not included in subsequent calculations. It is the first The layer generates latent state output based on category word embeddings, while Then it means the first The set of general prompts for all learnable text tasks in the layer, during propagation, the input The output embedding will be discarded and will not be passed to the next layer; each The subsequent Transformer layer serves as the task-specific knowledge extraction module; the task-specific knowledge extraction module uses filtered text task-specific prompts to model task specificity, as shown in formula (3):

[0017] (3);

[0018] Where L represents the total number of layers in the text encoder. This represents the set of task-specific cues for the learnable text obtained from filtering in the i-th layer; during the encoding process, the semantic information of the input sentence is progressively abstracted through the layer-by-layer propagation of the Transformer network. Finally, the cues from the last hidden state that coincide with the end symbol are selected. corresponding vector As a global semantic representation of the entire sentence, the global semantic representation vector It will be projected into a lower-dimensional space, thus forming the text feature representation h of the entire sentence. text As shown in formula (4):

[0019] (4);

[0020] in This represents a linear projection layer that processes text features;

[0021] Step 2: Extract visual features and establish a cross-modal embedding space; extract features from the input image using a pre-trained visual encoder to obtain a global visual embedding vector; then the global visual embedding vector is mapped to a unified cross-modal representation space, aligning with the text features in the same dimension;

[0022] Specifically, this involves introducing a pre-trained visual encoder. General hints for learnable image tasks and Image task-specific prompts ,in , This represents the embedding dimension of the image encoder; image task-specific cues and text task-specific cues are combined through a gating fusion mechanism to form a hybrid task-specific cue pool. The mixed-task-specific cue pool, after undergoing a dual-modal collaborative filtering mechanism, yields cuees adapted to the current task. ,in Indicates the first option selected from the suggestion pool. One tip;

[0023] The specific process of the image encoder is as follows: First, the image undergoes block embedding processing and is coupled with a... The tags are combined to form an embedding vector. Then, the embedding vector General tips for image tasks Vertical splicing constructs a sequence ,in This represents a set of general cues for the image task; the sequence is then fed into the Transformer coding layer in the image encoder for further processing; [Definition] For the first in the image encoder The forward propagation process of the first layer is shown in formula (5):

[0024] (5);

[0025] For until the The forward propagation process of the subsequent layers is shown in Equation (6):

[0026] (6);

[0027] No. The output of the layer prompts will accumulate to the 1st layer. In subsequent Transformer layers, specific hints will be provided based on the filtered hybrid task-specific prompts adapted to the current task. ,in Represents a set of hybrid task-specific prompts The first in A cue vector is used to model the features of the image for the current task category, as shown in formula (7):

[0028] (7);

[0029] in, Indicates the first The set of learnable hybrid task-specific cues obtained from filtering in each incremental task; after processing by the image encoder, the cues from the last hidden state that match... corresponding vector To represent the entire image, a vector It is projected into a lower-dimensional space, thereby generating a visual feature representation h of the entire image. img As shown in formula (8):

[0030] (8);

[0031] in This represents a linear projection layer that processes image features;

[0032] Step 3: Implement dual-modal gated fusion; Design a lightweight fusion module BPGF based on gated attention mechanism, which adaptively allocates weights for text task-specific cues and image task-specific cues according to input features; Specifically, a learnable gated weight generation network generates fusion weight coefficients, and the different modal cues are weighted and summed to obtain the fused hybrid task-specific cues M.

[0033] Specifically, given text task-specific prompts Image task-specific cues ,in It is the number of text-specific prompts. and represent the embedding dimensions of the text encoder and the image encoder, respectively; before cue fusion, the text task-specific cues are projected onto the same dimensions as the image task-specific cues according to formula (9):

[0034] (9);

[0035] The fusion mechanism of the gated weight generation network is calculated as follows: the text task-specific cue and the image task-specific cue are concatenated in the last dimension to obtain the joint cue. ,in Joint reminder The input is fed into a gated weight generation network, which consists of a linear layer and a sigmoid activation function to generate gated weights. The calculation process is shown in formula (10):

[0036] (10);

[0037] in, It is the weight matrix of the linear layer. It is a bias term. It's the Sigmoid function, which restricts the weights to the range [0,1]; the generated gated weights are used. The text-based task-specific cues and image-based task-specific cues are weighted and summed according to formula (11) to obtain the fused hybrid task-specific cues. :

[0038] (11);

[0039] in, This indicates element-wise multiplication.

[0040] Step 4: Perform bimodal cue collaborative filtering; propose a semantic similarity-based cue collaborative filtering mechanism BPCF, which automatically selects the optimal set of cue most relevant to the current task by calculating the cosine similarity between the hybrid task-specific cue and textual and image cue features after L2 normalization. ;

[0041] Incremental learning phase Assuming the current batch of image data is [size missing] The corresponding image input batch is denoted as Define text input batches , Size is the incremental stage and the total number of categories encountered in all previous tasks. ,in It contains text descriptions of all categories from both the current and historical tasks; firstly, it utilizes the pre-trained CLIP model's text encoder and image encoder to extract text descriptions from... and Extract the aggregated representation of the last hidden state as a textual clue. and image clues Specifically, as shown in formula (12):

[0042] (12);

[0043] in, This represents the text encoder in the pre-trained CLIP model. This represents the image encoder in the pre-trained CLIP model. This represents the aggregate output of the encoder's last hidden state; for a text encoder, it corresponds to the end-of-line character. The hidden state vector, for the image encoder, corresponds to the global label. The hidden state vector.

[0044] Then, regarding the text clues... and image clues The hybrid task-specific cue M obtained in step 3 is then subjected to L2 normalization, as shown in formula (13):

[0045] (13);

[0046] in This means projecting textual cues onto the same dimension as image cues. This indicates L2 normalization. , and These represent the text cue features, image cue features, and hybrid task-specific cue features after L2 normalization, respectively.

[0047] According to formula (14), the cosine similarity between the hybrid task-specific cue and the text cue features and image cue features is calculated:

[0048] (14);

[0049] in, and These represent the text similarity matrix and the image similarity matrix, respectively.

[0050] Finally, the text similarity matrix and the image similarity matrix are fused using mean fusion to generate a comprehensive similarity score. The specific calculation method is shown in formula (15):

[0051] (15);

[0052] in, and These represent the batch sizes for text input and image input, respectively. Finally, based on formula (16) and the comprehensive similarity score, the most relevant prompts to the current task are selected from the prompt pool. A tip :

[0053] (16);

[0054] in Used to select the most similar One index, This indicates that the corresponding prompt is in the prompt pool. The set of indices in;

[0055] Step 5: Perform few-shot incremental learning and parameter optimization based on bimodal fusion; in each incremental stage, freeze the text encoder and image encoder backbone networks of the pre-trained CLIP model, and only update the general prompts for the text task. Text-based task-specific prompts General tips for image tasks Image task-specific prompts The weights of the fusion layer are optimized by jointly improving the cue learning and bimodal fusion module through cross-entropy loss and orthogonal cue protection loss.

[0056] The incremental learning of few-shot classes based on dual-modal fusion specifically includes the following steps:

[0057] Step s1: Input image data Text description of the corresponding category The text features are extracted from the input text encoder and image encoder, respectively. With visual features ;

[0058] General hints for text encoders to concatenate text in the input sequence With text task-specific prompts Text features are obtained after multi-layer Transformer encoding. : ;

[0059] in It is a linear projection matrix. This is the global representation of the last hidden state.

[0060] The image encoder adds general task hints after image patch embedding. Through interlayer accumulation and image task-specific cues After enhancement, visual features are obtained. : ;

[0061] in It is a linear mapping matrix. This represents the global features of the final layer of the visual encoder.

[0062] Step s2: Select the task-specific text prompts obtained in step s1. Image task-specific cues The input is fed into the lightweight fusion module BPGF based on the gated attention mechanism for adaptive weighted fusion of features between modalities;

[0063] Step s2.1: First, project the text task-specific cue onto the same dimension as the image cue: ;

[0064] in It is a linear projection matrix;

[0065] Step s2.2: Combine the projected text task-specific cues with the image task-specific cues to obtain a joint cue sequence;

[0066] ;

[0067] The input gate weight generation network is used to calculate the gate weights via a linear layer and a sigmoid function:

[0068] ;

[0069] in , For network parameters, For the Sigmoid function, This indicates a task-specific cue from the projected text. Image task-specific cues A combined prompt formed by piecing together elements.

[0070] Step s2.3: Based on the gate weights calculated in step s2.2, the two modal cues are weighted and fused to obtain hybrid task-specific cues: ;

[0071] in This indicates element-wise multiplication. (This is a task-specific hint after merging.) Feature modeling will be used for the incremental phase.

[0072] Step s3: Develop the hybrid task-specific hints output in step s2. The input is fed into the semantic similarity-based prompt collaborative filtering mechanism BPCF, which selects the optimal prompt set based on task features.

[0073] Step s3.1: Extract image cues using the frozen CLIP model With textual clues :

[0074] ;

[0075] in, and These represent the frozen CLIP text encoder and image encoder, respectively.

[0076] Step s3.2: For , , Perform L2 normalization and calculate cosine similarity: ;

[0077] in, This represents the hybrid task-specific cue matrix obtained in step s2. and These represent the text cue feature matrices and image cue feature matrices extracted by the frozen CLIP text encoder and image encoder, respectively. This represents the matrix transpose operation. and These represent the similarity score matrices between task-specific cues and textual or image cues, respectively.

[0078] Step s3.3: Obtain the bimodal collaborative score S by combining text and image similarity. fusion : ;

[0079] Select the highest score These prompts form the optimal set of prompts: ;

[0080] Step s4: Select the optimal hint set from step s3. With visual features By concatenating the classification layer of the input model, category prediction is achieved through similarity matching;

[0081] Step s5: Hint at using a joint loss function during the learning and training of the bimodal fusion module:

[0082] in, For cross-entropy loss, Orthogonal hints for loss protection:

[0083] ;

[0084] in This represents the matrix transpose operation. This represents the identity matrix with the same dimensions as the suggested features. It is the Frobenius norm. This is the balance coefficient;

[0085] Step s6: After training is complete, input the new task samples into the model, use the fusion features to calculate the similarity with the text prototypes of each category, and output the prediction results;

[0086] Step 6: Generate classification prediction results; During the inference stage, the image and text features fused by dual-modality fusion are input into the classification layer of the model, and the target category prediction is obtained by calculating the similarity between the fused features and the text prototypes of each category.

[0087] On the other hand, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the incremental learning method based on prompting and multimodal fusion.

[0088] Thirdly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the aforementioned incremental learning method based on prompting and multimodal fusion.

[0089] The beneficial effects of adopting the above technical solution are as follows:

[0090] This invention provides a novel incremental learning method based on prompting and multimodal fusion, which has the following advantages:

[0091] (1) More efficient cross-modal feature fusion capability: This invention realizes dynamic weighting of visual and textual prompts through a dual-modal prompt gating fusion module, and adaptively allocates modal weights at different task stages, thereby achieving fine-grained information complementarity at the semantic level, effectively improving the model's ability to discriminate small sample categories and generalize.

[0092] (2) More accurate semantic filtering and feature matching: The dual-modal cue collaborative filtering module can dynamically select the optimal cue combination based on the similarity between image and text features, significantly reducing irrelevant cue interference, ensuring the model's stable focus on key semantics in incremental tasks, and improving the model's feature extraction efficiency and accuracy in complex tasks.

[0093] (3) Enhanced knowledge retention and parameter efficiency: By freezing the pre-trained backbone network and updating only the cue parameters and fusion layer weights, the model can expand new categories without destroying the existing knowledge structure, thus achieving efficient continuous learning of parameters and effectively alleviating the catastrophic forgetting problem in class incremental learning.

[0094] (4) Better feature independence and model stability: This invention introduces a cue independence enhancement mechanism to maintain the orthogonality and independence of different task cue in the semantic space, avoid semantic overlap between cue, and improve the robustness and learning stability of the model in multi-task scenarios.

[0095] (5) Wider applicability and promotion potential: The dual-modal fusion framework proposed in this invention is not only applicable to incremental learning tasks with few samples, but can also be extended to various visual language fusion scenarios such as cross-modal retrieval, image understanding, and video analysis, and has high application value and engineering feasibility.

[0096] In summary, by introducing a bimodal cue interaction and dynamic fusion mechanism, this invention significantly improves cross-modal feature representation and incremental learning performance while ensuring efficient model training. In particular, it demonstrates stronger adaptability and generalization ability under small sample and multi-task conditions, which has important research significance and practical application potential. Attached Figure Description

[0097] Figure 1 A flowchart illustrating an incremental learning method based on prompting and multimodal fusion, provided in an embodiment of the present invention;

[0098] Figure 2 A schematic diagram of the overall framework of BMC-CLIP provided in an embodiment of the present invention;

[0099] Figure 3 A schematic diagram of the lightweight fusion module BPGF provided in an embodiment of the present invention;

[0100] Figure 4This is a schematic diagram of the Prompt Collaborative Filtering (BPCF) mechanism provided in an embodiment of the present invention. Detailed Implementation

[0101] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0102] Example 1:

[0103] First, we describe the problem and define the notation for few-shot incremental learning. In the few-shot incremental learning framework, given a labeled training set corresponding to the incremental stage sequence... , of which The stage training set is defined as Here, Indicates the first The number of training samples provided in each stage. This is the final incremental stage. Specifically: The base class training set contains a large number of samples, belongs to the base class stage, and satisfies... ;when hour, The training set is a small sample set for the new class, which is in the incremental stage and satisfies... ( For small sample sizes, usually ).make Indicates training set For the corresponding set of category labels, FSCIL must satisfy the following constraints:

[0104] (1) Mutual exclusion of categories: ;

[0105] (2) Base class dominance: The base class size is significantly larger than the new class, i.e. ;

[0106] (3) New class balance: The size of new classes is consistent in each incremental stage, that is ;

[0107] For the evaluation phase, the only requirement is to take into account the current incremental phase. We'll use all the classes encountered previously to calculate incremental phase performance. Consider a labeled evaluation set. and model Then the stage can be calculated as follows: The accuracy of the assessment: ;

[0108] Among them is The number of evaluation examples It is an indicator function.

[0109] On the one hand, this invention provides a class of incremental learning methods based on prompting guidance and multimodal fusion, such as... Figure 1 As shown, it includes the following steps:

[0110] Step 1: Construct semantically enhanced text representations; To address the problem of insufficient semantic expression of categories, the category labels are extended based on a language model to generate text descriptions containing visual attributes and semantic features; The extended text sequence is input into a text encoder, and multi-layer semantic modeling is performed through learnable general text task prompts and text task-specific prompts to obtain discriminative text feature vectors; This process can use prior information from natural language to supplement the deficiencies of visual features and provide high-level semantic constraints for subsequent cross-modal fusion.

[0111] Specifically, while incremental learning methods based on the CLIP architecture introduce a text modality, their text input is often limited to simple labels (e.g., "apple") or basic descriptive phrases (e.g., "a photo of {label}"). This approach fails to fully utilize the deep semantic features in the text modality and also limits the full potential of the CLIP model's semantic reasoning ability. Therefore, this study proposes a simple improvement method: enriching the text input by adding Visual Characteristics Description (VCD) to the category labels. For example, expanding the description of "apple" to "apple: round shape with smooth red or green skin and a small stem." This enhanced text description not only provides additional background knowledge and explanatory information but also helps the model understand image content more accurately in incremental learning scenarios with only a small number of samples. The text encoder introduced... A learnable general prompt for text tasks. and A learnable text task-specific prompt. ,in , It is the embedding dimension of the text encoder; task-general cues are shared across different tasks and are present in the text encoder of the CLIP model. The CLIP text encoder uses general cues for knowledge transfer, while task-specific cues are selected from a pool of task-specific cues and applied in subsequent layers after layer k+1. This strategy leverages general cues to facilitate knowledge transfer while also improving performance for specific tasks through specific cues.

[0112] The specific process of the text encoder is as follows: First, the GPT-2 model is used to generate enhanced visual feature descriptions for the category labels, and these descriptions are passed as input to the text encoder. Then, the tokenizer of the CLIP model is used to process the visual feature descriptions, converting them into text tokens. The tokens are mapped to corresponding embedding vectors through the embedding layer. To effectively preserve the order information of the tokens, positional encoding is added to the generated embedding vectors. The embedding vectors with positional information are... General hints for text tasks By splicing, the resulting form is: The sequence, where This represents the set of general text task hints, with a total of L general text hints. a indivual, and These represent the first and last general text prompts, respectively. The sequence is fed into the Transformer encoding layer of the text encoder for processing. The normal Transformer encoding layer only processes... In our method, and They will be processed together by the Transformer encoding layer, and defined. For the text encoder's first The forward propagation process of the first layer is shown in formula (1):

[0113] (1);

[0114] For up to the The subsequent layers of the layer are used as general knowledge extraction modules for the task. The forward propagation process of the general knowledge extraction module for the task is shown in formula (2):

[0115] (2);

[0116] in, This indicates a vertical concatenation operation. The underscore (_) indicates that the output at that position is discarded and not included in subsequent calculations. It is the first The layer generates latent state output based on category word embeddings, while Then it means the first A set of common prompts for all learnable text tasks within a layer; during propagation, the input... The output embedding will be discarded and will not be passed to the next layer; each The subsequent Transformer layer serves as the task-specific knowledge extraction module; the task-specific knowledge extraction module uses filtered text task-specific prompts to model task specificity, as shown in formula (3):

[0117] (3);

[0118] Where L represents the total number of layers in the text encoder. This represents the set of task-specific cues for the learnable text obtained through filtering in the i-th layer. During encoding, the semantic information of the input sentence is progressively abstracted through layer-by-layer propagation in the Transformer network. Finally, the cues from the last hidden state that correspond to the end-of-line character are selected. corresponding vector As a global semantic representation of the entire sentence, the global semantic representation vector It will be projected into a lower-dimensional space, thus forming the text feature representation h of the entire sentence. text As shown in formula (4):

[0119] (4);

[0120] in This represents a linear projection layer that processes text features;

[0121] In this embodiment, as shown Figure 2 As shown, this invention employs a semantically enhanced prompt generation strategy on the text side of the model to fully mine the high-level semantic information contained in the category labels, thereby improving the model's discrimination ability under small sample conditions. Specifically, it includes the following sub-steps:

[0122] Step 1.1: Category Semantic Expansion: For each target category label, a pre-trained language model is used to automatically generate descriptive phrases containing visual attributes, structural features, and hyponymous semantic relationships. This description expresses the key features of the category in natural language form; for example, the label "bird" is expanded to "a small animal with feathers, wings, and a beak that can fly." In this way, the text input not only includes the category name but also introduces a semantic context that guides the extraction of visual features.

[0123] Step 1.2: Text Encoding and Hint Injection: The expanded text description is input into the text encoder. Several learnable task-general and task-specific hints are concatenated at the beginning of the input sequence to construct an input sequence with adjustable semantic expression capabilities. The text encoder adopts a multi-layer Transformer structure and uses a self-attention mechanism to model the features of the hint vector and category description, extracting global semantic information and generating a high-dimensional semantic embedding representation.

[0124] Step 1.3: Semantic Feature Extraction and Representation Output: In the hidden state of the last layer output of the text encoder, the feature vector corresponding to the terminator [EOS] is selected as the sentence-level global representation and mapped to a unified embedding space through a linear mapping layer to form the final text feature representation. This representation serves as the semantic benchmark for subsequent visual feature alignment and cross-modal fusion.

[0125] Through the above steps, this invention can introduce learnable prompt structures and rich semantic prior information at the language level, enabling the model to have stronger semantic expression and task adaptation capabilities, laying the foundation for subsequent cross-modal feature fusion.

[0126] Step 2: Extract visual features and establish a cross-modal embedding space; feature extraction is performed on the input image using a pre-trained visual encoder to obtain a global visual embedding vector; subsequently, the global visual embedding vector is mapped to a unified cross-modal representation space, aligning with the text features in the same dimension. This process ensures consistency in the semantic spaces of the two modalities, laying the foundation for dynamic interaction between modalities.

[0127] Introduction of pre-trained visual encoders A learnable general prompt for image tasks (Image Task - General Prompt) and Image Task-Specific Prompt ,in , This represents the embedding dimension of the image encoder; image task-specific cues and text task-specific cues are combined through a gating fusion mechanism to form a mixed task-specific prompt pool. The mixed-task-specific cue pool, after undergoing a dual-modal collaborative filtering mechanism, yields cuees adapted to the current task. ,in Indicates the first option selected from the suggestion pool. A tip.

[0128] The specific process of the image encoder is as follows: First, the image undergoes block embedding processing and is coupled with a... The tags are combined to form an embedding vector. Then, the embedding vector General tips for image tasks Vertical splicing constructs a sequence ,in This represents a set of general cues for the image task; the sequence is then fed into the Transformer coding layer in the image encoder for further processing; [Definition] For the first in the image encoder The forward propagation process of the first layer is shown in formula (5):

[0129] (5);

[0130] For until the The forward propagation process of the subsequent layers is shown in Equation (6):

[0131] (6);

[0132] The CPE-CLIP study found that employing different strategies to propagate cues across hierarchical layers has significant benefits. Following this line of thought, a cumulative approach is adopted in this phase for processing general cues for image tasks. Specifically, the first... The output of the layer prompts will accumulate to the 1st layer. In subsequent Transformer layers, specific hints will be provided based on the filtered hybrid task-specific prompts adapted to the current task. ,in Represents a set of hybrid task-specific prompts The first in A cue vector is used to model the features of the image for the current task category, as shown in formula (7):

[0133] (7);

[0134] in, Indicates the first The set of learnable hybrid task-specific cues obtained from filtering in each incremental task; after processing by the image encoder, the cues from the last hidden state that match... corresponding vector To represent the entire image, a vector It is projected into a lower-dimensional space, thereby generating a visual feature representation h of the entire image. img As shown in formula (8):

[0135] (8);

[0136] in This represents a linear projection layer that processes image features;

[0137] In this embodiment, visual features are extracted and a cross-modal embedding space is established. A pre-trained visual encoder is used to extract features from the input image to obtain a global visual embedding vector, such as... Figure 2 As shown;

[0138] Step 2.1: Image segmentation and embedding generation: Denote the input image as... Where H and W represent the image height and width, respectively. The image is divided into image patches of size p × p, resulting in... Each image patch is mapped to a feature vector through a linear projection layer. Stack all image blocks sequentially to form the initial image sequence. A global classification marker [CLS] is added to the front of the sequence, and a learnable positional encoding is superimposed. This yields the input sequence;

[0139] ;

[0140] Step 2.2: Task-Specific Hint Injection: To enhance the model's transferability across different tasks, a hint is added to the beginning of the input sequence. Learnable Image Task-General Prompts:

[0141] ;

[0142] Concatenating it with the input sequence yields the updated sequence:

[0143] ;

[0144] This approach allows the model to learn cross-task shared visual semantic features during the shallow encoding stage.

[0145] Step 2.3: Inter-layer cue accumulation and feature encoding: The visual encoder uses... In a layered Transformer architecture, the forward propagation of each layer is defined as follows:

[0146] ;

[0147] Here, MSA represents multi-head self-attention mechanism, and LN represents layer normalization operation. In the... After the layer outputs, the output of the general prompt is added as a residual connection to the input of the next layer, so that the prompt information accumulates in deeper layers:

[0148] ;

[0149] This structure enables inter-layer delivery and deep fusion of task-independent prompts;

[0150] Step 2.4: Task-Specific Cue Injection and Enhanced Representation: In the higher-level stages of the encoder (typically the first step) (layers and after), introduce Image Task-Specific Prompts:

[0151] ;

[0152] By concatenating it to the beginning of the previous layer's output sequence, we obtain: ;

[0153] Task-specific cues can be adaptively updated based on the current task distribution during model training, enhancing the ability to represent new category features.

[0154] Step 2.5: Global Feature Extraction and Projection Mapping: Extract the feature vector corresponding to the [CLS] marker in the output sequence of the last layer of the visual encoder. via linear mapping matrix Projecting onto the cross-modal embedding space yields a global representation of the visual modality:

[0155] This feature is semantically similar to textual features. Alignment makes the two modes comparable in the same space.

[0156] Through the above steps, this invention achieves hierarchical feature extraction from local image patches to global semantics on the visual modality side, and completes task-adaptive feature enhancement through inter-layer interaction between general and specific cues. The obtained visual features... Text features Sharing a unified embedding dimension provides basic semantic support for subsequent bimodal fusion and collaborative optimization.

[0157] Step 3: Implement bimodal gated fusion; design a lightweight fusion module BPGF (Bimodal Prompt Gated Fusion) based on a gated attention mechanism to adaptively allocate weights for text-based task-specific cues and image-based task-specific cues according to input features; specifically, a learnable gated weight generation network generates fusion weight coefficients, and the different modal cues are weighted and summed to obtain the fused hybrid task-specific cues M; this mechanism enables the model to flexibly adjust the modal contribution ratio at different task stages, thereby achieving fine-grained cross-modal information fusion at the semantic level.

[0158] Specifically, given text task-specific prompts Image task-specific cues ,in It is the number of text-specific prompts. and represent the embedding dimensions of the text encoder and the image encoder, respectively; before cue fusion, the text task-specific cues are projected onto the same dimensions as the image task-specific cues according to formula (9): (9);

[0159] The fusion mechanism of the gated weight generation network is calculated as follows: the text task-specific cue and the image task-specific cue are concatenated in the last dimension to obtain the joint cue. ,in Joint reminder The input is fed into a gated weight generation network, which consists of a linear layer and a sigmoid activation function to generate gated weights. The calculation process is shown in formula (10): (10);

[0160] in, It is the weight matrix of the linear layer. It is a bias term. It's the Sigmoid function, which restricts the weights to the range [0,1]; the generated gated weights are used. The text-based task-specific cues and image-based task-specific cues are weighted and summed according to formula (11) to obtain the fused hybrid task-specific cues. :

[0161] ;

[0162] in, This indicates element-wise multiplication.

[0163] In this embodiment, as shown Figure 3 As shown, the bimodal cue gating fusion module adaptively allocates fusion weights for visual and textual cues based on the semantic distribution of the input features, thereby achieving dynamic complementarity of cross-modal information at the feature layer. Specifically, it includes the following sub-steps:

[0164] Step 3.1: Feature Input and Dimension Alignment: Denote the text task-specific cue representation obtained in Step 1 as... ;

[0165] The image task-specific cue obtained in step 2 is denoted as... ;

[0166] in, and These represent the number of text prompts and image prompts, respectively. and These represent the respective feature dimensions. Since the feature dimensions of the two modes may be inconsistent, a linear transformation matrix is ​​first used... Perform a projection transformation on the text prompt to make it have the same feature dimension as the image prompt: ;

[0167] To ensure numerical stability, the transformed cue matrix is ​​normalized row by row:

[0168] ;

[0169] in, This represents the L2 normalization operation of a vector.

[0170] Step 3.2: Joint Cue Construction: Concatenate the normalized text cue and image cue according to the sample dimension to form a joint cue representation: ;

[0171] This joint representation is used to calculate cross-modal fusion weights and serves as the input for subsequent gating mechanisms.

[0172] Step 3.3: Gated Weight Generation: A feedforward gated weight generation network is used to learn the weight distribution of different modal features in the fusion process. Let the parameters of the gated weight generation network be: ;

[0173] The formula for calculating the gated output is: ;

[0174] in, It is the Sigmoid activation function. This represents a vector of all ones with the same dimension as the input. The output gating matrix. This indicates the weight ratio of each semantic channel in the fusion process.

[0175] Step 3.4: Weighted Fusion and Cue Output: To ensure a one-to-one fusion of the two modal cues, the first two cues from the text cues and the image cues are taken. Each pair is paired. A weighted calculation is performed on each pair to obtain the fusion suggestion set.

[0176] ;

[0177] in, This represents element-wise multiplication. The merged hint set can be written as:

[0178] ;

[0179] Step 3.5: Regularization and Output Stability: To prevent the gating weights from being overly biased towards a certain mode, a temperature coefficient is introduced. Control the smoothness of the Sigmoid function and apply L2 regularization to the weight parameters:

[0180] ;

[0181] in This is the regularization balance coefficient. This term is added to the total loss as an additive to improve the numerical stability of the fusion process.

[0182] Through the above sub-steps, this step achieves dynamic weighting and adaptive fusion of text and image prompts within a unified embedding space, resulting in a fused prompt. By integrating complementary information from visual and linguistic modalities, it possesses stronger semantic discrimination capabilities and task adaptability, providing high-quality input for the next step of collaborative filtering.

[0183] Step 4: Perform bimodal prompt collaborative filtering; based on gated fusion, a semantic similarity-based prompt collaborative filtering mechanism BPCF (Bimodal Prompt Collaborative Filtering) is proposed. By calculating the cosine similarity between the hybrid task-specific prompts after L2 normalization and the textual and image cue features, the optimal set of prompts most relevant to the current task is automatically selected. This mechanism can dynamically select the optimal combination of prompts based on the semantic distribution of the task, suppress redundant prompts, and ensure that the model focuses on key features in the incremental stage.

[0184] Incremental learning phase Assuming the current batch of image data is [size missing] The corresponding image input batch is denoted as Define text input batches , Size is the incremental stage and the total number of categories encountered in all previous tasks. ,in It contains text descriptions of all categories from both the current and historical tasks; firstly, it utilizes the pre-trained CLIP model's text encoder and image encoder to extract text descriptions from... and Extract the aggregated representation of the last hidden state as a textual clue. and image clues Specifically, as shown in formula (12):

[0185] (12);

[0186] in, This represents the text encoder in the pre-trained CLIP model. This represents the image encoder in the pre-trained CLIP model. This represents the aggregate output of the encoder's last hidden state; for a text encoder, it corresponds to the end-of-line character. The hidden state vector, for the image encoder, corresponds to the global label. The hidden state vector.

[0187] Then, regarding the text clues... and image clues The hybrid task-specific cue M obtained in step 3 is subjected to L2 normalization to eliminate the influence of feature scale, as shown in formula (13):

[0188] (13);

[0189] in This means projecting textual cues onto the same dimension as image cues. This indicates L2 normalization. , and These represent the text cue features, image cue features, and hybrid task-specific cue features after L2 normalization, respectively.

[0190] According to formula (14), the cosine similarity between the hybrid task-specific cue and the text cue features and image cue features is calculated:

[0191] (14);

[0192] in, and These represent the text similarity matrix and the image similarity matrix, respectively.

[0193] Finally, the text similarity matrix and the image similarity matrix are fused using mean fusion. This process can be viewed as bimodal collaboration to generate a comprehensive similarity score. The specific calculation method is shown in formula (15):

[0194] (15);

[0195] in, and These represent the batch sizes for text input and image input, respectively. Finally, based on formula (16) and the comprehensive similarity score, the most relevant prompts to the current task are selected from the prompt pool. A tip :

[0196] (16);

[0197] in Used to select the most similar One index, This indicates that the corresponding prompt is in the prompt pool. The set of indices in the [database]. The prompts integrate information from both text and image modalities, along with the rich prior knowledge inherent in the pre-trained CLIP model. This allows the model to learn more task-specific features from small samples, effectively addressing the challenge of insufficient samples. The selection process for text-specific prompts is similar to that for mixed-task-specific prompts, but filtering is based solely on the similarity calculated with the text cues.

[0198] like Figure 4 As shown, after obtaining the fused prompts, this step uses a bimodal prompt collaborative filtering module to select the set of prompts most relevant to the current task based on semantic similarity within a unified embedding space. This reduces redundant information and improves the discriminativeness and stability of the prompts. Specifically, it includes the following sub-steps:

[0199] Step 4.1: Input and Feature Acquisition: Denote the fusion cue set output from Step 3 as...

[0200] ;

[0201] in, Indicates the number of fusion prompts. The feature dimension is defined as follows. The frozen CLIP pre-trained model is used to extract image semantic cues and text semantic cues related to the task category, denoted as:

[0202] ;

[0203] in, and These are visual and text encoding functions, respectively, and their output vectors are both in a unified semantic space. middle.

[0204] Step 4.2: Feature Normalization and Similarity Calculation: To eliminate scale differences between modalities, L2 normalization is performed on the cue set and semantic cues.

[0205] ;

[0206] in Then, the cosine similarity matrices between the clues and the image cues, and the text cues, were calculated respectively: ;matrix ,in This represents the number of task categories.

[0207] Step 4.3: Collaborative Score Fusion: Weighted average of similarity information from visual and linguistic modalities is performed to obtain a bimodal comprehensive similarity matrix:

[0208] ;

[0209] in, These are modal weighting coefficients, used to control the proportion of contribution from visual and linguistic similarity. Each row... Indicates the first The correlation between each cue and the semantic center of each category.

[0210] Step 4.4: Optimal Hint Selection: To focus on the key semantics of the task, the scores of each row are sorted, and the top suggestions are selected. The optimal set of prompts consists of several prompts: ;

[0211] in The operation selects the corresponding subset of prompts based on the fusion score. This set retains the most relevant hints to the current task semantics while filtering out low-relevance or redundant hints, thereby improving the model's generalization ability and feature utilization efficiency.

[0212] Step 4.5: Output and Interface: The optimal set of hints output in this step. This will serve as input for subsequent incremental training phases, guiding the model to optimize parameters on new task data. To further maintain the discriminative nature of the cues, orthogonal constraints can be applied to the selected cues during the training phase, ensuring they remain linearly independent in the semantic space.

[0213] Through the above sub-steps, this step implements a cross-modal collaborative selection mechanism based on fused prompts, enabling the model to automatically select the set of prompts with the highest discriminative power and task relevance. This maintains the compactness and stability of the feature space in multi-task sequence learning, providing optimized input for subsequent incremental learning and parameter updates.

[0214] Step 5: Perform few-shot incremental learning and parameter optimization based on bimodal fusion; in each incremental stage, freeze the text encoder and image encoder backbone networks of the pre-trained CLIP model, and only update the general prompts for the text task. Text-based task-specific prompts General tips for image tasks Image task-specific prompts The weights of the fusion layer are optimized by jointly improving the cue learning and bimodal fusion module through cross-entropy loss and orthogonal cue protection loss. This ensures that task-specific cues maintain semantic independence at different stages, thereby reducing knowledge interference between new and old tasks. This process achieves efficient continuous learning of parameters while effectively mitigating catastrophic forgetting.

[0215] The incremental learning of few-shot classes based on dual-modal fusion specifically includes the following steps:

[0216] Step s1: Input image data Text description of the corresponding category The text features are extracted from the input text encoder and image encoder, respectively. With visual features ;

[0217] General hints for text encoders to concatenate text in the input sequence With text task-specific prompts Text features are obtained after multi-layer Transformer encoding. :

[0218] ;

[0219] in It is a linear projection matrix. This is the global representation of the last hidden state.

[0220] The image encoder adds general task hints after image patch embedding. Through interlayer accumulation and image task-specific cues After enhancement, visual features are obtained. :

[0221] ;

[0222] in It is a linear mapping matrix. This represents the global features of the final layer of the visual encoder.

[0223] Step s2: Select the task-specific text prompts obtained in step s1. Image task-specific cues The input is fed into the lightweight fusion module BPGF based on the gated attention mechanism for adaptive weighted fusion of features between modalities;

[0224] Step s2.1: First, project the text task-specific cue onto the same dimension as the image cue:

[0225] ;

[0226] in It is a linear projection matrix;

[0227] Step s2.2: Combine the projected text task-specific cues with the image task-specific cues to obtain a joint cue sequence;

[0228] ;

[0229] The input gate weight generation network is used to calculate the gate weights via a linear layer and a sigmoid function:

[0230] ;

[0231] in , For network parameters, For the Sigmoid function, This indicates a task-specific cue from the projected text. Image task-specific cues A combined prompt formed by piecing together elements.

[0232] Step s2.3: Based on the gate weights calculated in step s2.2, the two modal cues are weighted and fused to obtain hybrid task-specific cues:

[0233]

[0234] in This indicates element-wise multiplication. (This is a task-specific hint after merging.) Feature modeling will be used for the incremental phase.

[0235] Step s3: Develop the hybrid task-specific hints output in step s2. The input is fed into the semantic similarity-based prompt collaborative filtering mechanism BPCF, which selects the optimal prompt set based on task features.

[0236] Step s3.1: Extract image cues using the frozen CLIP model With textual clues :

[0237] ;

[0238] in, and These represent the frozen CLIP text encoder and image encoder, respectively.

[0239] Step s3.2: For , , Perform L2 normalization and calculate cosine similarity:

[0240] ;

[0241] in, This represents the hybrid task-specific cue matrix obtained in step s2. and These represent the text cue feature matrices and image cue feature matrices extracted by the frozen CLIP text encoder and image encoder, respectively. This represents the matrix transpose operation. and These represent the similarity score matrices between task-specific cues and textual or image cues, respectively.

[0242] Step s3.3: Obtain the bimodal collaborative score S by combining text and image similarity. fusion : ;

[0243] Select the highest score These prompts form the optimal set of prompts: ;

[0244] This set is used for subsequent task feature modeling and classification reasoning.

[0245] Step s4: Select the optimal hint set from step s3. With visual features The classification layer of the input model is concatenated, and category prediction is achieved through similarity matching. In each incremental stage, the model only updates the cue parameters and the gating fusion layer, and freezes the backbone network to ensure knowledge preservation and parameter efficiency.

[0246] Step s5: Hint at using a joint loss function during the learning and training of the bimodal fusion module:

[0247] in, Cross-entropy loss is used to optimize classification accuracy. To protect against loss through orthogonal cues, the independence between task-specific cues is constrained: ;

[0248] in This represents the matrix transpose operation. This represents the identity matrix with the same dimensions as the suggested features. It is the Frobenius norm. The balancing coefficient is used. Joint optimization ensures that the prompts for each task remain orthogonal in the semantic space, reduces cross-task interference, and improves incremental stability.

[0249] Step s6: After training is complete, input the new task samples into the model, calculate the similarity with the text prototypes of each category using the fused features, and output the prediction results. This stage allows for adaptation to new categories without retraining the backbone network, achieving efficient continuous learning and cross-task inference.

[0250] The training and optimization process of the few-shot incremental learning method based on bimodal fusion described in this embodiment is as follows:

[0251] d1: The cue parameters obtained from the training in the previous task phase Parameters of the gating fusion layer Used as the initial parameters for the current task, and the new task dataset. Input model. The model only updates gradients for the cue layer and gating layer, while freezing the backbone parameters of the pre-trained visual encoder and text encoder, thereby achieving efficient task transfer and knowledge preservation of parameters.

[0252] d2: Input image data and text data Visual features are extracted by passing the text encoder and the visual encoder respectively. and text features The optimal set of prompts obtained in step 4 will be used for filtering. Concatenate the resulting data with the input sequence to obtain the enhanced embedding:

[0253] ;

[0254] After multiple layers of self-attention interaction, the bimodal aligned fusion features are obtained:

[0255] ;

[0256] d3: Calculate the similarity matrix between the fused features and the semantic prototypes of each category. Assume the current task includes... There are several categories, and their semantic text prototypes are: The predicted score is:

[0257] ;

[0258] After softmax normalization, the class prediction probability distribution is obtained:

[0259] ;

[0260] d4: Calculate the classification loss function based on the prediction results and the true labels. :

[0261] ;

[0262] in The number of task samples. The true label is one-hot encoded. To maintain the independence between different task cues, a cue orthogonality constraint is introduced:

[0263] ;

[0264] in It is the identity matrix. This represents the Frobenius norm.

[0265] d5: Combining the above two points, construct the total loss function:

[0266]

[0267] in This is the balancing coefficient, used to control the weight ratio of orthogonal constraints in the total loss.

[0268] d6: Update the prompt parameters and gating fusion layer parameters separately using the backpropagation algorithm:

[0269] ;

[0270] in is the learning rate. Since the visual and text encoders are frozen, their parameters remain unchanged during optimization.

[0271] d7: In the mission After training is completed, the updated set of prompt parameters is saved as the initialization parameters for the next task stage, so as to achieve a smooth transition and knowledge inheritance in the continuous learning process.

[0272] Through the above steps, this invention adjusts only a few parameters of the prompting layer and the gating layer at each task stage, which can quickly adapt to new tasks while maintaining the performance of existing tasks. This effectively alleviates the catastrophic forgetting problem in incremental learning and significantly improves the parameter efficiency and generalization ability of the model.

[0273] Step 6: Generate classification prediction results; During the inference stage, the image and text features fused by dual-modal fusion are input into the classification layer of the model, and the target category prediction is obtained by calculating the similarity. This process can complete the recognition of new and old task categories without retraining the backbone network, and has high generalization and scalability.

[0274] The method provided in this embodiment calculates a similarity matrix based on the visual features of the input image and the semantic features of the text descriptions for each category during the model inference stage, and selects the category corresponding to the highest score as the predicted output. The model does not need to retrain the backbone network at this stage; it can complete task discrimination and result output solely based on the optimized cue parameters and fusion layer weights, achieving a lightweight and efficient inference process. The few-shot incremental learning method based on bimodal fusion proposed in this invention can establish a stable semantic alignment relationship between visual and linguistic modalities. Through semantically enhanced text representation, cue-gated fusion, and collaborative filtering mechanisms, this method exhibits high feature discrimination ability and generalization performance under multi-task, incremental learning, and few-shot conditions. The core innovation of this invention lies in combining the cue structure with a cross-modal fusion strategy to form a learning framework that can adaptively adjust modal weights and is task-sensitive. This framework not only maintains the knowledge transfer capability of the pre-trained model but also improves learning efficiency and robustness under new task conditions, providing an efficient and feasible solution for multimodal fusion and few-shot learning.

[0275] Example 2:

[0276] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0277] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the incremental learning method based on prompting and multimodal fusion described in various embodiments of this application.

[0278] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes, on which computer programs are stored. When the computer program is executed by a processor, it can implement the various steps of the aforementioned incremental learning method based on prompting and multimodal fusion.

[0279] Example 3:

[0280] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned incremental learning method based on prompting and multimodal fusion.

[0281] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0282] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0283] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the methods disclosed herein and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A class incremental learning method based on prompt guidance and multi-modal fusion, characterized in that, Includes the following steps: Step 1: Construct semantically enhanced text representations; expand category labels based on language models to generate text sequences containing visual attributes and semantic features; input the expanded text sequences into a text encoder, and perform multi-layer semantic modeling through learnable general text task prompts and text task-specific prompts to obtain discriminative text feature vectors; Step 2: Extract visual features and establish a cross-modal embedding space; extract features from the input image using a pre-trained visual encoder to obtain a global visual embedding vector; Subsequently, the global visual embedding vector is mapped to a unified cross-modal representation space, aligning with the text features in the same dimension; The pre-trained visual encoder introduces one learnable image task-agnostic hint and one image task-specific hint wherein , denotes the embedding dimension of the image encoder. Step 3: Implement dual-modal gated fusion; design a lightweight fusion module BPGF based on gated attention mechanism, and adaptively allocate weights for text task-specific cues and image task-specific cues according to input features; Specifically, a learnable gating weight generation network is used to generate fusion weight coefficients, and weighted summation is performed on different modal cues to obtain the fused hybrid task-specific cue M. Step 4: Perform bimodal cue collaborative filtering; A prompt collaborative filtering mechanism BPCF based on semantic similarity is proposed, and the cosine similarity between the mixed task-specific prompt after L2 normalization and the text clue feature and the image clue feature is calculated to automatically filter the optimal prompt set most relevant to the current task ; Wherein, the pre-trained CLIP model is used to extract the aggregated representation of the last hidden state from the text input batch and the image input batch respectively as the text clue and the image clue, and then the L2 normalization is performed on the text clue and the image clue and the mixed task-specific prompt M to obtain the L2 normalized mixed task-specific prompt, the text clue feature and the image clue feature. Step 5: Small sample class incremental learning and parameter optimization based on dual-modal fusion are performed; in each incremental stage, the text encoder and image encoder backbone network of the pre-trained CLIP model are frozen, and only the text task general prompt , text task specific prompt , image task general prompt , image task specific prompt and fusion layer weights are updated to the input image data and text data extract visual features and text features ; the optimal prompt set filtered in step 4 is spliced into the input sequence to obtain an enhanced embedding: After multi-layer self-attention interaction, a fusion feature aligned with the double modalities is obtained, and the prompt learning and double-modal fusion module are jointly optimized by cross-entropy loss and orthogonal prompt protection loss. The joint optimization of the cross-entropy loss and orthogonal cue protection loss is specifically achieved through a joint loss function: ； wherein, is the cross-entropy loss, is the orthogonal prompt protection loss: ； wherein denotes a matrix transpose operation, denotes an identity matrix of the same dimension as the prompt feature dimension, is the Frobenius norm, is a balancing coefficient; Step 6: Generate classification prediction results; During the inference stage, the image and text features fused by dual-modality fusion are input into the classification layer of the model, and the target category prediction is obtained by calculating the similarity between the fused features and the text prototypes of each category.

2. The incremental learning method based on prompting and multimodal fusion according to claim 1, characterized in that, Step 1 specifically involves: enriching the text input by adding visual feature descriptions to the category labels, and introducing a text encoder. A general hint for a learnable text task and A learnable text task-specific cue ,in , It is the embedding dimension of the text encoder; task-general cues are shared across different tasks and are placed before the text encoder. Layers are used, while task-specific cues are filtered from the task-specific cue pool and applied in subsequent layers after the (k+1)th layer of the text encoder. The specific process of the text encoder is as follows: First, the GPT-2 model is used to generate enhanced visual feature descriptions for the category labels, and these descriptions are passed as input to the text encoder. Then, the tokenizer of the CLIP model is used to process the visual feature descriptions, converting them into text tokens. The tokens are mapped to corresponding embedding vectors through the embedding layer. Position encoding is added to the generated embedding vectors, resulting in embedding vectors with positional information. General hints for text tasks By splicing, the resulting form is: The sequence, where This represents the set of general text task hints, with a total of L general text hints. a indivual, Indicates the Lth a A general text prompt, the sequence is fed into the Transformer encoding layer of the text encoder for processing, defining... For the text encoder's first The forward propagation process of the first layer is shown in formula (1): （1）； For the first For the subsequent layers of the layer, it is taken as the task general knowledge extraction module, and the forward propagation process of the task general knowledge extraction module is shown in formula (2): （2）； in, This indicates a vertical concatenation operation. An underscore ("_") indicates that the output at that position is discarded and not included in subsequent calculations. It is the first The layer generates latent state output based on category word embeddings, while Then it means the first The set of general prompts for all learnable text tasks in the layer, during propagation, the input The output embedding will be discarded and will not be passed to the next layer; each The subsequent Transformer layer serves as the task-specific knowledge extraction module; the task-specific knowledge extraction module uses filtered text task-specific prompts to model task specificity, as shown in formula (3): （3）； Where L represents the total number of layers in the text encoder. This represents the set of task-specific prompts for the learnable text obtained from the filtering in the i-th layer; during the encoding process, the semantic information of the input sentence is gradually abstracted through the layer-by-layer propagation of the Transformer network; finally, the prompt is selected from the last hidden state and the end symbol. corresponding vector As a global semantic representation of the entire sentence, the global semantic representation vector It will be projected into a lower-dimensional space, thus forming the text feature representation h of the entire sentence. text As shown in formula (4): （4）； wherein represents a linear projection layer that processes the text features.

3. The incremental learning method based on prompting and multimodal fusion according to claim 2, characterized in that, The step 2 is specifically: the image task-specific prompt and the text task-specific prompt are combined through a gating fusion mechanism to form a mixed task-specific prompt pool After the mixed task-specific prompt pool is screened through a double-modal collaborative filtering mechanism, a prompt suitable for the current task is obtained , wherein represents the first K prompts selected from the prompt pool ​ The specific process of the image encoder is as follows: First, the image undergoes block embedding processing and is coupled with a... The tags are combined to form an embedding vector. Then, the embedding vector General tips for image tasks Vertical splicing constructs a sequence ,in This represents a set of general cues for the image task; the sequence is then fed into the Transformer coding layer in the image encoder for further processing; [Definition] For the first in the image encoder The forward propagation process of the first layer is shown in formula (5): （5）； For until the The forward propagation process of the subsequent layers is shown in Equation (6): （6）； No. The output of the layer prompts will accumulate to the 1st layer. In subsequent Transformer layers, specific hints will be provided based on the filtered hybrid task-specific prompts adapted to the current task. ,in Represents a set of hybrid task-specific prompts The first in A cue vector is used to model the features of the image for the current task category, as shown in formula (7): （7）； wherein, represents the set of learnable mixed task-specific hints filtered out in the th incremental task; after being processed by the image encoder, the last hidden state corresponding to is selected to represent the whole image, and the vector is projected into a lower-dimensional space to generate the visual feature representation h of the whole image. img As shown in equation (8): （8）； wherein represents a linear projection layer that processes image features.

4. The incremental learning method based on prompting and multimodal fusion according to claim 3, characterized in that, Step 3 specifically involves: providing task-specific prompts in the given text. Image task-specific cues ,in It is the number of text-specific prompts. and represent the embedding dimensions of the text encoder and the image encoder, respectively; before cue fusion, the text task-specific cues are projected onto the same dimensions as the image task-specific cues according to formula (9): （9）； The fusion mechanism of the gated weight generation network is calculated as follows: the text task-specific cue and the image task-specific cue are concatenated in the last dimension to obtain the joint cue. ,in Joint reminder The input is fed into a gated weight generation network, which consists of a linear layer and a sigmoid activation function to generate gated weights. The calculation process is shown in formula (10): （10）； wherein, is a weight matrix of the linear layer, is a bias term, is a Sigmoid function that limits the weights between [0, 1]; using the generated gating weights The text task-specific prompt and the image task-specific prompt are weighted and summed according to formula (11) to obtain a fused mixed task-specific prompt : （11）； wherein represents an element-wise multiplication.

5. The incremental learning method based on prompting and multimodal fusion according to claim 4, characterized in that, Step 4 specifically involves: during the incremental learning phase Assuming the current batch of image data is [size missing] The corresponding image input batch is denoted as Define text input batches , Size is the incremental stage and the total number of categories encountered in all previous tasks. ,in It contains text descriptions of all categories from both the current and historical tasks; firstly, it utilizes the pre-trained CLIP model's text encoder and image encoder to extract text descriptions from... and Extract the aggregated representation of the last hidden state as a textual clue. and image clues Specifically, as shown in formula (12): （12）； wherein, represents a text encoder in the pre-trained CLIP model, represents an image encoder in the pre-trained CLIP model, represents an aggregated output of the last layer hidden states of the encoder, for the text encoder, corresponding to the end-of-sentence hidden state vector, for the image encoder, corresponding to the global token hidden state vector; Then the text cues and image cues and the mixed task-specific prompt M obtained in step 3 are normalized in L2 norm as shown in equation (13): （13）； wherein represents projecting the textual cues into the same dimension as the image cues, represents L2 normalization processing, , and respectively represent the L2 normalized textual cue features, image cue features, and mixed task-specific prompt features. According to formula (14), the cosine similarity between the hybrid task-specific cue and the text cue features and image cue features is calculated: （14）； wherein, and respectively represent the text similarity matrix and the image similarity matrix; Finally, the text similarity matrix and the image similarity matrix are averaged to generate a comprehensive similarity score The specific calculation method is shown in formula (15): （15）； wherein, and respectively represent the text input and image input batch size; finally, according to formula (16), the prompt most relevant to the current task is selected from the prompt pool according to the comprehensive similarity score prompt : （16）； in Used to select the most similar One index, This indicates that the corresponding prompt is in the prompt pool. The set of indices in the database.

6. The class-incremental learning method based on prompt guidance and multi-modal fusion according to claim 5, characterized in that, Step 5, the few-shot incremental learning based on bimodal fusion, specifically includes the following steps: Step s1: input image data with corresponding category text description into a text encoder and an image encoder, respectively extracting text features and visual features ; Text encoder concatenates text task general prompt in input sequence with text task specific prompt , obtains text feature after multi-layer Transformer encoding : ; in It is a linear projection matrix. This is the global representation of the last hidden state. The image encoder adds general task hints after image patch embedding. Interlayer accumulation and image task-specific cues After enhancement, visual features are obtained. : ; in It is a linear mapping matrix. These are the global features of the last layer of the visual encoder; Step s2: Select the task-specific text prompts obtained in step s1. Image task-specific cues The input is fed into the lightweight fusion module BPGF based on the gated attention mechanism for adaptive weighted fusion of features between modalities; Step s3: Develop the hybrid task-specific hints output in step s2. The input is fed into the semantic similarity-based prompt collaborative filtering mechanism BPCF, which selects the optimal prompt set based on task features; Step s4: Select the optimal hint set from step s3. With visual features By concatenating the classification layer of the input model, category prediction is achieved through similarity matching; Step s5: Train the cue learning and bimodal fusion modules using a joint loss function; Step s6: After training is complete, input the new task samples into the model, use the fusion features to calculate the similarity with the text prototypes of each category, and output the prediction results.

7. The incremental learning method based on prompting and multimodal fusion according to claim 6, characterized in that, Step s2 includes the following steps: Step s2.1: First, project the text task-specific cue onto the same dimension as the image cue: ; in It is a linear projection matrix; Step s2.2: Combine the projected text task-specific cues with the image task-specific cues to obtain a joint cue sequence; ； The input gate weight generation network is used to calculate the gate weights via a linear layer and a sigmoid function: ； in , For network parameters, For the Sigmoid function, This indicates a task-specific cue from the projected text. Image task-specific cues A combined prompt representation formed by splicing together; Step s2.3: Based on the gating weights calculated in step s2.2 The two modal cues are weighted and fused to obtain hybrid task-specific cues: ; in Indicates element-wise multiplication; task-specific hints after fusion. Feature modeling will be used for the incremental phase.

8. The incremental learning method based on prompting and multimodal fusion according to claim 6, characterized in that, Step s3 includes the following steps: Step s3.1: Extract image cues using the frozen CLIP model With textual clues : ； in, and These represent the frozen CLIP text encoder and image encoder, respectively. Step s3.2: For , , Perform L2 normalization and calculate cosine similarity: ; in, This represents the hybrid task-specific cue matrix obtained in step s2. and These represent the text cue feature matrices and image cue feature matrices extracted by the frozen CLIP text encoder and image encoder, respectively. This represents the matrix transpose operation. and These represent the similarity score matrices between hybrid task-specific cues and textual and image cues, respectively. Step s3.3: Obtain the bimodal collaborative score S by combining text and image similarity. fusion : ; Select the highest score These prompts form the optimal set of prompts: .

9. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed, cause the processor to perform the incremental learning method based on prompting and multimodal fusion as described in any one of claims 1-8.

10. A computer program product, characterized in that, Includes a computer program or instructions that, when executed by a processor, implement the incremental learning method based on prompting and multimodal fusion as described in any one of claims 1-8.

Images