Small-sample incremental learning system and method based on graph neural network

Abstract

The invention belongs to but not limited to the technical field of graph neural networks, and particularly relates to a few-sample incremental learning system and method based on a graph neural network, and the method comprises the steps: a text encoder and a visual encoder respectively process text and image input, and a graph isomorphic network (GIN) is used for processing text and visual graph structure learnable prompts; an encoder-decoder structure of the ACGA generates an optimized representation. And finally, the feature compression module, the Hamiltonian module and the restoration module generate updated learnable prompts, and feed back the learnable prompts to the text and visual encoder for subsequent learning. The whole process emphasizes fusion and updating of multi-modal information. According to the method, the fusion proportion of the node features and the neighborhood information is dynamically adjusted through the learnable parameters, the modeling capability of a complex graph structure is enhanced, and the feature expression capability under the condition of few samples is improved.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of graph neural network technology, and in particular relates to a few-sample class incremental learning system and method based on graph neural network. Background Art

[0002] Few-Shot Class Incremental Learning (FSCIL) is a key research direction in the field of continuous learning. It aims to enable models to continuously learn new categories using a small number of samples while retaining previously learned knowledge. In dynamic environments, models must continuously learn new information while retaining previously acquired knowledge. The core challenge of FSCIL is how to effectively learn new categories without forgetting old ones given limited samples. Existing technologies have the following shortcomings:

[0003] 1. Imbalance between stability and plasticity: Traditional methods (such as prototype learning and attention mechanisms) struggle to balance the relationship between new and old categories in dynamic incremental scenarios, which can easily lead to catastrophic forgetting. Prototype learning uses feature vectors of a small number of samples to represent category prototypes for classification. However, in complex FSCIL data, precisely defining prototypes is difficult, resulting in poor differentiation between similar categories. Attention mechanisms focus on relevant input data, but interference from new data can disrupt the distribution of attention, leading to forgetting of features from old categories.

[0004] 2. Data scarcity: Existing graph neural network (GNN) methods fail to fully consider the dynamic category increment characteristics under conditions of few samples, making them unable to effectively distinguish similar categories, resulting in reduced classification accuracy. When dealing with data scarcity and dynamic category changes, GNNs often fail to fully capture new category information, resulting in reduced classification accuracy.

[0005] 3. Inadequate dynamic feature modeling: Traditional GNN methods typically use static graph construction techniques, which cannot adapt to the dynamic changes in category relationships. When faced with visually or semantically similar categories, model performance degrades significantly, and it is difficult to capture the relationship between new and old categories that evolves over time, resulting in low knowledge transfer efficiency.

[0006] In view of the above analysis, the technical problems that need to be solved urgently in the existing technology are:

[0007] 1. Imbalance between stability and plasticity: Traditional methods (such as prototype learning and attention mechanisms) struggle to balance the relationship between new and old categories in dynamic incremental scenarios, which can easily lead to catastrophic forgetting. Prototype learning uses feature vectors of a small number of samples to represent category prototypes for classification. However, in complex FSCIL data, precisely defining prototypes is difficult, resulting in poor differentiation between similar categories. Attention mechanisms focus on relevant input data, but interference from new data can disrupt the distribution of attention, leading to forgetting of features from old categories.

[0008] 2. Data scarcity: Existing graph neural network (GNN) methods fail to fully consider the dynamic category increment characteristics under conditions of few samples, making them unable to effectively distinguish similar categories, resulting in reduced classification accuracy. When dealing with data scarcity and dynamic category changes, GNNs often fail to fully capture new category information, resulting in reduced classification accuracy.

[0009] 3. Inadequate dynamic feature modeling: Traditional GNN methods typically use static graph construction techniques, which cannot adapt to the dynamic changes in category relationships. When faced with visually or semantically similar categories, model performance degrades significantly, and it is difficult to capture the relationship between new and old categories that evolves over time, resulting in low knowledge transfer efficiency. Summary of the Invention

[0010] In response to the problems existing in the prior art, the present invention provides a small-sample incremental learning system and method based on graph neural networks.

[0011] The present invention is implemented as follows: a small-sample incremental learning system based on graph neural network, comprising:

[0012] The CLIP encoder, including a text encoder, a visual encoder, and a learnable hint generation unit, is used to construct an embedding space for cross-modal semantic alignment. By jointly training the text and visual encoders, it achieves an aligned mapping between text descriptions and image features, and introduces learnable hints in the text and visual encoders.

[0013] Dynamic graph construction module, used for graph node definition, adjacency matrix generation and node feature update;

[0014] Hamiltonian graph network and energy conservation module, used for state initialization, feature compression and system initialization, Hamiltonian energy function and state update, as well as state recovery and energy conservation constraints;

[0015] The adversarial constrained graph autoencoder module, which includes a graph convolutional encoder, a structure reconstruction decoder, and an adversarial regularization unit, is used to generate latent representations, reconstruct the graph structure, and introduce a discriminator.

[0016] The incremental module is used during the category increment phase. The model receives new category samples and adds their corresponding node features to the dynamic graph. The node features of the old categories remain unchanged, thus preserving historical knowledge and ensuring that the model does not forget the learned categories when learning new categories.

[0017] The training module is used to adopt the total loss function to perform a weighted sum of classification loss, reconstruction loss, adversarial loss and energy loss.

[0018] Furthermore, the CLIP encoder specifically includes:

[0019] 1. Multimodal alignment model: The CLIP encoder, based on contrastive learning, includes a symmetric text and visual encoder based on an improved Transformer structure. Its core is to construct an embedding space for cross-modal semantic alignment. By jointly training the text and visual encoders, it achieves an aligned mapping between text descriptions and image features.

[0020] 2. Learnable Hint Insertion: To enhance the expressiveness and adaptability of the model, learnable hints are introduced into the text and visual encoders. These hints are inserted into text and visual embeddings to guide the model to focus on key features, improve the model's ability to recognize new categories, and reduce catastrophic forgetting.

[0021] 3. Specific structure:

[0022] 1) Text Encoder: This consists of a word embedding matrix, positional encoding, and a multi-layer Transformer block. The input discrete tokens are mapped to continuous vectors. After adding the positional encoding, feature interaction is performed through a multi-head attention mechanism and a feedforward neural network. Finally, semantic projection is applied to obtain a normalized text embedding.

[0023] Word embedding representation: discrete tokens are mapped to continuous variables

[0024] Add position encoding: X←X+P T (n),

[0025] Feature Interaction:

[0026] Semantic Projection:

[0027] 2) Visual Encoder: This is symmetrical in structure to the text encoder, but takes a sequence of image blocks as input. The image blocks are linearly projected and a [CLS] token is added. After passing through the Transformer block, the features corresponding to the [CLS] token are extracted as visual embeddings.

[0028] Input image block sequence, the processing flow is symmetrical: X = [x [CLS] ;EV(p1),...,E V (p m )]+P V (m+1)

[0029] Extract [CLS] features:

[0030] 3) Learnable Cue Generation: Textual cues and visual cues are generated through learnable parameter matrices, respectively, and inserted into the textual and visual embeddings, directly replacing the input of each layer.

[0031] Text prompts each layer insert: X = [X [CLS] , g (l) , X tokens ]

[0032] Visual hint: Each layer is inserted: X = [X [CLS] , g V (k) , X patches ]

[0033] Furthermore, the dynamic graph construction module specifically includes:

[0034] 1. Graph node definition: Using textual and visual learnable cues as nodes, we construct a dynamic graph to capture the semantic associations between textual and visual modalities.

[0035] 2. Adjacency Matrix Generation: The adjacency matrix is ​​constructed by calculating the cosine similarity matrix and applying a sparseness-inducing threshold for binarization. Symmetry and normalization are used to ensure the symmetry of the adjacency matrix, and an attention mechanism can be introduced to enhance the model's expressiveness. The adjacency matrix construction formula is as follows:

[0036] Cosine similarity matrix: sim_matrix i，j =cos(x i , x j )

[0037] Binarization processing:

[0038] Symmetrization and normalization:

[0039]

[0040] Introducing the attention mechanism:

[0041] 3. Node Feature Update: Node feature matrices are updated using a graph isomorphism network (GIN). GIN updates the features of the current node by aggregating the features of neighboring nodes, thereby capturing the complex patterns and dependencies in graph-structured data. Specifically, the feature update process for each node at each layer involves neighbor feature aggregation, combining learnable coefficients, and multi-layer perceptron (MLP) transformations.

[0042] GIN layer formula:

[0043] Neighbor aggregation and feature fusion:

[0044] Furthermore, the Hamiltonian graph network and energy conservation module specifically include:

[0045] 1. State initialization: Receive the node feature matrix and adjacency matrix from the dynamic graph construction module, use the adjacency matrix to aggregate neighbor information, and splice the node features with the aggregated neighbor features to form the initial state.

[0046] aggregated=A·X, state=[X, aggregated];

[0047] 2. Feature compression and system initialization: The initial state is compressed into a low-dimensional space through a linear layer to obtain the compressed feature vector, which is then assigned to the generalized coordinates and generalized momentum of the system at the same time.

[0048] compressed=W compress state+b compress ;

[0049] 3. Hamiltonian Energy Function and State Update: A multilayer perceptron (MLP) is used to simulate the Hamiltonian energy function and calculate the state update rule based on the Hamiltonian equation. The symplectic Euler method is used to numerically solve the state update equation to ensure energy conservation in the system.

[0050]

[0051] 4. State recovery and energy conservation constraints: The updated state is restored to its original dimension through a linear layer to obtain the final node feature update result. To ensure energy conservation, an energy conservation loss function is introduced to calculate the energy difference between the initial and final states and use it as part of the optimization objective.

[0052]

[0053] Energy conservation loss:

[0054] Furthermore, the Adversarial Constrained Graph Autoencoder (ACGA) module specifically includes:

[0055] 1. Graph Convolutional Encoder: An improved Graph Isomorphism Network (GIN) is used as the encoder to fuse node features and topology information through a multi-layer message passing mechanism to generate a latent representation.

[0056] Z (l+1) =GINLayer(Z (l) , A);

[0057] 2. Structure reconstruction decoder: Reconstructs the adjacency matrix through the inner product of the potential representation and uses the Sigmoid function to simulate the edge existence probability to achieve reconstruction of the graph structure.

[0058]

[0059] Reconstruction loss:

[0060] 3. Adversarial Regularization Module: This module introduces a discriminator to force the latent representation to match the prior distribution. The discriminator consists of two fully connected layers and uses the Wasserstein distance as the adversarial loss function to enhance training stability.

[0061]

[0062] Fighting Loss:

[0063] Furthermore, the training module specifically includes:

[0064] The model's training process uses a total loss function, which is a weighted sum of classification loss, reconstruction loss, adversarial loss, and energy loss. By optimizing the total loss function, the model effectively balances the classification task with graph structure regularization, improving its generalization and robustness.

[0065] Classification loss:

[0066] Total loss function:

[0067] Another object of the present invention is to provide a graph neural network-based few-shot incremental learning method using the graph neural network-based few-shot incremental learning system, comprising:

[0068] S1, text and visual encoders process text and image inputs respectively;

[0069] S2, using graph isomorphism networks to process graph-structured learnable cues from text and vision;

[0070] S3, ACGA’s encoder-decoder structure generates optimized representations;

[0071] S4, the feature compression, Hamiltonian module, and restoration module generate updated learnable cues, which are fed back into the text and visual encoders for subsequent learning.

[0072] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the said small-sample incremental learning method based on graph neural network.

[0073] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the graph neural network-based few-sample incremental learning method.

[0074] Another object of the present invention is to provide an information data processing terminal, which includes the aforementioned graph neural network-based few-sample incremental learning system.

[0075] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0076] First, the present invention improves performance through the following improvements:

[0077] 1. Graph Isomorphism Network (GIN): Dynamically adjusts the fusion ratio of node features and neighborhood information through learnable parameters, enhancing the ability to model complex graph structures and improve feature expression capabilities under few-sample conditions.

[0078] 2. Hamiltonian Graph Network Energy Conservation (HGN-GC): Based on Hamiltonian dynamics modeling feature evolution, it stabilizes the training process through energy conservation constraints, reduces feature conflicts between new and old categories, and mitigates catastrophic forgetting.

[0079] 3. Adversarial Constrained Graph Autoencoder (ACGA): Combines adversarial training with topology reconstruction to enforce alignment of latent space distributions, avoid overfitting, and preserve topological relationships between categories.

[0080] 4. Parameter efficiency: Freeze the CLIP backbone network and train only a small number of modules, significantly reducing computing resource requirements.

[0081] Second, although there has been research progress in the field of graph neural networks at home and abroad, there is no technical solution to systematically integrate dynamic feature fusion, Hamiltonian dynamics energy conservation constraints, adversarial training and topology reconstruction, and efficient parameter optimization. In the present invention, the graph isomorphism network (GIN) uses an innovative mechanism to dynamically adjust the fusion ratio with learnable parameters, the Hamiltonian graph network energy conservation (HGN-GC) is a stable training method based on dynamics, the adversarial constrained graph autoencoder (ACGA) combines the unique architecture of adversarial training and topology reconstruction, and the parameter-efficient CLIP backbone network freezing strategy, all of which break through the boundaries of traditional technology in their respective dimensions. The organic combination of these technologies fills the technical gaps in many aspects such as complex graph structure modeling, stable training of dynamic feature evolution, topological relationship maintenance, and low-resource efficient training, opening up new paths for the development of graph neural network technology.

[0082] In the field of graph neural network research, there are two common technical prejudices. First, it is believed that improving model performance must rely on a large amount of training data and complex model structures, ignoring the potential of dynamic feature fusion and optimized training mechanisms; second, it is believed that freezing the backbone network will cause a significant decline in model performance, and people dare not try parameter-efficient optimization strategies. This invention breaks these prejudices. The graph isomorphism network (GIN) achieves efficient feature expression under the condition of few samples through learnable parameters, proving that the optimized fusion mechanism can break through data limitations; Hamiltonian graph network energy conservation (HGN-GC) and adversarial constrained graph autoencoder (ACGA) improve model performance through innovative training methods without over-reliance on complex structures; the parameter efficiency strategy only trains a small number of modules to achieve performance improvement while freezing the CLIP backbone network, proving that parameter optimization can guarantee model effects under resource-constrained conditions, providing a new thinking direction for the development of graph neural network technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a structural diagram of a few-shot incremental learning system based on a graph neural network provided by an embodiment of the present invention;

[0084] Figure 2 This is a flow chart of a few-shot incremental learning method based on graph neural networks provided by an embodiment of the present invention;

[0085] Figure 3 1 is a schematic diagram of the framework of ACHG-CLIP provided by an embodiment of the present invention;

[0086] Figure 4 These are the effect diagrams provided by the embodiments of the present invention; (a) comparison of the effects of different mapping methods; (b) comparison of the effects of adding different modules. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0088] like Figure 1 As shown in the figure, the present invention proposes the "Adversarially Constrained Hamiltonian Graph-CLIP Framework (ACHG-CLIP)", which includes the following core modules:

[0089] (1) CLIP encoder

[0090] 1. Multimodal alignment model: The CLIP encoder, based on contrastive learning, includes a symmetric text and visual encoder based on an improved Transformer structure. Its core is to construct an embedding space for cross-modal semantic alignment. By jointly training the text and visual encoders, it achieves an aligned mapping between text descriptions and image features.

[0091] 2. Learnable Hint Insertion: To enhance the expressiveness and adaptability of the model, learnable hints are introduced into the text and visual encoders. These hints are inserted into text and visual embeddings to guide the model to focus on key features, improve the model's ability to recognize new categories, and reduce catastrophic forgetting.

[0092] 3. Specific structure:

[0093] 1) Text Encoder: This consists of a word embedding matrix, positional encoding, and a multi-layer Transformer block. The input discrete tokens are mapped to continuous vectors. After adding the positional encoding, feature interaction is performed through a multi-head attention mechanism and a feedforward neural network. Finally, semantic projection is applied to obtain a normalized text embedding.

[0094] Word embedding representation: discrete tokens are mapped to continuous variables

[0095] Add position encoding: X←X+P T (n),

[0096] Feature Interaction:

[0097] Semantic Projection:

[0098] 2) Visual Encoder: This is symmetrical in structure to the text encoder, but takes a sequence of image blocks as input. The image blocks are linearly projected and a [CLS] token is added. After passing through the Transformer block, the features corresponding to the [CLS] token are extracted as visual embeddings.

[0099] Input image block sequence, the processing flow is symmetrical: X = [x [CLS] ;EV(p1),...,EV(p m )]+P V (m+1)

[0100] Extract [CLS] features:

[0101] 3) Learnable Cue Generation: Textual cues and visual cues are generated through learnable parameter matrices, respectively, and inserted into the textual and visual embeddings, directly replacing the input of each layer.

[0102] Text prompts each layer insert: X = [X [CLS] , g (l) , X tokens ]

[0103] Visual hint: Each layer is inserted: X = [X [CLS] , gV (l) , X patches ]

[0104] (2) Dynamic graph construction module

[0105] 1. Graph node definition: Using textual and visual learnable cues as nodes, we construct a dynamic graph to capture the semantic associations between textual and visual modalities.

[0106] 2. Adjacency Matrix Generation: The adjacency matrix is ​​constructed by calculating the cosine similarity matrix and applying a sparseness-inducing threshold for binarization. Symmetry and normalization are used to ensure the symmetry of the adjacency matrix, and an attention mechanism can be introduced to enhance the model's expressiveness. The adjacency matrix construction formula is as follows:

[0107] Cosine similarity matrix: sim_matrix i，j =cos(x i , x j )

[0108] Binarization processing:

[0109] Symmetrization and normalization:

[0110]

[0111] Introducing the attention mechanism:

[0112] 3. Node Feature Update: Node feature matrices are updated using a graph isomorphism network (GIN). GIN updates the features of the current node by aggregating the features of neighboring nodes, thereby capturing the complex patterns and dependencies in graph-structured data. Specifically, the feature update process for each node at each layer involves neighbor feature aggregation, combining learnable coefficients, and multi-layer perceptron (MLP) transformations.

[0113] GIN layer formula:

[0114] Neighbor aggregation and feature fusion:

[0115] (3) Hamiltonian Graph Network and Energy Conservation (HGN-GC) Module

[0116] 1. State initialization: Receive the node feature matrix and adjacency matrix from the dynamic graph construction module, use the adjacency matrix to aggregate neighbor information, and splice the node features with the aggregated neighbor features to form the initial state.

[0117] aggregated=A·X,state=[X,aggregated]

[0118] 2. Feature compression and system initialization: The initial state is compressed into a low-dimensional space through a linear layer to obtain the compressed feature vector, which is then assigned to the generalized coordinates and generalized momentum of the system at the same time.

[0119] compressed=W compress state+b compress

[0120] 3. Hamiltonian Energy Function and State Update: A multilayer perceptron (MLP) is used to simulate the Hamiltonian energy function and calculate the state update rule based on the Hamiltonian equation. The symplectic Euler method is used to numerically solve the state update equation to ensure energy conservation in the system.

[0121]

[0122] 4. State recovery and energy conservation constraints: The updated state is restored to its original dimension through a linear layer to obtain the final node feature update result. To ensure energy conservation, an energy conservation loss function is introduced to calculate the energy difference between the initial and final states and use it as part of the optimization objective.

[0123]

[0124] Energy conservation loss:

[0125] (4) Adversarial Constrained Graph Autoencoder (ACGA) Module

[0126] 1. Graph Convolutional Encoder: An improved Graph Isomorphism Network (GIN) is used as the encoder to fuse node features and topology information through a multi-layer message passing mechanism to generate a latent representation.

[0127] Z (l+1) =GINLayer(Z (l) ,A)

[0128] 2. Structure reconstruction decoder: Reconstructs the adjacency matrix through the inner product of the potential representation and uses the Sigmoid function to simulate the edge existence probability to achieve reconstruction of the graph structure.

[0129]

[0130] Reconstruction loss:

[0131] 3. Adversarial Regularization Module: This module introduces a discriminator to force the latent representation to match the prior distribution. The discriminator consists of two fully connected layers and uses the Wasserstein distance as the adversarial loss function to enhance training stability.

[0132]

[0133] Fighting Loss:

[0134] (V) Incremental modules

[0135] During the category increment phase, the model receives new category samples and adds their corresponding node features to the dynamic graph. The node features of the old categories remain unchanged, thus preserving historical knowledge and ensuring that the model does not forget the learned categories when learning new categories.

[0136] (6) Training module

[0137] The model's training process uses a total loss function, which is a weighted sum of classification loss, reconstruction loss, adversarial loss, and energy loss. By optimizing the total loss function, the model effectively balances the classification task with graph structure regularization, improving its generalization and robustness.

[0138] Classification loss:

[0139] Total loss function:

[0140] Most existing CLIP-based multimodal models rely on static global alignment strategies, making it difficult to capture fine-grained semantic relationships between text and visual segments. Traditional parameter fine-tuning can easily lead to catastrophic forgetting and rapid degradation of old knowledge when scenes or categories dynamically expand. Furthermore, existing graph neural networks often ignore the conservation of system energy, making them prone to stability issues such as embedding drift, gradient explosion, and vulnerability to adversarial perturbations in the later stages of training, limiting the model's generalization performance in open environments. These bottlenecks collectively constitute three core technical pain points: imprecise cross-modal semantics, unstable incremental learning, and unreliable graph structure representation.

[0141] This paper first uses an improved Transformer as the backbone, introducing learnable cues for both text and visual modalities, and simultaneously trains symmetric encoders within an end-to-end contrastive learning framework. Learnable cues are directly inserted at each layer, actively labeling key semantic regions and visually salient patches, significantly improving the discriminability of the cross-modal embedding space. The system then constructs a dynamic graph using these cues as nodes. Through cosine similarity adaptive sparsification and attention-weighted normalization, it depicts the dynamic associations between text fragments and image patches in real time, fundamentally addressing the problems of fine-grained semantic mismatch and the difficulty of describing time-varying semantics in static graphs.

[0142] After obtaining the dynamic graph, the present invention maps the node states into a Hamiltonian graph network: first, the aggregated node-neighborhood features are compressed into low-dimensional generalized coordinates and momentum. A multilayer perceptron is then used to simulate the Hamiltonian energy function. The state evolution equation is solved using the Symplectic-Euler method, and the initial and final state energies are constrained to be equal using an energy conservation loss. This mechanism explicitly maintains system energy balance during feature updates, avoiding representation drift and gradient instability, and significantly improving the robustness of cross-modal features during long-term training and inference phases.

[0143] Finally, the present invention superimposes an adversarial constrained graph autoencoder on top of the energy conservation representation: the encoding end uses an improved GIN to aggregate topological and semantic information, the decoding end reconstructs the adjacency matrix with inner products, and the discriminator uses the Wasserstein distance to force the potential distribution to fit the prior. During the training phase, the classification loss, graph structure reconstruction loss, adversarial regularization loss, and energy conservation loss are uniformly weighted and optimized; in the category increment link, only nodes are added to the new category and the features of the old nodes are frozen, realizing knowledge accumulation rather than replacement. Through this complete bottom-up technical route of "learnable prompts → dynamic semantic graphs → Hamiltonian conservation evolution → adversarial regularization → incremental updates", the present invention solves the outstanding problems of the existing technology in terms of cross-modal alignment accuracy, incremental learning stability, and graph representation reliability.

[0144] like Figure 2 As shown in Figure 2, the incremental learning method for few-shot classes based on graph neural networks includes:

[0145] S1, text and visual encoders process text and image inputs respectively;

[0146] S2, using graph isomorphism networks to process graph-structured learnable cues from text and vision;

[0147] S3, ACGA’s encoder-decoder structure generates optimized representations;

[0148] S4, the feature compression, Hamiltonian module, and restoration module generate updated learnable cues, which are fed back into the text and visual encoders for subsequent learning.

[0149] Example 1: CIFAR-100 dataset

[0150] Configuration: 60 base classes, 40 incremental classes, divided into 8 increments (5 classes / time, 5 samples / class).

[0151] Results: The average accuracy was 82.30%, and the PD value (catastrophic forgetting index) was 9.72.

[0152]

[0153] Example 2: CUB200 dataset

[0154] Configuration: 100 base classes, 100 incremental classes, divided into 10 increments (10 classes / time, 5 samples / class).

[0155] Results: The average accuracy was 69.54% and the PD value was 17.67.

[0156]

[0157] Example 3: MINIIMAGENET dataset

[0158] Configuration: 60 base classes, 40 incremental classes, divided into 8 increments (5 classes / time, 5 samples / class).

[0159] Results: Average accuracy was 85.05%, PD value was 8.42

[0160]

[0161] like Figure 3 Figure 1 shows the framework of ACHG-CLIP. The text and visual encoders on the left process text and image inputs, respectively. The center uses a graph isomorphism network (GIN) to process graph-structured learnable cues from text and vision. The right side shows the encoder-decoder structure of ACGA, which is used to generate optimized representations. Finally, feature compression, Hamiltonian modules, and restoration modules generate updated learnable cues, which are fed back into the text and visual encoders for further learning. The entire process emphasizes the fusion and updating of multimodal information.

[0162] An application embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of a few-sample incremental learning method based on a graph neural network.

[0163] An application embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of a few-sample incremental learning method based on a graph neural network.

[0164] An application embodiment of the present invention provides an information data processing terminal, which includes a few-sample class incremental learning system based on a graph neural network.

[0165] 1. Specific application fields or related products of the present invention.

[0166] The present invention can be used to develop a medical image-assisted diagnosis system based on the technology of the present invention. Medical image data, such as lesion tissue in CT and MRI images, can be abstracted into graph structure data, with different tissue regions as nodes and the relationships between regions as edges.

[0167] The Graph Isomorphism Network (GIN) can effectively process complex lesion structures. When only a small amount of typical case imaging data is obtained, it can also dynamically adjust the feature fusion ratio and extract representative lesion features to assist doctors in diagnosing rare diseases or early-stage diseases. Hamiltonian Graph Network Energy Conservation (HGN-GC) ensures that the model will not forget the common disease features that have been mastered due to the addition of new disease imaging data for training, thereby maintaining stable diagnostic performance. The Adversarial Constrained Graph Autoencoder (ACGA) ensures that the model does not over-learn the noise or local features in the training data, forces the distribution of different disease imaging features in the latent space to be aligned, and accurately distinguishes similar symptoms. The high parameter efficiency allows the system to run on ordinary workstations in hospitals, making it convenient for doctors to call it at any time, accelerating the diagnosis process, and improving medical efficiency.

[0168] 2. Relevant evidence of the technical effects obtained by the embodiments of the present invention.

[0169] 1. Theoretical Analysis Evidence

[0170] 1) Effectiveness of Graph Isomorphic Networks (GIN)

[0171] In traditional graph neural networks, the fusion ratio of node features and neighborhood information is fixed, which makes it difficult to adapt to changes in complex graph structures. GIN dynamically adjusts the fusion ratio through learnable parameters. From a mathematical perspective, it can be expressed as where ε is the learnable parameter (k) , which can adaptively adjust the weights of node features and neighborhood features based on the graph structure. When processing small amounts of data, this mechanism can avoid biased feature extraction and improve the accuracy of feature representation compared to traditional methods.

[0172] 2) Stability of Hamiltonian Graph Network Energy Conservation (HGN-GC)

[0173] Characteristic evolution equation based on Hamiltonian dynamics Energy conservation constraints ensure minimal fluctuations in system energy during training. When dynamically updating data, HGN-GC stabilizes feature evolution, minimizing the probability of catastrophic forgetting and ensuring the model's ability to continuously learn features from both new and old categories from a dynamic perspective.

[0174] 3) Anti-overfitting capability of Adversarial Constrained Graph Autoencoder (ACGA)

[0175] ACGA enforces distribution alignment in the latent space through the dual mechanism of adversarial training and topology reconstruction. The optimization objective of the discriminator D is Combined with the topology reconstruction loss, the overfitting error of the model on the training data is reduced, the generalization error is reduced, and the topological relationship between categories is effectively maintained.

[0176] In terms of resource utilization, freezing the CLIP backbone network drastically reduces the number of trainable parameters, significantly alleviating graphics memory and computing bottlenecks. Server GPU utilization has dropped significantly, and single-round training time has been reduced from tens of minutes to less than half, simultaneously reducing overall energy consumption and deployment costs. This "backbone freezing + lightweight incremental modules" strategy provides a more economical and feasible engineering path for large-scale cross-modal incremental learning.

[0177] Simulation experiments further validated the contributions of each submodule: ① In a comparison of graph construction strategies, GIN, based on feature homogeneity aggregation, consistently ranked first, followed by GraphSAGE and GCN. GAT, which relies solely on attention, saw its accuracy decline rapidly as the session progressed, demonstrating that maintaining sufficient node structural homogeneity is crucial for long-term stability. ② In module ablation tests, both "HGN-GC" and "ACGA" alone were able to slowly suppress forgetting, but the complete model (learnable hints + dynamic graph + Hamiltonian conservation + adversarial regularization) maintained an average accuracy lead of ≈5% after ten incremental sessions, validating the effectiveness of energy conservation and adversarial regularization in synergizing against catastrophic forgetting.

[0178] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0179] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A few-shot incremental learning system based on graph neural network, characterized by: include: The CLIP encoder, which includes a text encoder, a visual encoder, and a learnable cue generation unit, is used to construct an embedding space for cross-modal semantic alignment. By jointly training the text and visual encoders, it achieves an aligned mapping between text descriptions and image features, and introduces learnable cues into the text and visual encoders. Dynamic graph construction module, used for graph node definition, adjacency matrix generation and node feature update; Hamiltonian graph network and energy conservation module, used for state initialization, feature compression and system initialization, Hamiltonian energy function and state update, as well as state recovery and energy conservation constraints; The adversarial constrained graph autoencoder module, which includes a graph convolutional encoder, a structure reconstruction decoder, and an adversarial regularization unit, is used to generate latent representations, reconstruct the graph structure, and introduce a discriminator. The incremental module is used to receive new category samples during the category increment phase and add their corresponding node features to the dynamic graph. The node features of the old categories remain unchanged, thereby preserving historical knowledge and ensuring that the model does not forget the learned categories when learning new categories. The training module is used to adopt the total loss function to perform a weighted sum of classification loss, reconstruction loss, adversarial loss and energy loss.

2. The graph neural network-based few-shot incremental learning system according to claim 1, wherein: CLIP encoders specifically include: Multimodal alignment model: The CLIP encoder, based on contrastive learning, includes a text and visual symmetric encoder with an improved Transformer structure. Its core is to construct an embedding space for cross-modal semantic alignment, and to achieve an aligned mapping between text descriptions and image features by jointly training text and visual encoders. Learnable Hint Insertion: To enhance the expressiveness and adaptability of the model, learnable hints are introduced into the text and visual encoders. These hints are inserted into the text and visual embeddings to guide the model to focus on key features, improve the model's ability to recognize new categories, and reduce catastrophic forgetting. Specific structure: 1) Text Encoder: This consists of a word embedding matrix, positional encoding, and a multi-layer Transformer block. The input discrete tokens are mapped to continuous vectors. After adding the positional encoding, feature interaction is performed through a multi-head attention mechanism and a feedforward neural network. Finally, semantic projection is applied to obtain a normalized text embedding. Word embedding representation: discrete tokens are mapped to continuous variables Add position encoding: Feature Interaction: Semantic Projection: 2) Visual Encoder: This is symmetrical in structure to the text encoder, but takes a sequence of image blocks as input. The image blocks are linearly projected and a [CLS] token is added. After passing through the Transformer block, the features corresponding to the [CLS] token are extracted as visual embeddings. Input image block sequence, the processing flow is symmetrical: X = [x [CLS] ;E V (p1), ..., E V (p m )]+P V (m+1); Extract [CLS] features: 3) Learnable Cue Generation: Textual and visual cues are generated using learnable parameter matrices, respectively, and inserted into the textual and visual embeddings, directly replacing the input of each layer. Text prompts each layer insert: X = [X [CLS] , g (k) , X tokens ]; Visual hint: Each layer is inserted: X = [X [CLS] , g V (k) , X patches ].

3. The graph neural network-based few-shot incremental learning system according to claim 1, wherein: The dynamic graph construction module specifically includes: Graph node definition: Using textual and visual learnable cues as nodes, we construct a dynamic graph to capture the semantic associations between textual and visual modalities. Adjacency matrix generation: The adjacency matrix is ​​constructed by calculating the cosine similarity matrix and applying a sparse-inducing threshold for binarization. In addition, symmetrization and normalization operations are used to ensure the symmetry of the adjacency matrix. At the same time, the attention mechanism can be introduced to enhance the expressiveness of the model. The adjacency matrix construction formula is as follows: Cosine similarity matrix: sim_matrix i，j =cos(x i , x j ) Binarization processing: Symmetrization and normalization: Introducing the attention mechanism: Node feature update: Graph isomorphism networks are used to update the node feature matrix. GIN updates the features of the current node by aggregating the features of neighboring nodes, thereby capturing the complex patterns and dependencies in graph-structured data. Specifically, the feature update process for each node at each layer includes neighbor feature aggregation, combined with learnable coefficients, and multi-layer perceptron (MLP) transformation. GIN layer formula: Neighbor aggregation and feature fusion:

4. The graph neural network-based few-shot incremental learning system according to claim 1, wherein: The Hamiltonian graph network and energy conservation module specifically includes: State initialization: Receive the node feature matrix and adjacency matrix from the dynamic graph construction module, use the adjacency matrix to aggregate neighbor information, and concatenate the node features with the aggregated neighbor features to form the initial state; aggregated=A·X, state=[X, aggregated]; Feature compression and system initialization: Compress the initial state into a low-dimensional space through a linear layer to obtain the compressed feature vector, which is then assigned to the generalized coordinates and generalized momentum of the system. compressed=W compress ·state+b compress ; Hamiltonian energy function and state update: A multi-layer perceptron is used to simulate the Hamiltonian energy function and calculate the state update rule based on the Hamiltonian equation. The symplectic Euler method is used to numerically solve the state update equation to ensure energy conservation of the system. State recovery and energy conservation constraints: The updated state is restored to its original dimension through a linear layer to obtain the final node feature update result. To ensure energy conservation, an energy conservation loss function is introduced to calculate the energy difference between the initial and final states and use it as part of the optimization objective. Energy conservation loss:

5. The graph neural network-based few-shot incremental learning system according to claim 1, wherein: The Adversarial Constrained Graph Autoencoder (ACGA) module specifically includes: Graph Convolutional Encoder: uses an improved graph isomorphism network as the encoder, fuses node features and topology information through a multi-layer message passing mechanism to generate a latent representation; Z (l+1) =GINLayer(Z (l) ,A); Structure reconstruction decoder: reconstructs the adjacency matrix through the inner product of the potential representation and uses the Sigmoid function to simulate the edge existence probability to achieve reconstruction of the graph structure; Reconstruction loss: Adversarial Regularization Module: Introduces a discriminator to force the potential representation to match the prior distribution. The discriminator consists of two fully connected layers and uses Wasserstein distance as the adversarial loss function to enhance training stability. Fighting Loss:

6. The graph neural network-based few-shot incremental learning system according to claim 1, wherein: The training modules include: The model training process uses a total loss function, which is a weighted sum of classification loss, reconstruction loss, adversarial loss, and energy loss. By optimizing the total loss function, the model can effectively balance the classification task and graph structure regularization, improving the model's generalization ability and robustness. Classification loss: Total loss function:

7. A graph neural network-based few-shot incremental learning method using the graph neural network-based few-shot incremental learning system according to any one of claims 1 to 6, characterized in that: include: S1, text and visual encoders process text and image inputs respectively; S2, using graph isomorphism networks to process graph-structured learnable cues from text and vision; S3, the encoder-decoder structure of ACGA generates optimized representations; S4, the feature compression, Hamiltonian module, and restoration module generate updated learnable cues, which are fed back into the text and visual encoders for subsequent learning.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the graph neural network-based few-sample incremental learning method as described in claim 7.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the graph neural network-based few-sample incremental learning method as described in claim 7.

10. An information data processing terminal, comprising the graph neural network-based few-sample incremental learning system according to any one of claims 1 to 6.

Images