Prototype mutual prompt enhanced zero-shot text attribute graph learning method and system
By combining dual-view graph pre-training and multi-prototype prompting mechanisms with GNN and LLM, the problem of balancing cross-modal information of text and images and the transfer bias in cross-domain zero-shot scenarios is solved, achieving stable performance and generalization ability across tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHENGDU HEERKANG MEDICAL TECHNOLOGY CO LTD
- Filing Date
- 2026-03-27
- Publication Date
- 2026-06-09
AI Technical Summary
In cross-domain and cross-task zero-shot scenarios, existing technologies struggle to effectively generalize graph neural networks (GNNs) and large language models (LLMs). Single prompting strategies cannot take into account the complex cross-modal information of text and images and are prone to accumulating transfer bias.
A prototype-based mutual prompting enhancement method is adopted. By combining GNN and LLM, multiple prototype embeddings are generated through dual-view graph pre-training and multi-prototype prompting mechanism, and weights are calculated through router model to achieve deep fusion and dynamic adaptation of cross-modal information of text and image.
It effectively covers complex information across text and images, reduces zero-sample transfer bias, and improves the model's stability and generalization ability in different cross-modal scenarios.
Smart Images
Figure CN122174174A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a zero-shot text attribute graph learning method and system based on prototype mutual prompting enhancement. Background Technology
[0002] Text attribute graphs (TAGs), as a data form that integrates node / edge text descriptions with graph structures, have significant application value in fields such as social network analysis and e-commerce recommendation. Their core requirement is to extract the synergistic value of structural dependencies and semantic information from the graph. Currently, graph neural networks (GNNs) are the mainstream method for processing graph-structured data. They aggregate neighborhood information through message passing mechanisms to achieve node representation learning. However, these models are typically trained on specific datasets or tasks, and their performance drops sharply when generalizing across domains and tasks, making them difficult to adapt to zero-shot scenarios.
[0003] The emergence of Large Language Models (LLMs) has provided new possibilities for zero-shot learning. Leveraging massive amounts of pre-trained knowledge, they can effectively generalize to unseen tasks. However, LLMs are specifically designed for sequential text modeling, and when directly processing graph data, they easily lose geometric and topological information, failing to capture complex structural dependencies between nodes. To combine the advantages of both, existing research has developed three approaches: LLM-enhanced models use LLMs to replace traditional embedding methods to enrich the feature space, but the final prediction still relies on GNNs, limiting generalization; LLM-aligned models map graph and text modalities to a shared space, but only achieve model transfer within the same graph, lacking cross-scene adaptability; and LLM-predictive models convert graphs into text input LLMs, but lose key structural information, resulting in poor performance in tasks such as link prediction.
[0004] Chinese patent application CN202410089276.3 discloses a zero-shot learning method and system for cross-modal image-text fusion. This patent focuses on zero-shot tasks in cross-modal image-text fusion, proposing a modal alignment scheme based on a single prompt template. The aim is to map image features to the semantic space of a language model through fixed-format text prompts. Its core design involves using a unified prompt template, "<image features> corresponds to which text category? Candidate category: {candidate set}", to convert the visual feature vector of the image into a text description embedding, which is then input into a large language model (LLM) for classification. However, from a technical perspective, the expressive power of a single prompt is limited by the fixed template structure: for densely connected graphs, it is necessary to highlight local neighborhood structures; for sparse graphs, it is necessary to emphasize global topological features. A single prompt cannot accommodate both scenarios, and it is difficult to dynamically adjust the encoding granularity of graph structure information. Simultaneously, in zero-shot transfer, the difference in feature distribution between the source and target domains amplifies the initial bias. For example, when a single-cue model trained on an academic citation network (source domain) is transferred to an e-commerce product graph (target domain), the structural features of the "citation relationship" in the source domain are fundamentally different from the structural features of the "co-purchase relationship" in the target domain. Single-cue models struggle to adapt to this structural transfer, leading to accumulated bias. Furthermore, the single-cue strategy is essentially a one-way "graph → text" mapping; the semantic feedback of an LLM (Local Level Model) is insufficient to optimize the encoding of graph features, causing new biases generated during the transfer process to accumulate continuously, ultimately leading to the model converging to a local optimum. In summary, single-cue strategies are inadequate for covering complex cross-modal information between graphs and text, and are prone to bias accumulation during zero-sample transfer.
[0005] Therefore, there is an urgent need for a zero-shot text attribute graph learning method and system based on prototype mutual prompting enhancement that can cover complex cross-modal information of text and images and reduce the accumulation of zero-shot transfer bias. Summary of the Invention
[0006] This invention provides a zero-shot text attribute graph learning method and system based on prototype mutual prompting enhancement, which can cover complex cross-modal information of text and graph and reduce the accumulation of zero-shot transfer bias.
[0007] To solve the above-mentioned technical problems, this application provides the following technical solution: A zero-shot text attribute graph learning method based on prototype mutual prompting enhancement includes the following steps: S1. Obtain the text attribute graph G=(V,E,A,X), where V is the node set, E is the edge set, A is the adjacency matrix, and X is the node text attribute matrix. S2. Generate a random mask matrix using an edge removal strategy. Masking the adjacency matrix A yields Random mask vectors are generated using a node feature masking strategy. Masking the node feature matrix X yields This generates two graphical views. and Input the two graph views into the GNN encoder to obtain the node embeddings. and ; S3. Calculate instance-perceptual contrast loss Constrain the similarity of the same node embedded in two views; S4. Perform PCA dimensionality reduction on the token embedding of the LLM, extract the first P principal components to form an aligned coordinate system C, and then... and Mapped to this coordinate system, calculate the information perception contrast loss. ; S5, Through Optimize the GNN encoder to obtain a pre-trained GNN; S6. Freeze the pre-trained GNN and LLM backbone networks. Extract the node structure embedding Z from the pre-trained GNN and generate K prototype embeddings through K linear projectors. This is injected as a soft hint into the LLM instruction, and the LLM outputs K expert hints. ; S7. Calculate the weights of each expert suggestion using the router model. ,in, The weighted hints are concatenated with the original node features, and then linearly projected to obtain the enhanced node features. ; S8. Freeze the LLM training-related projectors and routers, freeze the GNN training LLM hint adaptation layer, and then... Iterative optimization is performed 1 to 3 times, among which, The prototype embedding orthogonality constraint loss, This is due to the loss from expert weight balancing. S9. Input the unseen text attribute map into the pre-trained GNN to obtain the structural embedding, generate geometric cues through prototype projection, inject it into LLM, and output the zero-sample prediction result.
[0008] The basic principle and beneficial effects of the scheme are as follows: Through dual-view graph pre-training in steps S2-S5, deep fusion of cross-modal information between text and images is achieved from two dimensions: instance awareness and informativeness. In instance-aware pre-training, dual-view graphs are generated through edge removal and node feature masking strategies, combined with instance-aware contrastive loss. Constraining the embedding similarity of the same node across different views enables the GNN encoder to accurately capture the geometric topology of the graph. In information-aware pre-training, PCA dimensionality reduction is used to extract the principal components of the LLM token embeddings to construct an aligned coordinate system. The GNN node embeddings are then mapped to this semantic space and computation is performed. This approach achieves pre-alignment between structural representation and semantic space, breaking down the domain gap between text and image modalities. This dual-view training enables pre-trained GNNs to simultaneously possess structural capture capabilities and semantic adaptability, laying the foundation for cross-modal information fusion.
[0009] The prototype mutual prompting framework in steps S6-S8 generates multiple prototype embeddings through K linear projectors, overcoming the expression bottleneck of a single prompt. The node structure embeddings extracted by the GNN are projected to obtain K prototype embeddings. Each prototype focuses on a type of structure-semantic co-pattern, which is injected into LLM instructions as soft prompts, enabling LLM to understand graph structure information of different dimensions; conversely, the K expert prompts output by the LLM... By fusing the weights of the router model with the original node features, targeted semantic guidance is provided to the GNN, adapting to the semantic complexity of different text attributes. This multi-prototype, bidirectional interactive prompting mechanism covers diverse information associations across text and image modalities, solving the problem that a single prompt cannot simultaneously address structural granularity and semantic complexity. The weights calculated by the router model... Introducing Gaussian noise The model encourages the exploration of diverse prompts, enabling it to dynamically adapt to the information needs of different tasks and domains: in node classification tasks, weights are biased towards semantically relevant prototype prompts to highlight category-distinguishing features; in link prediction tasks, emphasis is placed on structurally related prototype prompts to strengthen relationship capture capabilities; and for dense e-commerce co-purchasing graphs and sparse academic citation networks, weight allocation can adaptively adjust the contribution ratio of structure and semantics. This dynamic adaptation capability ensures that the model can effectively extract core information in different cross-modal scenarios, achieving comprehensive coverage of complex information.
[0010] In this scheme, since the initial bias of zero-sample transfer learning stems from incomplete image-text modality mapping, dual-view pre-training is used to achieve pre-alignment of structure and semantics, enabling the GNN embedding to naturally adapt to the LLM semantic space and reduce basic mapping bias. Simultaneously, K prototype cues cover the main cross-modal association patterns, avoiding the initial bias caused by missing information in a single cue. This is because the bias of a single cue originates from uncovered cue information. Multi-prototype prompts are provided through... Complete coverage of the deviation term This significantly reduces the likelihood of bias at its source. The iterative optimization in step S8 corrects bias through bidirectional feedback: training the projector and router while freezing the LLM allows the GNN to better utilize the semantic cues of the LLM to optimize structural representations; training the adaptation layer while freezing the GNN allows the LLM to accurately understand the structural cues of the GNN, forming a closed loop for bias correction. Simultaneously, The orthogonality constraint ensures that the prototype embeddings are independent of each other, avoiding the superposition of deviations caused by redundant prompt information; The weighted loss mechanism forces expert suggestions to be used evenly, preventing the model from becoming overly reliant on a single suggestion and thus reducing its generalization ability. This combination of iterative correction and regularization effectively suppresses the amplification of biases during the transfer from the source domain to the target domain, enabling the model to maintain stable performance across datasets and tasks.
[0011] In step S9, zero-shot inference directly inputs the unseen text attribute graph into the pre-trained GNN. Geometric cues generated through prototype projection are then injected into the LLM output, eliminating the need for target domain labeled data and fine-tuning. This inference model avoids negative transfer of source domain knowledge during fine-tuning, reducing the accumulation of adaptation bias. Simultaneously, the dual-view alignment and multi-cue learning during pre-training enable the model to possess strong cross-scene generalization capabilities, directly adapting to the graph and text feature distribution of unseen domains, further reducing the accumulation of bias during the transfer process.
[0012] Furthermore, in S3, the instance-aware contrast loss The calculation formula is
[0013] in,
[0014]
[0015] τ is a temperature parameter.
[0016] The beneficial effects are: by adjusting the smoothness of similarity calculation by temperature parameters, the embedding of the same node in different views is more compact, and the embedding of different nodes is more distinguishable, effectively enhancing the GNN's ability to capture graph structure dependencies.
[0017] Furthermore, in S4, the informational perception contrast loss The calculation formula is
[0018] in, , Embedded dimension for LLM tokens.
[0019] The beneficial effects are: breaking the node independence constraint, aligning GNN representations with the LLM semantic space from the feature dimension, narrowing the gap between the two domains, improving zero-shot classification accuracy, and ensuring that GNN representations not only capture structure but also adapt to the semantic understanding paradigm of LLM.
[0020] Furthermore, in S8, the aforementioned The calculation formula is
[0021] in, for Norm, where I is the identity matrix.
[0022] The beneficial effects are as follows: by forcing the K prototype embeddings to be independent through orthogonality constraints, redundant or homogeneous cue information is avoided, each prototype captures a unique structure-semantic pattern, the similarity between prototypes is reduced, the cue collapse problem is alleviated, and the LLM receives more comprehensive graph information, thereby improving the semantic understanding breadth during zero-shot reasoning.
[0023] Furthermore, in S8, the aforementioned The calculation formula is
[0024] Where CV(·) is the coefficient of variation.
[0025] The beneficial effects are as follows: by constraining the coefficient of variation of expert prompt weights, each prompt expert is forced to be used evenly, avoiding the model's over-reliance on a single expert which leads to a decline in generalization ability, improving the link prediction AUC value, ensuring the model's balanced capture of different types of structural and semantic information, and enhancing adaptability to complex scenarios.
[0026] Furthermore, the value of K is between 3 and 5, and the optimal value is determined through cross-validation.
[0027] Furthermore, the GNN encoder uses a 2-layer GraphSAGE, with a hidden dimension of 4096 and an activation function of ReLU.
[0028] Furthermore, the learning rate for the iterative optimization is set to... The batch size is 2.
[0029] Furthermore, the zero-shot prediction results include the accuracy of node classification, the macro F1 score, and the AUC-ROC score of link prediction. Attached Figure Description
[0030] Figure 1 This is a flowchart of Example 1.
[0031] Figure 2 For parameter sensitivity analysis. Detailed Implementation
[0032] The following detailed description illustrates the specific implementation method: Example 1 This invention discloses a zero-shot text attribute graph learning method based on prototype mutual prompting enhancement, as shown in the appendix. Figure 1 As shown, a detailed explanation is given using the cross-dataset zero-shot node classification task of academic paper citation networks as an example: S1: Obtain the text attribute graph Get Text Attribute Graph The Arxiv dataset was selected as the source training graph. This dataset contains 169,343 nodes representing academic papers in the field of computer science and 1,166,243 edges representing citation relationships between papers. Node set Each node corresponds to an academic paper; Edge set In the middle, if the paper Cited papers ,but ; Adjacency Matrix ,like ,but ,otherwise ; Node text attribute matrix each line The 768-dimensional feature vector is obtained by encoding the corresponding paper title and abstract using a pre-trained language model (such as BERT), and contains semantic information such as the research topic and methods of the paper.
[0033] S2: Generate the graph view and obtain node embeddings Two graph views are generated using edge removal (RE) and node feature masking (MF) strategies. and : Edge removal strategy, setting edge mask probability Generate a random mask matrix Its elements Follows Bernoulli distribution This means each edge has a 20% probability of being masked. The adjacency matrix after masking is calculated using the Hadamard product. For example, in the original adjacency matrix If a paper has 100 citations and 200 citations, then... ,but The reference relationship is masked; Node feature masking strategy, setting feature mask probabilities Generate a random mask vector Its elements Follows Bernoulli distribution This means that each feature dimension has a 15% probability of being masked. The masked node feature matrix... For example, nodes eigenvectors The 30th eigenvalue is 0.8. ,but The corresponding dimension value becomes 0; Generate two view diagrams and ,in and The adjacency matrix is obtained by two independent edge masks. and The feature matrix is obtained by two independent feature masks; The two graph views are input into the GNN encoder, which uses a 2-layer GraphSAGE layer with a hidden dimension of 4096 and a ReLU activation function. Node embeddings are obtained by aggregating neighborhood information through message passing. and For example, nodes exist Embedded in It is a 4096-dimensional vector obtained by aggregating the features of its unmasked neighboring nodes in the masked image with its own features.
[0034] S3: Calculate instance-aware contrastive loss
[0035] Calculate instance-aware contrastive loss. The calculation formula is
[0036] in,
[0037]
[0038] τ is a temperature parameter; Constrain the similarity of the same node embedded in two views: temperature parameter Its reference comparison learning often uses the optimal value; Similarity function
[0039] For example, nodes exist Embedded in With Embedded in The dot product is ,but ; Comparison of loss terms
[0040] If node of ,but ; Instance-aware contrastive loss
[0041] If all nodes The average value is 4.0, then .
[0042] S4: Calculate the informational perceptual contrast loss
[0043] Calculate the information perception contrast loss. The calculation formula is
[0044] in, , Embedding dimension for LLM tokens; Aligning GNN embeddings with the LLM semantic space: Vicuna-7B-v1.5 was selected as the LLM, and its token embedding dimension The hidden dimension is consistent with that of GNN; PCA dimensionality reduction is performed on the token embedding of the LLM to extract the undefined tokens. The principal components form an aligned coordinate system. These principal components cover more than 95% of the variance of the token embeddings; Will and Mapping to this coordinate system, we get , ,in, ; Informational perceptual contrast loss
[0045] If for the 100th principal component dimension, , Then the loss term for this dimension is If the average loss across all dimensions is 0.3, then... .
[0046] S5: Optimize the GNN encoder to obtain the pre-trained GNN According to the formula
[0047] Calculate the total pre-training loss and substitute it into... , ,get The AdamW optimizer was used, and the learning rate was set. With a batch size of 512 and a training period of 60 epochs, the parameters of the GNN encoder are updated through backpropagation to obtain a pre-trained GNN. This model can simultaneously capture the structural relationships of paper citations and the semantic alignment information of the paper text.
[0048] S6: Generating Prototype Embeddings and Expert Hints Freeze the pre-trained GNN and Vicuna-7B-v1.5 backbone network, and train only the subsequent projection layers and routers: Pre-trained GNNs process the original image adjacency matrix and characteristic matrix Extracting node structure embedding This embedding integrates the paper's citation structure and semantic features; set up It is the optimal value determined through cross-validation, using 3 independent linear projectors. , , Embed the structure of each node Projection is embedded in 3 prototypes ,in For example, nodes (A paper titled "Graph Neural Networks") Focus on capturing "same-domain reference structure" Focusing on "method similarity semantics", It focuses on "cross-domain association patterns"; As attached Figure 2 As shown in the figure, the left side of the figure shows the impact of the number of tokens (K) on performance, and the right side shows the impact of the number of iterations on performance. The left and right figures of Figure 2 (K=3 is the best performance for most datasets) can be referenced here to provide visual support for parameter selection and illustrate the rationality of parameter settings. The prototype embedding is injected as a soft prompt into the LLM instruction, with the instruction format: "This is a representation of a paper with the following information: Title: {Paper Title}; Abstract: {Paper Abstract}. Question: To which subfield of computer science does this paper belong? Candidate categories: {Artificial Intelligence, Data Mining, Computer Vision, Natural Language Processing...}", where the replacement is... , , Corresponding token embedding; LLM outputs three expert tips based on the input command. , ,For example, This is a category judgment prompt that emphasizes the density of citation relationships. For "Category hints based on abstract keyword matching", This is a category suggestion for "combining cross-domain collaboration models".
[0049] S7: Calculate expert suggestion weights and generate enhanced node features The weights of each expert suggestion are calculated using a router model, and then fused to obtain the enhanced node features: The router model is a linear layer. Input node structure embedding Output a 3-dimensional fractional vector; Introducing Gaussian noise ,For example, , combine the fraction vector with Perform the Hadamard product to obtain the adjusted fractional vector; The weights are obtained by normalization using the Softmax function. ,if ,but The product is After Softmax normalization ; The weighted hints are then concatenated with the original node features: ,in , ; Through linear projection layer Processing yields enhanced node features. This feature integrates the semantics and structural information of the original text with LLM expert tips.
[0050] S8: Iterative Optimization According to the following loss formula The calculation formula is
[0051] in, for Norm, where I is the identity matrix; The calculation formula is
[0052] Where CV(·) is the coefficient of variation; And the learning rate for iterative optimization is set to The hyperparameter with a batch size of 2 is used for iterative optimization: Freeze LLM, train linear projector , , and router ; Prototype embedding orthogonality constraint loss ,in for Norm, It is a 3×3 identity matrix. Suppose that the nodes... of
[0053] but
[0054] norm is If the average value of all nodes is 0.2, then ; Expert weight equilibrium loss ,in , Let be the coefficient of variation (standard deviation / mean). For example, , , The mean is The standard deviation is ,but , ; Total loss ,set up Substitute , , ,get ; Using learning rate With a batch size of 2, perform one round of iterative training, update relevant parameters, and complete model optimization.
[0055] S9: Zero-Shot Inference For the missing text attribute graph, select the Cora dataset, which contains 25,120 nodes and 91,140 edges, with node labels representing 70 machine learning subdomains. Input this data into the pre-trained GNN: A pre-trained GNN processes the adjacency matrix and node text attribute matrix of the Cora dataset to obtain the structural embedding of each node; Prototype embeddings (geometric hints) are generated using three linear projectors and injected into Vicuna-7B-v1.5; LLM, based on geometric cues and node text information, outputs node classification results, including evaluation metrics such as node classification accuracy, macro F1 score, and link prediction AUC-ROC score: the node classification accuracy is 17.3%, and the macro F1 score is 16.5%. If this model is directly applied to the link prediction task on the Cora dataset, the output AUC-ROC score is above 0.55, achieving zero-shot transfer across tasks.
[0056] Example 2 Based on Embodiment 1, this embodiment provides a zero-shot text attribute graph learning system based on prototype mutual prompting enhancement, which is used to execute the method described in Embodiment 1. The system achieves efficient learning and reasoning of zero-shot text attribute graphs through the collaborative work of hardware devices and software modules.
[0057] The hardware configuration includes: computing devices, four NVIDIA A100 GPUs supporting parallel model training and inference, meeting the memory requirements of large-dimensional embedded computing and LLM deployment; storage devices, a 1TB SSD for storing datasets, pre-trained model weights, and intermediate results, ensuring fast data read and write speeds; processors, Intel Xeon Platinum 8375C CPUs with a clock speed of 3.0GHz, used for auxiliary computing tasks such as data preprocessing and parameter scheduling; and memory, 256GB DDR4 memory, supporting batch data loading and temporary computing storage to avoid data transfer bottlenecks.
[0058] The software modules include a data acquisition module, a graph / view generation module, a GNN encoding module, a loss calculation module, a GNN pre-training module, a prototype hint generation module, a weight calculation module, an iterative optimization module, and a zero-shot inference module. The functions of each module are as follows: The data acquisition module is used to obtain text attribute graphs. The system performs data preprocessing, such as missing value imputation, text encoding, and adjacency matrix sparsification. It reads raw data from public datasets like Arxiv and Cora, encodes text attributes into 768-dimensional feature vectors using the BERT model, and stores the adjacency matrix in a sparse matrix format to reduce memory usage. For example, when processing the Arxiv dataset, the paper title and abstract are concatenated and input into BERT; the output [CLS] vector is used as the node text feature. .
[0059] The graph view generation module performs edge removal and node feature masking strategies to generate two graph views. and Generating random mask matrices using Python's NumPy library. and mask vector Masking operations on the adjacency matrix and eigenvalue matrix are performed using the Hadamard product; for example, setting... , The numpy.random.binomial function is called to generate Bernoulli distribution mask data, ensuring that the two masking operations are performed independently, resulting in two different graph views.
[0060] The GNN encoding module is used to load the GNN encoder, encode the graph view, and output node embeddings. , and structural embedding A GraphSAGE model was built using the PyTorch framework. The first layer has an input dimension of 768 and an output dimension of 4096; the second layer has an input dimension of 4096 and an output dimension of 4096. The activation function used is ReLU. Mean aggregation is used for message passing, i.e., for nodes... The average value of the embeddings of neighboring nodes is aggregated and concatenated with its own embedding, and then linearly transformed to obtain the current layer embedding.
[0061] The loss calculation module is used to calculate instance-aware contrast loss. Information perception contrast loss Prototype embedding orthogonality constraint loss Expert weighting equilibrium loss The total loss is obtained by combining the results; the loss function is calculated using PyTorch's automatic differentiation mechanism, for example, In the calculation, the dot product between node embeddings is calculated in batches through matrix operations, and then substituted into... Obtain the similarity matrix, and then calculate the contrast loss using the torch.log and torch.sum functions; In the calculation, the sklearn.decomposition.PCA tool is called to reduce the dimensionality of the LLM token embedding, extract the principal components, and then complete the embedding mapping and loss calculation.
[0062] The GNN pre-training module is used to calculate the total pre-training loss.
[0063] Optimize the GNN encoder parameters to obtain a pre-trained GNN; Using the AdamW optimizer, set the learning rate. Weight decay The batch size is 512, and the training lasts for 60 epochs. An early stopping strategy is used during training: if the loss on the validation set does not decrease for 5 consecutive epochs, training is stopped and the optimal model weights are saved.
[0064] The prototype hint generation module is used to freeze the pre-trained GNN and LLM backbone networks and generate prototype embeddings. and expert advice Construct three independent linear projectors to embed the structure output by the GNN. The projection consists of three prototype embeddings; the LLM uses Vicuna-7B-v1.5, loaded via the Hugging Face Transformers library, and multi-GPU deployment is achieved by setting device_map="auto". The prototype embeddings are then appended as soft suggestions to the instruction text, and inputting the LLM yields expert hints. .
[0065] The weight calculation module is used to calculate expert hint weights through the router model. And generate enhanced node features. ; The router model is a single-hidden-layer neural network with an input dimension of 4096, a hidden dimension of 2048, and an output dimension of 3. The activation function is ReLU; Gaussian noise is generated. The torch.normal function is called, the torch.softmax function is used to calculate the weights, and finally the enhanced node features are obtained through matrix concatenation and linear projection.
[0066] The iterative optimization module is used to freeze the LLM and GNN backbone networks, train the relevant projectors and routers, and achieve iterative optimization of the model; it also sets the learning rate. Batch size 2, training 1 iteration; monitor total loss during training.
[0067] The changes in parameters are updated through backpropagation to ensure model convergence.
[0068] The zero-shot inference module receives unseen text attribute maps and outputs zero-shot prediction results, including the accuracy of node classification, macro F1 score, and AUC-ROC score for link prediction. It inputs unseen datasets (such as Cora) into a pre-trained GNN to generate structural embeddings and prototype hints, which are then injected into an LLM to obtain predicted labels. Evaluation metrics are calculated using Python's Scikit-learn library, where accuracy is the number of correctly classified nodes divided by the total number of nodes, macro F1 score is the average of the F1 scores for each category, and AUC-ROC score is calculated using the `roc_auc_score` function. Finally, it outputs a metric report and prediction result file.
[0069] The workflow is as follows: The data acquisition module loads and preprocesses the text attribute graph data, and outputs standardized data. The view generation module for... Perform a masking operation and output two view graphs. and The GNN encoding module encodes the graph view and outputs node embeddings. and Loss calculation module calculates and The GNN pre-training module optimizes and obtains a pre-trained GNN; the prototype prompt generation module, based on the pre-trained GNN and LLM, outputs prototype embeddings. and expert advice The weight calculation module calculates expert weights and generates enhanced node features. The iterative optimization module is based on the total loss. The model undergoes iterative training to update its parameters. The zero-shot inference module receives unseen text attribute graphs, executes the inference process, and outputs zero-shot prediction results and evaluation metrics. Through modular design, the method has been successfully implemented in engineering, flexibly adapting to zero-shot learning tasks in various scenarios such as academic citation networks and e-commerce product graphs, demonstrating good versatility and scalability.
[0070] The above are merely embodiments of the present invention. The invention is not limited to the fields covered by these embodiments. Commonly known structures and characteristics in the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are able to access all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.
Claims
1. A zero-shot text attribute graph learning method based on prototype mutual prompting enhancement, characterized in that, Including the following steps: S1. Obtain the text attribute graph G=(V,E,A,X), where V is the node set, E is the edge set, A is the adjacency matrix, and X is the node text attribute matrix. S2. Generate a random mask matrix using an edge removal strategy. Masking the adjacency matrix A yields Random mask vectors are generated using a node feature masking strategy. Masking the node feature matrix X yields This generates two graphical views. and Input the two graph views into the GNN encoder to obtain the node embeddings. and ; S3. Calculate instance-perceptual contrast loss Constrain the similarity of the same node embedded in two views; S4. Perform PCA dimensionality reduction on the token embedding of the LLM, extract the first P principal components to form an aligned coordinate system C, and then... and Mapped to this coordinate system, calculate the information perception contrast loss. ; S5, Through Optimize the GNN encoder to obtain a pre-trained GNN; S6. Freeze the pre-trained GNN and LLM backbone networks. Extract the node structure embedding Z from the pre-trained GNN and generate K prototype embeddings through K linear projectors. This is injected as a soft hint into the LLM instruction, and the LLM outputs K expert hints. ; S7. Calculate the weights of each expert suggestion using the router model. ,in, The weighted hints are concatenated with the original node features, and then linearly projected to obtain the enhanced node features. ; S8. Freeze the LLM training-related projectors and routers, freeze the GNN training LLM hint adaptation layer, and then... Iterative optimization is performed 1 to 3 times, among which, The prototype embedding orthogonality constraint loss, This is due to the loss from expert weight balancing. S9. Input the unseen text attribute map into the pre-trained GNN to obtain the structural embedding, generate geometric cues through prototype projection, inject it into LLM, and output the zero-sample prediction result.
2. The zero-shot text attribute graph learning method based on prototype mutual prompting enhancement according to claim 1, characterized in that, In S3, the instance-aware contrast loss The calculation formula is ； in, ； ； τ is a temperature parameter.
3. The zero-shot text attribute graph learning method based on prototype mutual prompting enhancement according to claim 2, characterized in that, In S4, the informational perceptual contrast loss The calculation formula is ； in, , Embedded dimension for LLM tokens.
4. The zero-shot text attribute graph learning method based on prototype mutual prompting enhancement according to claim 3, characterized in that, In S8, the The calculation formula is ； in, for Norm, where I is the identity matrix.
5. The zero-shot text attribute graph learning method based on prototype mutual prompting enhancement according to claim 4, characterized in that, In S8, the The calculation formula is ； Where CV(·) is the coefficient of variation.
6. The zero-shot text attribute graph learning method based on prototype mutual prompting enhancement according to claim 5, characterized in that, The value of K is between 3 and 5, and the optimal value is determined through cross-validation.
7. The zero-shot text attribute graph learning method based on prototype mutual prompting enhancement according to claim 6, characterized in that, The GNN encoder uses a 2-layer GraphSAGE with a hidden dimension of 4096 and an activation function of ReLU.
8. The zero-shot text attribute graph learning method based on prototype mutual prompting enhancement according to claim 7, characterized in that, The learning rate for the iterative optimization is set to The batch size is 2.
9. The zero-shot text attribute graph learning method based on prototype mutual prompting enhancement according to claim 8, characterized in that, The zero-shot prediction results include the accuracy of node classification, the macro F1 score, and the AUC-ROC score of link prediction.
10. A zero-shot text attribute graph learning system based on prototype mutual prompting enhancement, characterized in that, Used to perform the method according to any one of claims 1-9.