Decoupled graph visual language model construction method for multi-modal attributed graph learning

Abstract

The application discloses a decoupled graph visual language model construction method for multi-modal attribute graph learning, and belongs to the technical field of computers. The method constructs a decoupled graph visual language model for multi-modal attribute graphs. The node-in semantic decoupling module extracts task-related semantic factors for a target node as the input of a visual language model backbone network. The node-structure decoupling module extracts structure factors distinguishing the structure perspective of the target node and each node in the graph neighborhood. Then, the mixed expert gating mechanism is used to adaptively select task-related structure factors. The LoRA mixed expert module uses the structure factors to perceive and inject the task-related structure factors into the processing of the input hidden state of the visual language model backbone network. Thus, the task-related semantic factors and structure factors are combined to generate a prediction result for the graph learning task to be processed, which can effectively improve the prediction accuracy and reduce the computational overhead.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a method for constructing a decoupled graph visual language model for multimodal attribute graph learning. Background Technology

[0002] A multimodal-attributed graph (MAG) consists of nodes, edges, graph topology, and various modal attributes associated with the nodes. Therefore, learning a multimodal-attributed graph requires simultaneously utilizing semantic information such as images and text of the nodes and the structural relationships between the nodes to complete graph learning tasks such as node classification and link prediction.

[0003] In related technologies, in order to utilize the image and text pre-trained cross-modal alignment capability of the Vision-Language Model (VLM), current multimodal attribute graph prediction methods based on the Vision-Language Model typically concatenate the image and text attributes of the target node and its neighboring nodes into a long cue sequence, and then input it into the Vision-Language Model for prediction.

[0004] However, such schemes directly splice the multimodal information of neighboring nodes, resulting in high computational costs. They also suffer from two types of problems: intra-node semantic entanglement and inter-node structural entanglement, leading to low accuracy in multimodal attribute graph prediction. On the one hand, after image tokens and text tokens are mapped to a unified token space, shared semantics, modality-specific details, and noise signals are mixed together, making it difficult for the model to extract semantic factors directly related to the current task from the multimodal attributes of the target node. On the other hand, various relationship patterns, redundant information, and task-irrelevant information carried by different neighbors are mixed together during the neighborhood aggregation process, making it difficult for the model to identify truly useful structural factors. Summary of the Invention

[0005] The purpose of this application is to provide a method for constructing a decoupled graph visual language model for multimodal attribute graph learning, which can solve the problems of high computational cost and low accuracy of multimodal attribute graph prediction in related technologies.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a method for constructing a decoupled graph visual language model for multimodal attribute graph learning, the method comprising: A decoupled graph visual language model is constructed for processing the graph learning task associated with multimodal attribute graphs. The decoupled graph visual language model includes a multimodal coding module, an intra-node semantic decoupling module, an inter-node structural decoupling module, a LoRA hybrid expert module, and a visual language model backbone network. The multimodal encoding module encodes the multimodal attributes of the target node associated with the graph learning task to be processed and each node in the graph neighborhood in the multimodal attribute graph into a multimodal token sequence in a unified token space. The intra-node semantic decoupling module processes the multimodal token sequence of the target node into a latent semantic sequence, and based on the pre-learned relationship between different latent semantic factors and different graph learning tasks, selects each latent semantic factor associated with the graph learning task to be processed from the latent semantic sequence as the semantic factors related to each task. The inter-node structural decoupling module utilizes parallel graph neural network channels corresponding to multiple structural perspectives to process the multimodal token sequences of the target node and each node in the graph neighborhood into structural factors under multiple structural perspectives. The visual language model backbone network generates input hidden states based on the semantic factors related to each task; The LoRA hybrid expert module is based on a hybrid expert gating mechanism. It adaptively selects each structural factor associated with the learning task of the graph to be processed as the task-related structural factor. The task-related structural factors are then injected into the visual language model backbone network through the corresponding LoRA experts to process the input hidden state. Different LoRA experts have independent low-rank update paths corresponding to structural factors under different structural perspectives. The visual language model backbone network generates prediction results for the learning task of the graph to be processed based on the processing results of the input hidden state.

[0007] Secondly, embodiments of this application provide a decoupled graph visual language model for multimodal attribute graph learning, the model comprising: The multimodal encoding module is used to encode the multimodal attributes of the target node associated with the graph learning task to be processed and each node in the graph neighborhood in the multimodal attribute graph into a multimodal token sequence in a unified token space. The intra-node semantic decoupling module is used to process the multimodal token sequence of the target node into a latent semantic sequence, and based on the pre-learned relationship between different latent semantic factors and different graph learning tasks, to select each latent semantic factor associated with the graph learning task to be processed as the relevant semantic factor of each task from the latent semantic sequence. The inter-node structural decoupling module is used to process the multimodal token sequences of the target node and each node in the graph neighborhood into structural factors under multiple structural perspectives by utilizing the parallel graph neural network channels corresponding to multiple structural perspectives. A visual language model backbone network is used to generate input hidden states based on the semantic factors related to each task; and to generate prediction results for the learning task of the graph to be processed based on the processing results of the input hidden states. The LoRA hybrid expert module is used to adaptively select each structural factor associated with the learning task of the graph to be processed as the task-related structural factor based on the hybrid expert gating mechanism, and inject each task-related structural factor into the processing process of the input hidden state of the visual language model backbone network through the corresponding LoRA expert. Different LoRA experts have independent low-rank update paths corresponding to structural factors under different structural perspectives.

[0008] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the decoupled graph visual language model construction method for multimodal attribute graph learning as described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the decoupled graph visual language model construction method for multimodal attribute graph learning as described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the decoupled graph visual language model construction method for multimodal attribute graph learning as described in the first aspect.

[0011] In this embodiment, a decoupled graph visual language model for multimodal attribute graphs is constructed. It extracts task-related semantic factors from the target node using an intra-node semantic decoupling module, serving as input to the visual language model backbone network. It also extracts structural factors from the target node and its neighbors using an inter-node structural decoupling module, distinguishing structural perspectives. Then, based on a hybrid expert gating mechanism, task-related structural factors are adaptively selected. Finally, a structure factor-aware LoRA hybrid expert module injects these structural factors into the visual language model backbone network's processing of the input hidden state. This combines task-related semantic and structural factors to generate prediction results for the graph learning task. Therefore, this application avoids directly splicing multimodal information from numerous neighboring nodes into the input sequence of the visual language model backbone network and effectively solves the problems of intra-node semantic entanglement and inter-node structural entanglement. This allows for improved prediction accuracy and reduced computational overhead while preserving the original graph-text alignment capability of the visual language model backbone network. Attached Figure Description

[0012] Figure 1 A flowchart illustrating the implementation of a decoupled graph visual language model construction method for multimodal attribute graph learning, provided in this application embodiment; Figure 2 A schematic diagram of the overall framework of a decoupled graph visual language model for multimodal attribute graph learning provided in this application embodiment; Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] Among related technologies, there are currently three main types of multimodal attribute graph learning methods: Graph neural network-based methods typically begin by using a pre-trained unimodal encoder to extract image and text features separately. These features are then concatenated, averaged, or mapped to form a unified node representation. Finally, the graph neural network facilitates message passing and neighborhood information aggregation between nodes, completing the downstream graph learning task. However, these methods often rely on fixed pre-trained image / text embeddings, making it difficult to fully utilize the fine-grained cross-modal alignment and inference capabilities of the token space in visual language models.

[0016] Methods based on large language models typically first convert multimodal node attributes, such as images and text, into a unified representation that the large language model can process. Then, the graph structure is encoded into textual descriptions, structural representations, or graph-related instructions, which are then input into the large language model for graph task inference. While these methods leverage the language understanding and reasoning capabilities of large language models, they usually require additional modality conversion and graph-language alignment processes. For example, visual information needs to be converted into representations or instructions compatible with the large language model. Their multimodal understanding capabilities are generally weaker than visual language models with native graph-text pre-training alignment capabilities.

[0017] Methods based on visual language models use the visual language model as a predictor, taking the target node and several neighboring nodes' images, text, and task instructions as input, and then fine-tuning the visual language model according to the instructions. These methods can directly utilize the native image-text alignment capabilities of the visual language model, but they typically introduce neighbor information through cue splicing, such as directly mixing the multimodal content of the target node and neighboring nodes. They fail to explicitly decouple semantic factors within nodes from structural factors between nodes, resulting in task-related and redundant information entering long cue sequences. This reduces the model's fine-grained information selection ability and causes high time and memory overhead.

[0018] To address the aforementioned issues, this application provides a method for constructing a decoupled graph visual language model for multimodal attribute graph learning. This method uses the visual language model backbone network as a task predictor to directly leverage the native graph-text alignment capabilities of the visual language model. For a multimodal attribute graph containing image attributes, text attributes, and graph topology, the model first encodes the multimodal attributes (e.g., image and text) of the target node associated with the graph learning task and each node in the graph neighborhood into a unified multimodal token sequence. Then, a semantic decoupling module within each node extracts task-related semantic factors as input to the visual language model backbone network, and the method utilizes the inter-node structure... The decoupling module extracts multiple structural factors that distinguish each other, and then the LoRA hybrid expert module extracts task-related structural factors and injects them into the visual language model backbone network's processing of the input hidden state. In this way, the decoupled semantic information and structural information are injected into the task prediction result generation process of the visual language model backbone network, avoiding the direct splicing of the multimodal information of a large number of neighboring nodes into the input sequence of the visual language model backbone network. This effectively solves the problems of semantic entanglement within nodes and structural entanglement between nodes, thereby improving prediction accuracy and reducing computational cost while preserving the original image-text alignment capability of the visual language model backbone network.

[0019] The following description, in conjunction with the accompanying drawings, details a method for constructing a decoupled graph visual language model for multimodal attribute graph learning, provided by the embodiments of this application, through specific implementations and application scenarios.

[0020] See Figure 1 The diagram shown is an implementation flowchart of a method for constructing a decoupled graph visual language model for multimodal attribute graph learning provided in this application. The method may include the following steps: Step S101: Construct a decoupled graph visual language model for processing the graph learning task associated with multimodal attribute graphs. The decoupled graph visual language model includes a multimodal encoding module, an intra-node semantic decoupling module, an inter-node structural decoupling module, a LoRA hybrid expert module, and a visual language model backbone network.

[0021] In this embodiment, the multimodal attribute graph consists of nodes, edges, graph topology, and various modal attributes associated with the nodes. For example, in an e-commerce network, a node can represent a product, node attributes can include product images and text descriptions, and edges can represent joint purchase or joint browsing relationships. In a social network, a node can represent a post containing node attributes such as text and images, and edges can represent joint comment relationships.

[0022] This application utilizes a multimodal coding module to achieve encoding in a unified token space and uses a visual language model backbone network (such as Qwen2.5-VL-7B) as a task predictor to directly leverage its native graph-text alignment capabilities. Furthermore, it designs and trains intra-node semantic decoupling modules, inter-node structural decoupling modules, and LoRA hybrid expert modules to achieve effective modeling of multimodal semantic information and graph structure information. This results in the construction of a decoupled graph visual language model for multimodal attribute graph learning, which can be widely applied to e-commerce recommendation, product classification, social media analysis, multimodal knowledge graph reasoning, and other application scenarios to complete related graph learning tasks such as node classification and link prediction.

[0023] Step S102: The multimodal encoding module encodes the multimodal attributes of the target node associated with the graph learning task to be processed and each node in the graph neighborhood in the multimodal attribute graph into a multimodal token sequence in a unified token space.

[0024] In this embodiment, the multimodal encoding module can be directly constructed based on the original basic encoding components such as the visual encoder, visual projector, word segmenter and embedding layer of the traditional pre-trained visual language model framework, so as to process the multimodal attributes such as image attributes and text attributes of the target node and each node in the graph neighborhood (or all nodes in the graph topology) respectively, thereby constructing a multimodal token sequence under a unified token space.

[0025] For example, for the target node The multimodal coding module can utilize the visual encoder to process its image attributes. The processed image features are mapped to the language model's hidden space (i.e., the unified token space) using a visual projector to obtain the image token; then, the language model's tokenizer and embedding layer are used to process its text attributes. This process yields text tokens in the language model's hidden space; finally, the image tokens and text tokens are concatenated into a multimodal token sequence in a unified token space. The aforementioned process can be represented as follows:

[0026] in, and These represent image tokens and text tokens, respectively. Indicates the total number of multimodal tokens. This represents the number of hidden dimensions in the visual language model (i.e., the number of hidden dimensions in the backbone network of the visual language model). This indicates a multimodal coding module.

[0027] Step S103: The intra-node semantic decoupling module processes the multimodal token sequence of the target node into a latent semantic sequence, and based on the pre-learned relationship between different latent semantic factors and different graph learning tasks, selects each latent semantic factor associated with the graph learning task to be processed from the latent semantic sequence as the semantic factors related to each task.

[0028] In this embodiment of the application, the intra-node semantic decoupling module can utilize a pre-trained encoding network (e.g., The VAE encoder performs Latent Semantic Analysis (LSA) on the multimodal token sequence of the target node, thereby processing the multimodal token into a latent semantic sequence.

[0029] Then, the intra-node semantic decoupling module uses the pre-learned relationships between different latent semantic factors and different graph learning tasks. For example, it uses a multilayer perceptron that has pre-learned these relationships to select the latent semantic factors associated with the graph learning task to be processed from the latent semantic sequence as the relevant semantic factors for each task. This helps to suppress latent components that are irrelevant to the current task or have strong noise, thereby solving the problem of intra-node semantic entanglement.

[0030] As one possible implementation, the relationship between different latent semantic factors and different graph learning tasks (i.e., task relevance) can be learned through end-to-end backpropagation of downstream task losses. For example, a multilayer perceptron can generate a soft-select mask based on the latent semantic representation, and then use the selected semantic factors to enhance the input of the visual language model backbone network and participate in task prediction; thus, the task loss will prompt the model to increase the weight of semantic factors relevant to the current task and suppress irrelevant or noisy factors.

[0031] Step S104: The inter-node structural decoupling module uses the parallel graph neural network channels corresponding to multiple structural perspectives to process the multimodal token sequences of the target node and each node in the graph neighborhood into structural factors under multiple structural perspectives.

[0032] In this embodiment of the application, the inter-node structural decoupling module is configured with multiple parallel graph neural network channels. Different channels have different parameters to correspond to different structural perspectives, thereby focusing on different adjacency relationships and propagation patterns in the graph neighborhood of the target node, and thus outputting diverse structural factors that distinguish each other.

[0033] Step S105: The visual language model backbone network generates the input hidden state based on the semantic factors related to each task.

[0034] In this embodiment, unlike traditional methods that directly concatenate multimodal information such as neighboring images and text into the visual language model input, this application keeps the direct input of the visual language model backbone network as the multimodal attributes of the target node (i.e., task-related semantic factors). The neighboring multimodal information is then entered into the visual language model backbone network through structural factors and LoRA hybrid expert modules, thereby reducing the computational overhead caused by directly concatenating neighboring multimodal information.

[0035] Step S106: The LoRA hybrid expert module, based on the hybrid expert gating mechanism, adaptively selects each structural factor associated with the learning task of the graph to be processed as each task-related structural factor, and injects each task-related structural factor into the visual language model backbone network for processing the input hidden state through the corresponding LoRA expert. Here, different LoRA experts have independent low-rank update paths corresponding to structural factors under different structural perspectives.

[0036] In this embodiment, considering that related technologies typically use a unified representation or shared adaptation modules to introduce graph structures, it is difficult to perform specialized modeling for different relational patterns. Therefore, this application learns different structural factors through multiple topology-adaptive graph neural network channels, and further injects each structural factor into a corresponding LoRA expert, enabling the visual language model backbone network to adaptively select and combine task-related structural information.

[0037] Specifically, based on the diverse structural factors obtained, the LoRA hybrid expert module further adaptively selects task-related structural factors based on a hybrid expert gating mechanism, which can solve the problem of structural entanglement between nodes and enable the model to identify truly useful structural factors. Furthermore, the LoRA hybrid expert based on structural factor awareness introduces independent low-rank update paths corresponding to structural factors under different structural perspectives, which can add low-rank updates to the frozen linear layer weights of the visual language model backbone network. Thus, by leveraging expert specialization, structural factors under the corresponding structural perspectives are stably injected into the visual language model backbone network for processing the input hidden state.

[0038] Therefore, since related technologies typically encode multimodal attributes and graph structures uniformly, lacking separate modeling of multimodal semantics within nodes and graph structures between nodes, which easily leads to the entanglement of semantic and structural information, this application extracts task-related semantic factors through a semantic decoupling module within nodes, and extracts and filters diverse structural factors through a structural decoupling module between nodes and a LoRA hybrid expert module, thereby achieving separate processing of the two types of information entanglement problems.

[0039] Step S107: The visual language model backbone network generates prediction results for the learning task of the graph to be processed based on the processing results of the input hidden state.

[0040] In this embodiment of the application, the visual language model backbone network can generate prediction results (such as node classification results, link prediction results, etc.) for the graph learning task to be processed by utilizing subsequent network parts (such as feedforward network and subsequent Transformer block) based on the processing results of the input hidden state.

[0041] Therefore, this application extracts task-related semantic factors and diverse structural factors through intra-node semantic decoupling module and inter-node structural decoupling module, respectively, and uses LoRA hybrid expert with structural factor awareness to adaptively inject structural information into the visual language model backbone network, so that the visual language model backbone network can combine decoupled semantic information and structural information to generate prediction results, thereby improving the prediction effect of multimodal attribute graph.

[0042] As can be seen from the above technical solution, this application constructs a decoupled graph visual language model for multimodal attribute graphs. It extracts task-related semantic factors from the target node using an intra-node semantic decoupling module as input to the visual language model backbone network. It also extracts structural factors from the target node and its neighbors using an inter-node structural decoupling module, distinguishing structural perspectives. Then, based on a hybrid expert gating mechanism, it adaptively selects task-related structural factors. Finally, using a structure factor-aware LoRA hybrid expert module, it injects these task-related structural factors into the visual language model backbone network's processing of the input hidden state. This combines task-related semantic factors and structural factors to generate prediction results for the graph learning task. Therefore, this application avoids directly splicing the multimodal information of a large number of neighboring nodes into the input sequence of the visual language model backbone network and effectively solves the problems of intra-node semantic entanglement and inter-node structural entanglement. This allows it to improve prediction accuracy and reduce computational overhead while preserving the original graph-text alignment capability of the visual language model backbone network.

[0043] In some embodiments, the method further includes: The intra-node semantic decoupling module uses the multimodal token sequence of the target node as a query and the semantic factors related to each task as keys and values. It calculates enhanced semantic signals through a cross-attention module with learnable query, key and value projection, so that tokens of different modalities and positions can obtain targeted semantic enhancement. The intra-node semantic decoupling module injects the enhanced semantic signal into the input of the visual language model backbone network through residual connections, so that the visual language model backbone network generates the input hidden state based on the enhanced semantic signal after residual connections and the multimodal token sequence of the target node.

[0044] In this embodiment of the application, the intra-node semantic decoupling module uses the multimodal token sequence of the target node. As a query, based on task-related semantic factors As keys and values, enhanced semantic signals are computed through a cross-attention module with learnable query (Q), key (K), and value (V) projections. This allows different input tokens to retrieve different semantic factors, thereby enabling targeted semantic enhancement for multimodal tokens such as image tokens and text tokens, as well as tokens at different locations. For example, the aforementioned process can be represented as follows:

[0045] in, This indicates a cross-attention module. Indicates the total number of multimodal tokens. This represents the number of hidden dimensions in the visual language model.

[0046] Then, the intra-node semantic decoupling module will enhance the semantic signal. The input to the visual language model backbone network is injected through residual connections; exemplarily, this process can be represented as:

[0047] in, This indicates a learnable scaling parameter. This represents the multimodal token sequence of the target node.

[0048] Understandably, the results obtained using residual joins Alternative As input to the backbone network of the visual language model, it incorporates task-related semantic factors while preserving the original pre-trained multimodal representations.

[0049] In some embodiments, the intra-node semantic decoupling module processes the multimodal token sequence of the target node into a latent semantic sequence, including: The intra-node semantic decoupling module generates a factorized Gaussian posterior distribution for each token in the multimodal token sequence of the target node through an encoding network. The intra-node semantic decoupling module samples the factorized Gaussian posterior distribution of each Token to obtain the latent semantic factors of each Token. The intra-node semantic decoupling module constructs a latent semantic sequence based on the latent semantic factors of all tokens in the multimodal token sequence of the target node.

[0050] In this embodiment of the application, for the multimodal token sequence of the target node The Middle Each Token represents (Where d represents the number of hidden dimensions in the visual language model), the intra-node semantic decoupling module can first use a linear layer to project its features, making it suitable for the encoding network (e.g., - The input dimension of the VAE encoder; subsequently, the encoding network generates a factorized Gaussian posterior distribution based on the projected token representation. Specifically, it can be expressed as:

[0051] in, and Let each represent the posterior distribution. The mean and standard deviation, express The corresponding latent semantic factors, This represents the construction operation of a diagonal matrix.

[0052] Then, the factorized Gaussian posterior distribution of each token is sampled to obtain the latent semantic factors of each token. The latent semantic factors corresponding to all tokens constitute the latent semantic sequence. ,in, This represents the total number of multimodal tokens; for example, the sampling process can be represented as:

[0053] in, Indicates standard Gaussian noise. Represents the standard normal distribution. Represents the identity matrix. This indicates element-wise multiplication.

[0054] In some embodiments, the step of selecting each latent semantic factor associated with the graph learning task to be processed from the latent semantic sequence as task-related semantic factors based on the interaction between different pre-learned latent semantic factors and different graph learning tasks includes: The latent semantic sequence is processed sequentially using a multilayer perceptron and a sigmoid function to obtain a soft selection mask associated with the graph learning task to be processed. The multilayer perceptron contains learnable parameters, which are used to characterize the relationship between different latent semantic factors and different graph learning tasks. The latent semantic sequence is processed using the soft selection mask to obtain task-related semantic factors.

[0055] In this embodiment, the intra-node semantic decoupling module uses a multilayer perceptron with learnable parameters (which pre-learns the effects of different latent dimensions on different downstream tasks) and a sigmoid function to generate a soft selection mask to select task-related semantic factors and suppress latent components that are irrelevant to the current task or have strong noise; the aforementioned process can be specifically represented as follows:

[0056] in, Indicates a soft selection mask. This represents the selected task-related semantic factors. Represents a latent semantic sequence. This represents the Sigmoid function. This represents a multilayer perceptron.

[0057] In some embodiments, the structural factors from the plurality of structural perspectives are determined by the following formula:

[0058] in, Indicates the first Structural factors from a structural perspective; Indicates the first Parallel graph neural network channels corresponding to each structural perspective; This represents the node feature matrix adapted to the channel input of the parallel graph neural network. The node feature matrix is ​​constructed based on the multimodal token sequences of the target node and each node in the graph neighborhood. Indicates the number of dimensions of the structural factors; This represents the number of channels in a parallel graph neural network; This indicates the number of the target node and all nodes in its graph neighborhood (or all nodes in the multimodal attribute graph); express The adaptive edge weight matrix is ​​determined by the following formula:

[0059] in, Represents a node With nodes exist The corresponding edge weights; the original edge weights Used to characterize the nodes in the multimodal property graph With nodes Does an edge exist between them (i.e., an edge in the multimodal attribute graph)? (original edge weights). This represents the hyperparameters used to balance the original topology and adaptive edge weights; and express Learnable transformation parameters; and Represents a node With nodes exist The nodes in the matrix are represented by T; T represents the transpose operation of the matrix. This represents the Sigmoid function.

[0060] In this embodiment, the inter-node structural decoupling module can pool the multimodal tokens of each node to obtain the node-level representation of the channel input of the adaptive graph neural network, and then form a node feature matrix from all node representations. ,in, This represents the target node and the set of nodes in its graph neighborhood. express The number of nodes in the middle.

[0061] For example, the node representation of the target node as follows:

[0062] in, This represents the multimodal token sequence of the target node, where d represents the number of hidden dimensions in the visual language model. This indicates a pooling operation, which can specifically employ average pooling, special aggregated token pooling, or other operations that can convert a variable-length token sequence into a fixed-length node vector.

[0063] In some embodiments, the LoRA hybrid expert module, based on a hybrid expert gating mechanism, adaptively selects each structural factor associated with the graph learning task to be processed as a task-related structural factor, and injects each task-related structural factor into the visual language model backbone network through the corresponding LoRA expert during the processing of the input hidden state, including: The LoRA hybrid expert module is based on a hybrid expert gating mechanism. It calculates the expert gating distribution for each linear layer according to the input hidden state of each linear layer in the visual language model backbone network. The LoRA hybrid expert module adaptively selects each structural factor associated with the learning task of the graph to be processed as the task-related structural factor according to the expert gating distribution corresponding to each linear layer. The input hidden state of the visual language model backbone network and the task-related structural factors are processed by the corresponding LoRA experts and then combined to obtain the hybrid expert output corresponding to each linear layer. The visual language model backbone network generates prediction results for the learning task of the graph to be processed based on the processing results of the input hidden state, including: The visual language model backbone network uses the hybrid expert output corresponding to each linear layer as the processing result of the input hidden state output of each linear layer, and continues to generate prediction results for the learning task of the graph to be processed through subsequent network parts.

[0064] In the embodiments of this application, see Figure 2 The diagram illustrates the overall framework of a decoupled graph visual language model for multimodal attribute graph learning. The LoRA hybrid expert module, based on a hybrid expert gating mechanism, calculates the expert gating distribution according to the input hidden states of each linear layer in the visual language model backbone network, such as the input hidden states of each Transformer block (which are the hidden states passed from the previous layer in each Transformer Block, containing multimodal semantic information, task instruction information, and contextual information). For example, the aforementioned process can be represented as:

[0065] in, Indicates the distribution of expert gating. This represents the current hidden state of the input of the linear layer. This represents a learnable gated matrix. Indicates the number of LoRA experts. This represents the Softmax function.

[0066] For the input hidden state of the current linear layer in the backbone network of the visual language model (for example Figure 2 (output of Transformer Block 1), the The processing results output by a LoRA expert It can be represented as:

[0067] Where W represents the frozen original weights (such as the weights of the original linear layers in the feedforward network of the Transformer block of the visual language model backbone network, which are directly loaded by the pre-trained model (i.e., the visual language model backbone network). and Indicates the first A trainable low-rank matrix of LoRA experts Represents the scaling factor for Low-Rank Adaptation (LoRA), parameters Used to put the first Structural factors from the structural perspective corresponding to each LoRA expert Projected onto the input dimension of the current linear layer. Thus, each LoRA expert has an independent low-rank update path corresponding to a structure factor, thereby freezing the linear layer weights based on standard LoRA. Add low-rank updates.

[0068] Based on this, the final hybrid expert output corresponding to the current linear layer can be expressed as: :

[0069] in, express The Middle The weight of each LoRA expert.

[0070] Therefore, the LoRA hybrid expert module can adaptively select and combine task-related structural factors for different input hidden states, and stably inject them into the backbone network of the visual language model to process the input hidden state.

[0071] Subsequently, the visual language model backbone network can use the hybrid expert output as the output of the current adaptive linear layer, and continue to pass through subsequent network parts (such as feedforward networks and subsequent Transformer Blocks) to generate the prediction results of the downstream graph learning task.

[0072] In some embodiments, the decoupled graph visual language model is constructed using the following two-stage curriculum training strategy: In the first stage, the visual language model backbone network is frozen, the hybrid expert gating mechanism of the LoRA hybrid expert module is turned off, all LoRA experts are activated with equal weights, and the intra-node semantic decoupling module, the intra-node semantic decoupling module, and the LoRA hybrid expert module are jointly trained; wherein, the total training loss used in the first stage is constructed based on the task loss, structural decoupling loss, and semantic decoupling loss of the instruction fine-tuning in the first stage; In the second stage, the visual language model backbone network and the intra-node semantic decoupling module obtained in the first stage are frozen, the hybrid expert gating mechanism of the LoRA hybrid expert module is enabled, and the intra-node semantic decoupling module and the LoRA hybrid expert module are trained together; wherein, the total training loss used in the second stage is constructed based on the task loss and semantic decoupling loss of the second stage.

[0073] In this embodiment, considering the expert collapse problem that can easily occur when directly training multiple experts using related technologies, and the fact that directly stitching together neighboring images and text increases the length of the input sequence and computational overhead, this application adopts a two-stage training approach: first, expert specialization, and then expert combination, to improve training stability. Simultaneously, a compact structure factor is used to carry neighborhood information, thereby reducing training and inference overhead.

[0074] Specifically, in the first stage, the visual language model backbone network is frozen, and the hybrid expert gating mechanism of the LoRA hybrid expert module is disabled. All LoRA experts are activated with equal weights, and the intra-node semantic decoupling module, the intra-node semantic decoupling module (such as the parallel graph neural network channel), and the LoRA hybrid expert module (such as each LoRA expert) are trained jointly to develop specialized capabilities for extracting and processing structural factors from different structural perspectives. It is understood that this stage trains each LoRA expert equally to avoid premature gating bias towards a few LoRA experts who are not yet fully specialized. For example, the total training loss in the first stage... It can be represented as follows:

[0075] in, This indicates the task loss due to the fine-tuning of instructions in the first phase. and These represent the structural decoupling losses, respectively. and semantic decoupling loss The weight.

[0076] The second stage freezes the visual language model backbone network and the intra-node semantic decoupling modules obtained in the first stage (such as the parallel graph neural network channels). It then enables the hybrid expert gating mechanism of the LoRA hybrid expert module and jointly trains the intra-node semantic decoupling module and the LoRA hybrid expert module (such as the LoRA experts and the gating network). This allows the model to adaptively combine LoRA experts corresponding to different structural factors based on the current hidden state. Understandably, this stage learns expert combinations based on the extraction of structural factors from different structural perspectives and the preliminary specialization of corresponding LoRA experts, thereby reducing the risk of expert collapse and improving training stability. For example, the total training loss in the second stage... It can be represented as follows:

[0077] in, This indicates the mission losses in the second phase. Represents semantic decoupling loss The weight.

[0078] Therefore, this application completes structural factor learning and expert specialization through two-stage course training, and then learns expert combinations, which can reduce the risk of expert collapse and improve training stability. Moreover, using compact structural factors to carry neighborhood information can avoid directly splicing a large number of neighbor images and texts into the input of the visual language model backbone network, thereby reducing training and inference overhead.

[0079] In some embodiments, the semantic decoupling loss is determined by the following formula:

[0080] in, This represents the semantic decoupling loss; This represents the reconstruction loss and is used to preserve valid information in the token. This represents the KL divergence regularization term, used to promote the formation of independent and regular distributions of latent semantic factors; and express and The weight.

[0081] In this embodiment of the application, the intra-node semantic decoupling module can utilize a decoding network to decouple the latent semantic sequences. Reconstructing the original token space yields the reconstructed token sequence. Subsequently, the reconstruction loss can be calculated using the following formula. and KL divergence regularization term :

[0082] in, express The corresponding original multimodal token sequence, Represents the factorized Gaussian posterior distribution. Represents the standard normal distribution. This represents the KL divergence operation.

[0083] In some embodiments, the weight of the KL divergence regularization term is determined by the following formula:

[0084] in, This represents the weight of the KL divergence regularization term at the current training step t. Indicates the number of preheating steps. represents the upper limit of the weight of the KL divergence regularization term, and min represents the minimum value function.

[0085] In this embodiment of the application, in order to avoid the destruction of reconstruction by excessively strong KL constraints in the early stage of training, this application further adopts a weighted annealing strategy that increases with the number of training steps to determine the weight of the KL divergence regularization term.

[0086] In some embodiments, the structural decoupling loss is determined by the following formula:

[0087] in, This represents the structural decoupling loss. This represents the number of channels in a parallel graph neural network. and Let represent the structure factor output by each channel of the parallel graph neural network corresponding to the i-th and j-th structural perspectives, and Cov denote the covariance operation. This represents the number of hidden dimensions in the visual language model.

[0088] In this embodiment, to prevent multiple parallel graph neural network channels from converging to the same structural representation, a structural decoupling loss is constructed based on pairwise squared distance covariance to constrain the statistical dependence between different structural factors. Thus, by minimizing... This can reduce the dependence between different structural factors, thereby prompting each channel to extract decoupled neighborhood relationship patterns.

[0089] Based on the above embodiments, this application designs and trains a decoupled graph visual language model for multimodal attribute graphs. While preserving the native graph-text alignment and reasoning capabilities of the visual language model backbone network, it extracts task-related semantic factors from the multimodal tokens of the target node and structural factors that distinguish each other from the graph neighborhood through intra-node semantic decoupling modules and inter-node structural decoupling modules. Furthermore, it utilizes a structure factor-aware LoRA hybrid expert module and a two-stage curriculum training mechanism to inject semantic and structural factors into the visual language model backbone network in a parameter-efficient and stable manner. This improves the accuracy and training stability of multimodal attribute graph prediction and reduces the computational overhead caused by directly splicing neighbor multimodal information.

[0090] This application provides a decoupled graph visual language model for multimodal attribute graph learning, the model comprising: The multimodal encoding module is used to encode the multimodal attributes of the target node associated with the graph learning task to be processed and each node in the graph neighborhood in the multimodal attribute graph into a multimodal token sequence in a unified token space. The intra-node semantic decoupling module is used to process the multimodal token sequence of the target node into a latent semantic sequence, and based on the pre-learned relationship between different latent semantic factors and different graph learning tasks, to select each latent semantic factor associated with the graph learning task to be processed as the relevant semantic factor of each task from the latent semantic sequence. The inter-node structural decoupling module is used to process the multimodal token sequences of the target node and each node in the graph neighborhood into structural factors under multiple structural perspectives by utilizing the parallel graph neural network channels corresponding to multiple structural perspectives. A visual language model backbone network is used to generate input hidden states based on the semantic factors related to each task; and to generate prediction results for the learning task of the graph to be processed based on the processing results of the input hidden states. The LoRA hybrid expert module is used to adaptively select each structural factor associated with the learning task of the graph to be processed as the task-related structural factor based on the hybrid expert gating mechanism, and inject each task-related structural factor into the processing process of the input hidden state of the visual language model backbone network through the corresponding LoRA expert. Different LoRA experts have independent low-rank update paths corresponding to structural factors under different structural perspectives.

[0091] Optionally, the intra-node semantic decoupling module is further configured to use the multimodal token sequence of the target node as a query, and the various task-related semantic factors as keys and values, to calculate an enhanced semantic signal through a cross-attention module with learnable query, key, and value projection, so that tokens of different modalities and different positions can obtain targeted semantic enhancement; and to inject the enhanced semantic signal into the input of the visual language model backbone network through residual connections, so that the visual language model backbone network can generate the input hidden state based on the enhanced semantic signal after residual connections and the multimodal token sequence of the target node.

[0092] Optionally, the intra-node semantic decoupling module is further configured to generate a factorized Gaussian posterior distribution for each Token in the multimodal Token sequence of the target node through an encoding network; sample the factorized Gaussian posterior distribution of each Token to obtain the latent semantic factors of each Token; and construct a latent semantic sequence based on the latent semantic factors of all Tokens in the multimodal Token sequence of the target node.

[0093] Optionally, the intra-node semantic decoupling module is further configured to process the latent semantic sequence sequentially using a multilayer perceptron and a sigmoid function to obtain a soft selection mask associated with the graph learning task to be processed. The multilayer perceptron contains learnable parameters, which are used to characterize the relationship between different latent semantic factors and different graph learning tasks. The module also uses the soft selection mask to process the latent semantic sequence to obtain semantic factors related to each task.

[0094] Optionally, the structural factors from the multiple structural perspectives are determined by the following formula:

[0095] in, Indicates the first Structural factors from a structural perspective; Indicates the first Parallel graph neural network channels corresponding to each structural perspective; This represents the node feature matrix adapted to the channel input of the parallel graph neural network. The node feature matrix is ​​constructed based on the multimodal token sequences of the target node and each node in the graph neighborhood. Indicates the number of dimensions of the structural factors; This represents the number of channels in a parallel graph neural network; This represents the target node and the number of nodes in its graph neighborhood; express The adaptive edge weight matrix is ​​determined by the following formula:

[0096] in, Represents a node With nodes exist The corresponding edge weights; the original edge weights Used to characterize the nodes in the multimodal property graph With nodes Does an edge exist between them? This represents the hyperparameters used to balance the original topology and adaptive edge weights; and express Learnable transformation parameters; and Represents a node With nodes exist The nodes in the matrix are represented by T; T represents the transpose operation of the matrix. This represents the Sigmoid function.

[0097] Optionally, the LoRA hybrid expert module is further configured to, based on the hybrid expert gating mechanism, calculate the expert gating distribution corresponding to each linear layer according to the input hidden state of each linear layer in the visual language model backbone network; and, based on the expert gating distribution corresponding to each linear layer, adaptively select each structural factor associated with the learning task to be processed as each task-related structural factor, and process the input hidden state of the visual language model backbone network and each task-related structural factor respectively through the corresponding LoRA experts before combining them to obtain the hybrid expert output corresponding to each linear layer; The visual language model backbone network is also used to take the hybrid expert output corresponding to each linear layer as the processing result of the input hidden state output of each linear layer, and continue to generate prediction results for the learning task of the graph to be processed through subsequent network parts.

[0098] Optionally, the decoupled graph visual language model is constructed using the following two-stage course training strategy: In the first stage, the visual language model backbone network is frozen, the hybrid expert gating mechanism of the LoRA hybrid expert module is turned off, all LoRA experts are activated with equal weights, and the intra-node semantic decoupling module, the intra-node semantic decoupling module, and the LoRA hybrid expert module are jointly trained; wherein, the total training loss used in the first stage is constructed based on the task loss, structural decoupling loss, and semantic decoupling loss of the instruction fine-tuning in the first stage; In the second stage, the visual language model backbone network and the intra-node semantic decoupling module obtained in the first stage are frozen, the hybrid expert gating mechanism of the LoRA hybrid expert module is enabled, and the intra-node semantic decoupling module and the LoRA hybrid expert module are trained together; wherein, the total training loss used in the second stage is constructed based on the task loss and semantic decoupling loss of the second stage.

[0099] Optionally, the semantic decoupling loss is determined by the following formula:

[0100] in, This represents the semantic decoupling loss; This represents the reconstruction loss and is used to preserve valid information in the token. This represents the KL divergence regularization term, used to promote the formation of independent and regular distributions of latent semantic factors; and express and The weight.

[0101] Optionally, the weight of the KL divergence regularization term is determined by the following formula:

[0102] in, This represents the weight of the KL divergence regularization term at the current training step t. Indicates the number of preheating steps. represents the upper limit of the weight of the KL divergence regularization term, and min represents the minimum value function.

[0103] Optionally, the structural decoupling loss is determined by the following formula:

[0104] in, This represents the structural decoupling loss. This represents the number of channels in a parallel graph neural network. and Let represent the structure factor output by each channel of the parallel graph neural network corresponding to the i-th and j-th structural perspectives, and Cov denote the covariance operation. This represents the number of hidden dimensions in the visual language model.

[0105] As can be seen from the above technical solution, this application constructs a decoupled graph visual language model for multimodal attribute graphs. It extracts task-related semantic factors from the target node using an intra-node semantic decoupling module as input to the visual language model backbone network. It also extracts structural factors from the target node and its neighbors using an inter-node structural decoupling module, distinguishing structural perspectives. Then, based on a hybrid expert gating mechanism, it adaptively selects task-related structural factors. Finally, using a structure factor-aware LoRA hybrid expert module, it injects these task-related structural factors into the visual language model backbone network's processing of the input hidden state. This combines task-related semantic factors and structural factors to generate prediction results for the graph learning task. Therefore, this application avoids directly splicing the multimodal information of a large number of neighboring nodes into the input sequence of the visual language model backbone network and effectively solves the problems of intra-node semantic entanglement and inter-node structural entanglement. This allows it to improve prediction accuracy and reduce computational overhead while preserving the original graph-text alignment capability of the visual language model backbone network.

[0106] Optionally, this application embodiment also provides an electronic device, including a processor 110, a memory 109, and a program or instructions stored in the memory 109 and executable on the processor 110. When the program or instructions are executed by the processor 110, they implement the various processes of the above-described embodiment of the decoupled graph visual language model construction method for multimodal attribute graph learning, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0107] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0108] Figure 3 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0109] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0110] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0111] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of the decoupled graph visual language model construction method for multimodal attribute graph learning, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0112] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0113] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the decoupled graph visual language model construction method for multimodal attribute graph learning, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0114] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0117] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A decoupled graph visual language model construction method for multi-modal attribute graph learning, characterized in that, The method includes: A decoupled graph visual language model is constructed for processing the graph learning task associated with multimodal attribute graphs. The decoupled graph visual language model includes a multimodal coding module, an intra-node semantic decoupling module, an inter-node structural decoupling module, a LoRA hybrid expert module, and a visual language model backbone network. The multimodal encoding module encodes the multimodal attributes of the target node associated with the graph learning task to be processed and each node in the graph neighborhood in the multimodal attribute graph into a multimodal token sequence in a unified token space. The intra-node semantic decoupling module processes the multimodal token sequence of the target node into a latent semantic sequence, and based on the pre-learned relationship between different latent semantic factors and different graph learning tasks, selects each latent semantic factor associated with the graph learning task to be processed from the latent semantic sequence as the semantic factors related to each task. The inter-node structural decoupling module utilizes parallel graph neural network channels corresponding to multiple structural perspectives to process the multimodal token sequences of the target node and each node in the graph neighborhood into structural factors under multiple structural perspectives. The visual language model backbone network generates input hidden states based on the semantic factors related to each task; The LoRA hybrid expert module is based on a hybrid expert gating mechanism. It adaptively selects each structural factor associated with the learning task of the graph to be processed as the task-related structural factor. The task-related structural factors are then injected into the visual language model backbone network through the corresponding LoRA experts to process the input hidden state. Different LoRA experts have independent low-rank update paths corresponding to structural factors under different structural perspectives. The visual language model backbone network generates prediction results for the learning task of the graph to be processed based on the processing results of the input hidden state.

2. The method according to claim 1, characterized in that, The method further includes: The intra-node semantic decoupling module uses the multimodal token sequence of the target node as a query and the semantic factors related to each task as keys and values. It calculates enhanced semantic signals through a cross-attention module with learnable query, key and value projection, so that tokens of different modalities and positions can obtain targeted semantic enhancement. The intra-node semantic decoupling module injects the enhanced semantic signal into the input of the visual language model backbone network through residual connections, so that the visual language model backbone network generates the input hidden state based on the enhanced semantic signal after residual connections and the multimodal token sequence of the target node.

3. The method according to claim 1, characterized in that, The intra-node semantic decoupling module processes the multimodal token sequence of the target node into a latent semantic sequence, including: The intra-node semantic decoupling module generates a factorized Gaussian posterior distribution for each token in the multimodal token sequence of the target node through an encoding network. The intra-node semantic decoupling module samples the factorized Gaussian posterior distribution of each Token to obtain the latent semantic factors of each Token. The intra-node semantic decoupling module constructs a latent semantic sequence based on the latent semantic factors of all tokens in the multimodal token sequence of the target node.

4. The method according to claim 1, characterized in that, The relationship between different pre-learned latent semantic factors and different graph learning tasks is established by selecting each latent semantic factor associated with the graph learning task to be processed from the latent semantic sequence as the task-related semantic factors, including: The latent semantic sequence is processed sequentially using a multilayer perceptron and a sigmoid function to obtain a soft selection mask associated with the graph learning task to be processed. The multilayer perceptron contains learnable parameters, which are used to characterize the relationship between different latent semantic factors and different graph learning tasks. The latent semantic sequence is processed using the soft selection mask to obtain task-related semantic factors.

5. The method according to claim 1, characterized in that, The structural factors from the multiple structural perspectives are determined by the following formula: in, Indicates the first Structural factors from a structural perspective; Indicates the first Parallel graph neural network channels corresponding to each structural perspective; This represents the node feature matrix adapted to the channel input of the parallel graph neural network. The node feature matrix is ​​constructed based on the multimodal token sequences of the target node and each node in the graph neighborhood. Indicates the number of dimensions of the structural factors; This represents the number of channels in a parallel graph neural network; This represents the target node and the number of nodes in its graph neighborhood; express The adaptive edge weight matrix is ​​determined by the following formula: in, Represents a node With nodes exist The corresponding edge weights; the original edge weights Used to characterize the nodes in the multimodal property graph With nodes Does an edge exist between them? This represents the hyperparameters used to balance the original topology and adaptive edge weights; and express Learnable transformation parameters; and Represents a node With nodes exist The nodes in the matrix are represented by T; T represents the transpose operation of the matrix. This represents the Sigmoid function.

6. The method according to claim 1, characterized in that, The LoRA hybrid expert module, based on a hybrid expert gating mechanism, adaptively selects each structural factor associated with the learning task of the graph to be processed as a task-related structural factor, and injects each task-related structural factor into the visual language model backbone network through the corresponding LoRA expert in the process of processing the input hidden state, including: The LoRA hybrid expert module is based on a hybrid expert gating mechanism. It calculates the expert gating distribution for each linear layer according to the input hidden state of each linear layer in the visual language model backbone network. The LoRA hybrid expert module adaptively selects each structural factor associated with the learning task of the graph to be processed as the task-related structural factor according to the expert gating distribution corresponding to each linear layer. The input hidden state of the visual language model backbone network and the task-related structural factors are processed by the corresponding LoRA experts and then combined to obtain the hybrid expert output corresponding to each linear layer. The visual language model backbone network generates prediction results for the learning task of the graph to be processed based on the processing results of the input hidden state, including: The visual language model backbone network uses the hybrid expert output corresponding to each linear layer as the processing result of the input hidden state output of each linear layer, and continues to generate prediction results for the learning task of the graph to be processed through subsequent network parts.

7. The method according to any one of claims 1-6, characterized in that, The decoupled graph visual language model is constructed using the following two-stage course training strategy: In the first stage, the visual language model backbone network is frozen, the hybrid expert gating mechanism of the LoRA hybrid expert module is turned off, all LoRA experts are activated with equal weights, and the intra-node semantic decoupling module, the intra-node semantic decoupling module, and the LoRA hybrid expert module are jointly trained; wherein, the total training loss used in the first stage is constructed based on the task loss, structural decoupling loss, and semantic decoupling loss of the instruction fine-tuning in the first stage; In the second stage, the visual language model backbone network and the intra-node semantic decoupling module obtained in the first stage are frozen, the hybrid expert gating mechanism of the LoRA hybrid expert module is enabled, and the intra-node semantic decoupling module and the LoRA hybrid expert module are trained together; wherein, the total training loss used in the second stage is constructed based on the task loss and semantic decoupling loss of the second stage.

8. The method according to claim 7, characterized in that, The semantic decoupling loss is determined by the following formula: in, This represents the semantic decoupling loss; This represents the reconstruction loss and is used to preserve valid information in the token. This represents the KL divergence regularization term, used to promote the formation of independent and regular distributions of latent semantic factors; and express and The weight.

9. The method according to claim 8, characterized in that, The weight of the KL divergence regularization term is determined by the following formula: in, This represents the weight of the KL divergence regularization term at the current training step t. Indicates the number of preheating steps. represents the upper limit of the weight of the KL divergence regularization term, and min represents the minimum value function.

10. The method according to claim 7, characterized in that, The structural decoupling loss is determined by the following formula: in, This represents the structural decoupling loss. This represents the number of channels in a parallel graph neural network. and Let represent the structure factor output by each channel of the parallel graph neural network corresponding to the i-th and j-th structural perspectives, and Cov denote the covariance operation. This represents the number of hidden dimensions in the visual language model.

Images