Large language model training method and system based on knowledge graph enhancement

Abstract

The invention relates to the technical field of big language models, and discloses a big language model training method based on knowledge graph enhancement, comprising the following steps: S1, constructing a multi-source heterogeneous knowledge graph; s2, coding the mixed attention heterogeneity map; s3, bidirectionally mapping a pre-training task; and S4, position specific gating fusion. According to the big language model training method and system based on knowledge graph enhancement, a same proton graph is established for a structured triple and text entity description, nodes are connected across graph edges to form a heterogeneous graph, associated edges are established through entity linking and syntactic analysis, multi-source knowledge is modeled in a unified mode, and the problem of low fusion efficiency is solved; mixed attention coding adopts a layering mechanism, a semantic level calculates weights according to type compatibility, a node level calculates similarity aggregation features through cosine distance and path length, entity vectors are generated through pooling, map structures and semantics are explicitly learned, reasoning accuracy is improved, and the problem of knowledge understanding superficial layer is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and specifically to a large language model training method and system based on knowledge graph enhancement. Background Art

[0002] Large language model training is a process driven by large-scale text data and using self-supervised learning mechanisms to enable the model to learn language rules and semantic representations in order to achieve the ability to understand, generate and reason about natural language.

[0003] In the existing technology, large language models have two core technical problems in terms of knowledge utilization: first, the inefficient fusion of multi-source heterogeneous knowledge. Traditional methods can only utilize structured knowledge graphs or unstructured texts, and cannot unify the modeling of the two, resulting in one-sided entity semantic representation; second, knowledge understanding is shallow. The model often only injects knowledge as external features, and does not explicitly learn the structural relationships and semantic logic of the knowledge graph. As a result, factual errors or logical contradictions are easy to occur in tasks that require cross-entity relationship reasoning, and cannot meet high accuracy requirements. Therefore, a large language model training method and system based on knowledge graph enhancement is proposed. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a large language model training method and system based on knowledge graph enhancement, which has the advantages of improving the factual accuracy of the large language model, enhancing complex reasoning capabilities, and adapting to knowledge-intensive tasks in multiple fields. It solves two core technical problems in the existing technology of large language models in knowledge utilization: First, the fusion of multi-source heterogeneous knowledge is inefficient. Traditional methods can only use structured knowledge graphs or unstructured texts alone, and cannot unify the modeling of the two, resulting in one-sided entity semantic representation; second, knowledge understanding is shallow. The model often only injects knowledge as external features, and does not explicitly learn the structural relationships and semantic logic of the knowledge graph, which leads to factual errors or logical contradictions in tasks that require cross-entity relationship reasoning, and cannot meet high accuracy requirements.

[0006] (2) Technical solution

[0007] To achieve the above-mentioned purpose of improving the factual accuracy of large language models, enhancing complex reasoning capabilities, and adapting to knowledge-intensive tasks in multiple fields, the present invention provides the following technical solution: a large language model training method based on knowledge graph enhancement, comprising the following steps:

[0008] S1. Multi-source heterogeneous knowledge graph construction: The triple data of the structured knowledge graph and the entity description of the unstructured text are respectively constructed into homogeneous subgraphs. The corresponding entity nodes and text nodes are connected by cross-graph edges to form a heterogeneous graph containing three types of nodes: entity, relationship, and text. The entity nodes store entity ID, type label, and timestamp attributes, and the text nodes store sentence fragments and syntactic labels.

[0009] S2. Hybrid Attention Heterogeneous Graph Encoding: A hierarchical attention mechanism is used to encode heterogeneous graphs. First, the node type weights are calculated through semantic-level attention, then the weights of neighbors of the same type are calculated through node-level attention, and finally, an entity representation vector containing multi-source knowledge is generated through graph average pooling.

[0010] S3. Bidirectional Mapping Pre-training Task: Design a text-to-entity mapping task and an entity-to-text generation task. The former extracts the context vector of text entity mentions through self-attention pooling and maps it into knowledge embeddings. The latter converts the knowledge embeddings into natural language descriptions through a decoder, and optimizes model parameters through a joint loss function.

[0011] S4. Position-specific gating fusion: In the stage of fusion of knowledge representation and text representation, local gating and global gating units are introduced. Local gating generates attention weights based on the word position of the entity mention, and global gating generates fusion coefficients based on the overall semantics of the sentence. The two are weighted to control the strength of knowledge injection.

[0012] Preferably, in the step of constructing the multi-source heterogeneous knowledge graph:

[0013] A rule-based entity linking algorithm is used to link entity mentions in unstructured text to corresponding entity nodes in the structured knowledge graph through text string matching and alias dictionary mapping;

[0014] Add weight attributes to the relationship edges in the structured knowledge graph. The weight value is determined based on the relationship frequency and domain expert annotations.

[0015] The unstructured text is processed into sentences, and each sentence fragment is used as a text node. Through dependency syntactic analysis, a "description-entity" edge connection is established with the entity node.

[0016] Preferably, the semantic level attention mechanism in the hybrid attention heterogeneous graph encoding is specifically:

[0017] For the target entity node, calculate the type compatibility score between it and its neighboring nodes. The type compatibility is determined by the preset entity and relationship type matrix.

[0018] Generate an attention weight vector for each node type. The weight vector is obtained by performing a dot product operation between the target node representation and the type feature vector and then performing Softmax normalization.

[0019] According to the semantic level attention weights, the feature information of different types of neighbor nodes is aggregated.

[0020] Preferably, the node-level attention mechanism is specifically:

[0021] In the set of neighboring nodes of the same type, the semantic similarity between each node and the target entity node is calculated. The similarity is calculated by weighting the word embedding cosine distance and the graph path length.

[0022] Generate node-level attention weights through a multi-head attention mechanism and perform weighted aggregation on the features of neighboring nodes of the same type;

[0023] The aggregation results of the semantic level and the node level are spliced ​​and input into the feedforward neural network to generate the final entity representation vector.

[0024] Preferably, in the bidirectional mapping pre-training task:

[0025] The loss function for the text-to-entity mapping task is cosine similarity loss. The mapping accuracy is optimized by maximizing the similarity between text embeddings and knowledge embeddings. Text embeddings are generated by extracting the token encoding of entity mentions through self-attention pooling, and knowledge embeddings are the graph neural network outputs of the corresponding entity nodes.

[0026] The entity-to-text mapping task adopts masked language model loss to embed knowledge into the input decoder to generate a text sequence, and uses cross-entropy loss to constrain the semantic consistency of the generated text with the original entity description.

[0027] Preferably, the calculation process of the position-specific gated fusion is:

[0028] The local gating score is generated by the concatenation vector of the entity mention token representation and the knowledge representation through the hyperbolic tangent activation function;

[0029] The global gating score is generated by linearly transforming the encoded representation of the sentence start token;

[0030] The final fused representation is composed of a weighted combination of the original text representation, the local gating score, the global gating score, and the knowledge representation. The warm-up factor increases linearly with the training rounds to a preset value.

[0031] Preferably, a knowledge decoupling training strategy is also included:

[0032] In the pre-training phase, differentiated learning rates are used for high-frequency entities and long-tail entities, with the learning rate for high-frequency entities reduced by 50%;

[0033] During the fine-tuning phase, the knowledge graph embedding parameters are frozen and only the text encoding layer parameters of the large language model are updated;

[0034] In the reasoning phase, knowledge injection is disabled, and text generation relies solely on the knowledge semantics learned in the pre-training phase.

[0035] The large language model training system based on knowledge graph enhancement includes:

[0036] Heterogeneous graph construction module: This module includes an entity linking unit, a text sentence segmentation unit, and a graph storage unit. The entity linking unit achieves entity alignment through alias dictionary query, fuzzy matching algorithm, and conflict resolution components.

[0037] Hybrid Attention Encoding Module: This module includes a semantic-level attention submodule, a node-level attention submodule, and a graph pooling submodule. The graph pooling submodule uses a self-attention pooling method to generate a global representation vector for the entire graph.

[0038] Bidirectional mapping training module: This module includes an entity mapping submodule and a text generation submodule, which respectively implement text-to-entity mapping tasks and entity-to-text generation tasks;

[0039] Gated fusion module: includes local gating calculation unit and global gating calculation unit to realize location-specific gating fusion mechanism;

[0040] Training strategy module: includes learning rate scheduling unit and parameter freezing unit to implement knowledge decoupling training strategy.

[0041] Preferably, the entity linking unit of the heterogeneous graph construction module includes:

[0042] Alias ​​dictionary query component, which stores common aliases and synonyms of domain entities;

[0043] The fuzzy matching algorithm component uses the edit distance to calculate the similarity between text mentions and entity names, with the threshold set to 0.8;

[0044] The conflict resolution component resolves the ambiguity problem of multiple entities mentioned in the same text through a voting mechanism.

[0045] Preferably, the graph pooling submodule of the hybrid attention encoding module adopts a self-attention pooling method, specifically:

[0046] Perform multi-head self-attention calculation on the feature vectors of all nodes in the heterogeneous graph to generate node importance scores;

[0047] Perform weighted averaging of node features according to their importance scores to generate a global representation vector for the entire graph.

[0048] The global representation vector is concatenated with the entity node features as the knowledge-enhanced input of the large language model.

[0049] (3) Beneficial effects

[0050] Compared with the existing technology, the present invention provides a large language model training method and system based on knowledge graph enhancement, which has the following beneficial effects:

[0051] 1. This large language model training method and system based on knowledge graph enhancement builds a multi-source heterogeneous knowledge graph, constructs homogeneous sub-graphs for structured knowledge graph triples and unstructured text entity descriptions, and forms a heterogeneous graph by connecting entities and text nodes through cross-graph edges. The entity linking algorithm realizes the link between text mentions and entity nodes, and establishes "description-entity" edges through sentence segmentation and dependency syntax analysis. It uniformly models multi-source heterogeneous knowledge, solves the problem of inefficient fusion, and makes the entity semantic representation more comprehensive.

[0052] 2. This large language model training method and system based on knowledge graph enhancement uses a hierarchical attention mechanism for hybrid attention heterogeneous graph encoding. Semantic-level attention calculates node type weights, which are determined based on the type compatibility matrix; node-level attention calculates the weights of neighbors of the same type, and calculates semantic similarity through word embedding cosine distance and graph path length. After aggregating features, the two are averaged and pooled to generate entity representation vectors. This method explicitly learns graph structure and semantic logic, improves reasoning accuracy, and solves the problem of shallow knowledge understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the large language model training method of the present invention;

[0054] Figure 2 This is the architecture diagram of the large language model training system of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] See also Figure 1-2 ,The large language model training method based on knowledge graph enhancement includes the following steps:

[0057] S1. Multi-source heterogeneous knowledge graph construction: The triple data of the structured knowledge graph and the entity description of the unstructured text are respectively constructed into homogeneous subgraphs. The corresponding entity nodes and text nodes are connected by cross-graph edges to form a heterogeneous graph containing three types of nodes: entity, relationship, and text. The entity nodes store entity ID, type label, and timestamp attributes, and the text nodes store sentence fragments and syntactic labels.

[0058] S2. Hybrid Attention Heterogeneous Graph Encoding: A hierarchical attention mechanism is used to encode heterogeneous graphs. First, the node type weights are calculated through semantic-level attention, then the weights of neighbors of the same type are calculated through node-level attention, and finally, an entity representation vector containing multi-source knowledge is generated through graph average pooling.

[0059] S3. Bidirectional Mapping Pre-training Task: Design a text-to-entity mapping task and an entity-to-text generation task. The former extracts the context vector of text entity mentions through self-attention pooling and maps it into knowledge embeddings. The latter converts the knowledge embeddings into natural language descriptions through a decoder, and optimizes model parameters through a joint loss function.

[0060] S4. Position-specific gating fusion: In the stage of fusion of knowledge representation and text representation, local gating and global gating units are introduced. Local gating generates attention weights based on the word position of the entity mention, and global gating generates fusion coefficients based on the overall semantics of the sentence. The two are weighted to control the strength of knowledge injection.

[0061] The large language model training system based on knowledge graph enhancement includes:

[0062] Heterogeneous graph construction module: This module includes an entity linking unit, a text sentence segmentation unit, and a graph storage unit. The entity linking unit achieves entity alignment through alias dictionary query, fuzzy matching algorithm, and conflict resolution components.

[0063] Hybrid Attention Encoding Module: This module includes a semantic-level attention submodule, a node-level attention submodule, and a graph pooling submodule. The graph pooling submodule uses a self-attention pooling method to generate a global representation vector for the entire graph.

[0064] Bidirectional mapping training module: This module includes an entity mapping submodule and a text generation submodule, which respectively implement text-to-entity mapping tasks and entity-to-text generation tasks;

[0065] Gated fusion module: includes local gating calculation unit and global gating calculation unit to realize location-specific gating fusion mechanism;

[0066] Training strategy module: includes learning rate scheduling unit and parameter freezing unit to implement knowledge decoupling training strategy.

[0067] Example 1:

[0068] The multi-source knowledge fusion method based on heterogeneous graph neural networks requires data preparation first. It uses entity linking tools to establish corresponding relationships between entity mentions in the input text and entities in the structured knowledge graph, and extracts unstructured entity description text at the same time; it uses a personalized web page ranking algorithm to screen the key neighbor nodes of the target entity, sets the algorithm's damping coefficient and number of iterations, and retains highly correlated neighbor nodes to construct hybrid knowledge units.

[0069] In the process of constructing heterogeneous graphs, homogeneous graphs are constructed for structured graphs and unstructured texts respectively. The former uses the triple relationship between entities as edges, and the latter uses the keyword association in the text as edges. Then, the two types of homogeneous graphs are fused and cross-source edges are added to form a unified heterogeneous graph containing multiple node types such as entities, relationships, and descriptive texts.

[0070] The design of the hybrid attention heterogeneous graph convolutional network includes two-layer attention mechanisms: semantic level and node level. The semantic level attention is used to calculate the importance weights of different node types, and the node level attention is used to calculate the weights of neighboring nodes of the same type. The aggregate update of node representation is achieved through multi-layer graph convolution operations.

[0071] During knowledge fusion and injection, the textual representation of entity mentions is extracted from the middle layer of the pre-trained model and fused with the heterogeneous graph representation through a gating mechanism. Local gating and global gating are designed to control the intensity of knowledge injection and avoid noise interference.

[0072] The pre-training tasks include text-to-entity embedding mapping and entity-to-text generation. The former converts text representation into graph embedding through a neural network, and the latter uses relational modeling to generate text mentions and optimizes the model's two-way understanding of knowledge through a loss function.

[0073] Example 2:

[0074] The structured information utilization method based on knowledge context first screens the neighbor nodes of the target entity through an improved web page ranking algorithm, comprehensively considers the entity's frequency of occurrence in the corpus, semantic importance and connection density in the graph, sets frequency thresholds and calculation rules, and screens out neighbor entities that are more valuable for semantic enhancement.

[0075] The implementation of the hybrid attention mechanism is divided into two steps. First, the importance weight of the neighbor entity type is calculated, and the weight distribution is generated through the interaction between the target entity representation and the type aggregation representation. Then, the weight of the neighbor nodes of the same type is calculated, and the final attention weight is generated by combining the type weight and node representation to aggregate the semantic information of the neighbor nodes.

[0076] The knowledge context modeling pre-training task is based on the idea of ​​word embedding algorithm, and designs the relationship prediction task between entities and neighbors and the representation alignment task of entity mentions. The former optimizes the structural association between entities and neighbors through negative sampling, and the latter improves the consistency of entity mention text and graph embedding through the loss function.

[0077] In the model implementation, self-attention pooling is used to extract features of entity mentions, layer normalization technology is used to stabilize the training process, and the hybrid attention module is inserted into a specific layer of the pre-trained model to set the number of neighbor nodes and training hyperparameters such as learning rate and batch size.

[0078] Example 3:

[0079] The text-to-entity mapping training is based on the knowledge understanding enhancement method of bidirectional mapping pre-training. The text features of entity mentions are aggregated through self-attention pooling technology, and converted into representations in graph embedding space through a two-layer neural network. The prediction probability is calculated using cosine similarity, and negative samples are selected from other entities with similar semantics to the target entity. The mapping parameters are optimized through cross-entropy loss.

[0080] The entity-to-text mapping training utilizes the relational modeling algorithm, takes the interaction result of the head entity embedding and the relation embedding as the prediction of the tail entity embedding, introduces scaling factors and nonlinear transformations to generate text representations, uses the decoder module of the pre-trained model to generate entity mention text, shares the model weights and uses sampling soft maximization loss to optimize the generation effect.

[0081] In the joint training phase, the masked language model loss, entity link loss, and generation loss are combined, loss weight parameters are set, the model is trained using the optimizer, the learning rate and warm-up ratio are adjusted, the input sequence length and training cycle are controlled, and the performance of the model in knowledge reasoning tasks is ensured.

[0082] Example 4:

[0083] The decoupled knowledge learning method based on heterogeneous graph neural attention network first defines the screening criteria for long-tail entities, comprehensively considers entity frequency, semantic importance and connectivity, sets frequency threshold and calculation method, and identifies low-frequency but structurally important entities.

[0084] During the pseudo-token embedding injection process, a pseudo-embedding representation is constructed based on the head entity and tail entity relationship of the graph triple, and a hybrid representation is generated by combining the features of the entity description text, and the alignment of the representation space is achieved through the fully connected layer.

[0085] The relational knowledge decoder adopts the encoder structure of the pre-trained model, inputs the mixed representation and position encoding after initializing the weights, generates the probability distribution of text mentions through a multi-layer neural network, and uses cross-entropy loss to optimize the decoding process.

[0086] The decoupling design of the training phase enables the knowledge injection module in the pre-training phase. Only pre-training parameters are used in the fine-tuning and inference phases to reduce real-time computing overhead. The maximum length of the entity description text is set in the model implementation, the exponential moving average optimizer is used, and the batch size and learning rate parameters are adjusted.

[0087] Embodiment 5:

[0088] The contrastive learning enhancement method based on the domain implicit graph structure adopts hyperbolic space embedding to learn the hierarchical structure of entities, uses the hyperbolic geometric model to capture the hierarchical relationship between entities, optimizes the aggregation of similar entities and the separation of heterogeneous entities through the distance function, sets the training cycle and learning rate, and uses the optimizer to update the embedding parameters.

[0089] Multi-level negative sample construction is based on the biconnected components of the graph. Negative samples of different difficulty levels are generated according to the jump distance. Entities of the same category are prioritized to increase the difficulty of distinction. Paths are obtained and high-quality negative samples are screened through breadth-first search.

[0090] The contrastive learning loss function uses the information-noise contrast estimation method to calculate the cosine similarity between the target entity representation and the positive and negative sample representations. The loss weight is adjusted through the temperature hyperparameter. In the model implementation, a parallel connection between the contrastive learning module and the pre-trained model is designed, and layer normalization and residual connection techniques are used to set parameters such as embedding dimension and batch size.

[0091] In summary, the large language model training method and system based on knowledge graph enhancement, through the construction of multi-source heterogeneous knowledge graphs, constructs homogeneous sub-graphs of structured knowledge graph triples and unstructured text entity descriptions respectively, connects entities and text nodes through cross-graph edges to form a heterogeneous graph, and realizes the linking of text mentions and entity nodes through entity linking algorithm. The sentence and dependency syntax analysis establishes "description-entity" edges, uniformly models multi-source heterogeneous knowledge, solves the problem of inefficient fusion, and makes the entity semantic representation more comprehensive.

[0092] In addition, the large language model training method and system based on knowledge graph enhancement adopts a hierarchical attention mechanism for hybrid attention heterogeneous graph encoding. The semantic level attention calculates the node type weight, which is determined according to the type compatibility matrix; the node level attention calculates the weight of neighbors of the same type, and calculates the semantic similarity through the word embedding cosine distance and the graph path length. After aggregating the features of the two, the entity representation vector is generated by graph average pooling. The graph structure and semantic logic are explicitly learned to improve the reasoning accuracy and solve the problem of shallow knowledge understanding. It solves the two core technical problems of large language models in knowledge utilization in the existing technology: first, the inefficient fusion of multi-source heterogeneous knowledge. Traditional methods can only use structured knowledge graphs or unstructured texts, and cannot unify the modeling of the two, resulting in one-sided entity semantic representation; second, the shallow knowledge understanding. The model often only injects knowledge as external features, and does not explicitly learn the structural relationship and semantic logic of the knowledge graph, which leads to factual errors or logical contradictions in tasks that require cross-entity relationship reasoning, and cannot meet the high accuracy requirements.

[0093] The relevant modules involved in this system are all hardware system modules or functional modules that combine computer software programs or protocols with hardware in the existing technology. The computer software programs or protocols involved in the functional modules are themselves technologies that are well known to those skilled in the art and are not improvements to this system. The improvements to this system are the interaction or connection relationships between the modules, that is, improvements to the overall structure of the system to solve the corresponding technical problems to be solved by this system.

[0094] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A large language model training method based on knowledge graph enhancement, characterized in that: The following steps are involved: S1. Multi-source heterogeneous knowledge graph construction: The triple data of the structured knowledge graph and the entity description of the unstructured text are respectively constructed into homogeneous subgraphs. The corresponding entity nodes and text nodes are connected by cross-graph edges to form a heterogeneous graph containing three types of nodes: entity, relationship, and text. The entity nodes store entity ID, type label, and timestamp attributes, and the text nodes store sentence fragments and syntactic labels. S2. Hybrid Attention Heterogeneous Graph Encoding: A hierarchical attention mechanism is used to encode heterogeneous graphs. First, the node type weights are calculated through semantic-level attention, then the weights of neighbors of the same type are calculated through node-level attention, and finally, an entity representation vector containing multi-source knowledge is generated through graph average pooling. S3. Bidirectional Mapping Pre-training Task: Design a text-to-entity mapping task and an entity-to-text generation task. The former extracts the context vector of text entity mentions through self-attention pooling and maps it into knowledge embeddings. The latter converts the knowledge embeddings into natural language descriptions through a decoder, and optimizes model parameters through a joint loss function. S4. Position-specific gating fusion: In the stage of fusion of knowledge representation and text representation, local gating and global gating units are introduced. Local gating generates attention weights based on the word position of the entity mention, and global gating generates fusion coefficients based on the overall semantics of the sentence. The two are weighted to control the strength of knowledge injection.

2. The large language model training method based on knowledge graph enhancement according to claim 1 is characterized in that: In the multi-source heterogeneous knowledge graph construction step: A rule-based entity linking algorithm is used to link entity mentions in unstructured text to corresponding entity nodes in the structured knowledge graph through text string matching and alias dictionary mapping; Add weight attributes to the relationship edges in the structured knowledge graph. The weight value is determined based on the relationship frequency and domain expert annotations. Unstructured text is segmented into sentences, with each sentence fragment treated as a text node. Dependency analysis is used to establish a "description-entity" edge connection with the entity node.

3. The large language model training method based on knowledge graph enhancement according to claim 1 is characterized in that The semantic level attention mechanism in the hybrid attention heterogeneous graph encoding is specifically: For the target entity node, calculate the type compatibility score between it and its neighboring nodes. The type compatibility is determined by the preset entity and relationship type matrix. Generate an attention weight vector for each node type. The weight vector is obtained by performing a dot product operation between the target node representation and the type feature vector and then performing Softmax normalization. According to the semantic level attention weights, the feature information of different types of neighbor nodes is aggregated.

4. The large language model training method based on knowledge graph enhancement according to claim 1 is characterized in that The node-level attention mechanism is specifically: In the set of neighboring nodes of the same type, the semantic similarity between each node and the target entity node is calculated. The similarity is calculated by weighting the word embedding cosine distance and the graph path length. Generate node-level attention weights through a multi-head attention mechanism and perform weighted aggregation on the features of neighboring nodes of the same type; The aggregation results of the semantic level and the node level are spliced ​​and input into the feedforward neural network to generate the final entity representation vector.

5. The large language model training method based on knowledge graph enhancement according to claim 1 is characterized in that In the bidirectional mapping pre-training task: The loss function for the text-to-entity mapping task is cosine similarity loss. The mapping accuracy is optimized by maximizing the similarity between text embeddings and knowledge embeddings. Text embeddings are generated by extracting the token encoding of entity mentions through self-attention pooling, and knowledge embeddings are the graph neural network outputs of the corresponding entity nodes. The entity-to-text mapping task adopts masked language model loss to embed knowledge into the input decoder to generate a text sequence, and uses cross-entropy loss to constrain the semantic consistency of the generated text with the original entity description.

6. The large language model training method based on knowledge graph enhancement according to claim 1 is characterized in that The calculation process of the position-specific gated fusion is: The local gating score is generated by the concatenation vector of the entity mention token representation and the knowledge representation through the hyperbolic tangent activation function; The global gating score is generated by linearly transforming the encoded representation of the sentence start token; The final fused representation is composed of a weighted combination of the original text representation, the local gating score, the global gating score, and the knowledge representation. The warm-up factor increases linearly with the training rounds to a preset value.

7. The large language model training method based on knowledge graph enhancement according to claim 1 is characterized in that It also includes knowledge decoupling training strategies: In the pre-training phase, differentiated learning rates are used for high-frequency entities and long-tail entities, with the learning rate for high-frequency entities reduced by 50%; During the fine-tuning phase, the knowledge graph embedding parameters are frozen and only the text encoding layer parameters of the large language model are updated; In the reasoning phase, knowledge injection is disabled, and text generation relies solely on the knowledge semantics learned in the pre-training phase.

8. A large language model training system based on knowledge graph enhancement, characterized by: include: Heterogeneous graph construction module: This module includes an entity linking unit, a text sentence segmentation unit, and a graph storage unit. The entity linking unit achieves entity alignment through alias dictionary query, fuzzy matching algorithm, and conflict resolution components. Hybrid Attention Encoding Module: This module includes a semantic-level attention submodule, a node-level attention submodule, and a graph pooling submodule. The graph pooling submodule uses a self-attention pooling method to generate a global representation vector for the entire graph. Bidirectional mapping training module: This module includes an entity mapping submodule and a text generation submodule, which respectively implement text-to-entity mapping tasks and entity-to-text generation tasks; Gated fusion module: includes local gating calculation unit and global gating calculation unit to realize location-specific gating fusion mechanism; Training strategy module: includes learning rate scheduling unit and parameter freezing unit to implement knowledge decoupling training strategy.

9. The large language model training system based on knowledge graph enhancement according to claim 8 is characterized in that: The entity linking unit of the heterogeneous graph construction module includes: Alias ​​dictionary query component, which stores common aliases and synonyms of domain entities; The fuzzy matching algorithm component uses the edit distance to calculate the similarity between text mentions and entity names, with the threshold set to 0.8; The conflict resolution component resolves the ambiguity problem of multiple entities mentioned in the same text through a voting mechanism.

10. The large language model training system based on knowledge graph enhancement according to claim 8 is characterized in that: The graph pooling submodule of the hybrid attention encoding module adopts the self-attention pooling method, specifically: Perform multi-head self-attention calculation on the feature vectors of all nodes in the heterogeneous graph to generate node importance scores; Perform weighted averaging of node features according to their importance scores to generate a global representation vector for the entire graph. The global representation vector is concatenated with the entity node features as the knowledge-enhanced input of the large language model.

Images