Model training method, electronic equipment and computer readable storage medium

Abstract

The invention provides a model training method, electronic equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence, and the method is applied to training of a knowledge graph completion model.According to the application, a multi-head adapter is introduced into the knowledge graph completion model. The model can learn knowledge of a specific task while keeping rich general knowledge stored in the feedforward network; the multi-head adapter adaptively fuses the output of the adapter and the output of the feed-forward network through a gating mechanism, so that the expression capability and flexibility of the model are enhanced; according to the method, the tail entities of the positive samples in the triple data used in each batch of training are replaced by other correct tail entities in the knowledge graph to generate the additional loss function of the positive samples, so that more positive samples can be introduced, the model can learn more diversified positive samples, and the representativeness and generalization ability of embedding representation are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a model training method, an electronic device, and a computer-readable storage medium in the field of artificial intelligence technology. Background Art

[0002] A knowledge graph is a structured data form used to represent entities and the relationships between them. It stores information in the form of triples (head entity h, relation r, tail entity t) and is widely used in fields such as search engines, recommendation systems, and question-and-answer systems.

[0003] Although knowledge graphs have been widely used, due to limitations in data collection, data quality issues, and the dynamic changes in the real world, knowledge graphs are usually incomplete. Therefore, how to complete knowledge graphs has become an urgent problem to be solved. Summary of the Invention

[0004] Embodiments of this application provide a model training method, an electronic device, and a computer-readable storage medium. On the one hand, by introducing a multi-head adapter into the knowledge graph completion model in embodiments of this application, the model can learn task-specific knowledge while maintaining the rich general knowledge stored in the feed-forward network. The multi-head adapter adaptively fuses the outputs of the adapter and the feed-forward network through a gating mechanism, enhancing the model's expressive ability and flexibility. On the other hand, by replacing the tail entity of the positive sample in the triple data used for each batch of training with other correct tail entities in the knowledge graph to generate an additional positive sample loss function, more positive samples can be introduced, enabling the model to learn more diverse positive samples, thereby improving the representativeness and generalization ability of the embedding representation. On the further hand, applying the trained knowledge graph completion model to complete the knowledge graph improves the accuracy of using the knowledge graph model to identify the correct tail entity and ensures the accuracy of the tail entity added to the knowledge graph.

[0005] In a first aspect, a model training method is provided, which is applied to the training of a knowledge graph completion model. The knowledge graph completion model includes: u first encoders, u second encoders, a first pooling layer, a second pooling layer, a query embedding layer, and a key embedding layer, where u≥2; the u first encoders are connected in series in sequence, and the u second encoders are connected in series in sequence; for each of the u first encoders and the u second encoders, the encoder includes a multi-head attention layer, a first addition and normalization layer, a feed-forward network, a multi-head adapter, a gating layer, and a second addition and normalization layer; the multi-head attention layer, the first addition and normalization layer, the feed-forward network, the gating layer, and the second addition and normalization layer are connected in series in sequence, and the first addition and normalization layer, the multi-head adapter, and the gating layer are connected in series in sequence; the multi-head adapter includes a linear layer, a connection layer, and an output layer, the linear layer, the connection layer, and the output layer are connected in sequence, there are v linear layers, the input ends of the v linear layers are connected to the output end of the first addition and normalization layer, and the output end of the output layer is connected to the input end of the gating layer, where v≥2; the input ends of the multi-head attention layer and the first addition and normalization layer in the first first encoder are connected to the output end of the query embedding layer, and the output end of the second addition and normalization layer in the u-th first encoder is connected to the input end of the first pooling layer; the input ends of the multi-head attention layer and the first addition and normalization layer in the first second encoder are connected to the output end of the key embedding layer, and the output end of the second addition and normalization layer in the u-th second encoder is connected to the input end of the second pooling layer; the model training method includes: grouping k triple data obtained based on the knowledge graph to obtain f groups of triple data; where each group of triple data includes z triple data, and each triple data includes a head entity, a relationship, and a tail entity, z = k / f, k≥2, 1 < f < k; determining z input data corresponding to each group of triple data; where, for each input data corresponding to each group of triple data, each input data includes query input data and key input data, the query input data includes the head entity and relationship of one triple data in each group of triple data, the key input data includes the tail entities of the z triple data respectively, and the z input data corresponding to each group of triple data are used for one batch of training of the knowledge graph completion model; for each batch of training of the knowledge graph completion model, the query input data in one input data corresponding to each group of triple data is input into the query embedding layer in sequence, and at the same time, the key input data in the one input data corresponding to each group of triple data is input into the key embedding layer, the head entity-relationship pair is encoded by the u first encoders, and the head entity-relationship pair embedding vector is output by the first pooling layer, and the tail entities are encoded by the u second encoders, and k tail entity embedding vectors are output by the second pooling layer;Train the knowledge graph completion model according to the head entity-relationship pair embedding vector, the k tail entity embedding vectors, and the target loss function; wherein, the target loss function includes a self-positive sample loss function and other positive sample loss functions, and the self-positive sample loss function refers to a loss function that regards the triple data marked as positive samples in each batch of training data as positive samples, and the other positive sample loss function is a loss function that generates additional positive samples by replacing the tail entity of the positive sample in the triple data used for each batch of training with other correct tail entities in the knowledge graph.;

[0006] In an embodiment of the present application, by adopting the above model training scheme, on the one hand, by designing a multi-head adapter for the feed-forward network in each encoder included in the knowledge graph completion model, while retaining the original general knowledge stored in the feed-forward network, this multi-head adapter learns new knowledge specific to the knowledge graph completion task and only trains a small number of parameters. This design alleviates the problem of information pattern conflict and effectively solves the problem that the feed-forward network learns similar features between adjacent encoder layers. On the other hand, by proposing an other positive sample loss function and using the target loss function composed of the self-positive sample loss function and the other positive sample loss function to optimize the model. Since the self-positive sample loss function only considers the triple data marked as positive samples in the triple data used for the current batch of training, and the other positive sample loss function generates additional positive samples by replacing the tail entity of the positive sample in the triple data used for the current batch of training with other correct tail entities in the knowledge graph, more positive samples are introduced during the model training process, which can help the model learn more diverse positive samples to discover more potential correct relationships and avoid the model's over-reliance on existing positive samples. This method corrects the bias problem in head-relationship pair embedding learning, not only improving the representativeness of the embedding representation but also enhancing the generalization ability of the model. On the other hand, applying the trained knowledge graph completion model to complete the knowledge graph improves the accuracy of using the knowledge graph model to identify the correct tail entity and ensures the accuracy of the tail entity added to the knowledge graph.

[0007] In a possible implementation manner, before determining the z input data corresponding to each group of triple data, the model training method further includes: freezing the parameters of all feed-forward networks in the knowledge graph completion model.

[0008] In a possible implementation, the encoding of the entity-relationship pairs by the u first encoder encoding heads includes: for each of the u first encoders, dividing the output result of the first addition and normalization layer of the first encoder into v sub-results; inputting the v sub-results into the v linear layers of the first encoder one by one, and each linear layer performs a linear transformation on the input sub-result to obtain the transformation result of the input sub-result in the subspace corresponding to each linear layer; wherein, the subspaces corresponding to each linear layer are different; inputting the v transformation results into the connection layer of the first encoder to map the v transformation results to the same space to obtain the output result of the connection layer of the first encoder; inputting the output result of the connection layer of the first encoder into the output layer of the first encoder, and the gating layer of the first encoder fuses the output result of the output layer of the first encoder and the output result of the feed-forward network of the first encoder to obtain the output result of the gating layer of the first encoder; inputting the output result of the gating layer of the first encoder into the second addition and normalization layer of the first encoder until the output result of the second addition and normalization layer in the u-th first encoder is input into the first pooling layer.

[0009] In a possible implementation, the fusion of the output result of the output layer of the first encoder and the output result of the feed-forward network of the first encoder by the gating layer of the first encoder to obtain the output result of the gating layer of the first encoder includes:

[0010] e (l) = AddNorm(a (l) ·FFN (l) (X)+(1 - a (l) )·Adapter (l) (X));

[0011] Wherein, e (l) represents the output result of the gating layer of the first encoder, FFN (l) (X) represents the output result of the feed-forward network of the first encoder, Adapter (l) (X) represents the output result of the output layer of the first encoder, a (l) is a trainable parameter used to control the fusion degree of FFN (l) (X) and Adapter (l) (X), and l represents the first encoder.

[0012] In a possible implementation, training the knowledge graph completion model according to the head entity-relation pair embedding vector, the k tail entity embedding vectors, and the target loss function includes: inputting the head entity-relation pair embedding vector and the k tail entity embedding vectors into the target loss function to obtain the loss function value for training the knowledge graph completion model in each batch; judging, according to the loss function value obtained by training the knowledge graph completion model in each batch, whether there is a change trend that after the loss function value drops to the loss threshold and then exceeds the loss threshold; if not, continue training the knowledge graph completion model; if so, stop training the knowledge graph completion model; wherein, the representation formula of the target loss function is as follows:

[0013] L MPS =λL SPS +(1 - λ)L OPS

[0014]

[0015] L MSP represents the target loss function, L SPS represents the self-positive sample loss function, L OPS represents the other positive sample loss function, λ is a hyperparameter, and by adjusting λ, the model can achieve a balance between accuracy and generalization. e represents the embedding vector, S hr represents the similarity, t represents the true tail entity given the head entity h and the relation r, and N represents all the wrong tail entities in the triple data used for training in each batch γ represents the hyperparameter that controls the boundary separation, τ represents the hyperparameter that controls the sharpness of the probability distribution, and P represents the set of all other correct tail entities related to the head entity h under the relation r of.

[0016] In a possible implementation, after the training of the knowledge graph completion model is completed, the model training method further includes: obtaining search description information, and extracting a target head entity-relation pair from the search description information; inputting the target head entity-relation pair into the knowledge graph completion model to obtain p candidate tail entities; wherein each candidate tail entity carries a score value, and the score value is used to represent the similarity between the candidate tail entity and the target head entity-relation pair, p≥2; determining the target tail entity corresponding to the target head entity-relation pair from the p candidate tail entities according to the score value; associating the target tail entity with the target head entity-relation pair and storing it in the knowledge graph to complete the knowledge graph.

[0017] In a possible implementation, determining the target tail entity corresponding to the target head entity-relationship pair from the p candidate tail entities according to the score value includes: sorting the p candidate tail entities based on the descending order of the score value to obtain a first sorting result; obtaining the first q candidate tail entities in the first sorting result to obtain a second sorting result, where 1 < q ≤ p; determining whether the difference value between the score value of the first candidate tail entity and the score value of the second candidate tail entity in the second sorting result is less than a preset threshold; if so, determining the first candidate tail entity in the second sorting result as the target tail entity; if not, using a large language model to correct the score values of the candidate tail entities in the second sorting result to obtain a third sorting result; and determining the first candidate tail entity in the third sorting result as the target tail entity.

[0018] In a possible implementation, using the large language model to correct the score values of the candidate tail entities in the second sorting result to obtain a third sorting result includes: using the large language model to recommend at least one candidate tail entity corresponding to the target head entity-relationship pair from the second sorting result; adding scores to the score values of the at least one candidate tail entity to obtain corrected score values of the at least one candidate tail entity; and using the corrected score values of the at least one candidate tail entity to correspondingly update the score values of the at least one candidate tail entity in the second sorting result to obtain the third sorting result.

[0019] In a second aspect, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the electronic device executes the model training method in the first aspect or any possible implementation manner of the first aspect.

[0020] In a third aspect, a computer program product is provided, which includes: computer program code, when the computer program code runs on a computer, enabling the computer to execute the model training method in the first aspect or any possible implementation manner of the first aspect.

[0021] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program code, and when the computer program code runs on a computer, enabling the computer to execute the model training method in the first aspect or any possible implementation manner of the first aspect. Description of the Drawings

[0022] Figure 1 A schematic flowchart of a model training method provided by an embodiment of the present application is shown;

[0023] Figure 2 Shows the architecture diagram of a knowledge graph completion model provided by an embodiment of the present application;

[0024] Figure 3 Shows an exemplary comparative analysis data graph of the knowledge graph completion model provided by the present application and the baseline method;

[0025] Figure 4 Shows another exemplary comparative analysis data graph of the knowledge graph completion model provided by the present application and the baseline method;

[0026] Figure 5 Shows the centered alignment heat map of the feature similarity between different layers of the query encoder and the key encoder;

[0027] Figure 6 Shows the line graph of the contributions of the multi - head adapter at different layers;

[0028] Figure 7 Shows the schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0029] Next, the technical solutions in the present application will be clearly and elaborately described in conjunction with the accompanying drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B can represent A or B. The "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

[0030] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0031] A knowledge graph is a structured data form for representing entities and the relationships between them. It stores information in the form of triples (head entity h, relation r, tail entity t) and is widely used in fields such as search engines, recommendation systems, and question - answering systems. Although knowledge graphs have been widely applied, due to limitations in data collection, data quality issues, and the dynamic changes in the real world, knowledge graphs are usually incomplete. Therefore, the knowledge graph completion task has emerged, aiming to predict the third element in a triple based on two known elements through an algorithm model. In the knowledge graph completion task, many models have been proposed by researchers. These models can not only supplement the missing triples in the knowledge graph but also infer potential new knowledge in the graph.

[0032] Existing models for knowledge graph completion include: structure-based methods and text-based methods.

[0033] 1. Structure-based methods mainly utilize the existing graph structure to predict missing links by analyzing patterns and relationships between entities. These methods include translation models such as TransE (Translating Embeddings for Modeling Multi-relational Data), TransR (Knowledge Graph Embedding via Translating Embeddings with Relation-Specific Norms), etc., which represent relations as translation operations in the embedding space, and more complex models such as RotatE (Knowledge Graph Embedding by Relational Rotation in Complex Space), ComplEx (Complex Embeddings for Simple Link Prediction), etc., which capture a wider range of relational patterns including symmetry, antisymmetry, and inversion. Other notable models also include R-GCN (Relational Graph Convolutional Network), which extends graph neural networks to handle relational data, and ConvE (Convolutional 2D Knowledge Graph Embeddings) that combines convolutional neural networks for learning graph embeddings. Although these models have achieved remarkable success, they rely too much on structural information and often ignore the rich semantic information contained in the knowledge graph context.

[0034] 2. Text-based methods address this shortcoming by leveraging text descriptions related to entities and external information, enhancing the performance of models for the knowledge graph completion task. For example, KG-BERT (Knowledge Graph BERT) treats triples as text sequences and fine-tunes pre-trained language models for the knowledge graph completion task. LASS (Latent Alignment and Semantic Similarity) further integrates text and structural information to enhance representation learning. Additionally, GenKGC (Generative Knowledge Graph Completion) is a generative model that views knowledge graph completion as a sequence generation task and generates new triples using existing text information.

[0035] Despite the progress of text-based methods, there are still some unresolved issues. First, these methods typically use Transformer-based models as the backbone, where the Feed-Forward Network (FFN) is a key component. Although the FFN has a large number of trainable parameters, it often learns similar features between adjacent Transformer layers, limiting the model's ability to capture data diversity. Second, since the pre-training phase of Transformer-based models mainly relies on semantically consistent text data, fine-tuning with semantically inconsistent synthetic data may disrupt the rich general knowledge stored in the FFN, thus affecting knowledge graph completion performance. Finally, although the introduction of contrastive learning has improved the performance of knowledge graph completion models, their loss functions usually consider only triples containing the head entity, relation, and annotated tail entity as positive samples, which ignores other correct tail entities in the knowledge graph, resulting in biased learning of head-relation pair embeddings. Consequently, the generated embeddings lack representativeness and may only identify a small fraction of correct tail entities.

[0036] Based on the problems existing in the above related technologies, the embodiments of the present application provide a model training method, an electronic device, and a computer-readable storage medium. On the one hand, by introducing a multi-head adapter into the knowledge graph completion model in the embodiments of the present application, the model can learn task-specific knowledge while maintaining the rich general knowledge stored in the feed-forward network; the multi-head adapter adaptively fuses the outputs of the adapter and the feed-forward network through a gating mechanism, enhancing the expressive power and flexibility of the model. On the other hand, by replacing the tail entity of the positive sample in the triple data used for each batch of training with other correct tail entities in the knowledge graph to generate the loss function of additional positive samples, more positive samples can be introduced, enabling the model to learn more diverse positive samples, thereby improving the representativeness and generalization ability of the embedding representation.

[0037] The following is an embodiment of a model training method provided in the specification of the present application.

[0038] Figure 1 The schematic flowchart of a model training method provided by the embodiments of the present application is shown. As Figure 1 shown, the model training method provided by the embodiments of the present application is applied to the training of the knowledge graph completion model. Figure 2 The architecture diagram of a knowledge graph completion model provided by the embodiments of the present application is shown. As Figure 2 shown, C1 represents the first encoder, and C2 represents the second encoder. The knowledge graph completion model provided by the embodiments of the present application includes: u first encoders, u second encoders, a first pooling layer, a second pooling layer, a query embedding layer, and a key embedding layer, where u≥2.

[0039] Among them, both the first encoder and the second encoder are designed based on the bert-base-uncased model. The u first encoders are connected in series in sequence, and the output of the previous first encoder in two adjacent first encoders is connected to the input of the subsequent first encoder; the u second encoders are connected in series in sequence, and the output of the previous second encoder in two adjacent second encoders is connected to the input of the subsequent second encoder.

[0040] The internal structures of the u first encoders and the u second encoders are the same. For each encoder Ci among the u first encoders and the u second encoders, the encoder Ci includes a multi-head attention layer, a first addition and normalization layer (addition and normalization layer 1), a feed-forward network, a multi-head adapter, a gating layer, and a second addition and normalization layer (addition and normalization layer 2). In the encoder Ci, the multi-head attention layer, the first addition and normalization layer, the feed-forward network, the gating layer, and the second addition and normalization layer are connected in series in sequence, and the first addition and normalization layer, the multi-head adapter, and the gating layer are connected in series in sequence.

[0041] The multi-head adapter includes a linear layer, a connection layer, and an output layer. The linear layer, the connection layer, and the output layer are connected in sequence. There are v linear layers, and the input ends of the v linear layers are connected to the output end of the first addition and normalization layer. The output end of the output layer is connected to the input end of the gating layer, where v≥2.

[0042] The input ends of the multi-head attention layer in the first first encoder and the input end of the first addition and normalization layer are connected to the output end of the query embedding layer. The output end of the second addition and normalization layer in the u-th first encoder (the last first encoder) is connected to the input end of the first pooling layer. The input ends of the multi-head attention layer and the first addition and normalization layer in the first second encoder are connected to the output end of the key embedding layer. The output end of the second addition and normalization layer in the u-th second encoder (the last second encoder) is connected to the input end of the second pooling layer.

[0043] Both the first encoder and the second encoder play a role in encoding processing, but the objects they process are different. The first encoder is responsible for encoding the head entity-relation pair (query input data) to generate the embedding vector of the head entity-relation pair. The second encoder is responsible for encoding the tail entity (key input data) to generate the embedding vector of the tail entity. Both the first encoder and the second encoder adopt a similar architecture, including a multi-head attention layer, an addition and normalization layer, a feed-forward network layer, a multi-head adapter, and a gating layer. This design enables the model to effectively capture the complex dependencies in the input data and learn task-specific knowledge with fewer parameters through the multi-head adapter. Finally, the embedding vectors of the head entity-relation pair and the tail entity generated by the pooling layer are used for similarity calculation. Among them, in actual model training, n is selected to be 12, that is, both the first encoder and the second encoder have 12, and both the first encoder and the second encoder belong to the same text Transformer encoder.

[0044] As Figure 1 and Figure 2 shown in parts (A) and (C) of

[0045] S110: Group the k triple data obtained from the knowledge graph to get w groups of triple data.

[0046] Each group of triple data includes z triple data, and each triple data includes a head entity, a relation, and a tail entity, where z = k / f, k≥2, 1<f<k. For example, k = 10000, f = 10, that is, there are 10000 triple data. After grouping, 10 groups of triple data are obtained, and each group of triple data includes 1000 triple data, that is, z = 1000.

[0047] S120: Determine z input data corresponding to each group of triple data, and the z input data corresponding to each group of triple data are used for one batch of training of the knowledge graph completion model.

[0048] For each input data corresponding to each group of triple data, each input data includes query input data and key input data. The query input data includes the head entity and relationship of a triple data in each group of triple data, and the key input data includes the tail entities of z triple data respectively. For example, the first group of triple data includes triple data 1 - triple data 10. The query input data in each input data includes triple data 1 of triple data 1, and the key input data in this input data includes the tail entities of triple data 1 - triple data 10 respectively.

[0049] S130: For each batch of training of the knowledge graph completion model, sequentially input the query input data in one input data corresponding to each group of triple data into the query embedding layer, and at the same time input the key input data in one input data corresponding to each group of triple data into the key embedding layer. u first encoders encode the head entity - relationship pair, and the first pooling layer outputs the head entity - relationship pair embedding vector, and u second encoders encode the tail entities, and the second pooling layer outputs k tail entity embedding vectors.

[0050] For each group of triple data, select a triple data (h, r, t') from this group of triple data, that is, obtain the head entity h, the relationship r, and the candidate tail entity t'. The head entity h and the relationship r have the description d h , and the candidate tail entity t' has the description d t＇ , then, the input sequence Input hr of the query embedding layer t＇ and the input sequence Input

[0051] of the key embedding layer hr are defined by the following formula: h Input neighboor = [[CLS], h, d t＇

[0052] Input t＇ = [[CLS], t', d t＇ , t' neighboor , [SEP]] (2)

[0053] where, h neighboor represents the tail entity related to the head entity h, t' neighboor represents the head entity connected to the candidate tail entity t', and [CLS] and [SEP] represent special tokens in the input sequence.

[0054] After obtaining the query input data, represent the query input data through formula (1), input the query input data into the query embedding layer, and the query embedding layer combines the head entity and the relationship to obtain combined data, and converts the combined data into a query embedding vector. Then, use the query embedding vector as the input data of the first first encoder and input it into the first first encoder, so as to encode the head entity-relationship pair by u first encoders.

[0055] The process of encoding the head entity-relationship pair by u first encoders includes:

[0056] For each layer in the u first encoders, the multi-head attention layer captures the complex dependencies in the query input data based on the query embedding vector. It processes the input in parallel through multiple attention heads, and each attention head focuses on different feature subspaces, thereby enhancing the expressive power of the model. The first addition and normalization layer performs a residual connection between the output of the multi-head attention layer and the input, and performs layer normalization to stabilize the training process. The feed-forward network is a fully connected neural network used to further process the features after passing through the multi-head attention layer and the addition normalization layer. The multi-head adapter is an innovative design of the knowledge graph completion model, aiming to learn task-specific knowledge with fewer parameters while retaining the rich general knowledge stored in the feed-forward network. The adapter adaptively fuses the outputs of the adapter and the feed-forward network through a gating mechanism. The input ends of the v linear layers in the multi-head adapter are connected to the output end of the first addition and normalization layer, and the connection layer concatenates the outputs of the v linear layers. The output layer maps the concatenated output to the same dimension and passes it to the gating layer. The gating layer is used to control the degree of fusion between the output of the feed-forward network and the output of the adapter. The second addition and normalization layer is used to apply the residual connection and layer normalization again to ensure the stability of the model. The first pooling layer is used to perform a pooling operation on the outputs of the u first encoders to generate the final head entity-relationship pair embedding vector. The head entity-relationship pair embedding vector represents the joint representation of the head entity and the relationship and is used for subsequent similarity calculation.

[0057] After obtaining the key input data, represent the key input data through formula (2), input the key input data into the key embedding layer, and the key embedding layer converts each tail entity in the key input data into a key embedding vector to obtain k key embedding vectors, and use the k key embedding vectors as the input of the first second encoder.

[0058] For each layer in the second encoder, the multi-head attention layer captures the complex dependencies between tail entities based on the key embedding vectors. Similar to the first encoder, the multi-head attention layer processes the input in parallel through multiple attention heads to enhance the model's expressive power. The first addition and normalization layer is used to perform a residual connection between the output of the multi-head attention layer and the input, and layer normalization is applied to stabilize the training process. The feed-forward network is a fully connected neural network that further processes the features after passing through the multi-head attention layer and the addition and normalization layer. Similar to the first encoder, the multi-head adapter in the second encoder learns task-specific knowledge with fewer parameters while retaining the rich general knowledge stored in the feed-forward network. The input ends of the v linear layers in the multi-head adapter are connected to the output end of the first addition and normalization layer. The connection layer is used to concatenate the outputs of the v linear layers. The output layer is used to map the concatenated output to the same dimension and pass it to the gating layer. The gating layer is used to control the degree of fusion between the output of the feed-forward network and the output of the adapter. The second addition and normalization layer is used to apply the residual connection and layer normalization again to ensure the stability of the model. The second pooling layer is used to perform a pooling operation on the outputs of the u second encoders to generate the embedding vectors of the k tail entities. The embedding vectors of the k tail entities represent the feature representations of the corresponding tail entities of the k tail entities, which are used for subsequent similarity calculation.

[0059] S140: Train the knowledge graph completion model according to the head entity-relation pair embedding vector, the k tail entity embedding vectors, and the target loss function; wherein, the target loss function includes a self-positive sample loss function and other positive sample loss functions. The self-positive sample loss function refers to the loss function that regards the triple data marked as positive samples in each batch of training data as positive samples. The other positive sample loss function is a loss function that generates additional positive samples by replacing the tail entity of the positive sample in the triple data used for each batch of training with other correct tail entities in the knowledge graph.

[0060] By calculating the similarity between the head entity-relation pair embedding vector and the k tail entity embedding vectors, the tail entity most likely to match the head entity-relation pair embedding vector is determined from the tail entities corresponding to the k tail entity embedding vectors. Then, the target loss function composed of the self-positive sample loss function and other positive sample loss functions is used to optimize the model until the model training is completed.

[0061] In the embodiments of the present application, by adopting the above model training scheme, on the one hand, by designing a multi-head adapter for the feed-forward network in each encoder included in the knowledge graph completion model, while retaining the original general knowledge stored in the feed-forward network, this multi-head adapter learns new knowledge specific to the knowledge graph completion task and only trains a small number of parameters. This design alleviates the problem of information pattern conflict and effectively solves the problem that the feed-forward network learns similar features between adjacent encoder layers. On the other hand, by proposing an additional positive sample loss function and using the objective loss function composed of the self-positive sample loss function and the additional positive sample loss function to optimize the model. Since the self-positive sample loss function only considers the triple data marked as positive samples in the triple data used for training in the current batch, while the additional positive sample loss function generates additional positive samples by replacing the tail entity of the positive sample in the triple data used for training in the current batch with other correct tail entities in the knowledge graph. More positive samples are introduced during the model training process, which can help the model learn more diverse positive samples to discover more potential correct relationships and avoid the model over-relying on the existing positive samples. This method corrects the bias problem in the head-relation pair embedding learning, not only improving the representativeness of the embedding representation but also enhancing the generalization ability of the model. On the further hand, applying the trained knowledge graph completion model to complete the knowledge graph improves the accuracy of using the knowledge graph model to identify the correct tail entity and ensures the accuracy of the tail entity added to the knowledge graph.

[0062] In a possible implementation manner, before determining the z input data corresponding to each group of triple data, the above model training method further includes the following steps:

[0063] Freeze the parameters of all feed-forward networks in the knowledge graph completion model.

[0064] Since the present application designs a multi-head adapter for the feed-forward network in each encoder included in the knowledge graph completion model, before training the knowledge graph completion model, the parameters of all feed-forward networks in the knowledge graph completion model, that is, freeze the parameters of each feed-forward network in the u first encoders and the u second encoders. That is, during the process of training the knowledge graph completion model, the parameters of all feed-forward networks remain unchanged, and only each multi-head adapter in the knowledge graph completion model is trained, enabling the model to learn new knowledge related to the knowledge graph completion task with fewer parameters while maximizing the retention of the general knowledge in the feed-forward network. Fewer parameters improve the learning efficiency, enabling each encoder layer of the model to capture different features and better reflect the diversity of the data.

[0065] In a possible implementation manner, the encoding of the head entity-relation pair by the u first encoders includes the following steps:

[0066] For each of the u first encoders Ci, divide the output result of the first addition and normalization layer of the first encoder Ci into v sub-results;

[0067] Input the v sub-results into the v linear layers of the first encoder one by one, and each linear layer performs a linear transformation on the input sub-result to obtain the transformation result of the sub-result input to each linear layer in the corresponding subspace of each linear layer; among them, the subspaces corresponding to each linear layer are different;

[0068] Input the v transformation results into the connection layer of the first encoder to map the v transformation results to the same space, and obtain the output result of the connection layer of the first encoder;

[0069] Input the output result of the connection layer of the first encoder into the output layer of the first encoder, and the gating layer of the first encoder fuses the output result of the output layer of the first encoder and the output result of the feed-forward network of the first encoder to obtain the output result of the gating layer of the first encoder;

[0070] Input the output result of the gating layer of the first encoder into the second addition and normalization layer of the first encoder until the output result of the second addition and normalization layer in the u-th first encoder is input into the first pooling layer.

[0071] For each first encoder Ci, the present application introduces a Multi-Head Adapter mechanism to perform more fine-grained processing and feature learning on the output of the first encoder Ci. Specifically, the output result of the first addition and normalization layer in each first encoder Ci is divided into multiple sub-results, which are respectively input into different linear layers for linear transformation, and then these transformation results are mapped to the same space through the connection layer. Finally, the gating layer fuses the output of the feed-forward network and the output of the adapter. This design significantly enhances the expression ability, flexibility and adaptability of the model, and at the same time effectively alleviates the overfitting problem. The analysis of enhancing the model expression ability, improving the flexibility and adaptability of the model and alleviating the overfitting problem is as follows:

[0072] 1. Enhance the model expression ability

[0073] Regarding the design of the multi-head adapter: By dividing the output result of the first addition and normalization layer into v sub-results and inputting each sub-result into a different linear layer for linear transformation, the model can capture different features of the input data in different subspaces. The subspaces corresponding to each linear layer are different, which means that each linear layer can focus on learning specific types of features, thereby enhancing the expression ability of the model.

[0074] Regarding the role of the connection layer: The connection layer maps the v linearly transformed sub-results to the same space, enabling the effective combination of features in these different sub-spaces. This design not only preserves the unique information in each sub-space but also ensures that this information can be fully utilized in subsequent processing.

[0075] 2. Improve the flexibility and adaptability of the model

[0076] Regarding the gating mechanism: The gating layer controls the contribution ratio between the output of the feed-forward network and the output of the adapter by fusing them. This gating mechanism enables the model to dynamically adjust the weights of the feed-forward network and the adapter according to the task requirements, thereby learning task-specific knowledge while maintaining general knowledge. This improves the flexibility and adaptability of the model, especially when facing different domains or tasks.

[0077] Regarding lightweight and efficiency: Compared with traditional fully connected layers (such as feed-forward networks), the multi-head adapter realizes diverse learning of features by introducing a small number of parameters (linear layer, connection layer, and output layer). This method not only reduces the number of model parameters, lowers the computational complexity, but also improves the training efficiency, and is particularly suitable for large-scale knowledge graph completion tasks.

[0078] 3. Alleviate the overfitting problem

[0079] Regularization effect: The multi-head adapter increases the complexity of the model by introducing additional linear transformations and gating mechanisms, but at the same time introduces a certain degree of regularization effect. This regularization helps prevent the model from overfitting the training data and enhances its generalization ability, especially in the case of sparse or unbalanced data.

[0080] Diverse learning: By performing linear transformations in different sub-spaces, the model can learn more diverse feature representations, avoiding the common "feature collapse" problem in traditional feed-forward networks (i.e., adjacent Transformer layers learn similar features). The diverse feature representations enable the model to better capture complex patterns in the data, thereby improving the prediction accuracy.

[0081] In a possible implementation, the above process of the gating layer of the first encoder fusing the output result of the output layer of the first encoder and the output result of the feed-forward network of the first encoder to obtain the output result of the gating layer of the first encoder includes the following steps:

[0082] Since the present application adopts a multi-head mechanism in the adapter, each head can obtain diverse representations by learning in different spaces and having its own weight matrix, which enables each head to focus on different information subsets or patterns in the data. Taking the adapter in the first encoder Ci as an example for illustration, assume X represents the output result of the first addition and normalization layer, d model represents the dimension of the vector (e.g., d model = 512 dimensions), n represents the number of tokens in the query input sequence, X is sliced into m sub-results, m = v, denoted as [X1, X2,..., X m , where and each sub-result X i undergoes the following linear transformation:

[0083]

[0084] Subsequently, the concatenated into a complete representation obtained by each head in different subspaces, and through the transformation matrix and the bias term mapped to the same space, as shown in formula (4):

[0085]

[0086] To balance the challenges of the new knowledge learned by the balanced multi-head adapter and the general knowledge retained in the feed-forward network, this application solves this problem by designing a gating mechanism to dynamically balance the contributions of the frozen feed-forward network and the activated multi-head adapter. The gating mechanism is defined as follows:

[0087]

[0088] where, e (l) represents the output result of the gating layer of the first encoder Ci, FFN (l) (X) represents the output result of the feed-forward network of the first encoder Ci, Adapter (l) (X) represents the output result of the output layer of the first encoder Ci, α (l) is a trainable parameter used to control the degree of fusion of FFN (l) (X) and Adapter (l) (X), and l represents the first encoder Ci.

[0089] By designing the gating mechanism, it is ensured that the knowledge graph completion model has the ability to learn and generalize while retaining rich general knowledge, and can adapt to the knowledge graph completion task.

[0090] Furthermore, this application applies average pooling to the output of the last first encoder to generate the final embedding representation. The final embedding representation e is specifically as shown in formula (6):

[0091] e = Pooling(e (L) ) (6)

[0092] Among them, L represents the last first encoder, and through this process, the head entity-relationship pair embedding vector e can be obtained hr .

[0093] The above process of encoding the tail entity by u second encoders is the same as the above process of encoding the head entity-relationship pair by u first encoders, and for each second encoder, the processing process is the same as that of each first encoder. Similarly, finally, the tail entity embedding vector e can be obtained through the above formula (6) t＇ , which will not be elaborated here.

[0094] Although using contrastive learning significantly improves the performance of the knowledge graph completion model, however, traditional contrastive learning loss functions usually only regard the triples composed of the head entity, relationship, and the labeled tail entity in the batch as positive samples, ignoring many other correct tail entities in the knowledge graph. This limitation leads to the learned head-relationship pair embeddings can only identify some correct tail entities, resulting in biased learning. To solve this problem, this application proposes a multi-positive sample loss function (i.e., the target loss function), denoted as L MSP , L MSP contains two components.

[0095] The first component is the self-positive sample loss function, denoted as L SPS , L SPS regards the triples of the head entity, relationship, and labeled tail entity in each batch as positive samples, and at the same time regards the triples of the head entity, relationship, and all incorrect tail entities in each batch as negative samples. L SPS ensures that the model can accurately predict the most correct tail entity given the head entity and relationship. Its formula follows the form of the InfoNCE (Information Noise Contrastive Estimation) loss, and the definition of L SPS is as follows:[[]]

[0096]

[0097] where S hr represents similarity, e represents the embedding vector, t represents the true tail entity given the head entity h and relationship r, N represents all incorrect tail entities in the triple data used for each batch of training γ represents the hyperparameter that controls the boundary separation, and τ represents the hyperparameter that controls the sharpness of the probability distribution.

[0098] Although L SPSThis loss can effectively guide the model to predict the correct tail entity. However, in some cases, when the tail entity is actually correct but not labeled in the current batch, and there is a one-to-many mapping relationship between the head entity and the tail entity, they can also serve as valuable supplements. In this way, the model can learn the embeddings of head-relation pairs that are not fully related to the labeled tail entities in the batch, and then obtain a more general embedding. These embeddings can approach all correct tail entities, thus reflecting a deeper understanding of the knowledge graph structure information by the model. To achieve this goal, we introduce a second component to optimize the model.

[0099] The second component is the other positive sample loss function, denoted as L OPS , L OPS plays a key role in effectively solving the biased learning problem of head-relation pair embeddings and enhancing the generalization ability of the model. L OPS considers the triples composed of the head entity, the relation, and all other correct tail entities in the entire knowledge graph as positive samples, while considering the triples of the head entity, the relation, and all incorrect tail entities in the batch as negative samples. The definition of L OPS is as follows:

[0100]

[0101] where P represents the set of all other correct tail entities related to the head entity h under the relation r .

[0102] The self-positive sample loss function L SPS and the other positive sample loss function L OPS are combined in a weighted sum manner to obtain the target loss function L MSP , L MSP is expressed by the following formula:

[0103] L MPS = λL SPS + (1 - λ)L OPS (9)

[0104] where λ is a hyperparameter, and by adjusting λ, the model can achieve a balance between accuracy and generalization.

[0105] In a possible implementation, the above method for training the knowledge graph completion model based on the head entity-relation pair embedding vector, k tail entity embedding vectors, and the target loss function includes the following steps:

[0106] Input the head entity-relation pair embedding vector and k tail entity embedding vectors into the target loss function to obtain the loss function value of training the knowledge graph completion model for each batch;

[0107] Based on the loss function values obtained by training the knowledge graph completion model for each batch, determine whether there is a trend of the loss function value decreasing below the loss threshold and then exceeding the loss threshold;

[0108] If not, it means that the model training has not reached the expectation, and continue to train the knowledge graph completion model;

[0109] If so, it means that the model training has reached the expectation, and stop training the knowledge graph completion model.

[0110] On the one hand, by monitoring the change trend of the loss function value, especially the phenomenon of "decreasing below the loss threshold and then exceeding the loss threshold", overfitting of the model can be effectively prevented. On the other hand, by setting a reasonable loss threshold and monitoring mechanism, training can be stopped in time when the model reaches the expected performance, avoiding unnecessary waste of training time, thereby improving training efficiency.

[0111] After training the knowledge graph completion model, save the trained knowledge graph completion model, and then the knowledge graph completion model can be applied. The following is the application process of the knowledge graph completion model.

[0112] In a possible implementation, after the knowledge graph completion model is trained, the model training method further includes the following steps:

[0113] Obtain search description information, and extract target head entity-relationship pairs from the search description information;

[0114] Input the target head entity-relationship pairs into the knowledge graph completion model to obtain p candidate tail entities; where each candidate tail entity carries a score value, and the score value is used to represent the similarity between the candidate tail entity and the target head entity-relationship pair, and p≥2;

[0115] Determine the target tail entity corresponding to the target head entity-relationship pair from the p candidate tail entities according to the score value;

[0116] Associate the target tail entity with the target head entity-relationship pair and store it in the knowledge graph to complete the knowledge graph.

[0117] The search description information can be understood as a search term. The search description information can be text information input by the user on the keyboard, or text information recognized through pictures or voices. After obtaining the search description information, extract the target head entity-relationship pairs from the search description information, and then use the target head entity-relationship pairs as the input data of the knowledge graph completion model and input them into the knowledge graph completion model. The knowledge graph completion model infers and obtains p candidate tail entities and then outputs them. And each candidate tail entity among the p candidate tail entities carries a score value, and the score value is used to represent the similarity between the candidate tail entity and the target head entity-relationship pair.

[0118] The calculation formula for the similarity between the candidate tail entity and the target head entity-relationship pair is shown in formula (10):

[0119] S hr (t′) = e hr ·e t′ (10)

[0120] After obtaining p candidate tail entities, select the candidate tail entity with the highest score value from the p candidate tail entities as the target tail entity corresponding to the target head entity-relationship pair, that is, obtain the tail entity that best matches the target head entity-relationship pair. Then, associate the target tail entity with the target head entity-relationship pair and store it in the knowledge graph, thereby completing the knowledge graph, and thus adding a new tail entity corresponding to the target head entity-relationship in the knowledge graph. For example, the triple information 1 in the knowledge graph before completion is (head entity: "French fries", relationship "main raw material is", tail entity "potato"), and the triple information 2 is (head entity: "French fries", relationship "main raw material is", tail entity "potato"). The target head entity-relationship pair is "French fries" - "main raw material is". The target tail entity finally obtained based on the trained knowledge graph completion model of the present application is "potato", that is, a new triple information 3 can be obtained. Triple information 3 = (head entity: "French fries", relationship "main raw material is", tail entity "potato"), that is, the tail entity corresponding to the head entity: "French fries" - relationship "main raw material is" in the knowledge graph is further completed.

[0121] Determining the target tail entity corresponding to the target head entity-relationship pair from the p candidate tail entities through the score value representing the similarity between the candidate tail entity and the head entity-relationship pair improves the accuracy of using the knowledge graph model to identify the correct tail entity.

[0122] In a possible implementation manner, the above determining the target tail entity corresponding to the target head entity-relationship pair from the p candidate tail entities includes the following steps:

[0123] Based on the sorting from large to small of the score values, sort the p candidate tail entities to obtain the first sorting result;

[0124] Obtain the first q candidate tail entities in the first sorting result to obtain the second sorting result, 1 < q ≤ p;

[0125] Judge whether the difference value between the score value of the first candidate tail entity and the score value of the second candidate tail entity in the second sorting result is less than a preset threshold;

[0126] If so, determine the first candidate tail entity in the second sorting result as the target tail entity;

[0127] Otherwise, use a large language model to correct the score values of the candidate tail entities in the second sorting result to obtain a third sorting result;

[0128] Determine the first candidate tail entity in the third sorting result as the target tail entity.

[0129] After sorting the p candidate tail entities according to the score values from large to small to obtain a first sorting result, screening the first sorting result to obtain the top q candidate tail entities in the first sorting result to obtain a second sorting result, and determining the first candidate tail entity (i.e., the candidate tail entity with the highest score value) in the second sorting result as the target tail entity. This determination method is only applicable to the case where the difference value between the score values of the first candidate tail entity and the second candidate tail entity (i.e., the candidate tail entity with the second highest score value) in the second sorting result is large. For example, when the difference value is greater than or equal to a preset threshold, the first candidate tail entity (i.e., the candidate tail entity with the highest score value in the second sorting result) in the second sorting result is determined as the target tail entity. As shown in (A) and (C) of Figure 2 ; Figure 2 where m = q in Figure 1 the sorting result shown by the histogram of the second sorting result is: e t1 the corresponding candidate tail entity (the first candidate tail entity), e t2 the corresponding candidate tail entity (the second candidate tail entity),..., e tm the corresponding candidate tail entity (the mth candidate tail entity). If the difference value between the candidate tail entity corresponding to e t1 and the candidate tail entity corresponding to e t2 is large, then determine the candidate tail entity corresponding to e t1 as the target tail entity.

[0130] Among them, the difference value is represented as ΔS, the first candidate tail entity in the sorting result is represented as S1 * and the second candidate tail entity is represented as S2 * .

[0131] If the difference value between the score values of the first candidate tail entity and the second candidate tail entity in the second sorting result is very small, for example, when the difference value is less than the preset threshold, the knowledge graph completion model has difficulty in distinguishing between the correct tail entity and the wrong tail entity, resulting in a low reliability of the prediction result. To solve this problem, this application introduces a large language model (LLM). Since the large language model has extensive common sense knowledge and powerful reasoning ability, the large language model in the known input target head entity-relationship pair (such as Figure 2The e shown in part (C) hr ), based on its own powerful reasoning ability, the score values of the candidate tail entities in the second sorting result are corrected, that is, the score values of the candidate tail entities in the second sorting result are added using the large language model. For example, the large language model first performs semantic understanding and context analysis on the target head entity-relationship pair to ensure that the true intention of the query is understood, and then evaluates the degree of match between each candidate tail entity and the target head entity-relationship pair based on the extensive common sense knowledge base of the large language model. For those candidate tail entities that are highly related to the target head entity-relationship pair, the large language model will add points to their scores. After the score values of the candidate tail entities in the second sorting result are corrected using the large language model, the difference value between the two adjacent candidate tail entities in the second sorting result is enlarged, and then the candidate tail entities in the second sorting result are re-sorted according to the score values to obtain a third sorting result, and the first candidate tail entity in the third sorting result (i.e., the candidate tail entity with the highest score value in the third sorting result) is determined as the target tail entity.

[0132] In the ranking results of candidate tail entities, when the difference between a candidate tail entity with a high score and a candidate tail entity with the second lowest score is small, by adopting a large language model with strong reasoning ability to correct the score of the candidate tail entity in the ranking results, potential new knowledge or implicit relationships can be inferred, which is conducive to further improving the accuracy of predicting the target tail entity.

[0133] In a possible implementation, the above-mentioned use of the large language model to correct the score value of the candidate tail entity in the second ranking result to obtain the third ranking result includes the following steps:

[0134] Using a large language model to recommend at least one candidate tail entity corresponding to the target head entity-relationship pair from the second ranking result;

[0135] Adding points to the score value of at least one candidate tail entity to obtain a revised score value of at least one candidate tail entity;

[0136] The score value of at least one candidate tail entity of the second ranking result is correspondingly updated using the revised score value of at least one candidate tail entity to obtain a third ranking result.

[0137] If the difference between the score values of the first candidate tail entity and the second candidate tail entity in the second sorting result is less than a preset threshold, the large language model is guided by prompt engineering to recommend at least one candidate tail entity corresponding to the target head entity-relationship pair from the second sorting result, and the at least one candidate tail entity recommended by the large language model is a candidate tail entity that the large language model itself considers to be more matched with the target head entity-relationship pair, for example,Figure 1 e in t2 corresponding candidate tail entity, e t3 corresponding candidate tail entity and e t3 corresponding candidate tail entity, which are denoted as recommended tail entity a1, recommended tail entity a2, and recommended tail entity a3 in sequence. For each recommended tail entity a j , take w j ×S1 * as the bonus value of the score value to add points to the recommended tail entity a j score value. w j represents the empirical weight assigned to the recommended tail entity a j by the large language model. Obtain the corrected score value S′ j of the recommended tail entity a through the correction formula (11) of the score value. The correction formula (12) is as follows: hr (t′), and the correction formula (12) is as follows:

[0138]

[0139] where, Δ th represents a preset threshold.

[0140] After obtaining the corrected score values of at least one candidate tail entity, use the corrected score values of at least one candidate tail entity to update the score values of at least one candidate tail entity in the second sorting result, and re-sort the candidate tail entities in the second sorting result based on the score values to obtain the third sorting result. The candidate tail entity with the highest score value in the third sorting result is the finally predicted tail entity That is, obtain the target tail entity, the finally predicted tail entity is expressed as follows:

[0141]

[0142] As Figure 2 shown in the histogram in (C) of Figure 2 , the histogram Figure 2 represents the third sorting result. ε represents all entities in the knowledge graph, that is, the union of all head entities and all tail entities in the knowledge graph. The shaded parts d1 - d3 all represent the added bonus values, d2 > d3 > d1, and the score value S hr (t3) is the highest score value. The finally obtained target tail entity is the candidate tail entity corresponding to the score value S hr (t3) in the third sorting result.

[0143] Figure 2The shown model architecture belongs to a composite reasoning framework that combines the efficiency of small models (knowledge graph completion models) and the deep reasoning ability of large language models. By using small models to preliminarily screen candidate tail entities and large models to refine the candidate tail entities, this approach ensures a balance between processing speed and prediction accuracy. The framework improves the credibility of the model's prediction results by reordering the candidate tail entities and updating their similarities.

[0144] To illustrate the performance of the knowledge graph completion model (denoted as AdpKGC) of this application.

[0145] This application evaluated the performance of the knowledge graph completion model on multiple benchmark databases and compared it with strong baseline models. The experimental results show that the knowledge graph completion model has outstanding performance in the knowledge graph completion task. In addition, further experiments verified the effectiveness of the above model training method.

[0146] This application compared and analyzed the performance of the knowledge graph completion model with baseline methods on four commonly used datasets for knowledge graph completion tasks, namely WN18RR, FB15k-237, Wikidata5M-Trans, and Wikidata5M-Ind. The experimental results are as Figure 3 and Figure 3 shown. Figure 3 It shows an exemplary comparative analysis data graph of the knowledge graph completion model provided by this application and the baseline method. Figure 4 It shows another exemplary comparative analysis data graph of the knowledge graph completion model provided by this application and the baseline method. Figure 3 shows the model performance on the WN18RR and FB15k-237 datasets. Bold / underlined indicates the best / second-best model performance; Figure 4 shows the model performance on the Wikidata5M-Trans and Wikidata5M-Trans datasets. Bold / underlined indicates the best / second-best model performance.

[0147] The knowledge graph completion model of this application performs excellently on all evaluation metrics of each dataset. In the WN18RR dataset, which mainly describes the hierarchical relationships between words, text-based methods perform better than structure-based methods, and the knowledge graph completion model of this application performs best among all methods. The knowledge graph completion model exceeds the second-ranked SimKGC by 5.4% in MRR and improves by 8.2%, 3.2%, and 2.2% in Hit@1, Hit@3, and Hit@10 respectively. In the FB15k-237 dataset, where the relationship connections are relatively dense and structure-based methods perform well, the knowledge graph completion model of this application outperforms all baseline methods, improving by 9.2% in MRR and increasing by 12.0%, 7.2%, and 1.5% in Hit@1, Hit@3, and Hit@10 respectively. These results fully demonstrate that the high accuracy of the knowledge graph completion model of this application in the knowledge graph completion task benefits from the above three key innovations (the design of multi-head adapters, the design of other positive sample losses, and the introduction of large language models).

[0148] In the two settings of the Wikidata5M dataset, the knowledge graph completion model of this application achieves the best performance. In the transduction setting, kAdpKGC exceeds other methods by 3.6% in MRR and improves by 4.8%, 3.7%, and 0.9% in Hit@1, Hit@3, and Hit@10 respectively. Especially in the inductive setting under low-resource conditions, the knowledge graph completion model of this application improves by 6.7% in MRR and increases by 9.9%, 6.2%, and 1.0% in Hit@1, Hit@3, and Hit@10 respectively. These results demonstrate the excellent generalization ability of the knowledge graph completion model of this application.

[0149] As Figure 5 shown, Figure 5Shows the centered kernel alignment heatmap of the feature similarities between different layers of the query encoder and the key encoder. Heatmap a represents the feature similarity of the FFN of different layers of the query encoder, heatmap b represents the feature similarity of the MH adapter of different layers of the query encoder, heatmap c represents the feature similarity of the gated output of different layers of the query encoder, heatmap d represents the feature similarity of the FFN of different layers of the key encoder, heatmap e represents the feature similarity of the MH adapter of different layers of the key encoder, and heatmap f represents the feature similarity of the gated output of different layers of the key encoder. The abscissa and ordinate of heatmaps a - f represent the layer numbers. The values of the ordinate from top to bottom are 1 - 12, with an interval of 1; the values of the abscissa from left to right are 1 - 12, with an interval of 1. This application evaluates the feature similarities between different layers of the query encoder (the first encoder) and the key encoder (the second encoder). The multi - head adapter effectively reduces the similarities between layers and enhances the model's ability to capture more diverse and rich representations, which enables the knowledge graph completion model of this application to better understand and distinguish different features in the input data. Among them, Figure 5 In the centered kernel alignment (CKA) heatmap, the darker the color, the more similar the feature similarities of the feed - forward networks of different layers of the query encoder, the more similar the feature similarities of the multi - head adapters of different layers of the query encoder, the more similar the feature similarities of the gated outputs of different layers of the query encoder, the more similar the feature similarities of the feed - forward networks of different layers of the key encoder, the more similar the feature similarities of the multi - head adapters of different layers of the key encoder, and the more similar the feature similarities of the gated outputs of different layers of the key encoder.

[0150] As Figure 6 shown, Figure 6 Shows the line chart of the contributions of the multi - head adapter at different layers. This application evaluates the contribution ratio of the multi - head adapter to the relative feed - forward network in each encoder layer (Transformer layer) of the query encoder. The results show that the multi - head adapter has a more significant impact on the initial layers of the model, indicating that the gating mechanism can adaptively adjust the contribution of the multi - head adapter, enabling it to play a greater role when the output similarities of the feed - forward networks are relatively high in the early stage of the Transformer layer.

[0151] The embodiment of the present application provides a model training method. On the one hand, by introducing a multi-head adapter into the knowledge graph completion model, the model can learn task-specific knowledge while maintaining the rich general knowledge stored in the feed-forward network. The multi-head adapter adaptively fuses the outputs of the adapter and the feed-forward network through a gating mechanism, enhancing the expressive power and flexibility of the model. On the other hand, by replacing the tail entity of the positive sample in the triple data used for each batch of training with other correct tail entities in the knowledge graph to generate the loss function of additional positive samples, more positive samples can be introduced, enabling the model to learn more diverse positive samples, thereby improving the representativeness and generalization ability of the embedding representation. On the further hand, by using a large language model with strong reasoning ability to correct the score values of the candidate tail entities output by the knowledge graph completion model, potential new knowledge or implicit relationships can be inferred, which is beneficial to further improving the accuracy of predicting the target tail entity.

[0152] Figure 7 Fig. shows a schematic structural diagram of an electronic device provided by an embodiment of the present application, as Figure 7 shown, the electronic device 700 includes: a memory 701 and a processor 702. Among them, an executable program code 7011 is stored in the memory 701, and the processor 702 is configured to call and execute the executable program code 7011 to execute a model training method.

[0153] In this embodiment, the electronic device can be divided into functional modules according to the above method examples. For example, each functional module can be corresponding, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0154] The electronic device provided by this embodiment is used to execute the above model training method, so the same effect as the above implementation method can be achieved.

[0155] In the case of adopting an integrated unit, the electronic device may include a processing module and a storage module. Among them, the processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute relevant program codes and data, etc.

[0156] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits described in combination with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory.

[0157] This embodiment also provides a computer-readable storage medium, in which computer program code is stored. When the computer program code runs on a computer, it causes the computer to execute the above-related method steps to implement a model training method in the above embodiment.

[0158] This embodiment also provides a computer program product. When the computer program product runs on a computer, it causes the computer to execute the above-related steps to implement a model training method in the above embodiment.

[0159] In addition, the electronic device provided in the embodiment of this application can specifically be a chip, a component, or a module. The electronic device can include a processor and a memory connected to each other. Among them, the memory is used to store instructions. When the electronic device runs, the processor can call and execute the instructions to cause the chip to execute a model training method in the above embodiment.

[0160] Among them, the electronic device, the computer-readable storage medium, the computer program product, or the chip provided in this embodiment are all used to execute the corresponding model training method provided above. Therefore, the beneficial effects that can be achieved by them can refer to the beneficial effects in the corresponding model training method provided above, and will not be elaborated here.

[0161] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0162] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0163] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A model training method, characterized in that, Applied to the training of a knowledge graph completion model, the knowledge graph completion model includes: u first encoders, u second encoders, a first pooling layer, a second pooling layer, a query embedding layer, and a key embedding layer, where u≥2; The u first encoders are connected in series in sequence, and the u second encoders are connected in series in sequence; for each encoder among the u first encoders and the u second encoders, the encoder includes a multi-head attention layer, a first addition and normalization layer, a feed-forward network, a multi-head adapter, a gating layer, and a second addition and normalization layer; The multi-head attention layer, the first addition and normalization layer, the feed-forward network, the gating layer, and the second addition and normalization layer are connected in series in sequence, and the first addition and normalization layer, the multi-head adapter, and the gating layer are connected in series in sequence; The multi-head adapter includes a linear layer, a connection layer, and an output layer. The linear layer, the connection layer, and the output layer are connected in sequence. There are v linear layers, and the input ends of the v linear layers are connected to the output end of the first addition and normalization layer. The output end of the output layer is connected to the input end of the gating layer, where v≥2; The input ends of the multi-head attention layer and the first addition and normalization layer in the first encoder of the first one are connected to the output end of the query embedding layer, and the output end of the second addition and normalization layer in the u-th first encoder is connected to the input end of the first pooling layer; The input ends of the multi-head attention layer and the first addition and normalization layer in the first encoder of the second one are connected to the output end of the key embedding layer, and the output end of the second addition and normalization layer in the u-th second encoder is connected to the input end of the second pooling layer; The model training method includes: Group the k triple data obtained based on the knowledge graph to obtain f groups of triple data; where each group of triple data includes z triple data, and each triple data includes a head entity, a relation, and a tail entity, z = k / f, k≥2, 1<f<k; Determine z input data corresponding to each group of triple data; where for each input data corresponding to each group of triple data, each input data includes a query input data and a key input data. The query input data includes the head entity and the relation of one triple data in each group of triple data, and the key input data includes the tail entities of the z triple data respectively. The z input data corresponding to each group of triple data are used for one batch of training of the knowledge graph completion model; For each batch of training of the knowledge graph completion model, sequentially input the query input data in one input data corresponding to each group of triple data into the query embedding layer, and at the same time input the key input data in the one input data corresponding to each group of triple data into the key embedding layer. Encode the head entity-relation pair by the u first encoders, and output the head entity-relation pair embedding vector by the first pooling layer, and encode the tail entities by the u second encoders, and output k tail entity embedding vectors by the second pooling layer; Train the knowledge graph completion model according to the head entity-relationship pair embedding vector, the k tail entity embedding vectors, and the target loss function; Among them, the target loss function includes a self-positive sample loss function and other positive sample loss functions. The self-positive sample loss function refers to the loss function that regards the triple data marked as positive samples in each batch of training data as positive samples. The other positive sample loss function is a loss function that generates additional positive samples by replacing the tail entity of the positive sample in each batch of training data with other correct tail entities in the knowledge graph; After the knowledge graph completion model is trained, the model training method further includes: Obtain search description information, and extract the target head entity-relationship pair from the search description information; Input the target head entity-relationship pair into the knowledge graph completion model to obtain p candidate tail entities; where each candidate tail entity carries a score value, and the score value is used to represent the similarity between the candidate tail entity and the target head entity-relationship pair, p≥2; Determine the target tail entity corresponding to the target head entity-relationship pair from the p candidate tail entities according to the score value; Associate the target tail entity with the target head entity-relationship pair and store it in the knowledge graph to complete the knowledge graph.

2. The model training method according to claim 1, wherein Before determining the z input data corresponding to each group of triple data, the model training method further includes: Freeze the parameters of all feed-forward networks in the knowledge graph completion model.

3. The model training method according to claim 1, wherein The encoding of the head entity-relationship pair by the u first encoders includes: For each of the u first encoders, divide the output result of the first addition and normalization layer of the first encoder into v sub-results; Input the v sub-results into the v linear layers of the first encoder one by one. Each linear layer performs a linear transformation on the input sub-result to obtain the transformation result of the input sub-result in the subspace corresponding to each linear layer; where the subspaces corresponding to each linear layer are different; Input the v transformation results into the connection layer of the first encoder to map the v transformation results to the same space to obtain the output result of the connection layer of the first encoder; Input the output result of the connection layer of the first encoder into the output layer of the first encoder. The gating layer of the first encoder fuses the output result of the output layer of the first encoder and the output result of the feed-forward network of the first encoder to obtain the output result of the gating layer of the first encoder; Input the output result of the gating layer of the first encoder into the second addition and normalization layer of the first encoder until the output result of the second addition and normalization layer in the u-th first encoder is input into the first pooling layer.

4. The model training method according to claim 3, wherein The fusion of the output result of the output layer of the first encoder and the output result of the feed-forward network of the first encoder by the gating layer of the first encoder to obtain the output result of the gating layer of the first encoder includes: e (l) = AddNorm(a (l )·FFN (l) (X)+(1 - a (l) )·Adapter (l) (X)); Among them, e (l) represents the output result of the gating layer of the first encoder, FFN (l) (X) represents the output result of the feed-forward network of the first encoder, Adapter (l) (X) represents the output result of the output layer of the first encoder, a (l) is a trainable parameter used to control the fusion degree of FFN (l) (X) and Adapter (l) (X), and l represents the first encoder.

5. The model training method according to claim 1, wherein Training the knowledge graph completion model according to the head entity-relationship pair embedding vector, the k tail entity embedding vectors, and the target loss function includes: Inputting the head entity-relationship pair embedding vector and the k tail entity embedding vectors into the target loss function to obtain the loss function value for training the knowledge graph completion model in each batch; According to the loss function value obtained by training the knowledge graph completion model in each batch, determine whether there is a change trend that the loss function value drops below the loss threshold and then exceeds the loss threshold; If not, continue training the knowledge graph completion model; If so, stop training the knowledge graph completion model; Among them, the representation formula of the target loss function is as follows: L MPS = λL SPS +(1 - λ)L OPS L MSP represents the target loss function, L SPS represents the self-positive sample loss function, L OPS represents the other positive sample loss function, λ is a hyperparameter, and by adjusting λ, the model can achieve a balance between accuracy and generalization. e represents the embedding vector, S hr represents the similarity, t represents the true tail entity given the head entity h and the relation r, and N represents all the incorrect tail entities in the triple data used for each batch of training γ represents the hyperparameter that controls the boundary separation, τ represents the hyperparameter that controls the sharpness of the probability distribution, and P represents the set of all other correct tail entities related to the head entity h under the relation r of.

6. The model training method according to claim 1, wherein Determining the target tail entity corresponding to the target head entity-relationship pair from the p candidate tail entities according to the score value includes: Based on the sorting of the p candidate tail entities from large to small according to the score value, obtain the first sorting result; Obtain the first q candidate tail entities in the first sorting result to obtain the second sorting result, where 1 < q ≤ p; Determine whether the difference value between the score value of the first candidate tail entity and the score value of the second candidate tail entity in the second sorting result is less than the preset threshold; If so, determine the first candidate tail entity in the second sorting result as the target tail entity; If not, use the large language model to correct the score values of the candidate tail entities in the second sorting result to obtain the third sorting result; Determine the first candidate tail entity in the third sorting result as the target tail entity.

7. The model training method according to claim 6, wherein Using the large language model to correct the score values of the candidate tail entities in the second sorting result to obtain the third sorting result includes: Using the large language model to recommend at least one candidate tail entity corresponding to the target head entity-relationship pair from the second sorting result; Add points to the score values of the at least one candidate tail entity to obtain the corrected score values of the at least one candidate tail entity; Use the corrected score values of the at least one candidate tail entity to update the score values of the at least one candidate tail entity in the second sorting result to obtain the third sorting result.

8. An electronic device, characterized in that, The electronic device includes: A memory for storing executable program code; A processor for calling and running the executable program code from the memory, so that the electronic device executes the model training method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed, the model training method according to any one of claims 1 to 7 is implemented.

Images