Knowledge graph question answering method and system based on multi-hop combined graph fingerprint network

Abstract

The present disclosure relates to the fields of knowledge graphs and representation learning technology, and proposes a knowledge graph question-answering method and system based on a multi-hop combined graph fingerprint network. Given a multiple-choice question with multiple answer candidate options, the question and candidate options are text-encoded, and the encoded question and candidate options enter the graph encoder. The knowledge subgraph related to the question and candidate options is encoded through the multi-hop combined graph fingerprint network designed in this paper; the credibility score of the spliced text encoding and graph encoding is calculated; and the correct answer to the question is selected from the answer options given in the question. After the entities and relationships adjacent to the central entity node are combined through entity-relationship operations, the model can learn entity embedding and relationship embedding at the same time. Structural fingerprint attention is used to assist in generating node representations in the graph, and the high-order structural information in the graph is utilized to better generate node representations, thereby alleviating the over-smoothing problem of the question-answering model.

Description

Technical Field

[0001] The present disclosure relates to the technical fields related to knowledge graphs and representation learning, and more specifically, to a knowledge graph question-answering method and system based on a multi-hop combined graph fingerprint network. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Knowledge graph commonsense question answering (KG-CSQA) is a challenging learning task that aims to provide correct answers to commonsense questions. In recent years, KG-CSQA has been widely used in fields such as healthcare, finance, and government. The inventors discovered that for KG-CSQA, most existing models can effectively learn node representations but cannot effectively utilize the relationship information in the knowledge graph. Furthermore, the attention mechanism used in message passing suffers from over-smoothing.

[0004] Early knowledge graph-based commonsense question answering (KGQA) leveraged the structured knowledge in the KG to answer questions, fully exploring the deep connections between questions and entities and relations in the KG. Lin et al. designed KagNet, an algorithm that uses semantic information to generate subgraphs from an external KG to assist in reasoning about commonsense questions. This model generates subgraphs on the KG based on the question and answer, uses a graph convolutional neural network (GCN) for knowledge reasoning, encodes paths and captures relational information using LSTM, and finally uses a hierarchical attention mechanism to calculate the importance of paths and concepts for reasoning. However, as the number of hops increases, the paths grow exponentially, making the model difficult to scale. Feng et al. proposed a multi-hop relational reasoning model, MHGRN, which combines path-based methods with graph neural network methods. Using a relational attention mechanism, it aggregates information from different paths. However, MHGRN has certain drawbacks. The model's parameters increase with the number of relations, leading to over-parameterization. Furthermore, the model primarily utilizes node features when calculating attention, making it prone to oversmoothing.

[0005] As can be seen, graph attention networks introduce an attention mechanism into graph neural networks (GNNs), assigning different weights to different neighboring nodes of a central node, allowing the model to focus more on important neighboring node information and addressing the problem of large amounts of noise in graph data. However, these models are prone to oversmoothing as the number of network layers and iterations increases, and therefore cannot accurately infer the interactions between node structures. Summary of the Invention

[0006] In order to solve the above problems, the present disclosure proposes a knowledge graph question answering method and system based on a multi-hop combined graph fingerprint network. The method can analyze and understand the questions and obtain the correct answers in combination with the knowledge graph.

[0007] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0008] One or more embodiments provide a knowledge graph question answering method based on a multi-hop combined graph fingerprint network, comprising the following steps:

[0009] Perform text encoding on the target question and candidate option texts, and obtain the knowledge subgraph after matching based on the knowledge graph;

[0010] Through the trained multi-hop composite graph fingerprint network model, the knowledge subgraph related to the question and candidate options is graph encoded;

[0011] The text code and the image code are concatenated to calculate the credibility score, and the answer to the target question is determined based on the score;

[0012] Among them, the multi-hop combined graph fingerprint network model first uses the structural fingerprint attention mechanism to calculate the structural fingerprint attention score of the input knowledge subgraph; then it uses the entity-relationship combination operation on the knowledge subgraph, and updates the node features based on the attention score during the multi-hop information transmission process to realize the update of the knowledge subgraph and obtain the graph encoding result.

[0013] One or more embodiments provide a knowledge graph question answering system based on a multi-hop combined graph fingerprint network, including:

[0014] Text encoding module: configured to perform text encoding on the target question and candidate option texts, and obtain a knowledge subgraph based on knowledge graph matching;

[0015] Graph encoding module: configured to perform graph encoding on the knowledge subgraph related to the question and candidate options through the trained multi-hop combined graph fingerprint network model;

[0016] Answer output module: configured to calculate the credibility score by concatenating the text encoding and the image encoding, and determine the answer to the target question based on the score;

[0017] Among them, the multi-hop combined graph fingerprint network model first uses the structural fingerprint attention mechanism to calculate the structural fingerprint attention score of the input knowledge subgraph; then it uses the entity-relationship combination operation on the knowledge subgraph, and updates the node features based on the attention score during the multi-hop information transmission process to realize the update of the knowledge subgraph and obtain the graph encoding result.

[0018] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above method are completed.

[0019] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are completed.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] This paper uses a framework that fuses multiple relational information to simultaneously learn entity and relation embeddings, addressing the problem of over-parameterization of the model. Furthermore, structural fingerprint attention is used to assist in generating node representations in the graph. This leverages the high-order structural information in the graph to better generate node representations and alleviate the model's oversmoothing problem.

[0022] The advantages of the present disclosure and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0024] Figure 1 This is a schematic diagram of the knowledge graph question-answering model structure of Example 1 of the present disclosure;

[0025] Figure 2 This is a schematic diagram of the structural fingerprint of Example 1 of the present disclosure;

[0026] Figure 3 Schematic diagram of attention coefficients between nodes in Example 1 of the present disclosure;

[0027] Figure 4 is the structural fingerprint weight w of the embodiment 1 of the present disclosure vi Updated schematic diagram;

[0028] Figure 5 This is a comparison chart of the question-answering accuracy of the MHCGFN using this embodiment and the traditional MHGRN in Example 1 of the present disclosure. DETAILED DESCRIPTION

[0029] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0031] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0032] To address the issues described in the background, this paper proposes a knowledge graph question answering model based on a multi-hop composite graph fingerprint network (MHCGFN). This model improves its message passing by introducing CompGCN and employs a framework that integrates multiple relationship information to simultaneously learn entity and relationship embeddings, addressing the model's over-parameterization problem. Furthermore, MHCGFN uses structural fingerprint attention to assist in generating node representations in the graph. This utilizes high-order structural information in the graph to better generate node representations and alleviate the model's oversmoothing problem. This is illustrated below with a specific example.

[0033] Example 1

[0034] In the technical solutions disclosed in one or more embodiments, Figures 1 to 4 As shown in FIG, the knowledge graph question answering method based on the multi-hop combined graph fingerprint network includes the following steps:

[0035] Step 1: Encode the target question and candidate option texts, and obtain the knowledge subgraph after matching based on the knowledge graph;

[0036] Step 2: Use the trained multi-hop combined graph fingerprint network model to perform graph encoding on the knowledge subgraph related to the question and candidate options;

[0037] Step 3: Concatenate the text code and the image code to calculate the credibility score, and determine the answer to the target question based on the score.

[0038] Among them, the multi-hop combined graph fingerprint network model first uses the structural fingerprint attention mechanism to calculate the structural fingerprint attention score of the input knowledge subgraph; then it uses the entity-relationship combination operation on the knowledge subgraph, and updates the node features based on the attention score during the multi-hop information transmission process to realize the update of the knowledge subgraph and obtain the graph encoding result.

[0039] In this embodiment, given a multiple-choice question with multiple answer candidate options, the text encoder uses a trained large-scale pre-trained language model to encode the question and candidate options. Then, the encoded question and candidate options enter the graph encoder, and the knowledge subgraph related to the question and candidate options is encoded through the multi-hop combined graph fingerprint network model designed in this article. Finally, the credibility score of the spliced text encoding and graph encoding is calculated. The correct answer is selected from the answer options given for the question. After the entities and relationships adjacent to the central entity node are combined through entity relationship operations, the model can learn entity embedding and relationship embedding at the same time. Structural fingerprint attention is used to assist in generating node representations in the graph, and the high-order structural information in the graph is used to better generate node representations, thereby alleviating the over-smoothing problem of the question-answering model.

[0040] The above steps in this embodiment are implemented by constructing a knowledge graph question answering model (MHCGFN), which includes a text encoder and a multi-hop composition graph fingerprint network (Multi-hop Composition Graph Fingerprint Network).

[0041] The text encoder is used to encode the target question and candidate option texts.

[0042] The multi-hop composite graph fingerprint network is configured to encode the knowledge subgraph related to the question and candidate options. The multi-hop composite graph fingerprint network includes an entity-relationship composition operation module, a structural fingerprint attention mechanism module, a multi-hop messaging module, an aggregation module, and an output module.

[0043] Optionally, in step 1, a large pre-trained language model is used to encode the textual content of the question and the candidate answers corresponding to the question in the knowledge graph question answering task. The advanced large pre-trained language models RoBERTa-Large and AristoRoBERTaV7 are suitable. Using these two models, a vector representation of each question q and its corresponding candidate answer a is obtained, with question q denoted as q and candidate answer a denoted as a.

[0044] Optionally, concatenate q and a as a sentence vector c, which is used for the context vector representation of questions and answers; obtain the knowledge subgraph G related to questions and options by matching on the large knowledge graph ConceptNet as external knowledge s ;

[0045] Specifically, the knowledge subgraph related to the sentence vector c is defined as G s =(V,R,E,φ), where V represents the set of all entity nodes in the knowledge subgraph, and all entities in the set have a φ relationship with the sentence vector c. R represents the set of all relations in the knowledge subgraph, and E represents the set of edges therein.

[0046] The mapping function is:

[0047] φ(v):V→T={e q ,e a ,e ο}

[0048] Among them, the entity node v∈V,e q Indicates that v is an entity that exists in question q, e a Indicates that the entity node v is an entity that exists in option a, e ο Indicates entities that do not exist in the question and options. Finally, the sentence vector c is combined with G s Concatenate to calculate credibility score.

[0049] In step 2, the entity-relationship combination operation can be: the entities and relationships adjacent to the central entity node are embedded in the knowledge graph through subtraction or multiplication to realize the entity and relationship combination operation.

[0050] Optionally, the entity-relationship combination operation in CompGCN can be used to simultaneously learn entity and relationship information in the knowledge graph.

[0051] CompGCN (Composition-Based Multi-Relational Graph Convolutional Networks) multi-relational graph convolutional network.

[0052] In this embodiment, CompGCN is introduced to improve the message passing method of the model, and a framework that integrates multiple relationship information is adopted to simultaneously learn entity embedding and relationship embedding to solve the problem of over-parameterization of the model.

[0053] Knowledge graphs are typical multi-relational graphs, where each edge is directional. Most existing methods for processing such multi-relational graphs suffer from over-parameterization, which can easily lead to model overfitting. This problem is present in MHGRN, which uses a generalized R-GCN for multi-hop message passing. As the number of relationships increases, the number of model parameters increases rapidly. This embodiment improves model performance by simultaneously learning both entity and relationship information from the multi-relational graph to address this over-parameterization issue.

[0054] Optionally, entity-relationship combination operations are of the form The combination operator e v is the entity embedding, e r is the relation embedding, is the dimensional representation of entity and relationship vectors. Knowledge graph embedding techniques such as subtraction and multiplication can be used to implement entity-relationship combination operations.

[0055] When using subtraction or multiplication as the conversion operation, the formula is as follows:

[0056] Subtraction:

[0057] multiplication:

[0058] By combining entities and relationships adjacent to the central entity node through an entity-relationship combination operation, the model can simultaneously learn entity and relationship embeddings. This combination operation not only reduces the number of parameters in the graph neural network, but also enables the central node to better aggregate information from all its neighbors.

[0059] Specifically, after simultaneously learning the entity and relationship information in the knowledge graph using the entity-relationship combination operation in CompGCN, the node features are updated during the multi-hop information transmission process based on the attention score to achieve the update of the knowledge subgraph. The graph encoding result includes the following steps:

[0060] Step 21: Initialize the linear transformation of the node features of the acquired knowledge subgraph nodes. After initialization, the node type after mapping can be perceived. The formula is as follows:

[0061]

[0062] where μ and β are specific to the entity node v i The learnable parameters of Represents the input knowledge subgraph G s Node characteristics of the node.

[0063] Step 22: Based on the multi-hop attention score, the message passing of CompGCN is generalized, and the information after the entity-relation combination operation is passed through multi-hop messages on the path to obtain multi-hop information.

[0064] This step can be implemented in the multi-hop messaging module (Multi-hop Messaging).

[0065] Perform τ-hop message transmission, the formula is as follows:

[0066]

[0067] in, represents the multi-hop attention score calculated by the structural fingerprint attention mechanism;

[0068] represents a valid τ-hop relationship path;

[0069] is the normalization factor, calculated as follows:

[0070]

[0071] W σ(r) It is a learnable matrix specific to the relationship type. The definition formula of the learnable matrix is as follows:

[0072]

[0073] Among them, W nbr is the relationship type of neighbor nodes, W inv Indicates the inverse relationship, W self It is a self-circulating relationship type;

[0074] The relationship update function formula is as follows:

[0075]

[0076] Among them, W r is a learnable transformation matrix that projects relations into the same embedding space as nodes.

[0077] Step 24: Aggregate the scores passed through the structural fingerprint attention mechanism and the obtained multi-hop information. This can be implemented in the aggregation module. The aggregation function formula is as follows:

[0078]

[0079] Step 24: The result of the aggregation operation is transmitted to the output module and processed using a nonlinear activation function to obtain the output node embedding. The formula is as follows:

[0080]

[0081] Where L and L' are learnable model parameters, and δ represents a nonlinear activation function.

[0082] In some embodiments, structural fingerprint attention is used to sThe structural fingerprint proposed in ADSF is constructed on the basis of the above to solve the over-smoothing problem after multi-hop message transmission. The structural fingerprint can effectively alleviate the over-smoothing problem, improve the utilization of the structural information around the node, and make better use of G s The high-order structural information in .

[0083] Among them, ADSF stands for Adaptive Structural Fingerprint.

[0084] Specifically, the structural fingerprint diagram is as follows Figure 2 As shown:

[0085] For the knowledge subgraph G s , the structural fingerprint attention mechanism is used to calculate the structural fingerprint attention score of the input knowledge subgraph, including:

[0086] Step 2-1: Targeting the knowledge subgraph G s , by constructing an adaptive structural fingerprint, each node of the knowledge subgraph is placed in the context of a local receptive field formed by its hop-by-hop adjacent nodes;

[0087] Figure 2 The triangle in the figure represents any given central node i, and all k-hop neighbor nodes of the central node i are used to generate the subgraph, which is expressed as (V vi ,E vi ), V vi and E vi They represent the sets of nodes (solid circles) and edges (black solid lines) in the neighborhood respectively.

[0088] Given a node i, the structural fingerprint is defined as: is the contribution weight of the node. The contribution weight determines the importance of each node in the formed receptive field and also determines the effective "shape" and size of the receptive field. Figure 2 The relatively three-dimensional local structure shown fully expresses the structural information of node i.

[0089] Step 2-2: For the nodes in the receiving domain, the content similarity between the nodes Interaction with structural fingerprints Calculate the attention score for each node.

[0090] Given two nodes i and j, such as Figure 3 As shown, we need to calculate the attention coefficient between node i and node j. Each node i and j has its own features and structural fingerprints. In terms of content, the features of the two nodes will be used to calculate their content similarity; in terms of structure, the structural fingerprints of the two nodes will be used to evaluate the interaction between them. Interaction with structural fingerprints to calculate the attention score. It will then be used to update the node features in the message passing step.

[0091] Node v i and v j The content similarity definition formula is as follows:

[0092]

[0093] Where W is the transformation matrix.

[0094] v i Assign a non-negative weight This weight can determine v i The importance of v in the receptive field. i The structural fingerprint of By mapping node distances, according to v j To the central node v i The distance [1,2,...,τ] is used to calculate v j The weight level ω=[ω1,ω2,...,ω τ ], ω is non-negative and monotonic and can be optimized during the learning process.

[0095] Optionally, in the knowledge subgraph G s In the process, a random walk with restart is performed to update the structural fingerprint weight The structural fingerprint information of the node is determined by updating the structural fingerprint weight; the structural fingerprint weight The updated schematic is as follows Figure 4 As shown, the middle solid node represents the central node v i , the remaining surrounding nodes represent other nodes, with v j express:

[0096] Weight The update definition formula is as follows:

[0097]

[0098] in, It is through v i The adjacency matrix The transition probability matrix is obtained by normalizing the columns of , and γ∈[0,1] is used as a parameter to weigh the random walk and restart. represents a vector with all entries zero except the one corresponding to the central node, and Ι is the identity matrix.

[0099] In this embodiment, by Quantifies the distance between the central node and other nodes, and reflects the s structural details.

[0100] One possible implementation method is to use weighted Jaccard similarity to quantify the interaction between two node structure fingerprints, such as v i and v j The interaction between the structural fingerprints The interaction between two fingerprints is quantified using J(·,·), which is as follows:

[0101]

[0102] The similarity of the nodes Interaction with structural fingerprints Perform normalization processing, the formula is as follows:

[0103]

[0104]

[0105] Finally, the attention score is calculated. The attention score calculation formula is as follows:

[0106]

[0107] Here, η(·) and ρ(·) are transfer functions similar to sigmoid.

[0108] In step 3, the text encoding and graph encoding are concatenated. Specifically, a pooling operation is performed on the knowledge subgraph obtained after text encoding to obtain a pooled graph representation related to the answer entity; the pooled graph representation is connected to the question-answer vector representation.

[0109] The credibility score can be calculated using a multi-layer perceptron (MLP):

[0110] ζ(q,a)=MLP(c⊕g s )

[0111] According to the calculated credibility scores, the answer to the question with the highest credibility score is used as the answer to the target question.

[0112] A further technical solution also includes a process of training the multi-hop combined graph fingerprint network model:

[0113] Step S1: Obtain training data and construct a sample set;

[0114] We can use the datasets CommonsenseQA and OpenBookQA; and divide these two datasets into three subsets: training set, test set, and validation set.

[0115] Step S2: Obtain sample set data, and perform text encoding based on the given question q and the candidate answer option a to obtain the vector representation and knowledge subgraph;

[0116] Step S3: For the multi-hop combined graph fingerprint network, initialize node features using pre-trained weights;

[0117] Step S4: transfer the knowledge subgraph to the constructed multi-hop combined graph fingerprint network, and update the knowledge subgraph G through multi-hop message passing with structured fingerprint attention mechanism. s ;

[0118] Step S5: Update the knowledge subgraph G s Perform a pooling operation to obtain a pooled graph representation associated with the answer entity;

[0119] Step S6: The pooling graph is represented as g s Connect it with the question-answer vector representation c, use a multi-layer perceptron MLP to calculate the rationality score, and get the predicted answer;

[0120] Step S7: Calculate the cross entropy loss between the predicted answer and the true answer, and adjust the network parameters based on the cross entropy loss until the prediction accuracy requirement is met, that is, the cross entropy loss is less than the set value, to obtain a trained multi-hop combination graph fingerprint network.

[0121] Calculate the minimum cross entropy loss to maximize the correct answer The rationality score is as follows:

[0122]

[0123] During inference, argmax can be used a (ζ(q,a)) to predict the most credible answer.

[0124] To illustrate the effectiveness of the method in this embodiment, the method in this embodiment is compared with other baseline models on the CommonsenseQA and OpenBookQA datasets. The model proposed in this embodiment achieves ideal results. The traditional MHGRN method is a multi-hop graph relational network.

[0125] like Figure 5 As shown in the figure, the question-answering accuracy (IHdev-Acc) of the MHCGFN of this embodiment and the traditional MHGRN with different multi-hop numbers are compared on the CommonsenseQA (IH) dataset. The accuracy of the model of this embodiment is significantly higher than that of the traditional MHGRN method, and the accuracy is the highest when the number of hops is 4.

[0126] Example 2

[0127] Based on Example 1, this embodiment provides a knowledge graph question-answering system based on a multi-hop combined graph fingerprint network, including:

[0128] Text encoding module: configured to perform text encoding on the target question and candidate option texts, and obtain a knowledge subgraph based on knowledge graph matching;

[0129] Graph encoding module: configured to perform graph encoding on the knowledge subgraph related to the question and candidate options through the trained multi-hop combined graph fingerprint network model;

[0130] Answer output module: configured to calculate the credibility score by concatenating the text encoding and the image encoding, and determine the answer to the target question based on the score;

[0131] Among them, the multi-hop combined graph fingerprint network model first uses the structural fingerprint attention mechanism to calculate the structural fingerprint attention score of the input knowledge subgraph; then it uses the entity-relationship combination operation on the knowledge subgraph, and updates the node features based on the attention score during the multi-hop information transmission process to realize the update of the knowledge subgraph and obtain the graph encoding result.

[0132] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0133] Example 3

[0134] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps described in the method of embodiment 1 are completed.

[0135] Example 4

[0136] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps described in the method of embodiment 1 are completed.

[0137] The electronic devices proposed in the present disclosure can be mobile terminals and non-mobile terminals. Non-mobile terminals include desktop computers, and mobile terminals include smart phones (such as Android phones, IOS phones, etc.), smart glasses, smart watches, smart bracelets, tablets, laptops, personal digital assistants, and other mobile Internet devices that can communicate wirelessly.

[0138] It should be understood that in the present disclosure, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0139] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0140] During implementation, each step of the above method can be performed by hardware integrated logic circuits in a processor or by software instructions. The steps of the method disclosed in this disclosure can be directly implemented as execution by a hardware processor, or by a combination of hardware and software modules in the processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in a memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here. Those skilled in the art will appreciate that the units, i.e., algorithmic steps, of the various examples described in connection with the embodiments disclosed herein can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this disclosure.

[0141] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0142] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a division of logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0143] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0144] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

[0145] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Those skilled in the art should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A knowledge graph question answering method based on a multi-hop combined graph fingerprint network, characterized in that: The steps include: Perform text encoding on the target question and candidate option texts, and obtain the knowledge subgraph after matching based on the knowledge graph; Through the trained multi-hop composite graph fingerprint network model, the knowledge subgraph related to the question and candidate options is graph encoded; The text code and the image code are concatenated to calculate the credibility score, and the answer to the target question is determined based on the score; Among them, the multi-hop combined graph fingerprint network model first uses the structural fingerprint attention mechanism to calculate the structural fingerprint attention score of the input knowledge subgraph; then it uses the entity-relationship combination operation on the knowledge subgraph and updates the node features based on the attention score during the multi-hop information transmission process to achieve the update of the knowledge subgraph and obtain the graph encoding result; The subgraph generated by all k-hop neighbor nodes of the central node i is expressed as , and Represent the set of nodes and edges in the neighborhood respectively; given a node i, the structural fingerprint is defined as: , is the contribution weight of the node; The entity-relationship combination operation is to embed the entities and relationships adjacent to the central entity node into the knowledge graph through subtraction or multiplication to achieve the entity-relationship combination operation; The method of calculating the structural fingerprint attention score using the structural fingerprint attention mechanism includes: For the knowledge subgraph, by constructing an adaptive structural fingerprint, each node of the knowledge subgraph is placed in the context of the local receptive field formed by the hop-by-hop adjacent nodes; For the nodes in the receptive field, the attention score of each node is calculated through the interaction of content similarity and structural fingerprints between nodes; Based on the attention score, node features are updated during multi-hop information transmission to achieve the update of the knowledge subgraph. The process of obtaining the graph encoding result is as follows: Initialize the linear transformation of the node features of the acquired knowledge subgraph nodes; Based on the multi-hop attention score, the message passing of CompGCN is generalized, and the information after the entity-relation combination operation is passed on the path through multi-hop messages to obtain multi-hop information; Aggregate the scores passed through the structural fingerprint attention mechanism and the obtained multi-hop information; The result after the aggregation operation is processed with a nonlinear activation function to obtain the output node embedding.

2. The knowledge graph question answering method based on a multi-hop combined graph fingerprint network according to claim 1, characterized in that: Use a large pre-trained language model to encode the textual content of questions and their corresponding answer candidates.

3. The knowledge graph question answering method based on a multi-hop combined graph fingerprint network according to claim 1, characterized in that: The text encoding and graph encoding are spliced together. Specifically, a pooling operation is performed on the knowledge subgraph obtained after text encoding to obtain a pooled graph representation related to the answer entity; the pooled graph representation is connected to the question-answer vector representation.

4. The knowledge graph question answering method based on a multi-hop combined graph fingerprint network according to claim 1, characterized in that: A multi-layer perceptron (MLP) is used to calculate the credibility score.

5. The knowledge graph question answering method based on a multi-hop combined graph fingerprint network according to claim 1, characterized in that: In the knowledge subgraph, random walk with restart is used to update the structural fingerprint weight; The weighted Jaccard similarity is used to quantify the interaction between two node structure fingerprints.

6. The knowledge graph question answering method based on a multi-hop combined graph fingerprint network according to claim 1, characterized in that: The process of training the multi-hop composite graph fingerprint network model includes the following steps: Obtain training data and build a sample set; Obtain sample data, perform text encoding based on the given question and candidate answer options, and obtain vector representations and knowledge subgraphs; The knowledge subgraph is transferred to the constructed multi-hop combined graph fingerprint network, and the knowledge subgraph is updated through multi-hop message passing with a structured fingerprint attention mechanism; Perform a pooling operation on the updated knowledge subgraph to obtain a pooled graph representation related to the answer entity; Connect the pooled graph representation with the question-answer vector representation, use a multi-layer perceptron (MLP) to calculate the rationality score, and obtain the predicted answer; The cross entropy loss between the predicted answer and the true answer is calculated, and the network parameters are adjusted based on the cross entropy loss until the prediction accuracy requirements are met, thereby obtaining a trained multi-hop combined graph fingerprint network.

7. A knowledge graph question answering system based on a multi-hop combined graph fingerprint network based on the knowledge graph question answering method based on a multi-hop combined graph fingerprint network according to claim 1, characterized in that: include: Text encoding module: configured to perform text encoding on the target question and candidate option texts, and obtain a knowledge subgraph based on knowledge graph matching; Graph encoding module: configured to perform graph encoding on the knowledge subgraph related to the question and candidate options through the trained multi-hop combined graph fingerprint network model; Answer output module: configured to calculate the credibility score by concatenating the text encoding and the image encoding, and determine the answer to the target question based on the score; Among them, the multi-hop combined graph fingerprint network model first uses the structural fingerprint attention mechanism to calculate the structural fingerprint attention score of the input knowledge subgraph; Then, an entity-relationship combination operation is applied to the knowledge subgraph, and the node features are updated in the multi-hop information transmission process based on the attention score to achieve the update of the knowledge subgraph and obtain the graph encoding result.

8. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of any one of the methods of claims 1 to 6 are completed.

9. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the steps of any one of the methods of claims 1 to 6.

Images