Question and answer method and system based on knowledge graph and large language model and storage medium

Abstract

The embodiment of the invention provides a question answering method and system based on a knowledge graph and a large language model and a storage medium, and relates to the technical field of natural language processing. The method comprises the steps of obtaining question information and a knowledge graph of a field to which the question information belongs; performing n-hop reasoning operation on the question information based on the knowledge graph to obtain a knowledge summary corresponding to each hop; n is an integer greater than 1; based on the knowledge summary and the question information corresponding to each hop, determining answer information corresponding to the question information through a large language model; the ith-hop reasoning operation comprises the steps of determining a knowledge clue corresponding to the ith hop based on a knowledge summary and question information corresponding to the (i-1) th hop; the knowledge clue corresponding to the ith hop comprises an ith main entity and an ith tail entity having a relationship with the ith main entity; the i main entity is the (i-1) tail entity in the knowledge clue of the (i-1) hop; and determining the knowledge summary corresponding to the ith hop based on the knowledge clue corresponding to the ith hop, the knowledge summary corresponding to the (i-1) th hop and the problem information.

Description

Technical Field

[0001] This application relates to the technical field of natural language processing, and particularly to a question-answering method, system, and storage medium based on a knowledge graph and a large language model. Background Art

[0002] With the development of technology, large language models (LLMs) have gradually acquired the ability to understand user intentions and context, and can generate customized answers according to the personalized needs of users. Users can describe existing problems in natural language and obtain answers to the problems through LLMs. This convenience enables users to obtain the required information more easily and improves the efficiency of information acquisition.

[0003] Since knowledge in the real world is constantly growing and updating, and after the training of LLMs is completed, the process of knowledge accumulation has ended. Therefore, the scope of knowledge is limited by the data used to train the LLMs, resulting in the "hallucination" problem when LLMs answer questions, that is, LLMs can give certain answers but there are incorrect contents among them. Summary of the Invention

[0004] This application provides a question-answering method, system, and storage medium based on a knowledge graph and a large language model, which is used to improve the answer accuracy of large language models when answering questions.

[0005] In the first aspect of the embodiments of this application, a question-answering method based on a knowledge graph and a large language model is provided, including: obtaining question information and a knowledge graph of the field to which the question information belongs, where the knowledge graph includes multiple entities and the relationships between the entities; performing an n-hop reasoning operation on the question information based on the knowledge graph to obtain the knowledge summary corresponding to each hop; n is an integer greater than 1; constructing a first prompt message based on the knowledge summary corresponding to each hop and the question information, and inputting the first prompt message into the large language model to determine the answer message corresponding to the question information; where the i-th hop reasoning operation in the n-hop reasoning operation, 1 < i ≤ n, includes the following steps: determining the knowledge clue corresponding to the i-th hop through the knowledge graph based on the knowledge summary corresponding to the (i - 1)-th hop and the question information; where the knowledge clue corresponding to the i-th hop includes the i-th main entity and the i-th tail entity having a relationship with the i-th main entity; the i-th main entity is the (i - 1)-th tail entity in the knowledge clue of the (i - 1)-th hop; determining the knowledge summary corresponding to the i-th hop based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the question information.

[0006] The method provided in the embodiment of the present application performs n-hop reasoning operations on question information based on the knowledge graph, and performs reasoning in each reasoning operation based on the knowledge summary corresponding to the previous jump, so that the information in the knowledge clue of the current jump is obtained based on the previous reasoning result, thereby ensuring that valid information is not lost during the reasoning process. In addition, new knowledge related to the question is supplemented during the reasoning process, thereby solving the problem of knowledge dependency and making the answer of the large language model more accurate and clear.

[0007] In combination with the first implementation method of the first aspect, based on the knowledge summary and question information corresponding to the i-1 hop, the knowledge clue corresponding to the i-th hop is determined through the knowledge graph, including: constructing second prompt information based on the relationship corresponding to the i-th main entity, the knowledge summary and question information corresponding to the i-1 hop; inputting the second prompt information into the large language model, and determining the i-th relationship in the relationship corresponding to the i-th main entity that matches the knowledge summary and question information corresponding to the i-1 hop through the large language model; determining the knowledge clue corresponding to the i-th layer based on the i-th main entity, the i-th relationship, and the i-th tail entity that has the i-th relationship with the i-th main entity.

[0008] In combination with the second implementation method of the first aspect, in the knowledge clue corresponding to the i-th layer, for each i-th relationship, each i-th tail entity that has the i-th relationship with the i-th main entity is represented in the form of natural language.

[0009] In combination with the third implementation method of the first aspect, the knowledge summary corresponding to the current jump is determined based on the knowledge clues corresponding to the i-th jump, the knowledge summary corresponding to the i-1-th jump, and the question information, including: constructing third prompt information based on the knowledge clues corresponding to the i-th jump, the knowledge summary corresponding to the i-1-th jump, and the question information; inputting the third prompt information into the large language model, and determining the knowledge summary corresponding to the i-th jump through the large language model.

[0010] In combination with the fourth implementation method of the first aspect, the first-hop reasoning operation in the n-hop reasoning operation includes the following steps: based on the keywords in the question information, determining one or more first main entities matching the keywords in the entities of the knowledge graph; in the relationship corresponding to the first main entity, determining the first relationship matching the question information, and obtaining the first knowledge clue, the first knowledge clue including: the first main entity, the first relationship, and the first tail entity having a first relationship with the first main entity; based on the first knowledge clue and the question information, determining the knowledge summary corresponding to the first jump.

[0011] Combined with the fifth implementation manner of the first aspect, based on the keywords in the question information, determining one or more first main entities that match the keywords in the entities of the knowledge graph includes: encoding the keywords in the question information to obtain keyword vectors; encoding the entities in the knowledge graph to obtain encoded vectors; and taking the entities corresponding to the encoded vectors whose similarity to the keyword vectors is higher than a preset similarity threshold as the first main entities.

[0012] Combined with the sixth implementation manner of the first aspect, in the relationships corresponding to the first main entities, determining a first relationship that matches the question information to obtain a first knowledge clue includes: constructing fourth prompt information based on the relationships corresponding to the first main entities and the question information; inputting the fourth prompt information into a large language model, and determining, through the large language model, a first relationship that matches the question information in the relationships corresponding to the first main entities; and determining the first knowledge clue based on the first main entity, the first relationship, and a first tail entity that has the first relationship with the first main entity.

[0013] Combined with the seventh implementation manner of the first aspect, constructing first prompt information based on the knowledge summary corresponding to each hop and the question information includes: describing the knowledge summary corresponding to each hop according to a hierarchical structure to form a target knowledge summary; and constructing the first prompt information based on the target knowledge summary and the question information.

[0014] The second aspect of the embodiments of the present application provides a question-answering system based on a knowledge graph and a large language model, including: a receiving device, a processing device, and a display screen; the receiving device is configured to receive question information input by a user; the processing device is configured to perform an n-hop reasoning operation on the question information based on the knowledge graph in the field to which the question information belongs to obtain a knowledge summary corresponding to each hop, where n is an integer greater than 1; construct first prompt information based on the knowledge summary corresponding to each hop and the question information, and input the first prompt information into the large language model to determine answer information corresponding to the question information; the display screen is configured to display the answer information corresponding to the question information; where the processing device's specific implementation of the i-th hop reasoning operation in the n-hop reasoning operation, 1 < i ≤ n, is as follows: determining a knowledge clue corresponding to the i-th hop through the knowledge graph based on the knowledge summary corresponding to the (i - 1)-th hop and the question information; where the knowledge clue corresponding to the i-th hop includes an i-th main entity and an i-th tail entity that has a relationship with the i-th main entity; the i-th main entity is the (i - 1)-th tail entity in the knowledge clue of the (i - 1)-th hop; and determining the knowledge summary corresponding to the i-th hop based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the question information.

[0015] A third aspect of an embodiment of the present application provides an electronic device, comprising: one or more processors; one or more memories; wherein the one or more memories are used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the question-answering method based on the knowledge graph and the large language model provided by the first aspect and its possible implementation methods.

[0016] The fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are run on a computer, the computer executes the question-answering method based on the knowledge graph and the large language model provided by the first aspect and its possible implementation methods.

[0017] The fifth aspect of the embodiments of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the question-answering method based on a knowledge graph and a large language model provided in the first aspect and its possible implementation methods.

[0018] Among them, the beneficial effects described in the second to fifth aspects can refer to the beneficial effect analysis of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0020] Figure 1 A schematic diagram of the structure of a question-answering system based on a knowledge graph and a large language model provided in an embodiment of the present application;

[0021] Figure 2 A method flow of a question-answering method based on a knowledge graph and a large language model provided in an embodiment of the present application Figure 1 ;

[0022] Figure 3 A method flow of a question-answering method based on a knowledge graph and a large language model provided in an embodiment of the present application Figure 2 ;

[0023] Figure 4 A method flow of a question-answering method based on a knowledge graph and a large language model provided in an embodiment of the present application Figure 3 ;

[0024] Figure 5 A method flow of a question-answering method based on a knowledge graph and a large language model provided in an embodiment of the present application Figure 4 ;

[0025] Figure 6 A method flow of a question-answering method based on a knowledge graph and a large language model provided in an embodiment of the present application Figure 5 ;

[0026] Figure 7 An overall flow chart of a question-answering method based on a knowledge graph and a large language model provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of another question-answering system based on a knowledge graph and a large language model provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of a question-answering device based on a knowledge graph and a large language model provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0032] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, when describing a pipeline, the "connected" and "connection" used in this application have the meaning of conduction. The specific meaning needs to be understood in conjunction with the context.

[0033] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0034] Since the scope of knowledge that a large language model can answer is limited by the data during training, LLM has a "hallucination" problem when answering questions, that is, LLM can give a certain answer but there are errors in it. In order to improve the accuracy of the answers given by the large language model in the related art, new knowledge data is usually used to fine-tune and update the parameters of LLM to adapt to specific downstream tasks. Since the number of parameters of LLM is huge (such as tens of billions or hundreds of billions), updating these parameters requires a lot of computing resources. The fine-tuning process requires high-performance computing devices, such as GPU or TPU, to accelerate the calculation. As the size of the model increases, the required computing resources also increase exponentially. In addition, due to the large size of the model, a large amount of intermediate data and gradient information may be generated during the fine-tuning process, further increasing the computational burden. Moreover, if the coverage of the fine-tuning data is not enough to cover all aspects of the model's previous learning, the model may lose the ability of the previous task when learning a new task. Or if the fine-tuning data is significantly different from the pre-training data, the model may adjust its weights to adapt to the new data distribution, thereby forgetting the information learned on the previous data set. That is, fine-tuning LLM is prone to catastrophic forgetting of knowledge.

[0035] The inventors found that if the knowledge graph is used to improve the LLM capability, it may not be able to handle complex knowledge dependency problems because the path can only select the most relevant adjacent entities. In the knowledge graph, the relationships between some entities may be indirect and need to be connected through multiple intermediate entities. When the number of these intermediate entities is large, the path-based method may have difficulty capturing these relationships because the path is too long. When constructing the reasoning path, the model may mistakenly identify relationships that are irrelevant or duplicate to the problem. These relationships will interfere with the model's reasoning process, resulting in inaccurate answers or inability to draw accurate conclusions. That is, the path-based method has the problem of knowledge dependency.

[0036] Based on this, an embodiment of the present application provides a question-answering method based on a knowledge graph and a large language model. When the question information is obtained, the method performs n-hop reasoning operations on the question information based on the knowledge graph of the field to which the question information belongs, and obtains the knowledge summary corresponding to each jump; wherein, when the i-th jump reasoning operation is executed, the knowledge clue corresponding to the current jump is determined based on the knowledge summary corresponding to the previous jump and the question information. Then, the knowledge summary corresponding to the current jump is determined based on the knowledge clue corresponding to the current jump, the knowledge summary corresponding to the previous jump, and the question information. After the multi-hop reasoning operation is completed, the first prompt information is constructed based on the knowledge summary and question information corresponding to each jump, and the first prompt information is input into the large language model to determine the answer information corresponding to the question information.

[0037] It can be seen that the method provided by the embodiment of the present application performs n-hop reasoning operations on the problem information, and in each hop reasoning operation, reasoning is performed based on the knowledge summary corresponding to the previous hop, so that the information in the knowledge clue of the current hop is obtained based on the previous reasoning result, thereby ensuring that valid information is not lost during the reasoning process. In addition, new knowledge related to the problem is supplemented during the reasoning process, which enhances the reliability and accuracy of the reasoning, thereby solving the problem of knowledge dependence.

[0038] In some embodiments, the question-answering method based on the knowledge graph and the large language model provided in the embodiments of the present application can be applied to electronic devices, and the electronic devices can be personal computers (PCs), laptops, mobile devices, tablet computers, laptop computers and other electronic devices. The embodiments of the present application do not limit the specific form of the electronic devices. Alternatively, the question-answering method based on the knowledge graph and the large language model provided in the embodiments of the present application can also be applied to a single server, or a server cluster composed of multiple servers. In some implementations, the server cluster can be a distributed cluster server. The embodiments of the present application do not impose any restrictions on this.

[0039] In other embodiments, the question-answering method based on knowledge graph and large language model provided in the embodiments of the present application can be applied to Figure 1 The question answering system 100 based on the knowledge graph and the large language model is shown. Figure 1 As shown, the question-answering system 100 based on the knowledge graph and the large language model may include a receiving device 11, a processing device 12 and a display screen 13.

[0040] In the embodiment of the present application, the receiving device 11 is used to receive question information input by the user.

[0041] In some embodiments, the receiving device 11 may be a device such as a keyboard, a mouse, a touch screen, etc., so that the receiving device 11 can directly receive the question information input by the user. Alternatively, the receiving device 11 may be a communication interface, so that the receiving device 11 can receive the question information input and sent by the user in other devices. The specific form of the receiving device 11 is not limited in the embodiments of the present application.

[0042] In the embodiments of the present application, the processing device 12 is configured to perform an n-hop reasoning operation on the question information based on the knowledge graph of the field to which the question information belongs, to obtain the knowledge summary corresponding to each hop; n is an integer greater than 1; construct a first prompt message based on the knowledge summary corresponding to each hop and the question information, and input the first prompt message into the large language model to determine the answer information corresponding to the question information.

[0043] Among them, for the i-th (1 < i ≤ n) hop reasoning operation in the n-hop reasoning operation, the processing device 12 may specifically include: determining the knowledge clue corresponding to the i-th hop through the knowledge graph based on the knowledge summary corresponding to the (i - 1)-th hop and the question information; wherein, the knowledge clue corresponding to the i-th hop includes the i-th main entity and the i-th tail entity having a relationship with the i-th main entity; the i-th main entity is the (i - 1)-th tail entity in the knowledge clue of the (i - 1)-th hop; determining the knowledge summary corresponding to the i-th hop based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the question information.

[0044] It should be understood that the processing device in the embodiments of the present application can be used to execute computer programs / instructions to implement the steps of the following question-answering method based on the knowledge graph and the large language model. The specific functions of the processing device are not elaborated herein.

[0045] In the embodiments of the present application, the display screen 13 is configured to display the answer information corresponding to the question information.

[0046] After the processing device 12 infers the answer information corresponding to the question information, the answer information can be displayed through the display screen 13, so that the user can read the answer information.

[0047] It should be understood that the system structure of the question-answering system 100 based on the knowledge graph and the large language model is not limited in the embodiments of the present application. Among them, the receiving device 11, the processing device 12, and the display screen 13 may be integrated on one device or on different devices, and information interaction is realized through a wireless network. The question-answering system 100 based on the knowledge graph and the large language model may further include more or fewer components, which are not limited in the embodiments of the present application.

[0048] The following describes the question-answering method based on the knowledge graph and the large language model provided in the embodiments of the present application with reference to the accompanying drawings.

[0049] Please refer to Figure 2 The question-answering method based on the knowledge graph and the large language model provided in the embodiment of the present application comprises the following steps:

[0050] S101. Obtain question information and a knowledge graph of the field to which the question information belongs.

[0051] Among them, the knowledge graph includes multiple entities and the relationships between entities.

[0052] Entities are the basic units in the knowledge graph, representing objects or concepts with clear boundaries and independent existence. These objects or concepts can be concrete or abstract. Common entity types include but are not limited to: concepts (such as "physics", "economics", etc.), time (such as "2023", "202 BC", etc.), place names (such as "Beijing", "Shanghai", etc.). Figure 2 As shown, entities are usually represented in the form of nodes in the knowledge graph, and each node has a unique identifier (ID) to distinguish different entities.

[0053] Relationships describe a certain connection or interaction between entities. In knowledge graphs, relationships are usually used to connect different entities to form a complex network structure. Relationships can be binary (involving only two entities) or multivariate (involving multiple entities). Common relationship types include but are not limited to the following:

[0054] Attribute relationship: describes an attribute or feature of an entity, such as "Name A-Age-30 years old".

[0055] Hierarchical relations: describe the hierarchical or taxonomic relationship between entities, such as "cat-is-a-mammal".

[0056] Relationship: describes a certain association or connection between entities, such as "Name A-Friend-Name B".

[0057] Event relationship: describes the role or relationship of an entity in an event, such as "Name A-attended-2023-XX meeting".

[0058] like Figure 2 As shown, relationships are usually represented in the knowledge graph in the form of edges, and each edge has a clear direction and type to indicate the direction and nature of the relationship.

[0059] The embodiments of the present application do not restrict the acquisition of question information and the knowledge graph of the field to which the question information belongs. For example, as a feasible implementation method, a question input interface can be set in the electronic device, and the user can input question information through an input device such as a touch screen and a keyboard based on the question input interface, so that the electronic device can obtain the question information. As another feasible implementation method, the user can input voice information to the electronic device based on a microphone connected to the electronic device, so that the electronic device can perform voice recognition based on the voice information to obtain the question information.

[0060] As a feasible implementation method, knowledge graphs in multiple technical fields can be stored in electronic devices or other storage media. After obtaining the problem information, the field to which the problem information belongs can be determined, and then the knowledge graph corresponding to the field can be determined.

[0061] Exemplarily, as a feasible implementation method, the knowledge graph can be established in the following way: collect data in the field and convert the data into the Resource Description Framework (RDF) format, and then use a data management system (such as Virtuoso) to import the data and establish a knowledge graph. The embodiment of the present application does not limit the method of obtaining or establishing the knowledge graph.

[0062] It should be understood that the RDF format is a language and model for describing network resources. It provides a framework container, adopts a triple structure, and defines a set of formal methods through Extensible Markup Language (XML), providing a structural basis for machine semantic understanding.

[0063] S102: Perform n-hop reasoning operations on the question information based on the knowledge graph to obtain the knowledge summary corresponding to each hop.

[0064] Wherein, n is an integer greater than 1. It should be understood that Figure 2 In the example, the number of hops n=3 is taken, and the n-hop reasoning operation includes the first hop, the second hop, and the third hop. In actual applications, the number of hops of the reasoning operation can be set according to requirements.

[0065] One-hop reasoning refers to the process of inferring new entities or attributes related to a known entity through a single relationship jump in a knowledge graph. Usually, the relationship between questions and answers is complex and requires multi-hop reasoning.

[0066] An n-hop reasoning operation (or multi-hop reasoning operation) refers to the process of starting from a given main entity (starting entity) and performing n-step reasoning through a series of relationship chains (i.e., a series of attributes or edges) to find other entities or information related to the main entity. Take a two-step relationship chain as an example: main entity → relationship 1 → intermediate entity → relationship 2 → answer. It should be understood that multi-hop problems require the system to be able to understand and process complex relationship chains and perform in-depth reasoning in the knowledge graph, which places high demands on the system's understanding and reasoning capabilities.

[0067] Exemplarily, n-hop reasoning operations usually involve the following steps: the starting point of reasoning needs to be clarified, that is, the main entity; starting from the main entity, select the first relationship to be traversed, which can be an attribute of the main entity or an entity associated with it; find the next entity or information associated with the main entity through the selected relationship; if further reasoning is required (i.e. n>1), start from the entity obtained in the previous step, select the relationship again, and find the next entity or information associated with the entity. This process will be repeated n-1 times; after n steps of reasoning, one or more entities or information related to the main entity will be obtained, which are the results of reasoning.

[0068] Since there may be many relationships corresponding to one entity during the reasoning process, at each depth, some entities will be selected from the entities corresponding to the main entity before exploring the next depth. In related technologies, similarity is usually used to filter relationships, that is, similarity is filtered for all tail entities or relationships connected to the main entity. However, this method is likely to delete answer-related entities in the initial screening process, and the screening of entities or relationships in each subsequent reasoning process makes it difficult to find the correct answer.

[0069] For example, suppose the question information is: "Which NBA team won the NBA Finals championship with a score of 4-3 in the shortest time since the team was founded?" The main entity in this example is the NBA Finals. In the knowledge graph, the first entities connected are the finals of different years (such as: 1961 NBA Finals, 1962 NBA Finals...) and other entities. These entities only differ in the year. The similarity screening method cannot determine which one is the answer to the question (the answer-related entity is the 1952 NBA Finals). After screening, the correct answer may even be deleted.

[0070] In the knowledge graph, the relationships between some entities may be indirect and need to be connected through multiple intermediate entities. When the number of these intermediate entities is large, the path-based method may have difficulty capturing these relationships because the path is too long. In other words, in the subsequent reasoning process, no matter how many times the reasoning is performed, the final required entity cannot be determined, and there is a problem of knowledge dependency.

[0071] Based on this, in the n-hop reasoning operation provided in the embodiments of the present application, the knowledge summary corresponding to the previous hop is considered during the reasoning process, that is, the knowledge obtained from the previous reasoning is referred to in each reasoning process, so that the reasoning at the current level has a valid basis, thereby solving the knowledge dependence problem.

[0072] Exemplarily, please refer to Figure 3 , the i-th hop reasoning operation in the n-hop reasoning operation of the embodiments of the present application, where 1 < i ≤ n, includes the following steps:

[0073] S201. Based on the knowledge summary corresponding to the (i - 1)-th hop and the problem information, determine the knowledge clue corresponding to the i-th hop through the knowledge graph.

[0074] Among them, the knowledge clue corresponding to the i-th hop includes the i-th main entity and the i-th tail entity having a relationship with the i-th main entity; the i-th main entity is the i-th tail entity in the knowledge clue of the (i - 1)-th hop.

[0075] It should be understood that the i-th main entity is the starting point of the knowledge clue for the current hop. Taking the i-th tail entity in the knowledge clue of the (i - 1)-th hop as the i-th main entity, that is, starting from the tail entity of the knowledge clue of the previous hop, to determine the next entity connected to the current main entity through a specific relationship.

[0076] When determining the knowledge clue corresponding to the current hop, it is determined according to the knowledge summary corresponding to the previous hop and the problem information. It can be seen that this is an iterative and gradually in-depth reasoning process, and each hop is based on the previous reasoning result, aiming to gradually narrow the search scope and finally find the knowledge clue most relevant to the problem.

[0077] It can be understood that taking the knowledge summary of the previous hop as the basis for determining the knowledge clue can analyze in detail the entities, relationships, and their associations mentioned in the previous hop summary, so as to have a clear understanding of its content; taking the problem information as the basis for determining the knowledge clue enables re-examination of the original problem during the reasoning process of the current hop, especially those parts related to the current reasoning step, and identifying key terms, restrictive conditions, or specific requirements in the problem.

[0078] Specifically, as a feasible implementation manner, please refer to Figure 4 , S201 can be specifically implemented as:

[0079] S2011. Based on the relationship corresponding to the i-th main entity, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information, construct the second hint information.

[0080] Exemplarily, the second prompt information is: given a question, a main entity and summarized knowledge, please select three relationships that are most helpful in answering the question from the following relationships: <question>, <main entity>, <summarized knowledge>, <relationship>.

[0081] Among them, the question is the question information, the main entity is the i-1th tail entity in the i-1th hop of knowledge clues, the summarized knowledge is the knowledge summary corresponding to the i-1th hop, and the relationship is the relationship corresponding to the i-th main entity.

[0082] S2012. Input the second prompt information into the large language model, and determine through the large language model the i-th relationship in the relationship corresponding to the i-th main entity that matches the knowledge summary and question information corresponding to the i-1 hop.

[0083] The second prompt information is input into the large language model. With the help of the large language model's powerful understanding and knowledge integration capabilities, the most helpful relationship for answering the question is screened out from the relationships corresponding to the i-th main entity.

[0084] For example, when the question information is: "Which team won the NBA Finals championship with a score of 4-3 in the shortest time since the team was founded?", one of the main entities can be "NBA Finals", and the relationship corresponding to the i-th main entity can include: game, player, team, stadium, coach. Then, the second prompt information can be as follows:

[0085] Given a question and a main entity, please select up to 3 relationships from the following that are most helpful in answering the question;

[0086] <Question> Which team won the NBA Finals championship with the shortest score of 4-3 since its establishment?

[0087] <Primary Entity> NBA Finals;

[0088] <Relationship> Games, players, teams, venues, coaches...

[0089] Input it into the large language model, and the answer obtained from the large language model is: competition.

[0090] It is understandable that among the relationships corresponding to the i-th main entity, the most helpful relationship for answering the question is screened out, that is, the reasoning path is screened out in the knowledge graph. In related technologies, similarity is usually used for screening, and the similarity method tends to ignore the deep semantic relationship between entities. Similarity filtering may cause important key information to be discarded in advance because it only considers the surface similarity between texts (such as between entities and entities, between entities and questions), without a deep understanding of the complex semantics and context of the entities.

[0091] The solution provided in this application uses a large language model to determine the most helpful relationship among multiple relationships corresponding to the main entity. With the help of the semantic understanding ability of the large language model, it can consider the semantic relationship between entities at a deeper level, thereby obtaining more accurate relationships that are most helpful in answering questions, that is, obtaining a more accurate reasoning path.

[0092] S2013. Determine the knowledge clue corresponding to the i-th layer based on the i-th main entity, the i-th relationship, and the i-th tail entity having the i-th relationship with the i-th main entity.

[0093] After obtaining the i-th relationship matching the knowledge summary and question information corresponding to the i-1 hop, the knowledge clue corresponding to the i-th layer can be determined based on the i-th main entity, the i-th relationship and the i-th tail entity that has the i-th relationship with the i-th main entity.

[0094] Related technologies regard relations and entities as separate, such as the triples in KG: (head entity, relation 1, entity 1), (head entity, relation 1, entity 2), (head entity, relation 1, entity 3)... This form will cause content redundancy, which is not conducive to LLM's understanding of knowledge after input (LLM has a limit on the number of characters input, and the more characters input, the longer the response time).

[0095] In the solution provided in the embodiment of the present application, as a feasible implementation method, in the knowledge clues corresponding to the i-th layer, for each i-th relationship, each i-th tail entity that has the i-th relationship with the i-th main entity is represented in the form of natural language.

[0096] Exemplarily, the following form can be used in this application to represent knowledge clues: <head entity> and <entity 1, entity 2, entity 3...> have <relationship 1>. It should be understood that in this embodiment, the main entity and multiple tail entities of each relationship connection in the knowledge clue are represented in the form of natural language. Compared with the related art that expresses different tail entities separately, the present application scheme describes the tail entities with the same relationship together, which can effectively solve the problem of content redundancy and improve the ability of LLM to understand knowledge.

[0097] It can be seen from S2011-S2013 that the method provided by this embodiment does not use the similarity determination method to match the surface features between entities or between entities and questions when determining the knowledge clues corresponding to the i-th layer, but uses a large language model to understand the semantic connection and deep relationship between entities based on the previous hop knowledge summary and question information. By semantically fusing all entities associated with the main entity, the large language model can capture multi-level and complex relationships between entities, thereby forming more complete and accurate knowledge clues. This enables the system to more effectively utilize the full picture of the knowledge graph during multi-hop reasoning, thereby obtaining more accurate knowledge clues.

[0098] S202: Determine the knowledge summary corresponding to the i-th hop based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i-1)-th hop, and the question information.

[0099] It can be understood that knowledge summary is a systematic review and integration of retrieved knowledge, aiming to organize scattered information, concepts, principles or skills into an organized, easy-to-understand and memorable structure. It usually requires identifying core information from a large amount of learning materials and eliminating redundant or unimportant parts. And the extracted knowledge is classified and arranged according to logic or themes. This helps to form a clear knowledge framework that learners can understand and remember more easily. Then, the connection between knowledge is established based on the logical relationship, cause and effect relationship, time sequence, etc. between concepts. This step helps to deepen the understanding of knowledge and promote the transfer and application of knowledge. Finally, it is necessary to summarize the core content of knowledge in concise and clear language, etc., reflecting the essence of knowledge while avoiding being too lengthy or complicated.

[0100] Through detailed analysis of knowledge clues and association with previous knowledge, the reasoning process becomes more coherent and organized. This helps reduce jumps and uncertainties in reasoning and improves the reliability of conclusions. Through logical reasoning and integration of new knowledge, the knowledge summary of the current jump is more complete, accurate and logical. This helps to provide a solid foundation for subsequent analysis and use. In the process of verifying and adjusting the knowledge summary, new information or conclusions related to the problem information may be discovered, thereby promoting the updating and expansion of knowledge. This helps to maintain the timeliness and accuracy of the knowledge graph or database. By integrating the knowledge clues of the i-th jump, the knowledge summary of the i-1th jump, and the problem information, the knowledge summary corresponding to the current jump is determined, which helps to avoid information omissions or duplications and ensure the comprehensiveness and accuracy of the reasoning process.

[0101] As a feasible implementation, we can use the power of large language models to determine knowledge summaries. Figure 5 , S202 may include the following steps:

[0102] S2021. Construct third prompt information based on the knowledge clues corresponding to the i-th jump, the knowledge summary and question information corresponding to the i-1-th jump.

[0103] Exemplarily, as a feasible implementation method, the third prompt information may be:

[0104] Given a question, a main entity, and summarized knowledge, please summarize the information that is helpful in answering the question from the following knowledge clues and return it. <question><main entity><summary knowledge><knowledge clue>.

[0105] Among them, <question> is the question information, <main entity> is the main entity in the knowledge clue corresponding to the i-th hop; <summarized knowledge> is the knowledge summary corresponding to the i-1-th hop, and <knowledge clue> is the knowledge clue corresponding to the i-th hop.

[0106] S2022: Input the third prompt information into the large language model, and determine the knowledge summary corresponding to the i-th jump through the large language model.

[0107] After receiving the third prompt information, the large language model will review and deeply understand the knowledge clues corresponding to the i-th jump, including key information such as entities, relationships, and attributes, to ensure that the model can accurately capture the basis of important content in the clues. Then, based on the guidance of the third prompt information, the model will filter out the parts of the knowledge clues that are directly related to answering the question information, which may involve reorganizing, summarizing, or refining the clue content. After filtering out the relevant information, the model will perform logical reasoning to clarify how this information is related to the question information. Subsequently, the model will integrate this information into a coherent and concise knowledge summary to ensure that its content is both accurate and directly relevant. After obtaining a preliminary knowledge summary, the model will verify it to ensure that its content is consistent with the third prompt information and the question information, and then obtain the knowledge summary corresponding to the i-th jump.

[0108] It can be seen that the method provided in this embodiment, when determining the knowledge summary corresponding to the i-th jump, can effectively extract key content that is helpful for answering questions from complex knowledge clues by leveraging the excellent understanding ability of the large language model, thereby generating an accurate, concise and directly relevant knowledge summary. In addition, since the large language model has excellent generation and expression capabilities, the output knowledge summary is both logical and easy to understand.

[0109] It is understandable that LLM can integrate information from multiple entities based on understanding language and context, avoiding the loss of potential and valuable relationships and information in advance. By generating summarized knowledge, LLM can fuse information based on the contextual relationships between entities, which can retain more semantic details and help to derive answers more accurately. In addition, LLM can handle unseen entities and relationships, and can even infer some implicit knowledge based on the context, and can extract and fuse more information from the knowledge graph, thereby obtaining more accurate and flexible knowledge summaries.

[0110] Moreover, the solution of the present application adopts a method of determining knowledge summarization, and is not limited to a certain reasoning path during the reasoning process. Instead, it flexibly integrates different relationships and entity information through the comprehensive reasoning ability of LLM, thus avoiding the limitations brought by path dependence.

[0111] S103: construct first prompt information based on the knowledge summary and question information corresponding to each hop, and input the first prompt information into the large language model to determine answer information corresponding to the question information.

[0112] After obtaining the knowledge summary corresponding to each jump, the question information is combined with the knowledge summary of each jump, for example, they can be integrated into a coherent sentence or paragraph through connecting words, punctuation marks, etc., to obtain the first prompt information. Then, the constructed first prompt information is input into the large language model, so that the large language model generates answer information directly related to the question based on the input first prompt information.

[0113] As a feasible implementation method, in S103, first prompt information is constructed based on the knowledge summary and problem information corresponding to each hop, which can be specifically implemented as S1031-S1032:

[0114] S1031. Describe the knowledge summary corresponding to each hop according to the hierarchical structure to form a target knowledge summary.

[0115] The knowledge in each knowledge summary corresponds to a level, such as the knowledge in the first knowledge summary corresponds to the first level; the knowledge in the second knowledge summary corresponds to the second level...

[0116] For example, when the number of hops is 3, the target knowledge summary may be as follows:

[0117] 1. <Fill in the first level of the outline with the knowledge summarized in the first jump 1>;

[0118] 1.1.<Fill in the knowledge summarized in 2 jumps in the second level of the outline 1-1>;

[0119] 1.1.1.<Fill in the 3-step summary of knowledge 1-1-1 in the third level of the outline>;

[0120] 1.1.2.<Fill in the 3-step summary of knowledge 1-1-2 in the third level of the outline>;

[0121] 1.2.<Fill in the knowledge summarized in 2 jumps 1-2 in the second level of the outline>;

[0122] 1.2.1.<Fill in the 3-step summary of knowledge 1-2-1 in the third level of the outline>;

[0123] …

[0124] 2. <The first level of the outline is to fill in the knowledge summarized in the first jump 2>…

[0125] S1032. Construct first prompt information based on the target knowledge summary and the question information.

[0126] The first prompt information is constructed by combining the question information and the target knowledge summary. For example, they can be integrated into a coherent sentence or paragraph through conjunctions, punctuation marks, etc. to obtain the first prompt information.

[0127] It can be seen from the above S1031-S1032 that describing the knowledge summary corresponding to each jump according to the hierarchical structure can clearly express the results of each jump summary in the reasoning process through the hierarchical structure, which can increase the transparency and explainability of the answer, so that the user can better understand the source and basis of the answer. It is understandable that when the user can understand how the model makes decisions, and these decisions are reasonable and acceptable, the user's trust in the model will increase. Therefore, the solution of the present application can also enhance the user's trust and acceptance of the model by providing a clear and understandable natural language description.

[0128] It can be seen from the above S101-S103 that the method provided by the embodiment of the present application, when performing n-hop reasoning operations on question information, performs reasoning based on the knowledge summary corresponding to the previous hop in each reasoning operation, so that the information in the knowledge clue of the current hop is obtained based on the previous reasoning result, thereby ensuring that valid information is not lost during the reasoning process. In addition, new knowledge related to the question is supplemented during the reasoning process, thereby solving the problem of knowledge dependency and making the answer of the large language model more accurate and clear.

[0129] In some embodiments, when performing a first-hop reasoning operation on question information, since there is no corresponding previous hop for the first hop, the first hop may determine knowledge clues and knowledge summaries based on information included in the question information.

[0130] For details, please refer to Figure 6 , the first hop reasoning operation in the n-hop reasoning operation includes the following steps:

[0131] S501. Based on the keywords in the question information, determine one or more first main entities matching the keywords in the entities of the knowledge graph.

[0132] It is understandable that after obtaining the question information input by the user, the keywords in the question information can be determined first, and then multiple main entities can be determined based on the keywords. That is, before S501, the method provided in the embodiment of the present application also includes: extracting keywords in the question information. The keywords in the question information are usually nouns or noun phrases, which can represent the core concepts or objects in the question. The embodiment of the present application does not make specific limitations on the method of determining the keywords in the question information.

[0133] Exemplarily, as a feasible implementation method, the capability of a large language model may be used to determine keywords.

[0134] Specifically, the following prompt can be constructed and input into LLM to obtain the question keywords: "Please extract keywords from the following question: <question>". Then, based on the output text of the large language model, the keywords in the question information are determined.

[0135] As another feasible implementation method, a keyword extraction model can be used to determine keywords, such as using a TF-IDF model, a TextRank algorithm model, a Latent Dirichlet Allocation (LDA) algorithm model, etc. as a keyword extraction model, inputting the question information into the keyword extraction model to obtain the keywords in the question information.

[0136] After obtaining the keywords in the question information, one or more first main entities matching the keywords can be determined in the entities of the knowledge graph based on the keywords.

[0137] As a feasible implementation method, characters can be used for entity linking, that is, if the characters of the keyword and the entity in the knowledge graph are exactly the same, the entity can be determined as the first main entity.

[0138] As another feasible implementation method, since the keywords in the question information are edited by the user, they are usually in natural language and may not match the entities in the knowledge graph, so the first main entity can be determined by similarity matching. That is, S501 can be specifically implemented as follows:

[0139] S5011. Encode the keyword to obtain a keyword vector.

[0140] Encoding the keywords means converting the text data into numerical form for subsequent processing. For example, as a feasible implementation method, the keywords can be encoded by using an encoding model, such as MiniLM, Mpnet, etc. in Sentence Transformer, to obtain keyword vectors.

[0141] As another feasible implementation, one-hot encoding or character embeddings can be used to convert keywords into vectors.

[0142] S5012. Encode the entities in the knowledge graph to obtain encoding vectors.

[0143] The process of encoding entities in the knowledge graph can refer to the above-mentioned process of encoding keywords, and this application will not go into details here.

[0144] S5013: Take the entity corresponding to the encoding vector whose similarity with the keyword vector is higher than a preset similarity threshold as the first main entity.

[0145] After obtaining the keyword vector and the keyword vector, the similarity of the encoding vector and the keyword vector can be calculated respectively. It should be understood that the similarity can be specifically: cosine similarity, Euclidean distance, Manhattan distance, etc. Since cosine similarity can measure the degree of similarity between two vectors in direction, the cosine similarity of the encoding vector and the keyword vector can be calculated respectively.

[0146] Since the similarity between the encoding vector and the keyword vector can represent the degree of correlation between the keyword and the entity, the higher the similarity, the more relevant the keyword and the entity. Therefore, the entity corresponding to the encoding vector whose similarity with the keyword vector is higher than the preset similarity threshold can be taken as the first main entity.

[0147] S502: Determine a first relationship matching the question information in the relationship corresponding to the first main entity, and obtain a first knowledge clue.

[0148] The first knowledge clue includes: a first main entity, a first relationship, and a first tail entity having a first relationship with the first main entity.

[0149] In the knowledge graph, the first main entity is usually connected to multiple tail entities through multiple relationships. It is necessary to match the question information (such as keywords, phrases or sentences in the question) with the relationship network of the first main entity. This matching process may involve semantic understanding, entity linking and other technologies to ensure that the information in the question can accurately correspond to a certain relationship or attribute in the relationship network. After matching the question information, we need to select a relationship from the relationship network that best matches the question information as the first relationship. This selection process may be based on similarity calculation, logical reasoning or manual judgment.

[0150] As an exemplary implementation method, the first relationship matching the question information can be determined by using the capability of a large language model. S502 can be specifically implemented as follows S5021-S5023:

[0151] S5021. Construct fourth prompt information based on the relationship and question information corresponding to the first main entity.

[0152] Since the first-hop reasoning operation does not have a previous jump, there is no knowledge summarized in the previous layer, so only the question information needs to be combined to construct the fourth prompt information.

[0153] Exemplarily, the fourth prompt information may be: given a question and a main entity, please select M relationships that are most helpful in answering the question from the following relationships; <question>; <main entity>; <relationship>.

[0154] It should be understood that M is preset and can be determined according to needs; <question> is question information; <main entity> is the first main entity; and <relationship> is the relationship corresponding to the first main entity.

[0155] S5022: Input the fourth prompt information into the large language model, and determine, through the large language model, a first relationship matching the question information in the relationship corresponding to the first main entity.

[0156] The fourth prompt information is input into the large language model, and the relationship corresponding to the first main entity is screened with the help of the understanding ability of the large language model to obtain M first relationships matching the question information.

[0157] S5023. Determine a first knowledge clue based on the first main entity, the first relationship, and the first tail entity having a first relationship with the first main entity.

[0158] After obtaining the first relationship, all entities connected to each first relationship can be determined in the knowledge graph (one relationship may connect multiple entities), and then the first-hop knowledge clues can be constructed using the relationship and entity information.

[0159] As a feasible implementation method, the relationship and entity information can be directly input into a pre-built template to obtain the first knowledge clue.

[0160] For example, the template may look like this:

[0161] <Entity 1> and <Entity 2, Entity 3, ...> have a relationship <Relationship>.

[0162] Among them, <Entity 1> is filled with the name of the main entity, <Relationship> is filled with the name of the first relationship, and <Entity 2, Entity 3...> is filled with the first tail entity that has the first relationship with the main entity.

[0163] S503: Determine a knowledge summary corresponding to the first hop based on the first knowledge clue and the question information.

[0164] Since there may be multiple first relations in the first knowledge clue, and each relation corresponds to one or more tail entities, when generating a knowledge summary, a knowledge summary may be generated for each first relation in the first knowledge clue.

[0165] Specifically, for each first relationship, the fifth prompt information can be constructed by combining the question information input by the user and the knowledge clues corresponding to the first relationship in the first knowledge clues, and input into the large language model to aggregate the knowledge to form the knowledge summary corresponding to the first relationship. Then, the knowledge summary corresponding to all the first relationships is combined to determine the knowledge summary corresponding to the first jump.

[0166] Exemplarily, the fifth prompt information may be as follows: Given a question and a main entity, please summarize and return the information that is helpful in answering the question from the following knowledge clues. <question>; <main entity>; <knowledge clue>.

[0167] Among them, <question> is filled with question information; <main entity> is the first main entity; and <knowledge clue> is the knowledge clue corresponding to the first relationship in the first knowledge clue.

[0168] As can be seen from S501-S503, the solution provided by the embodiment of the present application, when performing the first jump reasoning operation on the question information, first determines the first main entity matched by the keyword in the question information in the entity of the knowledge graph, and then determines the first knowledge clue in combination with the first main entity and the question information, and then determines the knowledge summary corresponding to the first jump based on the first knowledge clue and the question information. It can be understood that the first jump reasoning operation is actually building a knowledge chain from the question information to the relevant entities in the knowledge graph. This chain not only helps to understand the background and intention of the question information, but also provides a basis for subsequent multi-hop reasoning. By determining the first main entity through keyword matching, the entity most relevant to the question information can be quickly located in the knowledge graph, avoiding the problem of blind search in large-scale knowledge graphs. Then, by combining the first main entity and the question information to determine the first knowledge clue, the search scope can be further narrowed, and the knowledge clues directly related to the question can be quickly obtained, which improves the efficiency of information retrieval. Finally, by determining the first knowledge clue and the knowledge summary corresponding to the first jump, the process of knowledge reasoning can be made clearer and more organized. This helps users or systems better understand the reasoning results and make corresponding decisions or actions.

[0169] In some embodiments, see Figure 7 The question-answering method based on the knowledge graph and the large language model provided in the embodiment of the present application includes the following process when determining the answer information corresponding to the question information input by the user:

[0170] S601: Obtain question information input by the user, and determine keywords in the question information.

[0171] The question keywords can be obtained by constructing a prompt and inputting LLM, or it can be implemented using a keyword extraction model, which is not limited in the embodiments of the present application.

[0172] For example, the question information may be: which team won the NBA Finals championship with a score of 4-3 in the shortest time since the team was founded, and the determined keyword is: NBA Finals.

[0173] S602: Entity linking between keywords in the question information and entities on the knowledge graph.

[0174] There are two ways to perform entity linking:

[0175] Use characters for entity linking: If the question keyword and the entity keyword characters on the knowledge graph are exactly the same, then the entity is determined to be the first main entity and added to the main entity set.

[0176] Use vectors for alignment: Use encoding models in Sentence Transformer such as MiniLM, Mpnet, etc. to encode question keywords to obtain question keyword vectors, use the same model to encode entities on the knowledge graph to obtain entity vectors, calculate the cosine similarity between keyword vectors and different entity vectors, and if the similarity is higher than a set threshold, determine that the entity is the first main entity and add this entity to the main entity set.

[0177] It should be understood that in order to avoid duplication of main entities, the main entities may be deduplicated after the main entity set is obtained.

[0178] Combining the above example, the keyword "NBA Finals" and the entity keyword characters on the knowledge graph are exactly the same, that is, the main entity is "NBA Finals".

[0179] S603: Execute the first-hop reasoning operation.

[0180] Starting from the first main entity obtained in S602, find all the connected relationships, build a prompt based on the question information, input the large language model to filter all the relationships connected to the first main entity, and obtain the filtered M relationships. Then determine all the entities connected to each of the M relationships (one relationship may be connected to multiple entities), and use the relationship and entity information to build a new knowledge clue for the current layer. This step directly inputs the relationship and entity information into the template built in advance to form the first-hop knowledge clue.

[0181] For each of the M relationships, a prompt is constructed by combining the question information and the knowledge clues of the first jump. The large language model is input to aggregate the knowledge to form the knowledge summary of the current layer and obtain the knowledge summary corresponding to the first jump.

[0182] In combination with the above example, the relationships of the main entity "NBA Finals" in the knowledge graph include: game, player, team, stadium, coach. Therefore, the following prompt can be constructed:

[0183] Given a question and a main entity, please pick up to 3 of the following relationships that are most helpful in answering the question.

[0184] <Question> Which team won the NBA Finals championship with the shortest score of 4-3 since its establishment?

[0185] <Primary Entity> NBA Finals;

[0186] <Relationship> Games, players, teams, venues, coaches...

[0187] The answer obtained by the large language model is: competition.

[0188] Furthermore, the knowledge clues for the first hop can be: <NBA Finals> is related to <1957 NBA Finals, 1958 NBA Finals, 1959 NBA Finals...> in terms of <games>.

[0189] Based on the knowledge clues of the first hop, etc., construct the prompt again as follows:

[0190] Given the question and the main entity, summarize the information helpful for answering the question from the following knowledge clues and return it.

[0191] · <Question> Which team won the NBA Finals championship 4-3 in the shortest time since its establishment;

[0192] · <Main entity> NBA Finals;

[0193] · <Knowledge clues> NBA Finals is related to 1957 NBA Finals, 1958 NBA Finals, 1959 NBA Finals... in terms of <games>;

[0194] (Other knowledge clues).

[0195] Get the answer from the large language model, that is, the knowledge summary of the first hop is: The event of the NBA Finals can be traced back to 1947.

[0196] S604. Perform the i-th hop reasoning operation.

[0197] Among them, the initial value of i is 2.

[0198] Take the tail entity in the (i - 1)-th hop as the main entity, and continue to sample the relationships of entity connections. At this time, the knowledge summarized in the previous hop needs to be considered, that is, construct a prompt according to the knowledge summary of the (i - 1)-th hop, the main entity of the i-th hop, and the question information, input it into the large language model to screen multiple relationships connected to the main entity of the i-th hop, and obtain multiple i-th relationships. Furthermore, based on all entities connected by each i-th relationship (one relationship may connect multiple entities), construct the knowledge clues of the i-th hop with relationship and entity information.

[0199] Combined with the above example, taking the second-hop reasoning operation as an example, the constructed prompt can be:

[0200] Given the question, the main entity, and the summarized knowledge, select at most 3 relationships that are most helpful for answering the question from the following relationships.

[0201] · <Question> Which team won the NBA Finals championship 4-3 in the shortest time since its establishment;

[0202] · <Main entity> NBA Finals;

[0203] <Summarized knowledge> The NBA Finals can be traced back to 1947;

[0204] ·<Relationship>Champion, score…

[0205] The answer obtained from the large language model is that the i-th relationship is: champion, score.

[0206] Furthermore, based on the above i-th relationship, the following knowledge clues can be constructed:

[0207] <1957 NBA Finals> is related to <Celtics> <Championship>.

[0208] The <1957 NBA Finals> has something to do with the <4-3> <score>.

[0209] The prompt is constructed by combining the question information, the knowledge summary of the i-1th jump, and the knowledge clues of the i-th jump. The large language model is input to aggregate the knowledge to form the knowledge summary of the current layer and obtain the knowledge summary corresponding to the i-th jump.

[0210] Combining the above examples, the constructed prompt can be:

[0211] Given the question, main entity, and summarized knowledge, please summarize the information that is helpful in answering the question from the following knowledge clues and return it;

[0212] <Question> Which team won the NBA Finals championship with the shortest score of 4-3 since its establishment?

[0213] <Primary Entity> NBA Finals;

[0214] <Summarized knowledge> The NBA Finals can be traced back to 1947;

[0215] <Knowledge Clue> The 1957 NBA Finals and the Celtics were related to the championship;

[0216] The 1957 NBA Finals has something to do with the score of 4-3... (other knowledge clues).

[0217] The answer obtained by the large language model, that is, the knowledge summary of the second jump, is: the Lakers won the NBA Finals in 1952, 1953, 1954, ..., and other years, and the Boston Celtics won the NBA Finals with a score of 4-3 in ..., 1951, 1952, ..., 2016, and other years.

[0218] S605: Determine whether i is equal to N.

[0219] It should be understood that N is the number of hops of the multi-hop reasoning operation, which can be determined according to requirements in actual application.

[0220] When i is equal to N, it indicates that the multi-hop reasoning operation has been completed, and S606 is executed; when i is less than N, i=i+1 is set, and the above S604 is repeatedly executed.

[0221] S606. Integrate all the summarized knowledge and form a summary outline according to the structure.

[0222] That is, the knowledge summary corresponding to each hop is described according to the hierarchical structure to form the target knowledge summary.

[0223] For example, when the question information is: Which team won the NBA Finals championship with a score of 4-3 in the shortest time since the team was founded? The resulting summary outline can be:

[0224] 1. The NBA Finals dates back to 1947.

[0225] 1.1. The Lakers won the NBA Finals in 1952, 1953, 1954, etc., and the Boston Celtics won the NBA Finals in...

[0226] 1.1.1 The Lakers were founded in 1947 and the Boston Celtics were founded in 1946.

[0227] 1.2. The score of the NBA Finals in 1951, 1952, 2016, etc. was 4-3.

[0228] S607: Use the large language model to obtain answer information for the question information.

[0229] A prompt is constructed by combining the question information input by the user and the summary outline, the prompt is input into the large language model, and the output of the large language model is used as the answer information of the question information.

[0230] It can be seen that the method provided in the embodiment of the present application does not represent information such as questions, relationships, entities, etc. with vectors during the reasoning process of the knowledge graph, but directly processes and integrates the information by using natural language, and forms summary knowledge by inputting it into the LLM, thereby avoiding the use of similarity.

[0231] The solutions in the related technologies are all based on similarity during the reasoning process. For example, the related patent proposes a question-answering method based on a large language model and a knowledge graph. In this patent, an entity similarity screening is performed when the initial subgraph is obtained, and then an entity similarity screening is performed again after each fusion of the upper layer and the lower layer. Finally, a similarity screening is performed on all entities and questions, and the most similar entities and relationships are selected as the final answer.

[0232] However, this similarity screening method may result in the deletion of answer-related entities during the initial screening process, and the subsequent screening of entities or relationships makes it difficult to find the correct answer. For example, the question that the user needs to answer is: "Which NBA team won the NBA Finals championship with a score of 4-3 in the shortest time since the team was founded?" The main entity in this example is the NBA Finals. The first thing connected in the knowledge graph is the finals of different years (such as: 1961 NBA Finals, 1962 NBA Finals...), a total of 78 entities, these entities only have different years, and the similarity screening method cannot determine which is the answer the question wants (the answer-related entity is the 1952 NBA Finals), and the correct answer may even be deleted after screening.

[0233] Another example is a related patent that proposes a resume screening method and system for the recruitment industry based on LLM and knowledge base retrieval. The original text, the summarized text, and the vector obtained after the text is embedded are stored in the vector database, and an index is established on the vector field. Then, the text similarity comparison is used to determine whether the user requirements and the document are related, and a fast similarity search is performed.

[0234] However, in the answer search process of the knowledge graph, the method provided by this application does not use vectors to represent information such as questions, relationships, and entities, but directly processes and integrates information by using natural language, and forms summary knowledge by inputting it into the LLM, thus avoiding the use of similarity. Similarity filtering may cause important key information to be discarded in advance, because it only considers the surface similarity between words, without in-depth understanding of the complex semantics and context of the entity, and may miss important information in the early entity screening stage.

[0235] LLM can integrate information from multiple entities based on understanding language and context, avoiding the loss of potential and valuable relationships and information in advance. By generating summarized knowledge, LLM can fuse information based on the contextual relationships between entities, which can retain more semantic details and help to derive answers more accurately. When processing information, LLM does not just match based on surface features, but can understand the semantic connections and deep relationships between entities. By semantically fusing all entities associated with the main entity, LLM can capture multi-level and complex relationships between entities, thereby forming a more complete and accurate knowledge summary, which makes it more effective to use the full picture of the knowledge graph in multi-hop reasoning. LLM can handle unseen entities and relationships, and can even infer some implicit knowledge based on the context. This enables more information to be extracted and fused from the knowledge graph when faced with new and complex problems, providing more accurate and flexible answers.

[0236] Moreover, the solution of the present application directly uses natural language processing and information integration to convert complex calculation processes and results into text or language descriptions that can be directly understood by humans, so as to be more in line with human thinking and understanding habits, and improve the intuitiveness and interpretability of information processing. It is understandable that when users can understand how the model makes decisions and these decisions are reasonable and acceptable, the user's trust in the model will increase. Therefore, the solution of the present application helps to improve the user's trust and acceptance of the model by providing clear and understandable natural language descriptions. Moreover, since the processing process of the solution of the present application does not involve complex mathematical operations and vector space transformations, the results are easier to explain and understand.

[0237] The present application also provides a question-answering system based on a knowledge graph and a large language model. Figure 8 As shown, the question-answering system 70 based on the knowledge graph and the large language model may include: a data collection and processing module 71, a keyword extraction and mapping module 72, a relationship sampling module 73, a knowledge clue construction module 74, a knowledge aggregation module 75, a judgment module 76, a summary outline module 77, and a prompt word generation and LLM answer module 78.

[0238] Among them, the data collection and processing module 71 is used to obtain the question information and knowledge graph input by the user. The keyword extraction and mapping module 72 is used to determine the keywords in the question information, and to determine multiple main entities in the knowledge graph based on the keywords; the relationship sampling module 73 is used to determine the most matching multiple relationships in the relationships corresponding to the main entities, and the knowledge clue construction module 74 is used to determine the knowledge clues corresponding to the current jump according to the relationships determined by the relationship sampling module 73; the knowledge aggregation module 75 is used to determine the knowledge summary corresponding to the current jump according to the knowledge clues corresponding to the current jump; the judgment module 76 is used to judge whether the number of jumps reaches the preset value (N); the summary outline module 77 is used to generate a summary outline based on all knowledge summaries; the prompt word generation and LLM answer module 78 is used to generate prompt information based on the summary outline and the question information input by the user, and input LLM to obtain the answer information corresponding to the question information input by the user.

[0239] The present application also provides a question-answering device based on a knowledge graph and a large language model. Fig. 9 The question-answering device 80 based on the knowledge graph and the large language model includes: an acquisition module 81, a reasoning module 82 and an input module 83.

[0240] It should be understood that the question-answering device 80 based on the knowledge graph and the large language model can be the processing device 12 in the question-answering system 100 based on the knowledge graph and the large language model in the above embodiment, or it can be the electronic device in the above embodiment, and the embodiments of the present application do not limit this.

[0241] Among them, the acquisition module 81 is used to acquire problem information and the knowledge graph of the field to which the problem information belongs. The knowledge graph includes multiple entities and the relationships between the entities.

[0242] The reasoning module 82 is used to perform an n-hop reasoning operation on the problem information based on the knowledge graph to obtain the knowledge summary corresponding to each hop; n is an integer greater than 1.

[0243] The input module 83 is used to construct a first prompt message based on the knowledge summary corresponding to each hop and the problem information, and input the first prompt message into the large language model to determine the answer message corresponding to the problem information.

[0244] Among them, the specific implementation of the i-th hop reasoning operation in the n-hop reasoning operation by the reasoning module 82, where 1 < i ≤ n, is as follows:

[0245] Based on the knowledge summary corresponding to the (i - 1)-th hop and the problem information, determine the knowledge clue corresponding to the i-th hop through the knowledge graph; among them, the knowledge clue corresponding to the i-th hop includes the i-th main entity and the i-th tail entity having a relationship with the i-th main entity; the i-th main entity is the (i - 1)-th tail entity in the knowledge clue of the (i - 1)-th hop.

[0246] Determine the knowledge summary corresponding to the i-th hop based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information.

[0247] In some embodiments, the reasoning module 82 is specifically configured to construct a second prompt message based on the relationship corresponding to the i-th main entity, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information; input the second prompt message into the large language model, and determine, through the large language model, the i-th relationship in the relationship corresponding to the i-th main entity that matches the knowledge summary corresponding to the (i - 1)-th hop and the problem information; determine the knowledge clue corresponding to the i-th layer based on the i-th main entity, the i-th relationship, and the i-th tail entity having the i-th relationship with the i-th main entity.

[0248] In some embodiments, in the knowledge clue corresponding to the i-th layer, for each i-th relationship, represent each i-th tail entity having the i-th relationship with the i-th main entity in the form of natural language.

[0249] In some embodiments, the reasoning module 82 is specifically configured to construct a third prompt message based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information; input the third prompt message into the large language model, and determine the knowledge summary corresponding to the i-th hop through the large language model.

[0250] In some embodiments, the reasoning module 82 performs the first-hop reasoning operation in the n-hop reasoning operation as follows: based on the keywords in the question information, one or more first main entities matching the keywords are determined in the entities of the knowledge graph; in the relationships corresponding to the first main entities, a first relationship matching the question information is determined to obtain a first knowledge clue, where the first knowledge clue includes: a first main entity, a first relationship, and a first tail entity having a first relationship with the first main entity; based on the first knowledge clue and the question information, a knowledge summary corresponding to the first jump is determined.

[0251] In some embodiments, the reasoning module 82 is specifically used to encode keywords in the question information to obtain keyword vectors; encode entities in the knowledge graph to obtain encoding vectors; and take the entity corresponding to the encoding vector whose similarity with the keyword vector is higher than a preset similarity threshold as the first main entity.

[0252] In some embodiments, the reasoning module 82 is specifically used to construct fourth prompt information based on the relationship corresponding to the first main entity and the question information; input the fourth prompt information into the large language model, and determine the first relationship matching the question information in the relationship corresponding to the first main entity through the large language model; determine the first knowledge clue based on the first main entity, the first relationship and the first tail entity that has a first relationship with the first main entity.

[0253] In some embodiments, the input module 83 is specifically used to construct the first prompt information based on the knowledge summary and problem information corresponding to each jump, including: describing the knowledge summary corresponding to each jump according to the hierarchical structure to form a target knowledge summary; constructing the first prompt information based on the target knowledge summary and problem information.

[0254] The present application also provides an electronic device. Fig.10 The electronic device 110 includes: one or more memories 111, one or more processors 112, a communication bus 113 and a communication interface 114. The processor 112 is connected to the memory 111 via the bus 113; the one or more memories 111 are used to store computer program codes, and the computer program codes include computer instructions; when the one or more processors 112 execute the computer instructions, the electronic device 110 executes the answer generation method or text extraction method provided in the above embodiment.

[0255] Optionally, the memory 111 may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a tape, a floppy disk, and an optical data storage device, etc. The embodiments of the present application do not impose any restrictions on this.

[0256] The processor 112 may be a central processing unit (CPU), a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof, and the embodiments of the present application do not impose any limitation on this.

[0257] The communication bus 113 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus 113 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Just one thick line is used, but it does not mean that there is only one bus or one type of communication bus.

[0258] The communication interface 114 uses any transceiver or other device for communicating with other devices or communication networks, such as control systems, radio access networks (RAN), wireless local area networks (WLAN), etc.

[0259] The embodiments of the present application also provide a computer program product including one or more instructions, where the one or more instructions are stored in a memory of a computer device and executed by a processor to complete the various processes of the above embodiments.

[0260] An embodiment of the present application also provides a computer-readable storage medium, which includes computer execution instructions. When the computer execution instructions are executed on a computer, the computer executes the answer generation method or text extraction method provided in the above embodiments.

[0261] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0262] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0263] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0264] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0265] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a device (which may be a single-chip microcomputer, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0266] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A question-answering method based on knowledge graph and large language model, characterized in that: Including: Obtain problem information and a knowledge graph of the field to which the problem information belongs, where the knowledge graph includes multiple entities and relationships between the entities; Perform an n-hop reasoning operation on the problem information based on the knowledge graph to obtain a knowledge summary corresponding to each hop; n is an integer greater than 1; Construct a first prompt message based on the knowledge summary corresponding to each hop and the problem information, and input the first prompt message into a large language model to determine an answer message corresponding to the problem information; Among them, the i-th hop reasoning operation in the n-hop reasoning operation, where 1 < i ≤ n, includes the following steps: Based on the knowledge summary corresponding to the (i - 1)-th hop and the problem information, determine a knowledge clue corresponding to the i-th hop through the knowledge graph; where the knowledge clue corresponding to the i-th hop includes an i-th main entity and an i-th tail entity having a relationship with the i-th main entity; the i-th main entity is the (i - 1)-th tail entity in the knowledge clue of the (i - 1)-th hop; Determine a knowledge summary corresponding to the i-th hop based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information.

2. The method according to claim 1, characterized in that The determining of the knowledge clue corresponding to the i-th hop through the knowledge graph based on the knowledge summary corresponding to the (i - 1)-th hop and the problem information includes: Construct a second prompt message based on the relationship corresponding to the i-th main entity, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information; Input the second prompt message into the large language model, and through the large language model, determine an i-th relationship in the relationship corresponding to the i-th main entity that matches the knowledge summary corresponding to the (i - 1)-th hop and the problem information; Based on the i-th main entity, the i-th relationship, and the i-th tail entity having the i-th relationship with the i-th main entity, determine the knowledge clue corresponding to the i-th layer.

3. The method according to claim 1, characterized in that Among the knowledge clues corresponding to the i-th layer, for each i-th relationship, represent each i-th tail entity having the i-th relationship with the i-th main entity in the form of natural language.

4. The method according to claim 1, characterized in that: The determining of the knowledge summary corresponding to the current hop based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information includes: Construct a third prompt message based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information; Input the third prompt message into the large language model, and through the large language model, determine the knowledge summary corresponding to the i-th hop.

5. The method according to claim 1, characterized in that The first hop reasoning operation in the n-hop reasoning operation includes the following steps: Based on the keywords in the problem information, determine one or more first main entities in the entities of the knowledge graph that match the keywords; In the relationships corresponding to the first main entities, determine a first relationship that matches the problem information to obtain a first knowledge clue, where the first knowledge clue includes: the first main entity, the first relationship, and a first tail entity having the first relationship with the first main entity; Based on the first knowledge clue and the problem information, determine a knowledge summary corresponding to the first hop.

6. The method according to claim 5, characterized in that Determining one or more first main entities that match the keywords in the problem information in the entities of the knowledge graph includes: Encoding the keywords in the problem information to obtain keyword vectors; Encoding the entities in the knowledge graph to obtain encoded vectors; Taking the entities corresponding to the encoded vectors whose similarity to the keyword vectors is higher than a preset similarity threshold as the first main entities.

7. The method according to claim 5, characterized in that Determining a first relationship that matches the problem information in the relationships corresponding to the first main entities to obtain a first knowledge clue includes: Constructing a fourth hint message based on the relationships corresponding to the first main entities and the problem information; Inputting the fourth hint message into the large language model, and determining, through the large language model, a first relationship that matches the problem information in the relationships corresponding to the first main entities; Determining the first knowledge clue based on the first main entity, the first relationship, and a first tail entity that has the first relationship with the first main entity.

8. The method according to any one of claims 1 to 7, characterized in that: Constructing a first hint message based on the knowledge summary corresponding to each hop and the problem information includes: Describing the knowledge summary corresponding to each hop according to a hierarchical structure to form a target knowledge summary; Constructing the first hint message based on the target knowledge summary and the problem information.

9. A question-answering system based on knowledge graph and large language model, characterized in that: Including: A receiving device, a processing device, and a display screen; The receiving device is configured to receive problem information input by a user; The processing device is configured to perform an n-hop reasoning operation on the problem information based on a knowledge graph in the field to which the problem information belongs to obtain a knowledge summary corresponding to each hop; n is an integer greater than 1; constructing a first hint message based on the knowledge summary corresponding to each hop and the problem information, and inputting the first hint message into the large language model to determine an answer message corresponding to the problem information; The display screen is configured to display the answer message corresponding to the problem information; Wherein, the specific implementation of the i-th hop reasoning operation in the n-hop reasoning operation by the processing device, 1 < i ≤ n, is as follows: Determining a knowledge clue corresponding to the i-th hop through the knowledge graph based on the knowledge summary corresponding to the (i - 1)-th hop and the problem information; wherein, the knowledge clue corresponding to the i-th hop includes an i-th main entity and an i-th tail entity that has a relationship with the i-th main entity; the i-th main entity is the (i - 1)-th tail entity in the knowledge clue of the (i - 1)-th hop; Determining a knowledge summary corresponding to the i-th hop based on the knowledge clue corresponding to the i-th hop, the knowledge summary corresponding to the (i - 1)-th hop, and the problem information.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions run on a computer, the computer is caused to execute the question-answering method based on a knowledge graph and a large language model according to any one of claims 1 to 8.

Images