Knowledge graph query method and device based on multi-language model result fusion

Abstract

The application discloses a kind of knowledge graph query method and device based on the result fusion of multiple large language models, the method includes: multiple query results generation: obtain including query question;Multiple large language models are used to generate query sentence under multiple model parameters, and each query sentence is used to query in knowledge graph;Multiple query results synthesis: the execution of the multiple query sentences generated by the same large language model after parameter adjustment is integrated to obtain model level query reply;For multiple model level query replies, exclude model level query replies with low semantic similarity, and combine the model level query replies that have not been excluded to obtain result fusion query reply.The above technical solution is used, and multiple query sentences under multiple model parameters of multiple large language models are beneficial to obtaining information from knowledge graph comprehensively, and the query reply is integrated and combined, which can effectively reduce and avoid noise in query reply.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph question answering technology, and in particular to a knowledge graph query method and apparatus based on the fusion of results from multiple language models. Background Technology

[0002] Knowledge graph-based question-answering systems have consistently faced the challenge of low accuracy when dealing with massive amounts of knowledge. Knowledge graphs, as complex data structures, contain rich information about entities and their relationships, efficiently supporting semantic understanding and reasoning in question-answering systems and ensuring the correctness of responses. However, while traditional knowledge graph-based question-answering systems perform exceptionally well with small datasets, their performance often degrades significantly when faced with massive amounts of entities and relationships. This performance degradation is primarily due to the fact that as the size of the knowledge graph increases, it may contain more noisy data and redundant information, leading to lower query accuracy.

[0003] Most current knowledge graph question answering systems rely on query languages ​​such as SPARQL to retrieve relevant knowledge from knowledge bases. However, when processing large-scale knowledge graphs, SPARQL statements generated using large models are often affected by errors in entity, relation, and order selection. These errors typically lead to no results or results containing a large amount of noisy information, severely impacting the performance and user experience of the question answering system.

[0004] Specifically, regarding large-scale knowledge graph question answering systems that generate query statements based on large language models, large language models possess strong language performance. Query statements generated based on large language models can fully leverage the performance of large-scale knowledge graph question answering systems, enabling the knowledge graph to output more comprehensive and accurate query results. However, using large language models also introduces additional uncertainties during the query statement generation process. This is because query statements generated by large language models are prone to introducing noise or deviating from the user's original intent, resulting in greater instability. Even minor errors can lead to significant noise in the query results or query failures, preventing the large language model from performing stably and thus hindering the full utilization of the large-scale knowledge graph question answering system. Summary of the Invention

[0005] Purpose of the invention: This invention provides a knowledge graph query method and apparatus based on the fusion of results from multiple large language models, aiming to solve the problem that existing technologies often introduce noise into query statements generated by large language models, or that there are deviations from the user's original intent, resulting in a lot of noise in the query results of large-scale knowledge graph question answering systems.

[0006] Technical Solution: This invention provides a knowledge graph query method based on the fusion of results from multiple large language models, comprising: Multiple query result generation: Step 1, obtaining basic factual information, including query questions and knowledge graph triples corresponding to the query questions; Step 2, using the basic factual information, requesting the generation of query statements from multiple large language models, each large language model generating corresponding query statements under multiple model parameters, using each query statement to query the knowledge graph, and returning corresponding multi-parameter-level query responses; Step 3, if the parameter-level query responses returned by the knowledge graph are null, they are excluded; Multiple query result synthesis: Step 4, synthesizing multiple query results from the same large language model... The process involves integrating multiple parameter-level query responses corresponding to multiple query statements of a certain type. The integration method is to determine the content duplication rate among multiple parameter-level query responses based on their semantics, and select parameter-level query responses with content variation that meets the trend and has more content volume as model-level query responses based on the content duplication rate. Step 5: For multiple model-level query responses, calculate their semantic similarity. If there is a model-level query response whose semantic similarity is lower than a similarity threshold with more than a preset number of other model-level query responses, then the corresponding model-level query response is excluded. The remaining model-level query responses are merged to obtain the result fused query response, which is then output.

[0007] Specifically, the knowledge graph triples corresponding to the query question are obtained through fuzzy retrieval based on the content of the query question.

[0008] Specifically, the query response includes the entity and its corresponding relationship, and / or the entity and its corresponding attribute information.

[0009] Specifically, step 2 further includes: after generating multiple structured query statements using the large language model, removing structured query statements with duplicate content.

[0010] Specifically, the model parameters include temperature parameters.

[0011] Specifically, step 3 further includes: if the parameter-level query response returned by the knowledge graph is null, then the corresponding query statement is excluded, the model parameters are readjusted, a query statement is generated again under the new model parameters, and the query is sent to the knowledge graph until the returned parameter-level query response does not contain null values.

[0012] Specifically, step 4 includes: arranging multiple parameter level query responses in ascending order of their corresponding temperature parameters; calculating the content repetition rate between adjacent parameter level query responses one by one from low to high; if the change in the content repetition rate calculated by consecutive standard quantities is less than the change threshold, then the corresponding parameter level query response is deemed to meet the change trend; and selecting the parameter level query response corresponding to the highest temperature parameter as the model level query response.

[0013] Specifically, in step 5, if there is a model-level query response whose semantic similarity is lower than the similarity threshold with at least two other model-level query responses, then the corresponding model-level query response will be excluded.

[0014] Specifically, in step 5, the similarity threshold is the content duplication rate between the model-level query response and the adjacent parameter-level query response in step 4.

[0015] This invention also provides a knowledge graph query device based on the fusion of results from multiple large language models, comprising: a query result generation unit and a query result synthesis unit, wherein: the query result generation unit is used to acquire basic factual information, the basic factual information including a query question and a knowledge graph triple corresponding to the query question; using the basic factual information, requesting the generation of query statements from multiple large language models, each large language model generating corresponding query statements under multiple model parameters, using each query statement to query the knowledge graph, and returning corresponding multi-parameter-level query responses; if the parameter-level query responses returned by the knowledge graph are null, they are excluded; the query result synthesis unit uses... For integrating multiple parameter-level query responses corresponding to multiple query statements of the same large language model, the integration method is as follows: based on the semantics of the parameter-level query responses, the content duplication rate among multiple parameter-level query responses is determined. Based on the content duplication rate, parameter-level query responses with content variation amplitude that meets the trend and larger content volume are selected as model-level query responses. For multiple model-level query responses, the semantic similarity between them is calculated. If there is a semantic similarity lower than the similarity threshold with more than a preset number of other model-level query responses, the corresponding model-level query response is excluded. The remaining model-level query responses are merged to obtain the result fused query response, which is then output.

[0016] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: under multiple model parameters of multiple language models, multiple query statements are used to query the knowledge graph, which is conducive to obtaining comprehensive information from the knowledge graph. Based on the content repetition rate, the changing trend of multiple query responses is determined and selected, which can effectively reduce and avoid noise in the query responses. Based on semantic similarity, query responses are excluded, which can also effectively reduce and avoid noise in the query responses. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the knowledge graph query method based on the fusion of results from multiple language models provided by this invention. Detailed Implementation

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0019] See Figure 1 This is a flowchart illustrating the knowledge graph query method based on the fusion of results from multiple language models provided by this invention.

[0020] In this embodiment of the invention, the generation of multiple query results includes steps 1 to 3.

[0021] In this embodiment of the invention, step 1 involves obtaining basic factual information, which includes a query question and a knowledge graph triple corresponding to the query question.

[0022] In this embodiment of the invention, the knowledge graph triple corresponding to the query question is obtained through fuzzy retrieval based on the content of the query question.

[0023] In practical implementation, the query problem in basic factual information needs to be queried from the knowledge graph, which can be expressed in natural language. Knowledge graph triples are obtained through fuzzy retrieval. Fuzzy retrieval allows for a broader collection of knowledge graph triple content, enabling the large language model to have better corpus to generate more suitable and accurate query statements.

[0024] In practical implementation, the content of knowledge graph triples and knowledge graph query responses includes entities and their corresponding relations, and / or entity and corresponding attribute information. Entities are the entity content, relations are the relationships between entities, and attribute information typically includes attributes and attribute values, generally reflecting the nature and extent of the entity content.

[0025] In this embodiment of the invention, step 2 involves using the basic factual information to request the generation of query statements from multiple large language models. Each large language model generates corresponding query statements under multiple model parameters. Each query statement is then used to query the knowledge graph, and the corresponding query responses at multiple parameter levels are returned.

[0026] In practical implementation, Large Language Models (LLMs) typically refer to pre-trained language models with billions or even hundreds of billions of parameters. Currently, LLMs have received widespread attention and in-depth research. Trained on massive amounts of unlabeled text through self-supervised learning (such as "next word prediction"), they possess powerful language understanding, reasoning, and generation capabilities, and can complete complex tasks through parameter tuning. LLMs convert natural language into structured query statements, offering advantages in cross-domain, spoken, multilingual, and low-sample scenarios. They can generate highly accurate and comprehensive structured query statements, facilitating the full retrieval of information from knowledge graphs.

[0027] In practice, generating corresponding query statements under multiple model parameters refers to adjusting the model parameters of the same large language model. Different query statements can be generated under different model parameters. Utilizing multiple query statements to query the knowledge graph can further improve the comprehensiveness of the information obtained from the knowledge graph.

[0028] In this embodiment of the invention, after generating multiple structured query statements using a large language model, the structured query statements with (complete) duplicate content are removed.

[0029] In practice, even though the model parameters of the large language model have been adjusted, the content of the generated structured query language may still be repetitive in some cases. In such cases, the corresponding structured query statements should be removed.

[0030] In practice, structured query statements can be, for example, SPARQL query statements.

[0031] In this embodiment of the invention, the model parameters include the temperature parameters of the large language model.

[0032] In practical implementation, the temperature parameter is a scalar hyperparameter that controls the randomness of the large language model's output. A lower temperature parameter results in lower randomness and greater stability in the large language model's output; conversely, a higher temperature parameter leads to higher randomness and greater dispersion in the output. Even under the same large language model, the same temperature parameter, and the same input, two outputs from the large language model may differ significantly. Therefore, multiple structured query statements can be generated under the same temperature parameter within the same large language model to improve comprehensiveness. The temperature parameter can be adjusted from 0.1 to 2, in increments of 0.1, for example, 0.1, 0.2, 0.3…2.

[0033] In this embodiment of the invention, in step 3, if the parameter level query response returned by the knowledge graph is empty, then (the empty parameter level query response) is excluded.

[0034] In this embodiment of the invention, the method further includes: if the parameter-level query response returned by the knowledge graph is null, then the corresponding query statement is excluded, the model parameters are readjusted, a query statement is generated again under the new model parameters, and a query is made to the knowledge graph until the returned parameter-level query response does not contain null values.

[0035] In practical implementation, for example, if a query generated by a large language model at a temperature parameter of 1.1 returns null values, then the corresponding temperature parameter (i.e., 1.1) can be fine-tuned to obtain new model parameters (not used in the initial generation process), such as decreasing or increasing by 0.05, generating new query statements with temperature parameters of 1.05 or 1.15, and querying the knowledge graph. If the returned query response still contains null values, the model parameters are adjusted again until the returned parameter-level query response does not contain null values. This improves the comprehensiveness and completeness of the information obtained from the knowledge graph, fully leveraging the role of large-scale knowledge graphs.

[0036] In this embodiment of the invention, the synthesis of multiple query results includes steps 4 and 5.

[0037] In this embodiment of the invention, step 4 involves integrating multiple parameter-level query responses corresponding to multiple query statements of the same large language model. The integration method is to determine the content duplication rate among multiple parameter-level query responses based on the semantics of the parameter-level query responses, and select the parameter-level query responses with content change amplitude that meets the change trend and with more content volume from the parameter-level query responses based on the content duplication rate as model-level query responses.

[0038] In this embodiment of the invention, multiple parameter level query responses are arranged in ascending order of their corresponding temperature parameters. The content repetition rate between two adjacent parameter level query responses is calculated one by one from low to high. If the change in the content repetition rate calculated by consecutive standard quantities is less than the change threshold, the corresponding parameter level query response is considered to meet the change trend. The parameter level query corresponding to the highest temperature parameter is selected as the model level query response.

[0039] In practical implementation, for example, the same large language model generates six query statements under temperature parameters of 0.1, 0.2, 0.3, 0.4, 0.5, and 6, corresponding to query responses 1 to 6 respectively. Since the temperature parameter determines the divergence of the generated query statements, the content volume of query responses 1 to 6 typically increases, indicating that the noise may also increase. Therefore, by calculating the semantics of each query response, and then calculating the content repetition rate between each pair of responses based on the semantics, including calculating the content repetition rate between query responses 6 and 5, between query responses 5 and 4, between query responses 4 and 3, and so on, if the change in the content repetition rate of consecutive standard quantities is less than the change threshold, it indicates that the trend is that the noise is decreasing and the content of the query responses is becoming more stable. For example, setting the consecutive standard quantity to 3 and the change threshold to 5%, the content repetition rate between query responses 6 and 5 is 70%, between query responses 5 and 4 is 77%, between query responses 4 and 3 is 79%, and between query responses 3 and 2 is 80%, indicating that there are three consecutive... The calculated change in content repetition rate is less than the change threshold, indicating that the corresponding query responses 5 to 2 meet the trend of content stabilization. Query response 5 (the query response corresponding to the highest temperature parameter) is selected as the model-level query response because, among the queries 5 to 2, the temperature parameter corresponding to query response 5 is 0.5, indicating that the corresponding query statement is more divergent and comprehensive. The content of query response 5 is also more comprehensive and complete. Although the noise content of query response 5 may be slightly higher than that of query responses 5 to 2, its content trend is already relatively stable, indicating that, from the overall content perspective, the noise data is already less, and the content of query response 5 is usually more comprehensive and complete. This is a balance made between the comprehensiveness of content and the noise content.

[0040] In this embodiment of the invention, step 5 involves calculating the semantic similarity between multiple model-level query responses. If a model-level query response has a semantic similarity lower than a preset number of other model-level query responses, the corresponding model-level query response is excluded. The model-level query responses that are not excluded are merged to obtain a result fusion query response, which is then output.

[0041] In this embodiment of the invention, if there is a model-level query response whose semantic similarity is lower than the similarity threshold with at least two other model-level query responses, the corresponding model-level query response will be excluded.

[0042] In practical implementation, for example, if there are model-level query responses 1 to 5, and the semantic similarity between query response 1 and the other 4 query responses is lower than the similarity threshold, and query responses 2 to 5 only have a semantic similarity lower than the similarity threshold with query response 1 (taking query response 2 as an example, query response 2 only has a semantic similarity lower than the similarity threshold with query response 1, but has a semantic similarity higher than the similarity threshold with query responses 3, 4, and 5 respectively), then only model-level query response 1 should be excluded. This indicates that the query statement generated by the corresponding large language model has certain deviations or problems, resulting in a significant difference between query response 1 output by the knowledge graph and other query responses 2 to 5.

[0043] In this embodiment of the invention, in step 5, the similarity threshold is the content repetition rate between the model-level query response in step 4 and the adjacent parameter-level query response (the one with the lower corresponding temperature parameter).

[0044] In practice, the similarity threshold is set to the content duplication rate in step 4. This is because the content duplication rate calculated in step 4 represents the extent to which there is the same content between multiple parameter-level query responses. Furthermore, the content duplication rate between the model-level query response (selected from multiple parameter-level query responses) and adjacent query responses indicates a reasonable value for the content duplication rate when the content of the query responses tends to be stable. Therefore, it can be used to measure the degree to which the semantic similarity between multiple model-level query responses should reach.

[0045] In specific implementation, according to the method provided by the present invention, each large language model has a corresponding model-level query response. This means that according to the above-mentioned similarity threshold being equal to the content duplication rate, each model-level query response corresponds to a content duplication rate. One of these multiple content duplication rates can be selected arbitrarily. If the highest value is selected, it means that the query response needs to be more stable in content and has less noise. If the lowest value is selected, it means that the query response needs to be more comprehensive in content.

[0046] In practice, the merged model-level query responses that have not been excluded are used to obtain the result fusion query response. This can be achieved by directly superimposing multiple model-level query responses, while removing any semantically redundant content.

[0047] In practical implementation, the query method provided by this invention first generates multiple structured query statements from multiple large language models under multiple model parameters. Multiple query statements are then used to query the knowledge graph to obtain multiple query responses. This allows for the comprehensive acquisition of information from the knowledge graph. Then, the query responses are integrated and merged based on factors such as content duplication rate to eliminate noise information and improve the reliability of the query response content.

[0048] This invention also provides a knowledge graph query device based on the fusion of results from multiple large language models, comprising: a query result generation unit and a query result synthesis unit, wherein: the query result generation unit is used to acquire basic factual information, the basic factual information including a query question and a knowledge graph triple corresponding to the query question; using the basic factual information, requesting the generation of query statements from multiple large language models, each large language model generating corresponding query statements under multiple model parameters, using each query statement to query the knowledge graph, and returning corresponding multi-parameter-level query responses; if the parameter-level query responses returned by the knowledge graph are null, they are excluded; the query result synthesis unit uses... For integrating multiple parameter-level query responses corresponding to multiple query statements of the same large language model, the integration method is as follows: based on the semantics of the parameter-level query responses, the content duplication rate among multiple parameter-level query responses is determined. Based on the content duplication rate, parameter-level query responses with content variation amplitude that meets the trend and larger content volume are selected as model-level query responses. For multiple model-level query responses, the semantic similarity between them is calculated. If there is a semantic similarity lower than the similarity threshold with more than a preset number of other model-level query responses, the corresponding model-level query response is excluded. The remaining model-level query responses are merged to obtain the result fused query response, which is then output.

[0049] In specific implementation, the methods, steps or functions executed by the execution unit of the knowledge graph query device based on the fusion of results from multiple language models provided by the present invention can refer to the knowledge graph query method based on the fusion of results from multiple language models provided by the present invention.

Claims

1. A knowledge graph query method based on multi-large language model result fusion, characterized in that, include: Multiple query results generation: Step 1: Obtain basic factual information, which includes the query question and the knowledge graph triple corresponding to the query question; Step 2: Using the aforementioned basic factual information, request the generation of query statements from multiple large language models. Each large language model generates corresponding query statements under multiple model parameters. Each query statement is used to query the knowledge graph, and the corresponding query responses at multiple parameter levels are returned. The model parameters include temperature parameters. Step 3: If the parameter-level query response returned by the knowledge graph is empty, then exclude it. Combining multiple query results: Step 4: For the same large language model, arrange multiple parameter level query responses in order of increasing temperature parameters. Calculate the content repetition rate between adjacent parameter level query responses one by one from low to high. If the change in the content repetition rate calculated by consecutive standard numbers is less than the change threshold, the corresponding parameter level query response is considered to meet the change trend. Select the parameter level query response corresponding to the highest temperature parameter as the model level query response. Step 5: For multiple model-level query responses, calculate the semantic similarity between them. If there is a model-level query response whose semantic similarity is lower than the similarity threshold with more than a preset number of other model-level query responses, exclude the corresponding model-level query response. Merge the model-level query responses that are not excluded to obtain the result fusion query response and output it. 2.The method of claim 1, wherein, The knowledge graph triple corresponding to the query question is obtained through fuzzy retrieval based on the content of the query question. 3.The method of claim 1, wherein, The query response includes the entity and its corresponding relationship, and / or the entity and its corresponding attribute information. 4.The method of claim 1, wherein, Step 2 further includes: after generating multiple structured query statements using the large language model, removing structured query statements with duplicate content. 5.The method of claim 1, wherein, Step 3 further includes: if the parameter-level query response returned by the knowledge graph is null, then the corresponding query statement is excluded, the model parameters are readjusted, a query statement is generated again under the new model parameters, and the query is sent to the knowledge graph until the returned parameter-level query response does not contain null values. 6.The method of claim 5, wherein, In step 5, if there is a model-level query response whose semantic similarity is lower than the similarity threshold with at least two other model-level query responses, then the corresponding model-level query response will be excluded. 7.The method of claim 6, wherein, In step 5, the similarity threshold is the content duplication rate between the model-level query response and the adjacent parameter-level query response in step 4.

8. A knowledge graph query device based on multi-large language model result fusion, characterized in that, include: The query result generation unit and the query result synthesis unit, wherein: The query result generation unit is used to obtain basic fact information, which includes the query question and the knowledge graph triple corresponding to the query question. Using the aforementioned basic factual information, multiple large language models are requested to generate query statements. Each large language model generates corresponding query statements under multiple model parameters. Each query statement is then used to query the knowledge graph, and the corresponding query responses at multiple parameter levels are returned. If the parameter-level query response returned by the knowledge graph is empty, it is excluded. The model parameters include temperature parameters. The query result synthesis unit is used to arrange multiple parameter-level query responses for the same large language model in ascending order of their corresponding temperature parameters. It calculates the content repetition rate between adjacent parameter-level query responses one by one from low to high. If the change in the content repetition rate calculated for a consecutive standard number of responses is less than a change threshold, the corresponding parameter-level query response is considered to meet a change trend. The parameter-level query response corresponding to the highest temperature parameter is selected as the model-level query response. For multiple model-level query responses, the semantic similarity between them is calculated. If there is a model-level query response whose semantic similarity is lower than a similarity threshold with more than a preset number of other model-level query responses, the corresponding model-level query response is excluded. The remaining model-level query responses are merged to obtain a result fusion query response, which is then output.

Images