An intelligent reply method and device integrating large language model and knowledge graph

Abstract

The present invention discloses an intelligent reply method and device integrating a large language model and a knowledge graph, including and judging whether a target user has a second question; if so, judging whether the time interval between the first question and the second question is within a first preset time length; if so, extracting a first keyword set for the first question and a second keyword set for the second question; judging whether the keyword repetition rate between the first keyword set and the second keyword set is greater than a repetition rate threshold; if so, filtering out a target knowledge graph from a preset knowledge graph database; filling a preset prompt word structure framework with a number of first search results to obtain a first text; obtaining a second text; inputting the second text into a preset large language model to obtain a target reply corresponding to the first question of the target user. The present invention belongs to the field of intelligent customer service. The present invention can improve the hit rate of answers from a large language model.

Description

Technical Field

[0001] The present invention relates to the field of intelligent customer service, and in particular to an intelligent reply method and device integrating a large language model and a knowledge graph. Background Art

[0002] An intelligent answering system that integrates big language models and knowledge graphs is a method that combines structured knowledge representation (such as knowledge graphs) with natural language processing techniques (such as big language models) to provide more accurate and relevant answers to users. Knowledge graphs can provide precise factual information and relationships between entities, while big language models are good at understanding and generating natural language. The two work together to improve the performance of question-answering systems.

[0003] Although the combination of knowledge graph and large language model can improve the accuracy of answers to a certain extent, it is difficult to answer more complex questions from multiple aspects and dimensions. Therefore, how to further improve the accuracy of answers from large language models is an urgent problem to be solved. Summary of the invention

[0004] The present invention solves the technical problem of low hit rate of large language models in the prior art by providing an intelligent reply method and device that integrates a large language model and a knowledge graph, and achieves the technical effect of improving the answer accuracy of the large language model.

[0005] In a first aspect, the present invention provides an intelligent reply method integrating a large language model and a knowledge graph, the method comprising:

[0006] Obtain the first question of the target user, and determine whether the target user has a second question, the first question is the current question, and the second question is the previous question before the first question;

[0007] If not, obtaining an answer to the first question based on a preset large language model; if so, determining whether the time interval between the first question and the second question is within a first preset time length;

[0008] If not, then obtaining the answer to the first question based on the preset large language model; if yes, then extracting the first keyword set of the first question and the second keyword set of the second question respectively based on the preset large language model;

[0009] Determining whether a keyword repetition rate between the first keyword set and the second keyword set is greater than a repetition rate threshold;

[0010] If not, then obtain the answer to the first question based on the preset large language model; if greater, then construct several search query statements based on the first keyword set and the second keyword set, and filter out the target knowledge graph from the preset knowledge graph database;

[0011] After searching in the target knowledge graph based on the plurality of search query statements, a plurality of first search results are obtained, and a preset prompt word structure framework is filled with the plurality of first search results to obtain a first text, wherein the preset prompt word structure framework includes: background introduction, auxiliary information, answer format and core question;

[0012] Fill the first question into the first text to obtain the second text;

[0013] The second text is input into the preset large language model to obtain a target answer corresponding to the first question of the target user.

[0014] Furthermore, the target knowledge graph is screened out from the preset knowledge graph database, including:

[0015] Determine the graph score of each knowledge graph from the preset knowledge graph database;

[0016] According to several graph scores, the knowledge graphs are sorted in descending order, and the knowledge graphs before the preset ranking threshold are taken as target knowledge graphs.

[0017] Further, determining the graph score of each knowledge graph in the preset knowledge graph database includes:

[0018]

[0019] in, is the graph score of the nth knowledge graph in the preset knowledge graph database, and Preset weights, k is the total number of keywords in the keyword set, u is the total number of search query statements, is the number of occurrences of the i-th keyword in the keyword set in the j-th search query statement, is the number of the ith keyword in the nth knowledge graph, The number of search query statements containing core keywords.

[0020] Further, inputting the second text into a preset large language model to obtain a target answer corresponding to the first question of the target user, further comprising:

[0021] According to the target knowledge graph, fine-tune the preset large language model to obtain a fine-tuned preset large language model;

[0022] The second text is input into the fine-tuned preset large language model to obtain a target answer corresponding to the first question of the target user.

[0023] Further, determining the keyword repetition rate between the first keyword set and the second keyword set includes:

[0024]

[0025] in, is the keyword repetition rate between the first keyword set and the second keyword set, is the number of keywords that are the same in the first keyword set and the second keyword set, is the number of keywords in the second keyword set, is the number of keywords in the first keyword set.

[0026] Furthermore, the preset large language model is fine-tuned according to the target knowledge graph to obtain a fine-tuned preset large language model, including:

[0027] Extract several training datasets from the target knowledge graph. The types of training datasets include: relationship entity datasets, named entity datasets, and answer datasets.

[0028] Training a preset large language model based on a number of training data sets to obtain a number of simulated outputs of the preset large language model, each simulated output corresponding to a standard output;

[0029] Determine output verification indicators based on several simulation outputs and several standard outputs;

[0030] When the output verification index meets the preset training requirements, the fine-tuned preset large language model is obtained.

[0031] Furthermore, output verification indicators are determined based on a number of simulation outputs and a number of standard outputs, including:

[0032] Determine an output repetition rate corresponding to the analog output according to the analog output and the standard output corresponding to the analog output;

[0033] Determining length parameters according to a number of simulation outputs and a number of standard outputs;

[0034] Determine the length similarity corresponding to the simulated output according to the simulated output and the standard output corresponding to the simulated output;

[0035] An output verification index is determined according to output repetition rates and length parameters corresponding to a number of simulation outputs and length similarities corresponding to a number of simulation outputs.

[0036] Furthermore, output verification indicators are determined, including:

[0037]

[0038]

[0039] in, To output verification indicators, is the length parameter, is the output repetition rate corresponding to the yth analog output, and All are preset weights. is the length similarity corresponding to the y-th simulation output, is the maximum length of the sentences in the y-th simulation output and the y-th standard output, is the longest identical sentence length between the yth simulation output and the yth standard output.

[0040] Further, according to the plurality of simulation outputs and the plurality of standard outputs, length parameters are determined, including:

[0041] ,or,

[0042]

[0043] in, is the length parameter, is the total number of words of several analog outputs, The total number of characters in several standard outputs.

[0044] In a second aspect, the present invention provides an intelligent reply device integrating a large language model and a knowledge graph, the device comprising:

[0045] A first judgment module is used to obtain a first question asked by a target user and judge whether the target user has a second question, wherein the first question is a current question and the second question is a previous question before the first question;

[0046] A second judgment module is used to obtain a response to the first question based on a preset large language model if the response does not exist; if the response does exist, determine whether the time interval between the first question and the second question is within a first preset time length;

[0047] an extraction module, configured to obtain a reply to the first question based on a preset large language model if the user is not present; and to extract a first keyword set for the first question and a second keyword set for the second question based on the preset large language model if the user is present;

[0048] A third judgment module is used to judge whether the keyword repetition rate between the first keyword set and the second keyword set is greater than a repetition rate threshold;

[0049] A screening module is used for obtaining an answer to the first question based on a preset large language model if the number is not greater than ; if the number is greater than , constructing a plurality of search query statements based on the first keyword set and the second keyword set, and screening out a target knowledge graph from a preset knowledge graph database;

[0050] A filling module is used to obtain a plurality of first search results after searching in the target knowledge graph based on a plurality of search query statements, and fill a preset prompt word structure framework with the plurality of first search results to obtain a first text, wherein the preset prompt word structure framework includes: background introduction, auxiliary information, answer format and core question;

[0051] A second text module, used for filling the first question into the first text to obtain a second text;

[0052] The output module is used to input the second text into the preset large language model to obtain the target answer corresponding to the first question of the target user

[0053] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0054] By judging whether there is a second question and whether the time interval is within the first preset time, the present invention can prepare relevant information or guidance in advance, so that the interaction process is smoother and more natural; by identifying the continuity of the questions, the user's intention can be better understood; by judging the continuous progressiveness, the strong correlation between the first question and the second question in time and space is emphasized; and the present invention can also emphasize the time correlation in a more humane way by reasonably setting the first preset time; by "judging whether there is a second question and judging the time interval", the accuracy of subsequent hits on the first question can be greatly improved.

[0055] The present invention sets a repetition rate threshold to determine the content relevance between the first question and the second question, thereby providing a basis for constructing more accurate and more relevant query statements, and can achieve more accurate screening of the preset knowledge graph database, thereby improving the hit rate of the answer.

[0056] The present invention does not limit the knowledge graph to only one by setting a preset ranking threshold. In other words, multiple replies can be obtained, and the hit rate can be improved by enriching the answers.

[0057] The present invention can quantify the relevance of each knowledge graph by combining the frequency of occurrence of keywords in each knowledge graph with the keyword distribution in the search query statement; by normalizing and logarithmizing the total number of keywords in all search query statements, it helps to smooth the score, reduce the impact of extreme values, and make the score more stable and reliable, so as to more accurately find the knowledge graph most relevant to the user query, thereby improving query efficiency and user experience.

[0058] By setting a preset prompt word structure framework, the present invention can help the large language model better understand the first question and generate a structured and rich reply; it not only directly answers the user's core question, but also provides more background and auxiliary information, thereby improving the user experience.

[0059] The present invention designs verification indicators through indicators such as word count, sentence length, sentence similarity, etc., which can better verify the training effect of the preset large language model and improve the hit rate of the preset large language model answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 A flowchart of an intelligent reply method that integrates a large language model and a knowledge graph provided by the present invention. DETAILED DESCRIPTION

[0062] The embodiment of the present invention solves the technical problem of low reply hit rate in the prior art by providing an intelligent reply method that integrates a large language model and a knowledge graph.

[0063] The technical solution of the present invention is to solve the above technical problems, and the overall idea is as follows:

[0064] A smart answering method integrating a large language model and a knowledge graph, the method comprising: obtaining a first question of a target user, and determining whether the target user has a second question, the first question being the current question, and the second question being the previous question before the first question; if not, obtaining an answer to the first question based on a preset large language model; if so, determining whether the time interval between the first question and the second question is within a first preset time length; if not, obtaining an answer to the first question based on a preset large language model; if so, extracting a first keyword set for the first question and a second keyword set for the second question respectively based on the preset large language model; determining whether a keyword repetition rate between the first keyword set and the second keyword set is greater than repetition rate threshold; if it is not greater than, obtaining the answer to the first question based on the preset large language model; if it is greater than, constructing several search query statements according to the first keyword set and the second keyword set, and screening out the target knowledge graph from the preset knowledge graph database; after searching in the target knowledge graph based on the several search query statements, obtaining several first search results, and filling the preset prompt word structure framework with the several first search results to obtain the first text, the preset prompt word structure framework includes: background introduction, auxiliary information, answer format and core question; filling the first question into the first text to obtain the second text; inputting the second text into the preset large language model to obtain the target answer corresponding to the first question of the target user.

[0065] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0066] First of all, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0067] Large Language Model (LLM) refers to a complex artificial intelligence system trained through machine learning that can understand and generate human language. Large language models (such as ChatGPT) are usually trained on large text datasets based on deep learning architectures (such as Transformer, etc.) to grasp various patterns and structures of language.

[0068] Knowledge graph is a structured way of representing knowledge. It organizes and displays entities (such as people, places, things) and the relationships between entities in the form of graphics. The core purpose of knowledge graph is to capture information in the real world and present it in a way that machines can understand and process. The structured approach makes knowledge graph very suitable for natural language processing.

[0069] The present invention provides Figure 1 An intelligent reply method integrating a large language model and a knowledge graph is shown, comprising steps S11 to S18:

[0070] Regarding step S11, a first question of the target user is obtained, and it is determined whether the target user has a second question, the first question is the current question, and the second question is the previous question before the first question.

[0071] The target user refers to a user who has a need to ask questions. A large language model can usually set multiple dialog boxes. When a first question and a second question exist at the same time, the first question and the second question in the present invention both refer to questions in the same dialog box.

[0072] In the large language model, the first question of the target user is obtained, and it is determined whether the first question has a previous question (that is, the second question) in the same dialog box.

[0073] Regarding step S12, if it does not exist, obtain the answer to the first question based on the preset large language model; if it does exist, determine whether the time interval between the first question and the second question is within the first preset time length.

[0074] If the first question does not have a previous question (that is, there is no second question), in other words, the first question is the first question in a dialog box in the large language model, then the first question can be answered directly based on the preset large language model to get a reply.

[0075] If there is a previous question (that is, the second question), determine whether the interval between the first question and the second question is within a first preset time period. The first preset time period can be determined based on the number of words in the answer to the second question, and can be set to t=Z / 300, where t is the first preset time period and Z is the number of words in the answer to the second question.

[0076] Regarding step S13, if not, the answer to the first question is obtained based on the preset large language model; if yes, the first keyword set of the first question and the second keyword set of the second question are respectively extracted based on the preset large language model.

[0077] If the interval between asking the first question and the second question is not within the first preset time, it means that the time correlation between the two questions is not strong, which may mean that the user has changed the topic or started a new query request. In this case, the answer to the first question is obtained directly based on the preset large language model.

[0078] If the interval between the first question and the second question is within a first preset time period, a first keyword set for the first question and a second keyword set for the second question are respectively extracted based on a preset large language model.

[0079] By judging whether there is a second question and whether the time interval is within the first preset time, the present invention can prepare relevant information or guidance in advance, so that the interaction process is smoother and more natural; by identifying the continuity of the questions, the user's intention can be better understood; by judging the continuous progressiveness, the strong correlation between the first question and the second question in time and space is emphasized; and the present invention can also emphasize the time correlation in a more humane way by reasonably setting the first preset time; by "judging whether there is a second question and judging the time interval", the accuracy of subsequent hits on the first question can be greatly improved.

[0080] Extracting keyword sets for the first question and the second question respectively based on a preset large language model means using a large language model to identify key information or core vocabulary in each question.

[0081] For example, if there are a first question and a second question at the same time, and both the first question and the second question are within the first preset time length, the first question is: "What is the weather in Beijing?"; the second question is: "Will it rain in Beijing tomorrow?".

[0082] The keyword set of the first question consists of two keywords {Beijing, weather}; the keyword set of the second question consists of three keywords {Beijing, tomorrow, rain}.

[0083] Regarding step S14, it is determined whether the keyword repetition rate between the first keyword set and the second keyword set is greater than a repetition rate threshold.

[0084] Determining a keyword repetition rate between the first keyword set and the second keyword set includes:

[0085]

[0086] in, is the keyword repetition rate between the first keyword set and the second keyword set, is the number of keywords that are the same in the first keyword set and the second keyword set, is the number of keywords in the second keyword set, is the number of keywords in the first keyword set.

[0087] The repetition rate threshold can be set in segments according to the total number of words in the first keyword set and the second keyword set. For example, when the total number is less than 10, it can be set to 10%, when it is 10-20, it can be set to 15%, and when it is 20-50, it can be set to 20%. The specific analysis can be carried out according to the actual situation.

[0088] For example, the keyword set of the first question consists of two keywords {Beijing, weather}; the keyword set of the second question consists of three keywords {Beijing, tomorrow, rain}, and the total number of keywords is 5. is 1, is 2, If is 3, 25% is greater than 10%.

[0089] Regarding step S15, if it is not greater than, the answer to the first question is obtained based on the preset large language model; if it is greater than, several retrieval query statements are constructed according to the first keyword set and the second keyword set, and the target knowledge graph is filtered out from the preset knowledge graph database.

[0090] If it is not greater than, it means that the first question and the second question are not necessarily highly correlated in content, and the answer to the first question can be obtained directly based on the preset large language model; if it is greater than, it means that the first question and the second question are highly correlated in content, and the keyword sets of the first question and the second question can be used to construct a search query statement with more accurate content and stronger search purpose.

[0091] The present invention sets a repetition rate threshold to determine the content relevance between the first question and the second question, thereby providing a basis for constructing a more accurate query statement in the subsequent text, thereby improving the hit rate of the answer.

[0092] Based on the keyword sets of the first question and the second question, several search query statements can be constructed, which can be implemented by a large language model. However, in order to construct search query statements more accurately, especially for knowledge graph queries, they can be implemented by a SPARQL query editor.

[0093] Take the above example again: the keyword set of the first question is {Beijing, weather}; the keyword set of the second question is {Beijing, tomorrow, rain}.

[0094]

Search query statement 1

[0095] The search query statement can be: PREFIX:<“xx website” / weather#>

[0096] SELECT?currentWeather, where #“xxwebsite” is the website that can be queried. The same applies to the following content, which will not be repeated here.

[0097] WHERE{:In the Beijing area of ​​the current coordinates: has Current Weather?current Weather.}

[0098] Among them, "in the Beijing area of ​​the current coordinates" is the URI representing the city of Beijing.

[0099] :has Current Weather: is a property that represents the "current weather".

[0100] Current Weather: is a variable used to store the current weather information in the query results.

[0101]

Search query statement 2

[0102] You can also:

[0103] PREFIX :<“xxwebsite” / weather#>

[0104] SELECT ?rainProbability

[0105] WHERE{In the Beijing area of ​​the current coordinates: hasRainProbabilityTomorrow ?rainProbability.}

[0106] Among them, "in the Beijing area of ​​the current coordinates" is the URI representing the city of Beijing.

[0107] hasRainProbabilityTomorrow: Attribute indicating the probability of rain tomorrow.

[0108] ?rainProbability is a variable used to store the probability of rainfall tomorrow in the query results.

[0109] PREFIX: <"xx URL" / weather#> declares the namespace prefix: mapped to "xx URL" / weather#, which can be used in queries to use a short prefix instead of the full URI.

[0110] SELECT?currentWeather and SELECT?rainProbability respectively specify the information that needs to be obtained from the knowledge graph.

[0111] WHERE clause: defines the specific conditions of the query, that is, to find the current weather related to Beijing and the probability of rainfall tomorrow.

[0112] It can be understood that the preset knowledge graph database includes several knowledge graphs, for example, it can include general knowledge graphs, knowledge graphs in the medical and health field, knowledge graphs in the financial field, knowledge graphs in the e-commerce field, knowledge graphs in the academic research field, knowledge graphs in the entertainment field, knowledge graphs in the geographic space field, and knowledge graphs in the social network field, etc.

[0113] Filter the target knowledge graph from the preset knowledge graph database, including:

[0114] Determine the graph score of each knowledge graph from a preset knowledge graph database; sort the knowledge graphs in descending order based on a number of graph scores, and take all knowledge graphs before a preset ranking threshold as target knowledge graphs.

[0115] The preset ranking threshold can be determined according to the user's query requirements. If the user needs to get a more detailed and rich answer as much as possible, the preset ranking threshold can be appropriately set to a larger parameter, such as 4, 5, 6, etc. It can be understood that the present invention does not limit the knowledge graph to only one by setting the preset ranking threshold. In other words, multiple replies can be obtained, and the hit rate can be improved by enriching the answers.

[0116] Determine the graph score of each knowledge graph in the preset knowledge graph database, including:

[0117]

[0118] in, is the graph score of the nth knowledge graph in the preset knowledge graph database, and Preset weights, k is the total number of keywords in the keyword set, u is the total number of search query statements, is the number of occurrences of the i-th keyword in the keyword set in the j-th search query statement, is the number of the ith keyword in the nth knowledge graph, The number of search query statements containing core keywords.

[0119] Preset Weights and It can be set according to the actual situation, and the sum of the two is 1. k is the total number of keywords in the keyword set. It can be understood that K refers to the total number of keywords after deduplication. For example, the number of keywords after deduplication of {Beijing, weather} and {Beijing, tomorrow, rain} in the above text is 4.

[0120] Taking the above example, in the query: "What is the probability of rain tomorrow in the Beijing area at the current coordinates?", the keyword tomorrow, then is the number of times “tomorrow” appears in the question “What is the probability of rainfall tomorrow in the Beijing area at the current coordinates?”, which is 1.

[0121] Taking Beijing as an example, It is the number of times "Beijing" appears in the knowledge graph.

[0122] Core keywords refer to the keywords repeated in the keyword set (as long as there are core keywords in the search query, they are all search query statements containing core keywords), that is, in {Beijing, weather} and {Beijing, tomorrow, rain}, "Beijing" is the core keyword. Taking the query statements provided above as an example, both query statements contain Beijing, so the number is 2.

[0123] By combining the frequency of occurrence of keywords in each knowledge graph with the distribution of keywords in the search query statements, the relevance of each knowledge graph can be quantified; by normalizing and logarithmizing the total number of keywords in all search query statements, it helps to smooth the scores, reduce the impact of extreme values, and make the scores more stable and reliable, so as to more accurately find knowledge graphs that are more relevant to user queries, thereby improving query efficiency and user experience.

[0124] Regarding step S16, after searching in the target knowledge graph based on several search query statements, several first search results are obtained, and the preset prompt word structure framework is filled with the several first search results to obtain the first text. The preset prompt word structure framework includes: background introduction, auxiliary information, answer format and core questions.

[0125] Send the above search query statements to the target knowledge graph database and obtain relevant data. Obtain data matching the query from the target knowledge graph, including entity information, relationship information, attribute values, etc.

[0126] Generally speaking, the search query needs to match the preset prompt word structure framework. For example, the preset prompt word structure framework includes background introduction. Taking the above {Beijing, weather} and {Beijing, tomorrow, rain} as examples, the constructed search query can include: "Please introduce the climate of Beijing", etc. By querying the climate background of Beijing, it can help the large language model better understand the question. Auxiliary information can provide additional relevant information to make the answer more comprehensive; the answer format refers to the format of presenting the answer.

[0127] It should be noted that at this point only the core questions remain unfilled.

[0128] By setting a preset prompt word structure framework, the large language model can better understand the first question and generate a structured and rich response. It not only directly answers the user's core question, but also provides more background and auxiliary information, improving the user experience.

[0129] Regarding step S17, the first question is filled into the first text to obtain the second text.

[0130] That is to say, the first question is filled into the "core question" in the preset prompt word structure framework to obtain the second text.

[0131] Regarding step S18, the second text is input into the preset large language model to obtain a target answer corresponding to the first question of the target user.

[0132] By inputting the entire second text into the preset large language model, the preset large model can more accurately understand the first question in the core question by understanding the instructions in the background introduction, and answer it in a reply format with the help of auxiliary information.

[0133] It is understandable that there may be multiple answers to the first question (since there may be multiple target knowledge graphs as mentioned above), which means that the content filled in the background introduction and auxiliary information may include multiple types, which may also lead to the possibility of multiple target answers for the first answer. The target answers can be sorted in descending order according to the scores of the target knowledge graphs, and the target answer corresponding to the target knowledge graph with the highest score can be set at the top of all target answers for users to choose and view.

[0134] Furthermore, in order to improve the hit rate of the answers of the preset large language model, the second text can be input into the preset large language model to obtain the target answer corresponding to the first question of the target user, which also includes: fine-tuning the preset large language model according to the target knowledge graph to obtain the fine-tuned preset large language model; inputting the second text into the fine-tuned preset large language model to obtain the target answer corresponding to the first question of the target user.

[0135] The fine-tuning can be LoRA fine-tuning, which has fewer trainable parameters and improves the fine-tuning efficiency.

[0136] According to the target knowledge graph, the preset large language model is fine-tuned to obtain the fine-tuned preset large language model, including: extracting a number of training data sets from the target knowledge graph, and the types of training data sets include: relationship entity data sets, named entity data sets and reply data sets; based on the several training data sets, the preset large language model is trained to obtain a number of simulated outputs of the preset large language model, each simulated output corresponds to a standard output; according to the several simulated outputs and the several standard outputs, an output verification indicator is determined; when the output verification indicator meets the preset training requirements, the fine-tuned preset large language model is obtained.

[0137] Relational entity datasets refer to those containing relationship information between entities, for example, Beijing hasCurrentWeather, sunny; named entity datasets refer to those containing named entities and their classifications, for example, Beijing [city], sunny [weather conditions]; answer datasets refer to those containing questions and corresponding answers, for example, Q: What is the weather in Beijing? A: Beijing is sunny today, with a temperature of 15°C to 25°C. The preset large language model is trained based on several training datasets. Each question in the training dataset corresponds to a simulated output (that is, a simulated response), and each question also corresponds to a standard output (standard answer).

[0138] Determine output verification indicators based on a number of simulation outputs and a number of standard outputs, including: determine the output repetition rate corresponding to the simulation output based on the simulation output and the standard output corresponding to the simulation output; determine the length parameter based on the simulation output and the standard output corresponding to the simulation output; determine the output verification indicators based on the output repetition rate and length parameter corresponding to the number of simulation outputs and the length similarity corresponding to the number of simulation outputs.

[0139] The output repetition rate refers to the ratio of the number of repeated words in the analog output and the standard output corresponding to the analog output to the maximum number of words between the two. For example, the analog output is 8 words, and the standard output corresponding to the analog output is 6 words. The number of repeated words between the two is 4, then the repetition rate is 4 / 6=66.67%.

[0140] Length similarity refers to the ratio of the length of the longest identical sentence between the simulated output and the corresponding standard output to the maximum length of the sentence between the simulated output and the corresponding standard output.

[0141] Determine output verification indicators, including:

[0142]

[0143]

[0144] in, To output verification indicators, is the length parameter, is the output repetition rate corresponding to the yth analog output, and All are preset weights (which can be determined according to actual conditions). is the length similarity corresponding to the y-th simulation output, is the maximum length of the sentences in the y-th simulation output and the y-th standard output, is the longest identical sentence length between the yth simulation output and the yth standard output.

[0145] Determine length parameters based on several simulated outputs and several standard outputs, including:

[0146] ,or,

[0147]

[0148] in, is the length parameter, is the total number of words of several analog outputs, The total number of characters in several standard outputs.

[0149] When the output verification index meets the preset training requirements, the fine-tuned preset large language model is obtained.

[0150] By designing verification indicators through indicators such as word count, sentence length, and sentence similarity, the training effect of the preset large language model can be better verified, thereby improving the hit rate of the preset large language model's answers.

[0151] In summary, the present invention can prepare relevant information or guidance in advance by judging whether there is a second question and whether the time interval is within the first preset time length, so that the interaction process is smoother and more natural; by identifying the continuity of the questions, the user's intentions can be better understood; through the judgment of continuous progression, the strong correlation between the first question and the second question in time and space is emphasized; and the present invention can also emphasize the time correlation in a more humane way by reasonably setting the first preset time length; by "judging whether there is a second question and judging the time interval", the accuracy of subsequent hits on the first question can be greatly improved.

[0152] The present invention sets a repetition rate threshold to determine the content relevance between the first question and the second question, thereby providing a basis for constructing a more accurate query statement and improving the hit rate of the answer.

[0153] The present invention does not limit the knowledge graph to only one by setting a preset ranking threshold. In other words, multiple replies can be obtained, and the hit rate can be improved by enriching the answers.

[0154] The present invention can quantify the relevance of each knowledge graph by combining the frequency of occurrence of keywords in each knowledge graph with the keyword distribution in the search query statement; by normalizing and logarithmizing the total number of keywords in all search query statements, it helps to smooth the score, reduce the impact of extreme values, and make the score more stable and reliable, so as to more accurately find the knowledge graph most relevant to the user query, thereby improving query efficiency and user experience.

[0155] By setting a preset prompt word structure framework, the present invention can help the large language model better understand the first question and generate a structured and rich reply; it not only directly answers the user's core question, but also provides more background and auxiliary information, thereby improving the user experience.

[0156] The present invention designs verification indicators through indicators such as word count, sentence length, sentence similarity, etc., which can better verify the training effect of the preset large language model and improve the hit rate of the preset large language model answer.

[0157] Based on the same inventive concept, the present invention provides an intelligent reply device integrating a large language model and a knowledge graph, comprising:

[0158] A first judgment module is used to obtain a first question asked by a target user and judge whether the target user has a second question, wherein the first question is a current question and the second question is a previous question before the first question;

[0159] A second judgment module is used to obtain a response to the first question based on a preset large language model if the response does not exist; if the response does exist, determine whether the time interval between the first question and the second question is within a first preset time length;

[0160] an extraction module, configured to obtain a reply to the first question based on a preset large language model if the user is not present; and to extract a first keyword set for the first question and a second keyword set for the second question based on the preset large language model if the user is present;

[0161] A third judgment module is used to judge whether the keyword repetition rate between the first keyword set and the second keyword set is greater than a repetition rate threshold;

[0162] A screening module is used for obtaining an answer to the first question based on a preset large language model if the number is not greater than ; if the number is greater than , constructing a plurality of search query statements based on the first keyword set and the second keyword set, and screening out a target knowledge graph from a preset knowledge graph database;

[0163] A filling module is used to obtain a plurality of first search results after searching in the target knowledge graph based on a plurality of search query statements, and fill a preset prompt word structure framework with the plurality of first search results to obtain a first text, wherein the preset prompt word structure framework includes: background introduction, auxiliary information, answer format and core question;

[0164] A second text module, used for filling the first question into the first text to obtain a second text;

[0165] The output module is used to input the second text into the preset large language model to obtain the target answer corresponding to the first question of the target user

[0166] Since the electronic device introduced in this embodiment is an electronic device used to implement the information processing method in the embodiment of the present invention, based on the information processing method introduced in the embodiment of the present invention, a person skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present invention is not described in detail here. As long as the electronic device used by a person skilled in the art to implement the information processing method in the embodiment of the present invention, it belongs to the scope of protection of the present invention.

[0167] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0172] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An intelligent reply method integrating a large language model and a knowledge graph, characterized in that: The method comprises: Obtaining a first question of a target user, and determining whether the target user has a second question, wherein the first question is a current question, and the second question is a previous question before the first question; If not, obtaining an answer to the first question based on a preset large language model; if so, determining whether the time interval between the first question and the second question is within a first preset time length; If not, obtaining an answer to the first question based on the preset large language model; if yes, extracting a first keyword set for the first question and a second keyword set for the second question respectively based on the preset large language model; Determining whether a keyword repetition rate between the first keyword set and the second keyword set is greater than a repetition rate threshold; If not, obtaining an answer to the first question based on the preset large language model; if greater than, constructing several search query statements based on the first keyword set and the second keyword set, and filtering out target knowledge graphs from a preset knowledge graph database, including: determining a graph score of each knowledge graph from the preset knowledge graph database; sorting the knowledge graphs in descending order based on several graph scores, and taking all knowledge graphs before a preset ranking threshold as the target knowledge graphs; After searching the target knowledge graph based on a plurality of search query statements, a plurality of first search results are obtained, and a preset prompt word structure framework is filled with the plurality of first search results to obtain a first text, wherein the preset prompt word structure framework includes: background introduction, auxiliary information, answer format, and core question; Filling the first question into the first text to obtain a second text; Inputting the second text into the preset large language model to obtain a target answer corresponding to the first question of the target user, further comprising: fine-tuning the preset large language model according to the target knowledge graph to obtain a fine-tuned preset large language model; inputting the second text into the fine-tuned preset large language model to obtain a target answer corresponding to the first question of the target user, comprising: extracting a plurality of training data sets from the target knowledge graph, wherein the types of the training data sets include: a relationship entity data set, a named entity data set, and a reply data set; training the preset large language model based on the plurality of training data sets to obtain a plurality of simulated outputs of the preset large language model, wherein each simulated output corresponding to a standard output; determining an output verification indicator according to a plurality of simulated outputs and a plurality of standard outputs; when the output verification indicator meets the preset training requirements, obtaining a preset large language model after fine-tuning, including: determining an output repetition rate corresponding to the simulated output according to the simulated output and the standard output corresponding to the simulated output; determining a length parameter according to the plurality of simulated outputs and the plurality of standard outputs; determining a length similarity corresponding to the simulated output according to the simulated output and the standard output corresponding to the simulated output; determining the output verification indicator according to the output repetition rate corresponding to the plurality of simulated outputs, the length parameter and the length similarity corresponding to the plurality of simulated outputs, including: determining the output verification indicator, including: in, To output verification indicators, is the length parameter, is the output repetition rate corresponding to the yth analog output, and All are preset weights. is the length similarity corresponding to the y-th simulation output, is the maximum length of the sentences in the y-th simulation output and the y-th standard output, is the longest identical sentence length between the yth simulation output and the yth standard output.

2. The intelligent reply method integrating a large language model and a knowledge graph as claimed in claim 1, characterized in that: Determining a keyword repetition rate between the first keyword set and the second keyword set includes: in, is the keyword repetition rate between the first keyword set and the second keyword set, is the number of keywords that are the same in the first keyword set and the second keyword set, is the number of keywords in the second keyword set, is the number of keywords in the first keyword set.

3. The intelligent reply method integrating a large language model and a knowledge graph as claimed in claim 1, characterized in that: Determine length parameters based on several simulated outputs and several standard outputs, including: ,or, in, is the length parameter, is the total number of words of several analog outputs, The total number of characters in several standard outputs.

Images