An intelligent question-answering method based on vertical domain knowledge

Abstract

The present invention discloses an intelligent question-answering method based on vertical domain knowledge, including the following steps: data division based on vertical domain data types, building several knowledge bases according to several data types, and saving data in the knowledge base; the knowledge base includes a question-answer pair knowledge base, a knowledge graph database and a document knowledge base; based on the knowledge of the vertical domain, LLM fine-tuning instructions are built to intentionally understand the vertical domain professional vocabulary and the content involved in user questions; based on the divided data types and the fine-tuning of LLM, an intelligent question-answering core component is built to output answers. The original data is divided into three types, and corresponding technical solutions are given respectively to solve them; and the general LLM and the data in the vertical domain are deeply integrated, and the LLM is used to understand the intention through fine-tuning and customization; by building the intelligent question-answering core component, three question-answering modules are given to provide more accurate and professional answers in the vertical field.

Description

Technical Field

[0001] The present invention relates to an intelligent question-answering method, and in particular to an intelligent question-answering method based on vertical domain knowledge. Background Art

[0002] Intelligence in intelligent question-answering is reflected in the ability to understand the user's intention and give the correct answer. Traditional intelligent question-answering is generally a search-based intelligent question-answering system, such as question-answer pairs and knowledge graphs. It usually identifies the user's intention through technologies such as name recognition, relationship extraction, text classification, part-of-speech analysis, and syntactic analysis, and then retrieves the answer from the knowledge base through template matching and other methods; this rule- and template-based method lacks flexibility and generalization ability, and is difficult to handle complex semantics and contexts such as questions that are more colloquial.

[0003] In recent years, with the development of deep learning technology, large language models such as the GPT series have begun to emerge. These models have learned rich language representation and semantic understanding capabilities through pre-training on massive amounts of text data; they use advanced architectures such as Transformer, have powerful natural language processing capabilities, can generate natural and fluent text, and have achieved remarkable results in various NLP tasks. Applying large language models to intelligent question and answer can overcome the advantages and disadvantages of traditional intelligent question and answer. First, large language models have powerful language understanding capabilities and can understand complex semantics and contexts, so as to understand users' questions more accurately; secondly, because these models are trained based on large-scale data, their knowledge coverage is wider, and they can cover knowledge in more fields, providing users with richer and more accurate answers; large language models also have the ability to generate text, and can dynamically generate natural and fluent language, not just matching predefined answers.

[0004] Since OpenAI released ChatGPT, the development of large language models (LLMs) has set off a technological revolution worldwide. As an engine driving the advancement of artificial intelligence technology, LLMs have achieved remarkable success in solving general natural language processing tasks. However, their versatility also makes them face some challenges in vertical applications. The widespread application of general LLMs has achieved success in many fields, but applications in vertical fields often require a higher degree of specialization and rigor. General LLMs have difficulty understanding the professional terms and complex contexts of specific industries. Therefore, in existing applications in vertical fields, the performance of general LLMs is limited to a certain extent.

[0005] Existing large-scale language models face some challenges in intelligent question answering in vertical fields. First, traditional general models often cannot meet the precise requirements of vertical fields, resulting in poor performance on specialized questions; second, the particularity and complexity of data in vertical fields make model training and fine-tuning more complicated; and, when dealing with complex questions in vertical fields, existing intelligent question answering systems cannot provide sufficiently accurate and in-depth answers. Summary of the invention

[0006] In order to solve the shortcomings of the above-mentioned technology, the present invention provides an intelligent question-answering method based on vertical field knowledge.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: an intelligent question answering method based on vertical field knowledge, comprising the following steps:

[0008] Step S1: divide the data based on the vertical field data type, build several knowledge bases according to several data types, and save the data in the knowledge base; the knowledge base includes the question-answer pair knowledge base, the knowledge graph database and the document knowledge base;

[0009] Step S2: construct LLM fine-tuning instructions based on the knowledge of the vertical field to understand the professional vocabulary of the vertical field and the content involved in the user's questions; LLM fine-tuning includes large language model instruction alignment training, embedding model training and additional citations;

[0010] Step S3: Based on the data types divided in step S1 and the fine-tuning of LLM in step S2, construct the intelligent question-answering core component to output the answer; the intelligent question-answering core component includes a question-answer pair module, a knowledge graph module, and a retrieval enhancement generation module.

[0011] Furthermore, in step S1, the vertical field data for fixed reply content is extracted from question-answer pairs based on the large language model LLM, and stored in the question-answer pair knowledge base in the form of one question and one answer; the vertical field data for structured knowledge is extracted from the artificially designed knowledge graph structure based on LLM, and stored in the knowledge graph database; the vertical field data for files is sliced ​​and stored in the document knowledge base; the paragraphs in the file are sliced, and the article corresponding to the paragraph is saved; the sentence level in the file is sliced, and the paragraph corresponding to the sentence is saved.

[0012] Furthermore, the large language model instruction alignment training in step S2 includes the following steps:

[0013] Step S21-1: Collect vertical field data and pre-process the vertical field data;

[0014] Step S21-2: construct instructions for the preprocessed vertical field data, and construct them into logical reasoning instructions, knowledge graph intent understanding instructions, or generation instructions based on the nature of the data;

[0015] Step S21-3: Adopt a low-rank adaptive method, and based on the internal low-rank characteristics of LLM, add a bypass matrix to simulate full parameter fine-tuning to achieve alignment training of knowledge graph intent understanding instructions, logical reasoning instructions, and generation instructions; the loss function of instruction alignment training is the cross entropy loss function, and the formula is:

[0016] ;

[0017] Where T is the length of the sequence; is the true output at time step t; The model predicts the output for time step t;

[0018] Step S21 - 4 : Setting instruction alignment training parameters and performing instruction alignment training for a large language model.

[0019] Furthermore, the logical reasoning instructions in step S21-2 are used to determine whether the retrieved question-answer pairs can answer the user's questions; the knowledge graph intention understanding instructions are used by the large language model to map the user's questions to the subgraph; and the generation instructions are used to generate answers from the user's questions and text paragraphs in a reasonable format.

[0020] Furthermore, the embedded model training in step S2 includes the following steps:

[0021] Step S22-1: Collecting data in the general field and preprocessing the data in the general field;

[0022] Step S22-2: Expand the preprocessed general domain data and preprocessed vertical domain data with positive and negative samples, divide the tasks according to the data source and data preprocessing method, and use the loss function of each task for mixed training; the tasks are divided into classification tasks, retrieval tasks, text similarity tasks, and text implication tasks;

[0023] Step S22-3: assign weights to the loss functions corresponding to each task according to the amount of data after task division; the sum of the weighted coefficients of each loss function is 1 to prevent the embedding model from overfitting the data of a certain task;

[0024] Step S22-4: Setting embedding model training parameters and performing embedding model training.

[0025] Furthermore, the task division in step S22-2 includes the following criteria:

[0026] 1. The query text is classified into the classification task, and the category corresponding to the query text is taken as the positive sample, and a category corresponding to a non-query text is randomly selected as the negative sample; the loss function of the classification task is:

[0027]

[0028] ;

[0029] Where N is the number of samples in a batch; text is the query text; The category corresponding to the query text is the positive sample; is the category corresponding to the non-query text, i.e., the negative sample; C is the set of all categories of the classification task; τ is the temperature coefficient; is the cosine similarity function; log is the logarithmic function;

[0030] Second, the text content with a query text length of less than 128 characters and a positive sample character length of greater than or equal to 512 characters is divided into the retrieval task; the positive sample is the text paragraph corresponding to the query text, and the negative sample randomly selects another paragraph of the paragraph corresponding to the query text; the loss function of the retrieval task is:

[0031]

[0032] ;

[0033] Where N is the number of samples in a batch; text is the query text; The paragraph corresponding to the query text is the positive sample; Other paragraphs of the paragraph where the query text is located are negative samples; is the data set of the current batch; τ is the temperature coefficient; is the cosine similarity function; log is the logarithmic function;

[0034] 3. Based on the text content in the vertical field, the large language model generates different phrases; based on each phrase, two sentences are randomly generated based on the large language model, and the two sentences are scored for relevance, with the score ranging from 1 to 5; the text content is divided into text similarity tasks according to the text similarity task format;

[0035] The loss function for the text similarity task is:

[0036] ;

[0037] in, is a hyperparameter; is the feature vector and The correlation score between is the cosine similarity function; log is the logarithmic function;

[0038] 4. Based on the text content in the vertical field, different short sentences are generated through a large language model; the large language model generates three short sentences sentence1, sentence2, and sentence3 based on each short sentence sentence0; sentence1, sentence2, sentence3 and sentence0 are respectively in the relationship of implication, neutrality, and contradiction; the text content is divided into text implication tasks according to the text implication task format;

[0039] The loss function for the text entailment task is:

[0040] ;

[0041] in, is sentence0; for sentence1; for sentence2; for sentence3; is a hyperparameter.

[0042] Further, The formula for the cosine similarity function is:

[0043] ;

[0044] Among them, x and y are both eigenvectors.

[0045] Furthermore, in step S2, the problem of inconsistent distribution of long text and short text in semantic space in embedding model training is solved based on the small-to-big strategy, including the following steps:

[0046] Step S23-1: Summarize the long text based on the small-to-big strategy, extract the key phrases, and retain the mapping relationship between the key phrases and the long text;

[0047] Step S23 - 2 : Retrieve key phrases and map the long text through the key phrases.

[0048] Furthermore, in step S2, the large language model is added with citations, including the following steps:

[0049] Step 24-1: The large language model will mark the paragraph-level content and article title of the generated content when answering the user's question according to the generated instructions;

[0050] Step S24-2: To ensure the credibility of the citation, the generated citation will be judged. If the judgment is valid, the current citation will be kept; if it is invalid, the current citation will be modified;

[0051] The methods for identifying citations include:

[0052] Step S24-2-1: Slice the generated content;

[0053] Step S24-2-2: Calculate the similarity between the sliced ​​text and the content segment based on a text similarity algorithm;

[0054] Step S24-2-3: Set the threshold to 0.8. If the calculated similarity is greater than 0.8, the content fragment can be used as a basis for citation and is judged to be valid; if the calculated similarity is less than or equal to 0.8, the content fragment cannot be used as a basis for citation and is judged to be invalid.

[0055] Furthermore, outputting the answer based on the intelligent question-answering core component in step S3 includes the following steps:

[0056] Step S31: Retrieve candidate knowledge from the question-answer pair knowledge base, and enter the retrieved candidate knowledge into the question-answer pair module; construct the tuple data retrieved from the candidate knowledge and the user's question into a logical reasoning instruction;

[0057] Step S32: The large language model determines the correctness of the search result according to the logical reasoning instructions; if the search result is correct, the answer is output; if the search result is wrong, the candidate knowledge is retrieved from the knowledge graph database, and the retrieved candidate knowledge, i.e., the subgraph, enters the knowledge graph module;

[0058] Step S33: construct the subgraph and the user's question into a knowledge graph intention understanding instruction;

[0059] Step S34: The large language model maps the user's intention to the subgraph of the knowledge graph according to the knowledge graph intention understanding instruction, and obtains and outputs the answer by searching on the subgraph with complete information; if the information of the subgraph is incomplete, the system actively interacts with the user and completes the missing information on the subgraph, and obtains and outputs the answer; if the answer cannot be found on the subgraph with complete information, candidate knowledge is retrieved from the document knowledge base, and the retrieved candidate knowledge, i.e., the key phrase, enters the retrieval enhancement generation module;

[0060] Step S35: The key phrase is mapped back to its own long text paragraph based on the small-to-big strategy, and the long text paragraph and the user's question are constructed into a generation instruction;

[0061] Step S36: The large language model generates an answer according to the generation instruction; the credibility of the citation is verified based on the similarity post-processing method, and the answer is output.

[0062] The present invention discloses an intelligent question-answering method based on vertical domain knowledge. In view of the complexity of vertical domain knowledge, the original data is divided into three types, and corresponding technical solutions are given for each type to solve the problem; and the general LLM and the data in the vertical domain are deeply integrated. Through fine tuning and customization, the LLM can understand the intention and better adapt to the needs of intelligent question-answering in a certain field, and can solve specialized problems; by constructing the intelligent question-answering core component, three question-answering modules are given for the three data types respectively, which can provide more accurate and professional answers in the vertical field. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of the present invention.

[0064] Figure 2 This is a flow chart of instruction alignment in the present invention.

[0065] Figure 3 The flowchart of the embedded model training in the present invention.

[0066] Figure 4 This is an internal flow chart of the intelligent question-answering core component of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0068] An intelligent question answering method based on vertical domain knowledge includes the following steps:

[0069] Step S1: divide the data based on the vertical field data type, build several knowledge bases according to several data types, and save the data in the knowledge base; the knowledge base includes the question-answer pair knowledge base, the knowledge graph database and the document knowledge base;

[0070] Step S2: construct LLM fine-tuning instructions based on the knowledge of the vertical field to understand the professional vocabulary of the vertical field and the content involved in the user's questions; LLM fine-tuning includes large language model instruction alignment training, embedding model training and additional citations;

[0071] Step S3: Based on the data types divided in step S1 and the fine-tuning of LLM in step S2, construct the intelligent question-answering core component to output the answer; the intelligent question-answering core component includes a question-answer pair module, a knowledge graph module, and a retrieval enhancement generation module.

[0072] Specifically, in step S1, the vertical field data for fixed reply content is extracted from question-answer pairs based on the large language model LLM, and stored in the question-answer pair knowledge base in the form of one question and one answer; the vertical field data for structured knowledge is extracted from the artificially designed knowledge graph structure based on LLM, and stored in the knowledge graph database; the vertical field data for files is sliced ​​and stored in the document knowledge base; the paragraphs in the file are sliced, and the article corresponding to the paragraph is saved; the sentence level in the file is sliced, and the paragraph corresponding to the sentence is saved.

[0073] The large language model instruction alignment training in step S2 includes the following steps:

[0074] Step S21-1: Collect vertical field data and pre-process the vertical field data;

[0075] Step S21-2: construct instructions for the preprocessed vertical field data, and construct them into logical reasoning instructions, knowledge graph intent understanding instructions, or generation instructions based on the nature of the data;

[0076] Step S21-3: Adopt a low-rank adaptive method, and based on the internal low-rank characteristics of LLM, add a bypass matrix to simulate full parameter fine-tuning to achieve alignment training of knowledge graph intent understanding instructions, logical reasoning instructions, and generation instructions; the loss function of instruction alignment training is the cross entropy loss function, and the formula is:

[0077] ;

[0078] Where T is the length of the sequence; is the true output at time step t; The model predicts the output for time step t;

[0079] Step S21 - 4 : Setting instruction alignment training parameters and performing instruction alignment training for a large language model.

[0080] The logical reasoning instructions in step S21-2 are used to determine whether the retrieved question-answer pairs can answer the user's questions; the knowledge graph intention understanding instructions are used by the large language model to map the user's questions to the subgraph; and the generation instructions are used to generate answers from the user's questions and text paragraphs in a reasonable format.

[0081] The embedded model training in step S2 includes the following steps:

[0082] Step S22-1: Collecting data in the general field and preprocessing the data in the general field;

[0083] Step S22-2: Expand the preprocessed general domain data and preprocessed vertical domain data with positive and negative samples, divide the tasks according to the data source and data preprocessing method, and use the loss function of each task for mixed training; the tasks are divided into classification tasks, retrieval tasks, text similarity tasks, and text implication tasks;

[0084] Step S22-3: assign weights to the loss functions corresponding to each task according to the amount of data after task division; the sum of the weighted coefficients of each loss function is 1 to prevent the embedding model from overfitting the data of a certain task;

[0085] Step S22-4: Setting embedding model training parameters and performing embedding model training.

[0086] The task division in step S22-2 includes the following criteria:

[0087] 1. The query text is classified into the classification task, and the category corresponding to the query text is taken as the positive sample, and a category corresponding to a non-query text is randomly selected as the negative sample; the loss function of the classification task is:

[0088]

[0089] ;

[0090] Among them, N is the number of samples in a batch; text is the query text The category corresponding to the query text is the positive sample; is the category corresponding to the non-query text, i.e., the negative sample; C is the set of all categories of the classification task; τ is the temperature coefficient; is the cosine similarity function; log is the logarithmic function;

[0091] Second, the text content with a query text length of less than 128 characters and a positive sample character length of greater than or equal to 512 characters is divided into the retrieval task; the positive sample is the text paragraph corresponding to the query text, and the negative sample randomly selects another paragraph of the paragraph corresponding to the query text; the loss function of the retrieval task is:

[0092]

[0093] ;

[0094] Where N is the number of samples in a batch; text is the query text; The paragraph corresponding to the query text is the positive sample; Other paragraphs of the paragraph where the query text is located are negative samples; is the data set of the current batch; τ is the temperature coefficient; is the cosine similarity function; log is the logarithmic function;

[0095] 3. Based on the text content in the vertical field, the large language model generates different phrases; based on each phrase, two sentences are randomly generated based on the large language model, and the two sentences are scored for relevance, with the score ranging from 1 to 5; the text content is divided into text similarity tasks according to the text similarity task format;

[0096] The loss function for the text similarity task is:

[0097] ;

[0098] in, is a hyperparameter; is the feature vector and The correlation score between is the cosine similarity function; log is the logarithmic function;

[0099] 4. Based on the text content in the vertical field, different short sentences are generated through a large language model; the large language model generates three short sentences sentence1, sentence2, and sentence3 based on each short sentence sentence0; sentence1, sentence2, sentence3 and sentence0 are respectively in the relationship of implication, neutrality, and contradiction; the text content is divided into text implication tasks according to the text implication task format;

[0100] The loss function for the text entailment task is:

[0101] ;

[0102] in, is sentence0; for sentence1; for sentence2; for sentence3; is a hyperparameter.

[0103] The formula for the cosine similarity function is:

[0104] ;

[0105] Among them, x and y are both eigenvectors.

[0106] In step S2, the problem of inconsistent distribution of long text and short text in the semantic space during embedding model training is solved based on the small-to-big strategy, which includes the following steps:

[0107] Step S23-1: Summarize the long text based on the small-to-big strategy, extract the key phrases, and retain the mapping relationship between the key phrases and the long text;

[0108] Step S23 - 2 : Retrieve key phrases and map the long text through the key phrases.

[0109] In step S2, the large language model is appended with citations, including the following steps:

[0110] Step 24-1: The large language model will mark the paragraph-level content and article title of the generated content when answering the user's question according to the generated instructions;

[0111] Step S24-2: To ensure the credibility of the citation, the generated citation will be judged. If the judgment is valid, the current citation will be kept; if it is invalid, the current citation will be modified;

[0112] The methods for identifying citations include:

[0113] Step S24-2-1: Slice the generated content;

[0114] Step S24-2-2: Calculate the similarity between the sliced ​​text and the content segment based on a text similarity algorithm;

[0115] Step S24-2-3: Set the threshold to 0.8. If the calculated similarity is greater than 0.8, the content fragment can be used as a basis for citation and is judged to be valid; if the calculated similarity is less than or equal to 0.8, the content fragment cannot be used as a basis for citation and is judged to be invalid.

[0116] Outputting the answer based on the intelligent question-answering core component in step S3 includes the following steps:

[0117] Step S31: Retrieve candidate knowledge from the question-answer pair knowledge base, and enter the retrieved candidate knowledge into the question-answer pair module; construct the tuple data retrieved from the candidate knowledge and the user's question into a logical reasoning instruction;

[0118] Step S32: The large language model determines the correctness of the search result according to the logical reasoning instructions; if the search result is correct, the answer is output; if the search result is wrong, the candidate knowledge is retrieved from the knowledge graph database, and the retrieved candidate knowledge, i.e., the subgraph, enters the knowledge graph module;

[0119] Step S33: construct the subgraph and the user's question into a knowledge graph intention understanding instruction;

[0120] Step S34: The large language model maps the user's intention to the subgraph of the knowledge graph according to the knowledge graph intention understanding instruction, and obtains and outputs the answer by searching on the subgraph with complete information; if the information of the subgraph is incomplete, the system actively interacts with the user and completes the missing information on the subgraph, and obtains and outputs the answer; if the answer cannot be found on the subgraph with complete information, candidate knowledge is retrieved from the document knowledge base, and the retrieved candidate knowledge, i.e., the key phrase, enters the retrieval enhancement generation module;

[0121] Step S35: The key phrase is mapped back to its own long text paragraph based on the small-to-big strategy, and the long text paragraph and the user's question are constructed into a generation instruction;

[0122] Step S36: The large language model generates an answer according to the generation instruction; the credibility of the citation is verified based on the similarity post-processing method, and the answer is output.

[0123] This embodiment provides an implementation method of intelligent question answering based on vertical domain knowledge; an intelligent question answering method based on vertical domain knowledge, such as Figure 1 The flowchart of an intelligent question answering method based on vertical domain knowledge is shown in FIG. , which specifically includes the following steps:

[0124] By analyzing the knowledge in various vertical fields, namely education, government affairs, tourism, and medical insurance, we summarized the common characteristics of vertical field knowledge and divided the vertical field knowledge into three types of data: question and answer pairs, service guides, and policy documents.

[0125] Step 1: Obtain the original vertical field data and divide the data; the vertical field focuses on specific tasks or fields to train specific data sets; according to the relevant messages in the vertical field, the services that each department is responsible for, and the policy notification documents of each department, the original vertical field data is divided into three data types.

[0126] Step 2: Construct three knowledge bases according to the three data types, namely, question-answer pair knowledge base, knowledge graph database, and document knowledge base;

[0127] Step 21: Extract the question-answer pairs based on the large language model, organize the vertical domain knowledge data into a question-answer format, and store them in the question-answer pair knowledge base; the vertical domain knowledge data mainly includes messages and fixed reply content;

[0128] Step 22: Organize the structured knowledge such as the service guides or service processes of each department into a knowledge graph based on a large language model; the knowledge graph can have multi-level relationships, such as the organizational structure of a department;

[0129] Step 23: Slice the policies, notices, announcements and other documents based on paragraph or sentence level division to form a document knowledge base; the sliced ​​paragraphs will retain the articles corresponding to the paragraphs, and if it is a sentence level division, the paragraphs corresponding to the sentences will be retained.

[0130] Step 3: If Figure 2 As shown in the flowchart of instruction alignment in an intelligent question-answering method based on vertical domain knowledge, vertical domain data is collected and preprocessed, and instructions for LLM model fine-tuning are constructed based on vertical domain knowledge to understand intent, so as to improve the understanding ability and professionalism of large language models in vertical domain knowledge;

[0131] Step 31: The current source of vertical field data is the government service platform of each province, and the collected data is in plain text format;

[0132] Step 32: The preprocessing method is: segment the text according to the delimiter; each sentence of the segmented text is passed through a large language model before constructing the instruction, and the large language model determines whether a valid instruction can be constructed, and the segmented text content is in plain text format as a valid instruction;

[0133] Step 33: Divide the valid instructions into three types, including knowledge graph intent understanding instructions, logical reasoning instructions, and generation instructions; the knowledge graph intent understanding instructions are intended to enable the large language model to map the user's question to the subgraph in the case of user questions and reference subgraphs; the logical reasoning instructions are intended to determine whether the retrieved question-answer pairs can answer the user's question, and also determine whether the citations generated by the large language model are correct; the generation instructions are intended to generate the user's answer in a reasonable format based on the user's question and reference content fragments;

[0134] Step 34: Perform instruction alignment training on the three instructions based on vertical domain knowledge and use them for fine-tuning of the large language model; adopt a low-rank adaptive method, and based on the inherent low-order characteristics of the large language model, add a bypass matrix to simulate full parameter fine-tuning; compared with other fine-tuning methods, such as Full Finetune, P-Tuning, Adapter Tuning, etc., the low-rank adaptive method reduces the number of parameters to be trained, and is faster and easier to train;

[0135] Step 34-1: The low-rank adaptive method introduces two matrices, namely matrix A and matrix B. Assume that the original matrix size of parameter W is , then the sizes of matrix A and matrix B are and , where r is the rank, which is usually less than 100; if a low-rank adaptive method with a rank of 4 is used, the shape of the matrix is

[0136] , greatly reducing the number of parameters that need to be trained; where d is the dimension of the parameter, x is the input feature, and h is the output feature.

[0137] Step 34-2: Use the loss function to measure the gap between the instruction alignment training prediction value and the actual value; the loss function is the cross entropy loss function, and the formula is:

[0138] ;

[0139] Where T is the length of the sequence; is the true output at time step t; The model predicts the output at time step t. The training parameters in instruction alignment training are: batch size is 8, learning rate is 5e-5, learning rate decay strategy is cosine, and the total number of training epochs is 3.

[0140] Step 4: Figure 3 As shown in the flowchart of embedding model training in an intelligent question-answering method based on vertical domain knowledge, general domain data is collected and preprocessed, general domain data and vertical domain data are divided into tasks, and the text content of the task is expanded with positive and negative samples. A variety of loss functions are used to train the embedding model to improve the ability to use professional terms and solve spoken questions; convert spoken questions into professional terms;

[0141] Step 41: Divide the tasks into classification tasks, retrieval tasks, text similarity tasks and text implication tasks based on the data source and data preprocessing method;

[0142] Step 41-1: Collect general domain data in the general domain dataset ThuNews. The general domain data is divided into 14 categories. The general domain data will be divided into classification tasks. The query text is the text content of the classification data, the positive sample is the type corresponding to the query text, and the negative sample is randomly selected from other categories of the category corresponding to the positive sample.

[0143] The loss function for the classification task is:

[0144]

[0145] ;

[0146] Where N is the number of samples in a batch; text is the query text; The category corresponding to the query text is the positive sample; is the category corresponding to the non-query text, i.e., the negative sample; C is the set of all categories of the classification task; τ is the temperature coefficient; is the cosine similarity function, and the formula is: ;

[0147] Among them, x and y are both eigenvectors.

[0148] Step 41-2: Assign the general domain data and vertical domain data whose character length of the query text text is less than 128 characters and whose character length of the positive sample text_pos is greater than or equal to 512 characters to the retrieval task; preprocess the retrieval task, the general domain data and vertical domain data used as the retrieval task are usually an article, the article is divided into multiple paragraphs, and different query texts are constructed for each paragraph; each paragraph is sent to the large language model, and a text generation task is given to the large language model, the large language model generates multiple different query texts, for example, the large language model generates a query text and the answer corresponding to the query; the positive sample corresponding to the query text is the text paragraph, and the negative sample is other text paragraphs in the paragraph where the query text is located;

[0149] The loss function for the retrieval task is:

[0150]

[0151] ;

[0152] Where N is the number of samples in a batch; text is the query text; The paragraph corresponding to the query text is the positive sample; Other paragraphs of the paragraph where the query text is located are negative samples; is the data set of the current batch; τ is the temperature coefficient; is the cosine similarity function, and the formula is:

[0153] ;

[0154] Among them, x and y are both eigenvectors.

[0155] Step 41-3: If the similarity scores of the two texts are between 1 and 5, it is a text similarity task; the data format of the text similarity task is (text1, text2, score), where text1 and text2 represent different texts, and score represents the similarity score of text1 and text2. The larger the score, the more similar the two texts are; preprocess the text similarity task: based on the text data of the current vertical field, let the large language model generate different short sentences, and for each pair of phrases, let the large language model randomly generate two sentences, and score the relevance of the two generated sentences. The score ranges from 1 to 5, and the higher the score, the more relevant it is;

[0156] The loss function for the text similarity task is:

[0157] ;

[0158] in, is a hyperparameter; is the feature vector and The correlation score between is the cosine similarity function, and the formula is:

[0159] ;

[0160] Among them, x and y are both eigenvectors.

[0161] Step 41-4: Based on the text data of the current vertical field, the large language model generates different short sentences. Based on the large language model referring to each short sentence sentence0, the large language model generates three sentences, namely sentence1, sentence2 and sentence3, where sentence1, sentence2, sentence3 and sentence0 are in the relationship of implication, neutrality and contradiction respectively; the data format of the text implication task is represented as (text3, text4, label), where text3 is the premise text and text4 is the hypothesis text. If the hypothesis text text4 can be inferred from the premise text text3, it indicates an implication relationship, and the corresponding label value is implication; if the premise text text3 and the hypothesis text text4 are conflicting, the label value is conflict; if the premise text text3 and the hypothesis text text4 are neither in an implication relationship nor in a conflict relationship, the label value is neutral;

[0162] The loss function for the text entailment task is:

[0163] ;

[0164] in, is sentence0; for sentence1; for sentence2; for sentence3; is a hyperparameter.

[0165] To prevent the embedding model from overfitting the data of a certain task, the loss function is weighted according to the amount of data in the task, and the sum of the weighted coefficients is 1. The training parameters for training the embedding model are: batch size is 16, learning rate is 1e-5, learning rate decay strategy is linear, and the total number of training epochs is 3.

[0166] Step 42: Since the embedding model has inconsistent distribution of long text and short text in the semantic space, based on the small-to-big strategy, the long text content will be summarized and key phrases extracted to achieve compression, and the relationship between the long text before and after compression will be retained.

[0167] The small-to-big strategy means that in the actual retrieval process, the query text entered by the user is usually short, while the documents that are seen are generally paragraph-level, semantically rich and relatively long. However, long texts are prone to noise interference in the embedding model, so the long text content needs to be compressed;

[0168] Step 421: Summarize the long text and extract key phrases;

[0169] Step 422: retaining the mapping relationship between the key phrase and the long text;

[0170] Step 42-3: When searching, instead of searching the long text, searching is performed on the key phrases; the key phrases are mapped to the long text.

[0171] Step 43: The content generated by the large language model contains citations. That is, the large language model will mark the paragraph-level content and article title of the generated content when answering the user's question according to the generation instruction to improve the credibility of the generated content. To ensure the credibility of the citation, the generated citation will be judged. If the judgment is valid, the current citation will be maintained; if it is invalid, the current citation will be corrected.

[0172] The method for identifying citations includes the following steps:

[0173] Step 43-1: Slice the generated content;

[0174] Step 43-2: Calculate the similarity between the sliced ​​text and the content segment based on a text similarity algorithm;

[0175] Step 43-3: Set the threshold to 0.8. If the calculated similarity is greater than 0.8, the content segment can be used as a basis for citation and is judged to be valid. If the calculated similarity is less than or equal to 0.8, the content segment cannot be used as a basis for citation and is judged to be invalid.

[0176] Step 5: Figure 4 As shown in the internal flow chart of the intelligent question-answering core component of an intelligent question-answering method based on vertical domain knowledge, the intelligent question-answering core component is constructed based on the fine-tuned LLM large language model and three data types. The intelligent question-answering core component includes a question-answer pair module, a knowledge graph module, and a retrieval enhancement generation module;

[0177] In the question-answer pair module, the question-answer pair data in the question-answer pair knowledge base is used to understand the user's question through LLM, and the question-answer pair database is searched for questions that are highly relevant to the user's question. The answers with high relevance are then judged by the large language model and directly returned to the user, and the answers are directly retrieved and output;

[0178] In the knowledge graph module, the service guide data in the knowledge graph database is used by LLM to understand the user's questions and retrieve the subgraphs with higher relevance in the knowledge graph. Based on the understanding of the questions, LLM will map the user's questions to the subgraph and get the answers by querying the paths in the subgraph. If it is found that nodes or edges on the graph are missing during the mapping process to the subgraph, the system will actively interact with the user to fully understand the user's intentions and get the answers through interactive retrieval.

[0179] In the retrieval enhancement generation module, the policy notification type files in the document knowledge base understand the user's questions through LLM, retrieve the file content related to the question in the document database, and conduct a comprehensive understanding so that it can correctly answer the user's questions and generate and output answers.

[0180] Step 51: Retrieve knowledge from the question-answer pair knowledge base. If candidate knowledge is retrieved, enter the question-answer pair module. In the question-answer pair module, the tuple data retrieved from the candidate knowledge and the user's question are constructed into a logical reasoning instruction; the tuple data includes a question and an answer;

[0181] Step 52: The large language model determines the correctness of the search result according to the instruction; if the judgment result is correct, the answer is output; if the judgment result is wrong, the search is performed from the knowledge graph knowledge base;

[0182] Step 53: Enter the knowledge graph module and construct the retrieved candidate knowledge, i.e., the subgraph, and the user's question into the intention understanding instruction of the knowledge graph;

[0183] Step 54: The large language model maps the user's intention to a subgraph of the knowledge graph according to the intention understanding instruction, and obtains the answer by searching on the subgraph;

[0184] Step 55: Determine whether it is necessary to interact with the user; if the subgraph information is complete and the answer can be obtained, the answer is output; if the subgraph information is incomplete, it is necessary to interact with the user and complete the missing information on the subgraph; if the answer is obtained on the subgraph after the information is complete, the answer is output; if the answer cannot be obtained on the subgraph after the information is complete, the search enhancement generation module will be entered;

[0185] Step 56: Enter the retrieval enhancement generation module and map the candidate knowledge, i.e., key phrases, back to the original long text paragraph through the small-to-big strategy;

[0186] Step 57: Construct the text paragraph and the user's question into a generation instruction;

[0187] Step 58: Verify the credibility of the citation through similarity post-processing, generate an answer based on a large language model, and output the answer.

[0188] The above implementation modes are not limitations of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the scope of the technical solution of the present invention also belong to the protection scope of the present invention.

Claims

1. An intelligent answering method based on vertical domain knowledge, characterized in that: The following steps are involved: Step S1: divide the data based on the vertical field data type, build several knowledge bases according to several data types, and save the data in the knowledge base; the knowledge base includes the question-answer pair knowledge base, the knowledge graph database and the document knowledge base; Step S2: construct LLM fine-tuning instructions based on the knowledge of the vertical field to understand the professional vocabulary of the vertical field and the content involved in the user's questions; LLM fine-tuning includes large language model instruction alignment training, embedding model training and additional citations; Step S3: Based on the data types divided in step S1 and the fine-tuning of LLM in step S2, construct the intelligent question-answering core component to output the answer; the intelligent question-answering core component includes a question-answer pair module, a knowledge graph module, and a retrieval enhancement generation module; The large language model instruction alignment training in step S2 includes the following steps: Step S21-1: Collect vertical field data and pre-process the vertical field data; Step S21-2: construct instructions for the preprocessed vertical field data, and construct them into logical reasoning instructions, knowledge graph intent understanding instructions, or generation instructions based on the nature of the data; Step S21-3: Adopt a low-rank adaptive method, and based on the internal low-rank characteristics of LLM, add a bypass matrix to simulate full parameter fine-tuning to achieve alignment training of knowledge graph intent understanding instructions, logical reasoning instructions, and generation instructions; the loss function of instruction alignment training is the cross entropy loss function, and the formula is: ； Where T is the length of the sequence; is the true output at time step t; The model predicts the output for time step t; Step S21-4: setting instruction alignment training parameters and performing instruction alignment training for a large language model; Embedding model training includes the following steps: Step S22-1: Collecting data in the general field and preprocessing the data in the general field; Step S22-2: Expand the preprocessed general domain data and preprocessed vertical domain data with positive and negative samples, divide the tasks according to the data source and data preprocessing method, and use the loss function of each task for mixed training; the tasks are divided into classification tasks, retrieval tasks, text similarity tasks, and text implication tasks; Step S22-3: assign weights to the loss functions corresponding to each task according to the amount of data after task division; the sum of the weighted coefficients of each loss function is 1 to prevent the embedding model from overfitting the data of a certain task; Step S22-4: setting embedding model training parameters and performing embedding model training; The task division in step S22-2 includes the following criteria:

1. The query text is classified into the classification task, and the category corresponding to the query text is taken as the positive sample, and a category corresponding to a non-query text is randomly selected as the negative sample; the loss function of the classification task is: ； Where N is the number of samples in a batch; text is the query text; The category corresponding to the query text is the positive sample; is the category corresponding to the non-query text, i.e., the negative sample; C is the set of all categories of the classification task; τ is the temperature coefficient; is the cosine similarity function; log is the logarithmic function; Second, the text content with a query text length of less than 128 characters and a positive sample character length of greater than or equal to 512 characters is divided into the retrieval task; the positive sample is the text paragraph corresponding to the query text, and the negative sample randomly selects another paragraph of the paragraph corresponding to the query text; the loss function of the retrieval task is: ； Where N is the number of samples in a batch; text is the query text; The paragraph corresponding to the query text is the positive sample; Other paragraphs of the paragraph where the query text is located are negative samples; is the data set of the current batch; τ is the temperature coefficient; is the cosine similarity function; log is the logarithmic function; 3. Based on the text content in the vertical field, the large language model generates different phrases; based on each phrase, two sentences are randomly generated based on the large language model, and the two sentences are scored for relevance, with the score ranging from 1 to 5; the text content is divided into text similarity tasks according to the text similarity task format; The loss function for the text similarity task is: ； in, is a hyperparameter; is the feature vector and The correlation score between is the cosine similarity function; log is the logarithmic function; 4. Based on the text content in the vertical field, different short sentences are generated through a large language model; the large language model generates three short sentences sentence1, sentence2, and sentence3 based on each short sentence sentence0; sentence1, sentence2, sentence3 and sentence0 are respectively in the relationship of implication, neutrality, and contradiction; the text content is divided into text implication tasks according to the text implication task format; The loss function for the text entailment task is: ； in, is sentence0; for sentence1; for sentence2; for sentence3; is a hyperparameter; Said The formula for the cosine similarity function is: ； Among them, x and y are both eigenvectors.

2. The intelligent answering method based on vertical domain knowledge according to claim 1 is characterized by: In step S1, the vertical field data is data of fixed reply content, and question-answer pairs are extracted based on the large language model LLM, and stored in the form of one question and one answer in the question-answer pair knowledge base; Vertical field data is structured knowledge data, which is extracted from the manually designed knowledge graph structure based on LLM and stored in the knowledge graph database; Slice the data of vertical field files and store them in the document knowledge base; Slice the paragraphs in the file and save the article corresponding to the paragraph; slice the sentence level in the file and save the paragraph corresponding to the sentence.

3. The intelligent answering method based on vertical domain knowledge according to claim 1 is characterized by: The logical reasoning instruction in step S21-2 is used to determine whether the retrieved question-answer pair can answer the user's question; the knowledge graph intention understanding instruction is used by the large language model to map the user's question to the subgraph; Generate instructions to generate answers from user questions and text paragraphs in a reasonable format.

4. The intelligent answering method based on vertical domain knowledge according to claim 1 is characterized in that: In step S2, the problem of inconsistent distribution of long text and short text in semantic space in embedding model training is solved based on the small-to-big strategy, including the following steps: Step S23-1: Summarize the long text based on the small-to-big strategy, extract the key phrases, and retain the mapping relationship between the key phrases and the long text; Step S23 - 2 : Retrieve key phrases and map the long text through the key phrases.

5. The intelligent answering method based on vertical domain knowledge according to claim 1 is characterized in that: In step S2, the large language model is added with citations, including the following steps: Step 24-1: The large language model will mark the paragraph-level content and article title of the generated content when answering the user's question according to the generated instructions; Step S24-2: To ensure the credibility of the citation, the generated citation will be judged. If the judgment is valid, the current citation will be kept; if it is invalid, the current citation will be modified; The methods for identifying citations include: Step S24-2-1: Slice the generated content; Step S24-2-2: Calculate the similarity between the sliced ​​text and the content segment based on a text similarity algorithm; Step S24-2-3: Set the threshold to 0.

8. If the calculated similarity is greater than 0.8, the content fragment can be used as a basis for citation and is judged to be valid; if the calculated similarity is less than or equal to 0.8, the content fragment cannot be used as a basis for citation and is judged to be invalid.

6. The intelligent answering method based on vertical domain knowledge according to claim 1 is characterized by: Outputting the answer based on the intelligent question-answering core component in step S3 includes the following steps: Step S31: Retrieve candidate knowledge from the question-answer pair knowledge base, and enter the retrieved candidate knowledge into the question-answer pair module; construct the tuple data retrieved from the candidate knowledge and the user's question into a logical reasoning instruction; Step S32: The large language model determines the correctness of the search result according to the logical reasoning instructions; if the search result is correct, the answer is output; if the search result is wrong, the candidate knowledge is retrieved from the knowledge graph database, and the retrieved candidate knowledge, i.e., the subgraph, enters the knowledge graph module; Step S33: construct the subgraph and the user's question into a knowledge graph intention understanding instruction; Step S34: The large language model maps the user's intention to the subgraph of the knowledge graph according to the knowledge graph intention understanding instruction, and obtains and outputs the answer by searching on the subgraph with complete information; if the information of the subgraph is incomplete, the system actively interacts with the user and completes the missing information on the subgraph, and obtains and outputs the answer; if the answer cannot be found on the subgraph with complete information, candidate knowledge is retrieved from the document knowledge base, and the retrieved candidate knowledge, i.e., the key phrase, enters the retrieval enhancement generation module; Step S35: The key phrase is mapped back to its own long text paragraph based on the small-to-big strategy, and the long text paragraph and the user's question are constructed into a generation instruction; Step S36: The large language model generates an answer according to the generation instruction; the credibility of the citation is verified based on the similarity post-processing method, and the answer is output.

Images