Training method and device of generative large language model based on knowledge base feedback

Abstract

The present invention discloses a training method and device for a generative large language model based on knowledge base feedback. The method utilizes a domain knowledge base and supervised fine-tuning training to optimize the large language model; adopts a reinforcement learning method based on knowledge base feedback, utilizes the domain knowledge base, constructs a reward model, scores and provides feedback on the answers generated by the large language model, and constitutes a reinforcement learning process. The innovation of the present invention is to apply knowledge graph technology to reward construction, so that the large language model can be fine-tuned through the automated process of knowledge engineering, replacing the reinforcement learning based on human feedback, which is conducive to saving a large amount of high cost of human feedback annotation. The correct answer to the domain application obtained based on deterministic knowledge reasoning can correct the key defect of the generative large language model in fabricating facts. It can be used in the scenario of constructing a large language model for industry vertical applications based on the domain knowledge graph, and has strong applicability.

Description

Technical Field

[0001] The present invention relates to the fields of knowledge engineering and large language model fine-tuning training, specifically to the fields of reinforcement learning and knowledge graph artificial intelligence technology, and more particularly to a method and device for training a generative large language model based on knowledge base feedback. Background Art

[0002] Large language models (LLMs) such as ChatGPT (Chat Generative Pre-trained Transformer) and GPT4 (Generative Pre-trained Transformer 4) can generate fluent human-like responses in the form of conversations for many downstream tasks, such as task-oriented dialogue and question answering. ChatGPT and GPT4 are chatbot programs developed by OpenAI.

[0003] Reinforcement Learning from Human Feedback (RLHF) optimizes language models based on human feedback using reinforcement learning. RLHF is an advanced AI system training method that combines reinforcement learning with human feedback. It is a method that creates a more robust learning process by incorporating the wisdom and experience of human trainers into the model training process. It involves using human feedback to create reward signals and then improving the model's behavior through reinforcement learning.

[0004] Supervised learning refers to the machine learning problem of learning a predictive model from labeled data. The labeled data represents the correspondence between input and output, and the predictive model produces the corresponding output for a given input. The essence of supervised learning is to learn the statistical laws that govern the mapping from input to output.

[0005] The RLHF method plays a key role in the training process of ChatGPT's large language model. Previously, traditional language models, primarily based on supervised training methods, typically only generate grammatically correct sentences but often fail to understand user intent, resulting in output that doesn't align with user expectations. To address this issue, we aim to introduce a method that uses human feedback to fine-tune the language model to better align with user intent. By fine-tuning the model using human feedback, we enable it to better follow various written instructions and generate more accurate, coherent, and readable output.

[0006] However, for engineering and algorithmic reasons, the cost of training and fine-tuning LLMs based on supervised learning or RLHF methods is too high, mainly because the required training set is huge (hundreds of millions to billions of documents, 10TB to 100TB of text), making manual annotation expensive. At the same time, when answering domain-specific questions, generative large language models trained on general language datasets inevitably suffer from hallucinations (i.e., nonsense) because the generative model's mechanism and the large language model only have knowledge of the current training corpus. For example, when asking questions such as "When was this company founded?", specific domain knowledge and real-time information are required, and cannot be generated simply by relying on the knowledge contained in the LLM parameters. They all need to rely on external knowledge bases to generate accurate results. Summary of the Invention

[0007] The purpose of the present invention is to provide a method and device for training a generative large language model based on knowledge base feedback to address the contradiction between the high cost of manual labeling in the existing technology and the key defect of large language models fabricating facts, which is in conflict with the need for accurate answers in field applications.

[0008] The objective of the present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a method for training a generative large language model based on knowledge base feedback, comprising the following steps:

[0009] (1) Using an open-source large language model as a basis, the large language model is self-supervised trained using unlabeled data to obtain a pre-trained large language model;

[0010] (2) extracting questions from the domain's question library, using prompts to retrieve the expected answers corresponding to the questions from the domain's knowledge base, and constructing a labeled dataset of the domain based on the questions and the expected answers corresponding to the questions; using the labeled data in the labeled dataset to perform supervised fine-tuning training on the pre-trained large language model obtained in step (1) to obtain a supervised fine-tuned large language model;

[0011] (3) using the questions in the labeled dataset of the domain constructed in step (2) to be input into the large language model trained by supervised fine-tuning obtained in step (2), obtaining multiple predicted answers and constructing multiple question-answer pairs together with the expected answers corresponding to the questions in the labeled dataset, and obtaining the topics to which the question-answer pairs belong; searching the questions in the knowledge graph of the domain to obtain matching semantic nodes; calculating similarity based on the matching semantic nodes and the multiple predicted answers and the expected answers, sorting the answer scores based on the similarity, and using the sorted answer scores to correct the reward model;

[0012] (4) The supervised fine-tuning large language model obtained in step (2) is trained with a proximal strategy optimization model in reinforcement learning based on the answer score output by the reward model obtained in step (3) to obtain a generative large language model.

[0013] Furthermore, the large language model includes the Lamma2 model of the autoregressive Transformer architecture, the Chinese-English bilingual pre-training model ChatGLM2-6B model, the Chinchilla-70B model and the PaLM-540B model.

[0014] Furthermore, in step (2), the process of extracting questions from the domain question library, retrieving the expected answers corresponding to the questions from the domain knowledge base using prompts, and constructing a domain labeled dataset based on the questions and the expected answers corresponding to the questions specifically includes:

[0015] According to the scenario of supervised fine-tuning training, questions are extracted from the question library of the domain, and the similar documents containing the question content are retrieved in the knowledge base of the domain through vector embedding. Then, the prompt of the large language model is used to embed the question and the retrieved similar documents into the prompt as the expected answer corresponding to the question; a labeled dataset of the domain is constructed based on the question and its corresponding expected answer. The style of the labeled dataset is question-answer pairs.<Question,Answer> .

[0016] Furthermore, the method for determining the expected answer corresponding to the question specifically includes:

[0017] First, determine whether the subject category of the question is included in the subject category scope pre-set in the domain knowledge base. If the subject category of the question is not included in the subject category scope pre-set in the domain knowledge base, manually supplement the expected answer corresponding to the question; if the subject category of the question is included in the subject category scope pre-set in the domain knowledge base, determine whether the expected answer corresponding to the question exists in the retrieved similar documents based on the data content pre-set in the domain knowledge base. If the expected answer corresponding to the question exists in the retrieved similar documents, determine the second matching degree between the question and the expected answer corresponding to the question in the similar documents. If the second matching degree is greater than or equal to the preset second matching degree threshold, directly use the expected answer corresponding to the question in the similar documents as the final expected answer corresponding to the question; if the second matching degree is less than the preset second matching degree threshold, manually correct the expected answer corresponding to the question in the similar documents and use it as the final expected answer corresponding to the question; if the expected answer corresponding to the question does not exist in the retrieved similar documents, use "I don't know" as the expected answer corresponding to the question.

[0018] Furthermore, the determination of whether the subject category of the question is included in the subject category category pre-set in the domain knowledge base specifically includes:

[0019] Find the entity node corresponding to the question in the domain knowledge base through the subject words in the question; the entity node includes four layers of key information: the first layer of key information is the question words, which are the obvious characteristics of the question; the second layer of key information is the question focus, which is the entity related to the question words; the third layer of key information is the subject words of the question, which are determined through named entity recognition; the fourth layer of key information is the central verb of the question, which is determined through part-of-speech tagging;

[0020] The question is converted into a question graph through the question words, question focus, question subject words and question central verb of the entity nodes;

[0021] The problem graph and the domain knowledge base graph are subjected to a first matching degree calculation through subgraph matching degree calculation, and it is determined whether the calculated first matching degree is less than a preset first matching degree threshold. If the first matching degree is less than the preset first matching degree threshold, it indicates that the subject category of the problem is not included in the subject category scope preset in the domain knowledge base; if the first matching degree is greater than or equal to the preset first matching degree threshold, it indicates that the subject category of the problem is included in the subject category scope preset in the domain knowledge base.

[0022] Furthermore, the step (3) includes the following sub-steps:

[0023] (3.1) Using the questions in the labeled dataset of the domain constructed in step (2), the questions are input into the large language model trained by supervised fine-tuning obtained in step (2), and multiple predicted answers are obtained. Together with the expected answers corresponding to the questions in the labeled dataset, multiple question-answer pairs for a question are formed;

[0024] (3.2) Using open domain datasets and domain datasets, we use knowledge extraction and fusion methods from knowledge engineering to build a domain knowledge graph. The domain knowledge graph contains the knowledge and entity category architecture.

[0025] (3.3) Obtain the entity category system in the knowledge graph that matches the domain through the topic, and obtain the topic to which the question and answer pair belongs;

[0026] (3.4) Search the entity category system of the constructed domain knowledge graph to determine whether the entity category system contains the topic to which the question-answer pair belongs. If the entity category system contains the topic to which the question-answer pair belongs, directly execute step (3.5); otherwise, directly execute step (3.6);

[0027] (3.5) Using a graph neural network model or a knowledge graph-based CLQA engine, search the domain knowledge graph for the question to obtain matching semantic nodes; calculate the semantic similarity between the matching semantic nodes and the expected answer and multiple predicted answers in the question-answer pair to obtain multiple similarities; sort the answer scores according to the similarities; use the sorted answer scores to modify the reward model to obtain a reward model trained based on the knowledge graph feedback;

[0028] (3.6) Send the question-answer pair to the user to manually determine the answer score and sort the answer scores; construct the best question-answer pair of question and answer based on the expected answer or predicted answer corresponding to the maximum score after sorting, and use the best question-answer pair to update the knowledge graph of the domain through human feedback.

[0029] Furthermore, in the step (3.5), sorting the answer scores according to similarity specifically includes:

[0030] Similarity is positively correlated with answer scores, and answer scores are sorted according to the size of similarity.

[0031] Furthermore, the step (4) specifically includes:

[0032] The predicted answer generated by the supervised fine-tuning trained large language model obtained in step (2) is input into the reward model obtained in step (3) to obtain an output answer score, and the answer score is added to the proximal policy optimization model in reinforcement learning for training, the policy parameters of the proximal policy optimization model are updated, and the supervised fine-tuning trained large language model is optimized according to the policy parameters to obtain a generative large language model.

[0033] A second aspect of an embodiment of the present invention provides a training device for a generative large language model based on knowledge base feedback, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned training method for a generative large language model based on knowledge base feedback.

[0034] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the program is used to implement the above-mentioned training method of a generative large language model based on knowledge base feedback.

[0035] The beneficial effects of the present invention are that the present invention combines the domain knowledge base with the large language model, reuses the existing knowledge base as training corpus, and performs model fine-tuning based on domain data in industry applications, providing a standard and feasible technical path for the application of large language models in specific fields that can build knowledge graphs; the present invention can greatly reduce costs, and based on the existing knowledge base of the enterprise or organization, it is beneficial to save the huge manual annotation costs of model fine-tuning when the large language model is applied in the field, as well as the cost of manual feedback sorting of results when establishing a reward model; the present invention combines the deterministic reasoning of knowledge engineering, and obtains accurate results by performing deterministic reasoning on the obtained knowledge graph, which helps to correct the key defects of the large language model's fabrication in order to obtain credible answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is an overall flow chart of the training method of a generative large language model based on knowledge base feedback of the present invention;

[0037] Figure 2 This is an example flow chart of the reinforcement learning with knowledge feedback (RLKF) method of the present invention;

[0038] Figure 3 A flow chart of a method for generating a high-quality labeled dataset in supervised fine-tuning training of the present invention;

[0039] Figure 4 Schematic diagram of the implementation process of the reward model in the present invention;

[0040] Figure 5 The figure is a structural diagram of the training device of the generative large language model based on knowledge base feedback of the present invention. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0042] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0043] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0044] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0045] See also Figure 1 The training method of the generative large language model based on knowledge base feedback of the present invention specifically includes the following steps:

[0046] (1) Using an open-source large language model as a basis, the large language model is self-supervised trained using unlabeled data to obtain a pre-trained large language model.

[0047] Furthermore, pre-trained large language pre-training models can be used, including the LLaMA2 model (model parameters range from 7B to 65B) with an autoregressive Transformer architecture published by Meta, the ChatGLM2-6B model, ChatGPT model, and GPT4 model, Chinese-English bilingual pre-training models published by Zhipu AI, etc. It should be understood that other open source large language models can also be used, such as Chinchilla-70B, PaLM-540B, the InternLM-7B model of the Shanghai Artificial Intelligence Laboratory's Shusheng Puyu large language model, the open source Chinese large model BELLE: Be Everyone's Large Languagemodel Engine, and the Baichuan-7B model of Baichuan Intelligence.

[0048] It should be noted that the data used to train the open source large language model is the general low-quality unlabeled corpus dataset corresponding to the large language model.

[0049] In the pre-training phase, the large language model is trained on large-scale general text data to learn the basic structure of the language and various common sense. The large language model pre-training is to predict the next word based on the previous word. Given a corpus of token series, Using the standard language model objective function, maximize the following likelihood function:

[0050]

[0051] Here, k is the context window size, and the conditional probability P uses a Transformer-structured neural network model with parameters θ, which are updated using stochastic gradient descent. Training stops after the iterative update step reaches the set number of training steps, or after the likelihood function value does not change by more than a set threshold between the nth and n-1th values of training for m consecutive times.

[0052] Typically, the pre-training phase of a large language model requires training data with trillions of tokens, context windows of 1K to 4K, thousands of GPU training cards, and months of training time.

[0053] (2) Extract questions from the domain's question library, use Prompt to retrieve the expected answers corresponding to the questions from the domain's knowledge base, and construct a labeled dataset of the domain based on the questions and the expected answers corresponding to the questions; use the labeled data in the labeled dataset to perform supervised fine-tuning (SFT) on the pre-trained large language model obtained in step (1) to obtain a supervised fine-tuned large language model.

[0054] It should be noted that the domain knowledge base participates in the process of obtaining the expected answer, which constitutes the labeler. The labeled data is trained with supervised fine-tuning to tune the large language model pre-trained in step (1).

[0055] In this embodiment, questions are extracted from the question library of the domain, and the expected answers corresponding to the questions are retrieved from the knowledge base of the domain using Prompt prompts. A labeled dataset of the domain is constructed based on the questions and the expected answers corresponding to the questions. Specifically, the following steps are performed: questions are extracted from the question library of the domain according to the scenario of supervised fine-tuning training as needed, and similar documents containing the question content are retrieved in the knowledge base of the domain through vector embedding. Then, the questions and the retrieved similar documents are embedded into the prompts using the Prompt prompts of the large language model as the expected answers corresponding to the questions; a labeled dataset of the domain is constructed based on the questions and the expected answers corresponding to the questions. The style of the labeled dataset is question-answer pairs.<Question,Answer> ,like Figure 3It should be noted that supervised fine-tuning training is used to transform the large language model from autoregressive prediction to dialogue form.

[0056] It should be understood that the current step (2) uses the knowledge base and knowledge graph to replace or partially replace the human feedback in the RLHF. Through the domain-specific knowledge base and knowledge graph reasoning engine, a process for optimizing the training of the large language model in a specific domain is obtained. Combining the domain knowledge base with the large language model, reusing the existing knowledge base as training corpus, and fine-tuning the model based on domain data in industry applications provides a standard and feasible technical path for the application of large language models in specific domains where knowledge graphs can be constructed. Based on the existing knowledge base, it is helpful to save the huge manual annotation cost of fine-tuning the model when applying the large language model in the field.

[0057] Furthermore, the method for determining the expected answer corresponding to the question specifically includes: judging whether the subject category of the question is included in the subject category category pre-set in the knowledge base of the domain, and finding the entity node corresponding to the question in the knowledge base of the domain through the subject words in the question, wherein the entity node includes four layers of key information. The first layer of key information is the question words, such as who, when, what, where, how, which, why, whom, whose, which are obvious features of the question; the second layer of key information is the focus of the question, such as name, time, place, extracting entities related to the question words; the third layer of key information is the subject words of the question, such as "meteorology and geology", which can be determined through named entity recognition (NER), and a question may have multiple subject words; the fourth layer of key information is the central verb of the question, which can be determined through part-of-speech (POS) recognition. Through the question words, question focus, question subject words and question central verb, the question can be converted into a question graph (QuestionGraph), and the question graph and the domain knowledge base graph are matched by subgraph matching calculation to perform a first matching calculation, and judge whether the calculated first matching is less than the preset first matching threshold. If the first matching is less than the preset first matching threshold, it means that the question exceeds the subject category of the domain knowledge base and no answer can be matched. It is necessary to manually supplement the expected answer corresponding to the question; if the first matching of the subject category of the question graph and the domain knowledge base graph is greater than or equal to the preset first matching threshold, the question corresponding to the question is judged according to the pre-set data content of the domain knowledge base. Whether the expected answer to the question exists in the retrieved similar documents. If the expected answer corresponding to the question exists in the retrieved similar documents, the second matching degree between the question and the expected answer corresponding to the question in the similar documents is judged. If the second matching degree is greater than or equal to the preset second matching degree threshold, the expected answer corresponding to the question in the similar documents is directly used as the final expected answer corresponding to the question; if the second matching degree is less than the preset second matching degree threshold, the expected answer corresponding to the question in the similar documents is manually corrected and used as the final expected answer corresponding to the question; if the expected answer corresponding to the question does not exist in the retrieved similar documents, "I don't know" is used as the expected answer corresponding to the question.

[0058] Furthermore, for the calculation of matching degree, an optional mode is to convert finding the correct answer among the candidate answers into a binary classification problem. Use the training data question-answer pairs to train a classifier to find the correct answer. The input feature vector of the classifier is composed of the question-candidate answer features. When in use, for each candidate answer, extract its features (assuming there are k features), and then combine them with each feature in the question (assuming there are m features), then k*m question-candidate answer features are obtained. Therefore, the input vector is a k*m-hot vector (that is, the k*m dimension is 1, and the other dimensions are 0). Logistic regression with L1 regularization can be used as a classifier to train a classifier of question-candidate answer features.

[0059] It should be understood that the greater the degree of match between a question and its corresponding expected answer, the better the question-answer<Question,Answer> The better the quality.

[0060] In this embodiment, the question-answer pairs in the constructed labeled dataset are used.<Question,Answer> The supervised fine-tuning training method is used to further train the pre-trained large language model. The SFT supervised fine-tuning training process is as follows:

[0061] For labeled datasets<Question,Answer> , which can be regarded as an input token sequence x 1 ,...,x m , corresponding to the label y. The input sequence is obtained through the pre-training model to obtain the activation coefficient in the Transformer network structure block By comparing with the coefficient W y Perform matrix multiplication to obtain the linear output layer to predict y and obtain the probability P of predicting y, which is expressed as:

[0062]

[0063] Therefore, the goal is to maximize the following function:

[0064]

[0065] In order to make the large language model generalizable and accelerate convergence, the following objective function is actually used:

[0066]

[0067] Among them, λ is the weight.

[0068] Then use the stochastic gradient descent method to train the parameters W yTraining stops when the iterative update step reaches the set number of training steps, or when the objective function value between the nth and n-1th training steps does not change by more than the set threshold for the objective function for m consecutive times. The result is a large language model trained through supervised fine-tuning, which is a proprietary large language model for domain adaptation.

[0069] (3) Use the questions in the labeled dataset of the domain constructed in step (2) and input them into the supervised fine-tuned large language model obtained in step (2) to obtain multiple predicted answers (Answer), and jointly construct multiple question-answer pairs with the expected answers corresponding to the questions in the labeled dataset, and obtain the topics to which the question-answer pairs belong; search the questions in the knowledge graph of the domain to obtain matching semantic nodes; calculate the similarity based on the matching semantic nodes and multiple predicted answers and expected answers, sort the answer scores based on the similarity, and use the sorted answer scores to modify the reward model (Reward Model), such as Figure 4 shown.

[0070] (3.1) Use the questions in the labeled dataset of the domain constructed in step (2) to input into the large language model trained by supervised fine-tuning obtained in step (2) to obtain multiple predicted answers, which together with the expected answers corresponding to the question in the labeled dataset constitute multiple question-answer pairs for a question, such as Figure 2 As shown in the figure, for example, the question in the labeled dataset in the geological field is "What is geology?", and the expected answer corresponding to this question in the labeled dataset is C. Through supervised fine-tuning, the trained large language model obtains three predicted answers A, B, and D. The expected answer C and multiple predicted answers A, B, and D together constitute multiple question-answer pairs for a question.

[0071] (3.2) Using high-quality open domain datasets and domain-specific datasets, and using knowledge extraction methods and knowledge fusion methods in knowledge engineering, a knowledge graph of the domain is constructed. The knowledge graph of the domain contains knowledge and entity category architecture.

[0072] It should be noted that high-quality open domain datasets and domain-specific datasets can be downloaded from open source websites, such as papers, patents, standards, reports and other documents, web pages, news, information, encyclopedias, Wikipedias, books and other texts, or open source websites such as GitHub, etc.

[0073] It should be understood that knowledge graphs can be constructed using knowledge extraction methods and knowledge fusion methods, which are commonly used methods in knowledge engineering.

[0074] (3.3) The entity category system in the knowledge graph that matches the domain is obtained through the topic, and the topic to which the question and answer pair belongs is obtained. For example, the question "When was our company established?" belongs to the topic of "XX Company", and "What is space geology?" belongs to the secondary topic of space geology in the geology topic.

[0075] (3.4) Search the entity category system of the constructed domain knowledge graph to determine whether the entity category system contains the topic to which the question-answer pair belongs. If the entity category system contains the topic to which the question-answer pair belongs, directly execute step (3.5); otherwise, directly execute step (3.6).

[0076] (3.5) Using a graph neural network (GNN) model or a knowledge graph-based CLQA (Complex Logical Question Answering) engine, the question is retrieved from the domain knowledge graph to obtain matching semantic nodes; the semantic similarity between the matching semantic nodes and the expected answer and multiple predicted answers in the question-answer pair is calculated to obtain multiple similarities; the answer scores are sorted according to the similarities; the reward model is modified using the sorted answer scores, and finally a reward model is obtained based on knowledge graph feedback training.

[0077] It should be understood that the GNN model is a commonly used model. The GNN model or CLQA engine can be used to search for questions in the knowledge graph and obtain matching semantic nodes.

[0078] Furthermore, the reward model accepts text input and outputs a score. Specifically, the text input of the reward model here is different answers given to the same question, and the output is a score of the answer quality. The reward model (RewardModel) removes the softmax layer of the last layer of the supervised micro-trained large language model, replacing it with a linear layer. The RM model inputs are the question and answer, and the output is a scalar, the answer score.

[0079] Furthermore, semantic similarity is a very common method for calculating similarity. By calculating the distance between words, which is reflected in the knowledge graph as the distance between nodes, the distance is converted into similarity. A long distance indicates a small similarity, and a close distance indicates a large similarity.

[0080] Furthermore, when sorting the answer scores based on similarity, specifically including: similarity is positively correlated with answer scores, and answer scores can be sorted based on the size of the similarity, that is, the answer scores corresponding to the expected answer or predicted answer with greater similarity are higher, and the answer scores corresponding to the expected answer or predicted answer with less similarity are lower. For example, answers with low similarity, opposite semantics, or answers not included in the knowledge graph will be given low scores, such as Figure 4 shown.

[0081] Furthermore, since the input of the reward model is a text, that is, the expected answer or predicted answer corresponding to the question, and the output is the corresponding answer score, after sorting, the answer score ranking corresponding to the output is obtained; after sorting the answer scores according to similarity, the answer score ranking corresponding to the similarity is obtained. The answer score ranking corresponding to the similarity may be different from the answer score ranking corresponding to the output. At this time, the reward model needs to be corrected according to the answer score ranking corresponding to the similarity, and finally a reward model based on the knowledge graph can be obtained.

[0082] The Pairwise Ranking Loss of the reward model is as follows:

[0083]

[0084] Among them, D is a dataset that sorts answers through knowledge graph feedback; x is the question in D, y∈{y0,y1} is the two answers selected from K answers, y i is a better answer, r θ (x, y) is the RM model to be trained, and the scalar score obtained for the input x and y. The training process is the same as the training method for supervised fine-tuning of the large language model in step (2). The parameters θ are trained using stochastic gradient descent. Training is stopped after the iterative update step reaches the set number of training times, or when the loss function does not change by more than the set loss threshold between the nth and n-1th values of the training for m consecutive times.

[0085] (3.6) Send the question-answer pair to the user to manually determine the answer score and sort the answer scores; construct the best question-answer pair of question and answer based on the expected answer or predicted answer corresponding to the maximum score after sorting, and use the best question-answer pair to update the knowledge graph of the domain through human feedback.

[0086] It should be understood that the current step (3) uses the knowledge base and knowledge graph to replace or partially replace the human feedback in the RLHF. Through the knowledge base and knowledge graph reasoning engine of a specific domain, a process for optimizing the training of a large language model in a specific domain is obtained. Combining the domain knowledge base with the large language model, reusing the existing knowledge base as training corpus, and fine-tuning the model based on domain data in industry applications provides a standard and feasible technical path for the application of large language models in specific domains where knowledge graphs can be constructed. Based on the existing knowledge base, it is helpful to save the cost of manually sorting the feedback results when establishing a reward model. Deterministic reasoning through the obtained knowledge graph to obtain accurate results helps to correct the key defects of the large language model's fabrication so as to obtain credible answers.

[0087] (4) The supervised fine-tuning large language model obtained in step (2) is trained with the Proximal Policy Optimization (PPO) model in reinforcement learning based on the answer score output by the reward model obtained in step (3) to obtain a generative large language model.

[0088] It should be understood that the proximal policy optimization model is a commonly used method in reinforcement learning, which can achieve small batch updates during training and solve the problem of difficult step size determination.

[0089] Specifically, the predicted answer generated by the supervised fine-tuning trained large language model obtained in step (2) is input into the reward model obtained in step (3) to obtain the output answer score rk, and the answer score is added to the proximal policy optimization model in reinforcement learning for training, the policy parameters of the proximal policy optimization model are updated, and the supervised fine-tuning trained large language model is optimized according to the policy parameters to obtain a generative large language model.

[0090] The overall reward-KL penalty value during training can be written as follows

[0091]

[0092] Among them, β is the weight coefficient, and the strategy learned in reinforcement learning and the original supervised fine-tuning model π SFT Calculate the KL divergence between the two and get the above formula. The purpose is to make the gap between the RL model and the previous model as small as possible. The scoring part r θ (x, y) makes the new model data score higher. During the reinforcement learning process, the SFT model and RM model in the previous two steps participate in the loop training, updating the parameters θ to obtain an RL policy model.

[0093] Typically, reinforcement learning sets a fixed number of training times, such as 1,000 or 10,000 times. If the overall reward function does not change by more than the set threshold between the nth and n-1th values of training for m consecutive times, it indicates that the model is stable.

[0094] It should be understood that the current step (4) creates a reinforcement learning loop where, in each training episode, the large language model takes several prompts from the training dataset and generates text; its output is then passed to the reward model, which provides a score to assess its consistency with human preferences; after updating, the large language model is subsequently updated to create outputs that score higher in the reward model.

[0095] The present invention uses the reinforcement learning with knowledge feedback (RLKF) method instead of the reinforcement learning from human feedback (RLHF) method. Based on high-quality open domain data or domain-specific databases, a domain-specific knowledge graph is constructed through knowledge extraction and knowledge fusion methods. A reward model used to generate answer scores for a large language model is constructed through knowledge graph pre-training. Different answer scores are given to different training results based on the training of the large language model. The reinforcement learning method optimized by the PPO proximal strategy is used to optimize the results so that the reward model obtains the maximum score. A large language model for vertical field applications, such as task-oriented dialogue and question answering, is obtained through the RLKF method. When applied in vertical fields, query questions are processed through vector embedding and prompt engineering processes, combined with knowledge base similar document search and matching, to obtain answers based on the knowledge base, thereby avoiding the defect of fabricating facts.

[0096] See also Figure 5 An embodiment of the present invention provides a training device for a generative large language model based on knowledge base feedback, comprising one or more processors and a memory coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the training method for a generative large language model based on knowledge base feedback in the above embodiment.

[0097] The embodiment of the training device of the generative large language model based on knowledge base feedback of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 5 As shown, this is a hardware structure diagram of any device with data processing capability where the training device of the generative large language model based on knowledge base feedback of the present invention is located. Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0098] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0099] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0100] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the training method of the generative large language model based on knowledge base feedback in the above embodiment is implemented.

[0101] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for training a generative large language model based on knowledge base feedback, characterized in that: The following steps are involved: (1) Using an open-source large language model as a basis, the large language model is self-supervised trained using unlabeled data to obtain a pre-trained large language model; (2) Extract questions from the domain's question library, use prompts to retrieve the expected answers corresponding to the questions from the domain's knowledge base, and build a labeled dataset for the domain based on the questions and the expected answers corresponding to the questions; Using the labeled data in the labeled data set to perform supervised fine-tuning training on the pre-trained large language model obtained in step (1) to obtain a supervised fine-tuned large language model; (3) using the questions in the labeled dataset of the domain constructed in step (2) to be input into the large language model trained by supervised fine-tuning obtained in step (2), obtaining multiple predicted answers and constructing multiple question-answer pairs together with the expected answers corresponding to the questions in the labeled dataset, and obtaining the topics to which the question-answer pairs belong; searching the questions in the knowledge graph of the domain to obtain matching semantic nodes; calculating similarity based on the matching semantic nodes and the multiple predicted answers and the expected answers, sorting the answer scores based on the similarity, and using the sorted answer scores to correct the reward model; (4) The supervised fine-tuning large language model obtained in step (2) is trained with a proximal strategy optimization model in reinforcement learning based on the answer score output by the reward model obtained in step (3) to obtain a generative large language model.

2. The method for training a generative large language model based on knowledge base feedback according to claim 1, characterized in that: The large language model includes the Lamma2 model with an autoregressive Transformer architecture, the Chinese-English bilingual pre-training model ChatGLM2-6B model, the Chinchilla-70B model, and the PaLM-540B model.

3. The method for training a generative large language model based on knowledge base feedback according to claim 1, characterized in that: In step (2), questions are extracted from the question library of the domain, and the expected answers corresponding to the questions are retrieved from the knowledge base of the domain using prompts. The labeled dataset of the domain is constructed based on the questions and the expected answers corresponding to the questions, which specifically includes: According to the scenario of supervised fine-tuning training, questions are extracted from the question library of the domain, and the similar documents containing the question content are retrieved in the knowledge base of the domain through vector embedding. Then, the prompt of the large language model is used to embed the question and the retrieved similar documents into the prompt as the expected answer corresponding to the question; a labeled dataset of the domain is constructed based on the question and its corresponding expected answer. The style of the labeled dataset is question-answer pairs.<Question,Answer> .

4. The method for training a generative large language model based on knowledge base feedback according to claim 3, characterized in that: The method for determining the expected answer corresponding to the question specifically includes: First, determine whether the subject category of the question is included in the subject category scope pre-set in the domain knowledge base. If the subject category of the question is not included in the subject category scope pre-set in the domain knowledge base, manually supplement the expected answer corresponding to the question; if the subject category of the question is included in the subject category scope pre-set in the domain knowledge base, determine whether the expected answer corresponding to the question exists in the retrieved similar documents based on the data content pre-set in the domain knowledge base. If the expected answer corresponding to the question exists in the retrieved similar documents, determine the second matching degree between the question and the expected answer corresponding to the question in the similar documents. If the second matching degree is greater than or equal to the preset second matching degree threshold, directly use the expected answer corresponding to the question in the similar documents as the final expected answer corresponding to the question; if the second matching degree is less than the preset second matching degree threshold, manually correct the expected answer corresponding to the question in the similar documents and use it as the final expected answer corresponding to the question; if the expected answer corresponding to the question does not exist in the retrieved similar documents, use "I don't know" as the expected answer corresponding to the question.

5. The method for training a generative large language model based on knowledge base feedback according to claim 4, characterized in that: The determination of whether the subject category of the question is included in the subject category category pre-set in the domain knowledge base specifically includes: Find the entity node corresponding to the question in the domain knowledge base through the subject words in the question; the entity node includes four layers of key information: the first layer of key information is the question words, which are the obvious characteristics of the question; the second layer of key information is the question focus, which is the entity related to the question words; the third layer of key information is the subject words of the question, which are determined through named entity recognition; the fourth layer of key information is the central verb of the question, which is determined through part-of-speech tagging; The question is converted into a question graph through the question words, question focus, question subject words and question central verb of the entity nodes; The problem graph and the domain knowledge base graph are subjected to a first matching degree calculation through subgraph matching degree calculation, and it is determined whether the calculated first matching degree is less than a preset first matching degree threshold. If the first matching degree is less than the preset first matching degree threshold, it indicates that the subject category of the problem is not included in the subject category scope preset in the domain knowledge base; if the first matching degree is greater than or equal to the preset first matching degree threshold, it indicates that the subject category of the problem is included in the subject category scope preset in the domain knowledge base.

6. The method for training a generative large language model based on knowledge base feedback according to claim 1, characterized in that: The step (3) includes the following sub-steps: (3.1) Using the questions in the labeled dataset of the domain constructed in step (2), the questions are input into the large language model trained by supervised fine-tuning obtained in step (2), and multiple predicted answers are obtained. Together with the expected answers corresponding to the questions in the labeled dataset, multiple question-answer pairs for a question are formed; (3.2) Using open domain datasets and domain datasets, we use knowledge extraction and fusion methods from knowledge engineering to build a domain knowledge graph. The domain knowledge graph contains the knowledge and entity category architecture. (3.3) Obtain the entity category system in the knowledge graph that matches the domain through the topic, and obtain the topic to which the question and answer pair belongs; (3.4) Search the entity category system of the constructed domain knowledge graph to determine whether the entity category system contains the topic to which the question-answer pair belongs. If the entity category system contains the topic to which the question-answer pair belongs, directly execute step (3.5); otherwise, directly execute step (3.6); (3.5) Using a graph neural network model or a knowledge graph-based CLQA engine, search the domain knowledge graph for the question to obtain matching semantic nodes; calculate the semantic similarity between the matching semantic nodes and the expected answer and multiple predicted answers in the question-answer pair to obtain multiple similarities; and sort the answer scores based on the similarities; Use the ranked answer scores to modify the reward model and obtain a reward model trained based on knowledge graph feedback. (3.6) Send the question-answer pair to the user to manually determine the answer score and sort the answer scores; construct the best question-answer pair of question and answer based on the expected answer or predicted answer corresponding to the maximum score after sorting, and use the best question-answer pair to update the knowledge graph of the domain through human feedback.

7. The method for training a generative large language model based on knowledge base feedback according to claim 6, characterized in that: In the step (3.5), sorting the answer scores according to similarity specifically includes: Similarity is positively correlated with answer scores, and answer scores are sorted according to the size of similarity.

8. The method for training a generative large language model based on knowledge base feedback according to claim 1, characterized in that: The step (4) specifically includes: The predicted answer generated by the supervised fine-tuning trained large language model obtained in step (2) is input into the reward model obtained in step (3) to obtain an output answer score, and the answer score is added to the proximal policy optimization model in reinforcement learning for training, the policy parameters of the proximal policy optimization model are updated, and the supervised fine-tuning trained large language model is optimized according to the policy parameters to obtain a generative large language model.

9. A training device for a generative large language model based on knowledge base feedback, comprising one or more processors and a memory, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the training method of a generative large language model based on knowledge base feedback according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that A program is stored thereon, which, when executed by a processor, is used to implement the training method of a generative large language model based on knowledge base feedback as described in any one of claims 1-8.

Images