Construction method of patent large model question answering system based on map enhancement generation

Abstract

The invention relates to a method for constructing a patent large model question-answering system based on map enhancement generation, which adopts an incremental pre-training strategy, integrates a large number of patent abstract texts into an LLM learning process, and can more accurately call and generate related contents in subsequent question-answering interaction. In order to further transform the text continuous writing type LLM into a form suitable for executing an accurate question and answer task and improve the understanding and following degree of the LLM on a specific instruction, an instruction fine tuning technology is applied, targeted optimization is carried out on the model through a massive instruction data set, and the adjusted LLM can accurately meet the knowledge answering requirement needed by a user.

Description

Technical Field

[0001] The present invention relates to a method for constructing a patent large model question-answering system based on graph enhancement generation. Background Art

[0002] Since the release of ChatGPT, Large Language Model (LLM) has gradually entered the public eye and has also become a key area of ​​academic research. As a general artificial intelligence, LLM can solve a variety of NLP downstream tasks, including text summarization, information extraction, translation, question and answer, etc. Through instruction fine-tuning and reinforcement learning based on human feedback, it can be more inclined to generate answers that are more inclined to human preferences in the field of question and answer. Although LLM can solve a variety of downstream problems, in some vertical fields, such as medical, legal, and financial, due to its lack of corresponding knowledge and the existence of hallucination problems, further research and training are needed. Although there has been considerable research on vertical fields, there is still little research on LLM in the field of patents. Therefore, this patent intends to explore the application of LLM in the field of question and answer in the field of patents.

[0003] The training of LLM can be roughly divided into three aspects: pre-training, supervised fine tuning (SFT), and reinforcement learning. In the pre-training stage, by injecting a large amount of text data, LLM can master a large amount of domain knowledge. At present, the academic community generally believes that this stage can be regarded as a kind of knowledge reserve, so a large amount of corpus is required. In the SFT stage, by fine-tuning on some labeled data, the model's question-answering ability can be effectively improved. For example, the instruction fine-tuning technology uses a small amount of instruction data to enable LLM to have a certain ability to follow instructions. Even if the instruction type has never appeared in the training set, LLM can understand and execute commands, showing a certain general intelligence. In the reinforcement learning stage, the model generates content that humans like more mainly through reinforcement learning RLHF based on human feedback.

[0004] Currently, the LLM field has produced many excellent large model examples, such as OPENAI's GPT-4, Zhipu Technology's ChatGLM, Alibaba Cloud's Qwen series models, and Shusheng Puyu's Intern2, etc. There are also many outstanding works in specific vertical fields, such as Code-LLAMA, which specializes in programming applications, and Chat-Doctor, which serves medical consultation.

[0005] In the research work of patent knowledge graph, most of the knowledge graphs use graph databases such as Neo4j as the underlying storage mechanism. Although this technical solution can efficiently mine and associate related entities and relationships with data query languages ​​such as Cypher, it cannot be ignored that non-professional users may find it difficult to use such professional search methods, which in turn limits their effective acquisition of patent cooperation information resources. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for constructing a patent large model question-answering system based on graph enhancement generation, thereby lowering the technical threshold for ordinary users to obtain relevant information.

[0007] The technical solution adopted by the present invention is: A method for constructing a patent large model question answering system based on graph enhancement generation, characterized in that it includes the following steps: S01: Incremental pre-training, using the Intern2-1.8b-base model and using patent abstracts as the dataset for incremental pre-training; S02: Instruction fine-tuning. In the patent field question-answering scenario, LLM is used to deal with patent-related questions. This scenario requires accurate recall of triple information closely related to the question from the patent knowledge graph. To ensure that the model can effectively understand these structured triple texts, instruction fine-tuning is used to guide the model to understand the instructions related to the triples. The Qwen-72b model is used to construct patent instruction fine-tuning data, which includes questions, triples, and answers, and is used to train the large model to use patent knowledge graph triples for patent question-answering. S03: Retrieval-enhanced generation, using patent knowledge graphs and vector recall to obtain the triples most relevant to user questions, and then inject user questions to allow the large model to obtain patent knowledge and generate answers.

[0008] Furthermore, in the incremental pre-training process, the patent abstract dataset is first processed, and the patent data is deduplicated using Minhash combined with the LSH algorithm.

[0009] Furthermore, when constructing the instruction fine-tuning database, the Cypher query language is used to randomly extract triple instances under each type of relationship from the patent knowledge graph. The Qwen-27B-chat model is used to generate a question for the subject, relationship and object for the extracted triples, and each component corresponding to the triple constitutes the answer to the corresponding question. Each triple generates 3 question-answer pairs. After data cleaning, the corresponding training set, validation set and test set are obtained.

[0010] Furthermore, a general-purpose Chinese instruction set is introduced in the process of constructing the instruction set.

[0011] Furthermore, in the retrieval enhancement generation process, encoding technology is used to encode and store each triple in the patent knowledge graph. When the user asks a question, the question is encoded in the same way, and a similarity matching algorithm is used to search for the triple that is most appropriate to the question. The system then merges the matched triple with the original question into a new question input, and then guides the generation of answer content in the LLM.

[0012] Furthermore, before performing similarity matching calculation, the structured triple information in the knowledge graph and the natural language questions raised by the user are mapped to the same continuous vector space.

[0013] Furthermore, a template-based triple translation method is used to transform triple data into unstructured text content; The triple translation method comprises: Build a template system to convert each triple structure into an understandable textual representation. In the process of building the template, the subject and object entity class information is included. Specifically, the entity class text is suffixed after the entity to ensure that the converted text fully reflects the intrinsic semantic characteristics of the original triple. For different relationship types, ChatGPT is used to obtain the synonymous vocabulary set of the corresponding relationship, thereby creating a series of different ways to describe the same relationship. When building an instance, a relationship expression word in the set is randomly selected and incorporated into the text generated by the template. After completing the conversion from triples to texts, these texts are vectorized: BGE encoding technology is used to convert text information into high-dimensional vector representation, thereby constructing a comprehensive set of triple vectors.

[0014] Furthermore, when a user asks a query question, the BGE encoding technology is used to vectorize the question. Then, with the help of the mathematical tool of cosine similarity, the similarity between the question vector and the triple vectors stored in the library is measured, and the triple that best matches the question vector and has the highest similarity is accurately identified. This highly similar triplet is selected as the key retrieval result as the basis for guiding the large language model to generate accurate answers.

[0015] The positive effect of the present invention is that the present invention constructs a patent large model question and answer system, and users can obtain relevant information of the patent through a simple question and answer format, which greatly reduces the technical threshold for ordinary users to obtain relevant information.

[0016] The present invention adopts an incremental pre-training strategy to integrate a large amount of patent abstract text into the learning process of LLM, so that relevant content can be more accurately called and generated in subsequent question-and-answer interactions. In order to further transform the text-continuation LLM into a form suitable for performing precise question-and-answer tasks and improve its understanding and compliance with specific instructions, the instruction fine-tuning technology is used to optimize the model with a massive instruction data set. The adjusted LLM can accurately meet the knowledge answer needs of users. Taking into account the factual accuracy challenges that large models may face, the so-called hallucination problem, the present invention adopts knowledge graph retrieval enhancement technology. When a user asks a question to the model, the system uses a vector similarity retrieval method to match the triple information that is most appropriate to the question from the knowledge graph, and cleverly integrates it into the context environment, ensuring that the LLM can effectively combine and utilize the actual data in the patent knowledge graph in the process of generating answers, thereby providing both real and effective response content. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Construct a flow chart for the present invention. DETAILED DESCRIPTION

[0018] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0019] As attached Figure 1 As shown, the present invention provides a method for constructing a patent large model question-answering system based on graph enhancement generation, aiming to lower the technical threshold for ordinary users to obtain relevant information.

[0020] The specific method is as follows: S01: Incremental Pre-training In order to balance training efficiency and model performance, this patent selected a model with 2B parameters for incremental pre-training research. The Intern2-1.8b-base model was used to perform incremental pre-training steps using the unique dataset of patent abstracts.

[0021] In terms of data processing, in order to further improve the quality and purity of training data, first of all, the efficient data deduplication solution proposed by slimpajam is adopted, and Minhash combined with the LSH algorithm is used to perform in-depth and comprehensive duplicate removal of patent data; secondly, a series of rule-based data cleaning methods can also be implemented to purify the content of the data set.

[0022] In order to comprehensively test the memory and comprehension capabilities of the trained model, the present invention also discloses an evaluation method that uses two complementary methods: a memory reproduction evaluation based on ROUGE_L and a patent multiple-choice question parsing benchmark test.

[0023] ROUGE_L-based memory reproduction evaluation: First, in order to explore the model's ability to remember the injected patent abstract text, the present invention adopts a method that uses the ROUGE_L score as a measurement indicator. Specifically, text samples randomly selected from the training set were selected as the test set. For each sample, the first 16 characters or the first half of the text are first provided to the model, and then the model is required to generate the remaining content. Next, the ROUGE_L score between the original text fragment and the model-generated fragment is calculated to quantitatively evaluate the model's memory effect on the patent knowledge stored in the training set.

[0024] Patent multiple-choice question parsing benchmark: To deeply examine the model's ability to understand patent knowledge, a patent multiple-choice question evaluation benchmark was constructed. The creation process of this benchmark includes using the powerful Qwen-72B-chat model to generate challenging question-answer pairs.

[0025] In the specific operation, the randomly selected patent abstract text is input into Qwen-72B-chat, and a prompt is set to require the model to generate question-answer pairs. Then a question bank and an answer bank are generated. After that, three interference options randomly selected from other answer banks are configured for each question, and after data cleaning, a test data set of patent multiple-choice questions is formed. Finally, these designed multiple-choice questions and their options are input into the tested model, and by analyzing the logits value output by the last layer of the model, the option with the highest probability is selected as the answer predicted by the model. It is expected that this method can accurately reflect the depth of cognition and accuracy of the trained model's understanding of the substantive content of the patent text.

[0026] Patent knowledge is injected into LLM through an incremental pre-training method, and the generation ability of the trained model is verified by two evaluation methods.

[0027] S02: Command fine-tuning In the patent field question-answering scenario of the present invention, LLM is used to deal with complex patent-related issues. This scenario requires accurate recall of triple information closely related to the question from the patent knowledge graph. To ensure that the model can effectively understand these structured triple texts, this study uses instruction fine-tuning to guide the model to understand instructions related to triples.

[0028] Regarding the construction of instruction data, first, with the help of Cypher query language, triple instances under each type of relationship are randomly extracted from the patent knowledge graph. For the extracted triples, the Qwen-27B-chat model is used to generate a question for the subject, relationship, and object respectively, and each component corresponding to the triple constitutes the answer to the corresponding question. In this way, each triple generates 3 question-answer pairs, and the corresponding training set, verification set, and test set are obtained after data cleaning. These question-answer pairs consist of three key elements: instructions (i.e., questions generated by LLM), input (triple content), and output (specific part of the triple). The prompts set in this process are as follows: "You are now a knowledge graph expert and linguistics expert.\nI will give you a triple, which is a concept in the knowledge graph, and contains three parts: <subject, relationship, object>;\nI hope you can help me generate three questions based on this triple, asking about the three parts of the triple respectively. For example:\nInput: <[Patent] Refrigeration and heating system and its terminal device, [Relationship] Advantages, [Advantage] Uniform indoor temperature>\nOutput:\n[\n{"type": "Query object", "question": "What are the advantages of the refrigeration and heating system and its terminal device", "answer": "Uniform indoor temperature"},\n{"type": "Query relationship", "question": "What is the relationship between the refrigeration and heating system and its terminal device and the uniform indoor temperature", "answer": "Advantages"},\n{"type": "Query subject", "question": "What patent has the advantage of uniform indoor temperature", "answer": "Refrigeration and heating systems and their terminal devices"},\n]\n#############The following is a triple. Please help me generate three questions, asking about the three parts of the triple respectively###############\n" .

[0029] In addition, given that existing research has pointed out that the diversity of instruction types is crucial to improving model performance, the present invention further introduces a general-purpose Chinese instruction set in the process of constructing the instruction set, aiming to enrich the instruction dimension of model training, thereby improving the comprehension and execution effect of the question-answering model.

[0030] In terms of evaluation methods, the similarity score between the model output and the standard answer is calculated based on the ROUGE-L indicator to quantify the performance of the evaluation model in the patent question-answering task.

[0031] Finally, we use the general domain instructions and the patent instruction dataset constructed in this study to fine-tune the base model obtained by incremental pre-training and obtain a large model that can understand general instructions and special triple instructions.

[0032] S03: Retrieval Enhancement Generation Given that LLM generally suffers from hallucination problems in text generation tasks, the prior art uses a retrieval-enhanced generation strategy to reduce errors and irrelevant components in the output content. Similarly, in the specific field of patent question answering, in order to significantly improve the accuracy and reliability of the model's answers, the present invention draws on and applies this technology to optimize the quality of LLM's answers.

[0033] In the instruction fine-tuning stage, the method adopted is: first, randomly select triples as basic training data, and build learning samples based on triple instructions in combination with the question context that the user may ask, in order to enable the model to internalize and understand these structured knowledge units. However, in the operation of the actual system, the processing flow is slightly different. The system first receives the actual question of the user, and the relevant triples are not pre-set, but need to be dynamically extracted from the green cooperative patent knowledge graph through the vector retrieval mechanism.

[0034] In specific operations, the coding technology is used to efficiently encode and store each triple in the constructed patent knowledge graph. When the user asks a question, the system will perform the same coding process on the question and use the similarity matching algorithm to search for the most relevant triple that is most appropriate to the question. Subsequently, the system will merge the matched triple with the original question as a new input, and then guide the generation of accurate and high-quality answer content in the LLM.

[0035] Before implementing the triple recall strategy, especially before performing similarity matching calculations, the key step is to map the structured triple information in the knowledge graph and the natural language questions raised by the user to the same continuous vector space. However, due to the inherent structural properties of triple data, conventional vectorization methods that directly apply unstructured text are not applicable. Therefore, the present invention proposes a template-based triple translation method. In terms of specific operations, in actual application scenarios, this paper first designs a template system to convert each triple structure into an understandable text representation.

[0036] The template system is constructed as follows: Considering the rich ontological attribute information contained in the triples, the subject and object entity type information is specially incorporated when constructing the template to ensure that the converted text fully reflects the intrinsic semantic characteristics of the original triples. In addition, in order to enhance the diversity and generalization ability of template expression, for different relationship types, ChatGPT is used to obtain a set of synonymous words for the corresponding relationship, thereby creating a series of different ways to describe the same relationship. When constructing an instance, the system randomly selects a relationship expression word from the set and incorporates it into the text generated by the template. In this way, for each type of triple, the entity type information can be retained while also introducing some diversity.

[0037] After completing the conversion from triples to text, the texts are then vectorized. In this process, the BGE encoding technology developed by the Zhiyuan Research Institute in 2023 and achieved SOTA in multiple benchmarks is used to effectively convert rich text information into high-dimensional vector representations, laying a solid foundation for subsequent recall and matching tasks. The triple encoding strategy aims to translate all triple structures in the knowledge graph one by one through the template text conversion method, and then convert their text representations into vectorized representations, thereby constructing a comprehensive set of triple vectors.

[0038] In the practical application phase, when facing the query questions raised by users, BGE encoding technology is also used to deeply vectorize the questions to ensure that they can be seamlessly mapped to the pre-constructed triple vector space. Then, with the help of the mathematical tool of cosine similarity, the similarity between the question vector and each triple vector stored in the library is measured. In this way, the triple that best matches the question vector and has the highest similarity can be accurately identified.

[0039] This highly similar triplet is selected as the key retrieval result, and then serves as an important basis for guiding the large language model to generate accurate answers.

[0040] The present invention uses patent data to incrementally pre-train a large model. Compared with the common large models in the prior art, the patent large model trained by this patent is specifically used to store patent knowledge, thereby improving the effect of patent question answering in downstream tasks. In addition, the present invention specifically constructs a data set for evaluating the ability to understand patent content. This method can effectively evaluate the ability of the large model to understand patent texts.

[0041] In terms of instruction fine-tuning, patent instruction fine-tuning data was specially constructed using Qwen-72b, which includes questions, triples and answers, and is used to train and evaluate the ability of large models to use patent knowledge graph triples to perform patent question and answering.

[0042] Since large models generally have hallucination problems, they are not capable of answering patent questions that are more professional and logical. In order to improve the accuracy of questions and answers, the present invention adopts a retrieval enhancement generation method. Specifically, the constructed patent knowledge graph is specially used to obtain the triples most relevant to the user's questions using a vector recall method. In addition, in order to further synchronize the similarity between text content and triples, the present invention designs a triple conversion template for converting structured triples into unstructured text content. Then inject user questions, so that the large model can obtain professional patent knowledge and generate professional and reasonable answers.

[0043] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a patent large model question answering system based on graph enhancement generation, characterized in that It includes the following steps: S01: Incremental pre-training, using the Intern2-1.8b-base model and using patent abstracts as the dataset for incremental pre-training; S02: Instruction fine-tuning. In the patent field question-answering scenario, LLM is used to deal with patent-related questions. This scenario requires accurate recall of triple information closely related to the question from the patent knowledge graph. To ensure that the model can effectively understand these structured triple texts, instruction fine-tuning is used to guide the model to understand the instructions related to the triples. The Qwen-72b model is used to construct patent instruction fine-tuning data, which includes questions, triples, and answers, and is used to train the large model to use patent knowledge graph triples for patent question-answering. S03: Retrieval-enhanced generation, using patent knowledge graphs and vector recall to obtain the triples most relevant to user questions, and then inject user questions to allow the large model to obtain patent knowledge and generate answers.

2. The method for constructing a patent large model question answering system based on graph enhancement generation according to claim 1 is characterized in that In the incremental pre-training process, the patent abstract dataset is first processed, and the Minhash combined with the LSH algorithm is used to remove duplicates from the patent data.

3. The method for constructing a patent large model question answering system based on graph enhancement generation according to claim 1 is characterized in that When constructing the instruction fine-tuning database, the Cypher query language is used to randomly extract triple instances under each type of relationship from the patent knowledge graph. The Qwen-27B-chat model is used to generate a question for the subject, relationship and object for the extracted triples, and each component corresponding to the triple constitutes the answer to the corresponding question. Each triple generates three question-answer pairs. After data cleaning, the corresponding training set, validation set and test set are obtained.

4. The method for constructing a patent large model question answering system based on graph enhancement generation according to claim 3 is characterized in that In the process of constructing the instruction set, the general-purpose Chinese instruction set is introduced.

5. The method for constructing a patent large model question answering system based on graph enhancement generation according to claim 1 is characterized in that In the retrieval enhancement generation process, encoding technology is used to encode and store each triple in the patent knowledge graph. When the user asks a question, the question is encoded in the same way, and a similarity matching algorithm is used to search for the triple that is most appropriate to the question. The system then merges the matched triple with the original question into a new question input, and then guides the generation of answer content in the LLM.

6. The method for constructing a patent large model question answering system based on graph enhancement generation according to claim 5 is characterized in that Before performing similarity matching calculations, the structured triple information in the knowledge graph and the natural language questions raised by the user are mapped to the same continuous vector space.

7. The method for constructing a patent large model question answering system based on graph enhancement generation according to claim 6 is characterized in that A template-based triple translation method is used to convert triple data into unstructured text content; The triple translation method comprises: Build a template system to convert each triple structure into an understandable textual representation. In the process of building the template, the subject and object entity class information is included. Specifically, the entity class text is suffixed after the entity to ensure that the converted text fully reflects the intrinsic semantic characteristics of the original triple. For different relationship types, ChatGPT is used to obtain the synonymous vocabulary set of the corresponding relationship, thereby creating a series of different ways to describe the same relationship. When building an instance, a relationship expression word in the set is randomly selected and incorporated into the text generated by the template. After completing the conversion from triples to texts, these texts are vectorized: BGE encoding technology is used to convert text information into high-dimensional vector representation, thereby constructing a comprehensive set of triple vectors.

8. The method for constructing a patent large model question answering system based on graph enhancement generation according to claim 7 is characterized in that When a user asks a query question, the BGE encoding technology is used to vectorize the question. Then, with the help of the mathematical tool of cosine similarity, the similarity between the question vector and the triple vectors stored in the library is measured, and the triplet that best matches the question vector and has the highest similarity is accurately identified. This highly similar triplet is selected as the key retrieval result as the basis for guiding the large language model to generate accurate answers.

Images