Construction method of patent large model question and answer system based on atlas enhancement generation

Abstract

The present application relates to a kind of construction methods of patent large model question and answer system based on atlas enhancement generation, adopt incremental pre-training strategy, a large number of patent abstract text is integrated into the learning process of LLM, can more accurately call and generate relevant content in subsequent question and answer interaction. In order to further transform text continuation type LLM into the form suitable for executing accurate question and answer task, and improve its understanding and following degree to specific instruction, instruction fine-tuning technology is used, and the model is optimized with a large number of instruction data sets, and the adjusted LLM can accurately meet the knowledge answer demand required by user.

Description

Technical Field

[0001] This invention relates to a method for constructing a patented large-scale model question-answering system based on graph enhancement generation. Background Technology

[0002] Since the release of ChatGPT, Large Language Models (LLMs) have gradually entered the public eye and become a key area of ​​academic research. As a form of general artificial intelligence, LLMs can solve various downstream NLP tasks, including text summarization, information extraction, translation, and question answering. Through instruction fine-tuning and reinforcement learning based on human feedback, they are more likely to generate answers that appeal to humans in the question answering domain. While LLMs can solve many downstream problems, further research and training are needed in certain vertical fields, such as medicine, law, and finance, due to a lack of relevant knowledge and the existence of illusion problems. Although significant research has been conducted in vertical fields, research on LLMs in the patent field remains limited. Therefore, this patent aims to explore the application of LLMs in question answering within the patent field.

[0003] The training of LLM can be broadly divided into three aspects: pre-training, supervised fine-tuning (SFT), and reinforcement learning. In the pre-training stage, by injecting a large amount of text data, the LLM can acquire a vast amount of domain knowledge. Currently, the academic community generally considers this stage as a kind of knowledge reserve, thus requiring a large corpus. In the SFT stage, fine-tuning on labeled data can effectively improve the model's question-answering ability. For example, instruction fine-tuning techniques, using a small amount of instruction data, enable the LLM to possess a certain degree of instruction-following ability. Even if the instruction type has not appeared in the training set, the LLM can understand and execute commands, demonstrating a certain degree of general intelligence. The reinforcement learning stage mainly uses reinforcement learning based on human feedback (RLHF) to enable the model to generate content that humans prefer.

[0004] Currently, the LLM field has given rise to numerous excellent large-scale model examples, such as OPENAI's GPT-4, Zhipu Technology's ChatGLM, Alibaba Cloud's Qwen series models, and Shusheng Puyu's Intern2, among other general-purpose LLMs. There are also many outstanding works in specific vertical fields, such as Code-LLAMA, which specializes in programming applications, and Chat-Doctor, which serves medical consultations.

[0005] In patent knowledge graph research, most knowledge graphs use graph databases such as Neo4j as the underlying storage mechanism. Although this technical solution can efficiently mine and associate related entities and relationships with data query languages ​​such as Cypher, it is undeniable that non-professional users may find such professional search methods quite difficult, thus limiting their effective access to patent cooperation information resources. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a method for constructing a patented large-scale model question-answering system based on graph enhancement generation, thereby reducing the technical threshold for ordinary users to obtain relevant information.

[0007] The technical solution adopted in this invention is:

[0008] A method for constructing a patent large-scale model question-answering system based on graph enhancement generation, characterized by the following steps:

[0009] S01: Incremental pre-training, using the Intern2-1.8b-base model and patent abstracts as the dataset for incremental pre-training;

[0010] S02: Instruction Fine-tuning. In the patent question-answering scenario, LLM is applied to handle patent-related questions. This scenario requires accurately recalling triple information closely related to the question from the patent knowledge graph. To ensure that the model can effectively understand these structured triple texts, instruction fine-tuning is used to guide the model to understand the instructions related to the triples. Patent instruction fine-tuning data, including questions, triples, and answers, is constructed using the Qwen-72b model to train the large model to use patent knowledge graph triples for patent question answering.

[0011] S03: Enhanced retrieval generation utilizes a patent knowledge graph and employs vector recall to obtain the triples most relevant to the user's question. These triples are then injected into the user's question, allowing the large model to acquire patent knowledge and generate an answer.

[0012] Furthermore, during the incremental pre-training process, the patent abstract dataset is first processed by using Minhash combined with the LSH algorithm to remove duplicates from the patent data.

[0013] Furthermore, during the construction of the instruction fine-tuning database, the Cypher query language is used to randomly extract triple instances under each type of relation from the patent knowledge graph. For each extracted triple, the Qwen-27B-chat model is used to generate a question for the subject, relation, and object respectively. Each component of the triple constitutes the answer to the corresponding question. Thus, each triple generates 3 question-answer pairs. After data cleaning, the corresponding training set, validation set, and test set are obtained.

[0014] Furthermore, a general-purpose Chinese instruction set is introduced during the instruction set construction process.

[0015] Furthermore, in the process of enhancing the retrieval, encoding technology is used to encode and store each triple in the patent knowledge graph. When a user asks a question, the question is encoded in the same way, and a similarity matching algorithm is used to search for the triple that best matches the question. Then, the system merges the matched triple with the original question to form a new question input, which then guides the generation of answer content in the LLM.

[0016] Furthermore, before performing similarity matching calculations, the structured triple information in the knowledge graph and the natural language questions raised by the user are mapped to the same continuous vector space.

[0017] Furthermore, a template-based triple translation method is adopted to transform triple data into unstructured text content;

[0018] The triplet translation method includes:

[0019] A template system is constructed to convert each triple structure into an understandable textual representation. When constructing the template, subject and object entity class information is incorporated, specifically by appending the entity category text to the entity to ensure that the converted text fully reflects the inherent semantic features of the original triple. For different relation types, ChatGPT is used to obtain a set of synonyms for the corresponding relation, thereby creating a series of different ways to describe the same relation. When constructing instances, a relation expression word from this set is randomly selected and incorporated into the text generated by the template.

[0020] After converting triples into text, the text is vectorized: BGE encoding technology is used to convert the text information into a high-dimensional vector representation, thereby constructing a comprehensive set of triple vectors.

[0021] Furthermore, when a user submits a query, the BGE encoding technique is used to vectorize the query. Then, using the mathematical tool of cosine similarity, the similarity between the query vector and the triple vectors stored in the database is measured. The triple that best matches the query vector and has the highest similarity is accurately identified. This highly similar triple is selected as the key search result and serves as the basis for guiding the large language model to generate accurate answers.

[0022] The positive effects of this invention are as follows: This invention constructs a patent large-scale model question-and-answer system, which allows users to obtain relevant patent information through a simple question-and-answer format, greatly reducing the technical threshold for ordinary users to obtain relevant information.

[0023] This invention employs an incremental pre-training strategy, integrating a large amount of patent abstract text into the LLM's learning process. This allows for more accurate retrieval and generation of relevant content in subsequent question-and-answer interactions. To further transform the text-based LLM into a form suitable for performing precise question-and-answer tasks and improve its understanding and adherence to specific instructions, instruction fine-tuning technology is used. The model is specifically optimized with a massive instruction dataset, and the adjusted LLM can accurately meet the user's knowledge-based answering needs. Considering the challenge of factual accuracy that large models may face, namely the so-called illusion problem, this invention adopts knowledge graph retrieval enhancement technology. When a user asks a question to the model, the system uses a vector similarity retrieval method to match the most relevant triplet information from the knowledge graph and cleverly integrates it into the context. This ensures that the LLM can effectively combine and utilize the actual data in the patent knowledge graph during the answer generation process, thereby providing both authentic and effective response content. Attached Figure Description

[0024] Figure 1 A flowchart for constructing this invention is provided. Detailed Implementation

[0025] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0026] As attached Figure 1 As shown, this invention provides a method for constructing a patented large-scale model question-answering system based on graph enhancement generation, aiming to reduce the technical threshold for ordinary users to obtain relevant information.

[0027] The specific method is as follows:

[0028] S01: Incremental Pre-training

[0029] To balance training efficiency and model performance, this patent selected a 2B model for incremental pre-training research. The Intern2-1.8b-base model was used, and the incremental pre-training process was performed using the unique dataset of patent abstracts.

[0030] In terms of data processing, to further improve the quality and purity of training data, firstly, the efficient data deduplication scheme proposed by slimpajam is adopted, and Minhash combined with LSH algorithm is used to perform deep and comprehensive duplicate removal processing on patent data; secondly, a series of rule-based data cleaning methods can also be implemented to purify the content of the dataset.

[0031] To comprehensively test the memory and comprehension capabilities of the trained model, this invention also discloses an evaluation method that employs two complementary approaches: a memory retrieval evaluation based on ROUGE_L and a patent multiple-choice question analysis benchmark test.

[0032] ROUGE_L-based Memory Reproduction Evaluation: This invention primarily explores the model's ability to remember injected patent abstract text. It employs a method using ROUGE_L scores as a metric. Specifically, text samples randomly selected from the training set are used as the test set. For each sample, the model is first provided with the first 16 characters or the first half of the text, and then asked to generate the remaining content. Next, the ROUGE_L score between the original text fragment and the model-generated fragment is calculated to quantitatively evaluate the model's performance in remembering patent knowledge stored in the training set.

[0033] Patent Multiple Choice Question Analysis Benchmark: To deeply examine the model's understanding of patent knowledge, a patent multiple choice question evaluation benchmark was constructed. The creation of this benchmark involved utilizing the powerful Qwen-72B-chat model to generate challenging question-answer pairs.

[0034] In practice, randomly selected patent abstract texts are input into Qwen-72B-chat, and prompts are set to require the model to generate question-answer pairs. This generates a question database and an answer database. Then, three randomized distractor options are assigned to each question from other answer databases, and after data cleaning, these constitute a test dataset of patent multiple-choice questions. Finally, these designed multiple-choice questions and their options are input into the model under test, and the option with the highest probability is selected as the model's predicted answer by analyzing the logits value of the model's last layer output. The aim is to accurately reflect the depth and accuracy of the trained model's understanding of the substantive content of the patent text.

[0035] Patent knowledge is injected into the LLM using an incremental pre-training method, and the generative capability of the trained model is verified using two evaluation methods.

[0036] S02: Command Fine-tuning

[0037] In the patent domain question-answering scenario of this invention, LLM is applied to handle complex patent-related questions. This scenario requires accurately recalling triple information closely related to the question from the patent knowledge graph. To ensure that the model can effectively understand these structured triple texts, this study uses an instruction fine-tuning method to guide the model to understand instructions related to the triples.

[0038] Regarding the construction of instruction data, firstly, using the Cypher query language, triple instances under each type of relation are randomly extracted from the patent knowledge graph. For each extracted triple, the Qwen-27B-chat model is used to generate a question for the subject, relation, and object respectively, and each component of the triple constitutes the answer to the corresponding question. In this way, each triple generates 3 question-answer pairs. After data cleaning, corresponding training, validation, and test sets are obtained. These question-answer pairs consist of three key elements: instruction (i.e., the question generated by LLM), input (the triple content), and output (the specific part of the triple). The prompts set in this process are as follows:

[0039] "You are now a knowledge graph expert and a linguistics expert.\nI will give you a triple, a concept in knowledge graphs, which contains three parts: <subject, relation, object>;\nI want you to generate three questions based on this triple, asking each of the three parts of the triple respectively. For example:\nInput: <

Patent

Relation

Advantages

[0040] Furthermore, given that existing research indicates the importance of instruction type diversity for improving model performance, this invention further incorporates a general-domain Chinese instruction set during the instruction set construction process. This aims to enrich the instruction dimensions for model training, thereby enhancing the understanding and execution performance of the question-answering model.

[0041] In terms of evaluation methodology, the similarity score between the model output and the standard answer is calculated based on the ROUGE-L index, thereby quantifying the model's performance in the patent question-answering task.

[0042] Finally, using general domain instructions plus the patent instruction dataset constructed in this study, the base model obtained through incremental pre-training is fine-tuned to obtain a large model capable of understanding both general instructions and specific triplet instructions.

[0043] S03: Enhanced Search Generation

[0044] Given the common illusion problem in LLM text generation tasks, existing technologies employ retrieval-enhanced generation strategies to reduce errors and irrelevant components in the output content. Similarly, in the specific field of patent question answering, this invention borrows and applies this technology to optimize the answer quality of LLM in order to significantly improve the accuracy and reliability of the model's answers.

[0045] During the instruction fine-tuning phase, the approach is as follows: first, triples are randomly selected as basic training data, and then learning samples based on triple instructions are constructed in conjunction with possible user-posed question scenarios. The aim is to enable the model to internalize and understand these structured knowledge units. However, in the actual operation of the system, the processing flow is slightly different. The system first receives the user's actual question, and the relevant triples are not pre-set; they need to be dynamically extracted from the green collaborative patent knowledge graph through a vector retrieval mechanism.

[0046] In practice, encoding techniques are used to efficiently encode and store each triple in the constructed patent knowledge graph. When a user asks a question, the system performs the same encoding process on the question and uses a similarity matching algorithm to search for the most relevant triples that match the question. Subsequently, the system merges the matched triples with the original question as a new input, which then guides the generation of accurate and high-quality answers within the LLM (Limited Learning Model).

[0047] Before implementing a triplet recall strategy, especially before performing similarity matching calculations, a key step is to map the structured triplet information in the knowledge graph to the user-generated natural language questions into the same continuous vector space. However, due to the inherent structural nature of triplet data, directly applying conventional vectorization methods for unstructured text is not suitable. Therefore, this invention proposes a template-based triplet translation method. In practical operation, in real-world application scenarios, this paper first designs a template system to convert each triplet structure into an understandable textual representation.

[0048] The template system is constructed as follows: Considering the rich ontological attribute information contained within triples, subject and object entity type information is specifically incorporated into the template construction to ensure that the converted text fully reflects the inherent semantic features of the original triples. Furthermore, to enhance the diversity and generalization ability of the template expression, ChatGPT is used to obtain a set of synonyms for different relation types, thereby creating a series of different ways to describe the same relation. When constructing instances, the system randomly selects a relation expression term from this set and integrates it into the text generated by the template. In this way, for each type of triple, entity type information is preserved while also introducing some diversity.

[0049] After converting triples to text, the next step is to vectorize this text. In this process, the BGE encoding technology, developed by the Beijing Academy of Artificial Intelligence in 2023 and achieving state-of-the-art (SOTA) results in multiple benchmark tests, is employed. This effectively transforms rich textual information into high-dimensional vector representations, laying a solid foundation for subsequent recall and matching tasks. The purpose of this triple encoding strategy is to translate all triple structures within the knowledge graph one by one using template text conversion methods, transforming their textual representations into vectorized representations, thereby constructing a comprehensive set of triple vectors.

[0050] In practical application, when faced with user queries, BGE encoding technology was used to deeply vectorize the queries, ensuring seamless mapping to a pre-constructed triplet vector space. Subsequently, cosine similarity was used to measure the similarity between the query vector and the various triplet vectors stored in the database. This method accurately identifies the triplet that best matches the query vector and has the highest similarity.

[0051] The highly similar triples were selected as key search results, and thus served as an important basis for guiding the large language model to generate accurate answers.

[0052] This invention utilizes patent data to incrementally pre-train a large model. Compared to common large models in existing technologies, the patent-trained large model is specifically designed to store patent knowledge, thereby improving performance in downstream tasks such as patent question answering. Furthermore, this invention specifically constructs a dataset for evaluating patent content comprehension capabilities. This method effectively assesses the large model's ability to understand patent texts.

[0053] In terms of instruction fine-tuning, patent instruction fine-tuning data was specially constructed using Qwen-72b, which includes questions, triples, and answers, to train and evaluate the ability of large models to perform patent question answering using patent knowledge graph triples.

[0054] Because large-scale models generally suffer from the illusion problem, their ability to answer highly specialized and logically rigorous patent questions is insufficient. To improve question-answering accuracy, this invention employs a retrieval-enhanced generation method. Specifically, it utilizes a constructed patent knowledge graph and employs vector recall to obtain the triples most relevant to the user's question. Furthermore, to further synchronize the similarity between text content and triples, this invention designs a triple conversion template to convert structured triples into unstructured text content. This is then injected into the user's question, allowing the large-scale model to acquire specialized patent knowledge and generate professional and reasonable answers.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a patent large-scale model question-answering system based on graph enhancement generation, characterized in that... It includes the following steps: S01: Incremental pre-training, using the Intern2-1.8b-base model and patent abstracts as the dataset for incremental pre-training; S02: Instruction Fine-tuning. In the patent question-answering scenario, LLM is applied to handle patent-related questions. This scenario requires accurately recalling triple information closely related to the question from the patent knowledge graph. To ensure that the model can effectively understand these structured triple texts, instruction fine-tuning is used to guide the model to understand the instructions related to the triples. Patent instruction fine-tuning data, including questions, triples, and answers, is constructed using the Qwen-72b model to train the large model to use patent knowledge graph triples for patent question answering. S03: Enhanced retrieval generation utilizes a patent knowledge graph and employs vector recall to obtain the triplet most relevant to the user's question. This triplet is then injected into the user's question, allowing the large model to acquire patent knowledge and generate an answer. When constructing the instruction fine-tuning database, the Cypher query language is used to randomly extract triple instances under each type of relation from the patent knowledge graph. For the extracted triples, the Qwen-27B-chat model is used to generate a question for the subject, relation, and object respectively. Each component of the triple constitutes the answer to the corresponding question. Thus, each triple generates 3 question-answer pairs. After data cleaning, the corresponding training set, validation set, and test set are obtained. In the process of enhanced retrieval and generation, encoding technology is used to encode and store each triple in the patent knowledge graph. When a user asks a question, the question is encoded in the same way, and a similarity matching algorithm is used to search for the triple that best matches the question. Then the system merges the matched triple with the original question to form a new question input, which then guides the generation of answer content in LLM. Before performing similarity matching calculations, the structured triple information in the knowledge graph and the natural language questions raised by the user are mapped to the same continuous vector space; A template-based triple translation method is used to transform triple data into unstructured text content; The triplet translation method includes: constructing a template system to convert each triplet structure into an understandable textual representation; When constructing the template, subject and object entity class information is incorporated. Specifically, the entity category text is appended to the entity to ensure that the converted text fully reflects the inherent semantic features of the original triples. For different relation types, ChatGPT is used to obtain a set of synonyms for the corresponding relation, thereby creating a series of different ways to describe the same relation. When constructing instances, a relation expression word from this set is randomly selected and incorporated into the text generated by the template. After completing the conversion from triples to text, these texts are vectorized: BGE encoding technology is used to convert the text information into a high-dimensional vector representation, thereby constructing a comprehensive set of triple vectors.

2. The method for constructing a patent large-scale model question-answering system based on graph enhancement generation according to claim 1, characterized in that... During the incremental pre-training process, the patent abstract dataset is first processed by using Minhash combined with the LSH algorithm to remove duplicates from the patent data.

3. The method for constructing a patent large-scale model question-answering system based on graph enhancement generation according to claim 1, characterized in that... A general-purpose Chinese instruction set is introduced during the instruction set construction process.

4. The method for constructing a patent large-scale model question-answering system based on graph enhancement generation according to claim 1, characterized in that... When a user submits a query, the query is vectorized using BGE encoding technology. Then, using the mathematical tool of cosine similarity, the similarity between the query vector and the triple vectors stored in the database is measured. The triple with the highest similarity to the query vector is accurately identified, and this triple is selected as the key search result, serving as the basis for guiding the large language model to generate accurate answers.

Images