Scientific and technological document question-answering method, device, storage medium, and electronic device

Abstract

This application belongs to the field of artificial intelligence technology and specifically relates to a method, device, storage medium, and electronic device for question-and-answering scientific and technological documents. The method includes a knowledge graph generation step, a large language model training step, and a knowledge graph-based question-and-answer step. The technical solution provided in this application can more accurately and efficiently implement large language model-based question-and-answering of scientific and technological documents, providing rapid, accurate, and professional answers to questions, providing new possibilities and directions for the further development of the field of artificial intelligence in theory and practice.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a scientific and technological document question-answering method, device, storage medium and electronic device. Background Art

[0002] With the rapid development of artificial intelligence and machine learning, language models have evolved from simple bag-of-words and N-gram models to more complex and powerful neural network models. Large language models (LLMs) have particularly attracted attention in this process. They not only excel in natural language processing (NLP) tasks but also demonstrate incredible potential in a variety of cross-domain applications. From generative text and dialogue systems to more complex tasks such as text summarization, machine translation, and sentiment analysis, large language models are gradually changing the way we interact with the digital world. Artificial intelligence technologies based on large language models can help people complete specific tasks more quickly and accurately, significantly reducing labor costs and improving work efficiency.

[0003] However, using general large language models for document question answering in the field of science and technology still has the following problems: First, although general large language models learn a wide range of knowledge, they often lack depth, and may lack professionalism when answering questions in the field of science and technology; second, general large language models update knowledge slowly and cannot grasp new concepts proposed in the latest published literature; in addition, general large language models may produce hallucinations and fabricate content that does not conform to the facts, resulting in the generation of incorrect answers.

[0004] Retrieval-enhanced generation technology can help large language models grasp the latest scientific and technological literature knowledge, but it places high demands on the relevance of document retrieval, and irrelevant document content may mislead large language models. Graph-based retrieval-enhanced generation technology can alleviate this problem to a certain extent. This technology is better able to capture and utilize the complex relationships between information fragments. Compared with flat document structures, graph structures support the system to reason along relationship chains, achieving more complex and in-depth logical analysis, and can more naturally represent hierarchical and non-hierarchical relationships, closer to the knowledge organization of the real world. However, graph databases that store knowledge graphs often only store entities and their relationships, lacking contextual information, resulting in large language models' answers to questions being relatively brief and lacking relevant information. In addition, despite providing fine-grained information, large language models may still give incorrect answers, and the reasoning capabilities of first-level large language models need to be further strengthened. Summary of the Invention

[0005] One of the purposes of this application is to provide a method, device, storage medium and electronic device for scientific and technological document question answering, so as to more accurately and efficiently realize scientific and technological document question answering based on a large language model.

[0006] To achieve the above-mentioned and other related purposes, in a first aspect, embodiments of the present application provide a scientific document question-and-answer method, comprising the following steps:

[0007] The knowledge graph generation step, based on the open knowledge graph, extracts triples from scientific documents to generate a scientific knowledge graph containing entity, relationship and paragraph information;

[0008] The large language model training step involves searching the scientific and technological knowledge graph for user questions in the training data and generating answers with thought chains. The generated answers are then judged as correct based on the standard answers in the training data and labeled as correct and incorrect answers. The large language model is then trained using the user questions and labeled answers as input.

[0009] In the question-answering step based on the knowledge graph, triples and paragraphs related to the user input text are extracted from the scientific and technological knowledge graph, and a large language model trained after prompt word input is constructed to generate answers.

[0010] In some embodiments, the knowledge graph generation step includes:

[0011] Extract triples from each paragraph of scientific documents and integrate the extracted triples into the open knowledge graph;

[0012] Identify the similarity of phrase pairs in the open knowledge graph and add synonym edges to phrase pairs whose similarity is above a predetermined threshold;

[0013] Combined with the paragraphs of scientific documents, add the corresponding paragraph edges in the open knowledge graph.

[0014] In some embodiments, the open knowledge graph is a schema-less open knowledge graph.

[0015] In some embodiments, in the large language model training step, user questions in the training data are retrieved in the science and technology knowledge graph to obtain relevant triples and their corresponding paragraphs. If there is no matching triple, the paragraph is directly matched; the large language model is used to generate an answer with a thought chain for each retrieved triple and its corresponding paragraph or directly matched paragraph.

[0016] In some embodiments, in the large language model training step, labeling the correct answers and the incorrect answers separately includes: scoring the correct answers, selecting the correct answer with the highest score, and randomly selecting an incorrect answer to label.

[0017] In some embodiments, the score is the sum of a first score and a second score, the first score being the score calculated by matching the triples extracted from the correct answer with the triples in the science and technology knowledge graph, based on the number of matching triples, and the second score being the score calculated by inputting the user question and the generated answer into a large language model, and using the large language model to score the relevance between the question and the answer.

[0018] In some embodiments, in the question-answering step based on the knowledge graph, prompt words are dynamically constructed.

[0019] In a second aspect, an embodiment of the present application provides a scientific document question-and-answer device, comprising:

[0020] The knowledge graph generation module is used to extract triples from scientific documents based on the open knowledge graph and generate a scientific knowledge graph containing entity, relationship and paragraph information;

[0021] The large language model training module is used to retrieve user questions from the training data in the scientific and technological knowledge graph and generate answers with thought chains. The generated answers are judged correct or incorrect based on the standard answers in the training data and correct and incorrect answers are labeled respectively. The user questions and labeled answers are used as input to train the large language model.

[0022] The knowledge graph-based question-answering module is used to extract triplets and paragraphs related to the user input text from the scientific and technological knowledge graph, and build a large language model trained on prompt word input to generate answers.

[0023] In a third aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, which implements any of the aforementioned methods when executed by a processor.

[0024] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, and the computer program implements any of the aforementioned methods when executed by the processor.

[0025] The above technical solutions provided in the embodiments of the present application can bring about the following technical effects: quickly, accurately and professionally answer questions in the field of science and technology, and provide new possibilities and directions for the further development of the field of artificial intelligence in theory and practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 The present invention is a flowchart of the steps of the scientific document question-and-answer method according to an embodiment of the present application.

[0028] Figure 2 The figure is a schematic diagram of the structure of a scientific document question-and-answer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present application through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in the present application can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0030] This application is at least intended to address the shortcomings of general large language models in scientific and technological documents, such as slow information updates, unprofessional and inaccurate answers to questions, and proposes a new scientific and technological document question-answering method that can more accurately and efficiently implement scientific and technological document question-answering based on large language models, answer questions quickly, accurately and professionally, and provide new possibilities and directions for the further development of the field of artificial intelligence in theory and practice.

[0031] Reference Figure 1 In one embodiment, the present application proposes a scientific document question-answering method, comprising the following steps:

[0032] S1, the knowledge graph generation step, extracts triples from scientific documents based on the open knowledge graph to generate a scientific knowledge graph containing entity, relationship, and paragraph information. The paragraph information at least includes the paragraph and its number.

[0033] In one example, the knowledge graph generation step may specifically include the following processing flow:

[0034] S11, extracts triples from each paragraph of scientific documents and integrates the extracted triples into a schema-free open knowledge graph.

[0035] Specifically, triples can be extracted using large language models, for example, to extract entity-relationship-entity / attribute value triples such as (laser, type, semiconductor laser). In fields such as knowledge graphs, databases, and data modeling, a schema is a structured definition of data structure, entity types, relationships, and constraints. It is used to standardize the organization and representation of data and ensure its consistency, understandability, and interoperability. A schema-free open knowledge graph is a knowledge graph that does not rely on predefined data structures or entity-relationship templates, allowing for the dynamic expansion of entity types, attributes, and relationships during data acquisition and storage.

[0036] S12, identifying the similarity of phrase pairs in the open knowledge graph, and adding synonym edges to phrase pairs whose similarity is higher than a predetermined threshold.

[0037] Specifically, the vector encoder and cosine similarity can be used to calculate the similarity of phrase pairs in the open knowledge graph to identify synonyms, and detect phrase pairs whose vector similarity is higher than a predetermined threshold. Synonym edges are added between these phrase pairs. Synonym edges can be expressed as (entity 1, synonym, entity 2), for example (tomato, synonym, tomato).

[0038] S13, combined with the paragraphs of scientific documents, adds the corresponding paragraph edges in the open knowledge graph.

[0039] For example, associating the triple (laser, type, semiconductor laser) with paragraph "P001" indicates that the fact originates from that paragraph. This allows the generated scientific and technological knowledge graph to contain entity, relationship, and paragraph information.

[0040] S2, the large language model training step, retrieves user questions in the training data in the science and technology knowledge graph and generates answers with thought chains. The correctness of the generated answers is judged based on the standard answers in the training data and the correct and wrong answers are labeled respectively. The user questions and labeled answers are used as input for training.

[0041] Specifically, user questions can be searched within the S&T knowledge graph using search-enhanced generative technology to obtain relevant triples and their corresponding paragraphs. If no matching triples exist, the paragraphs are directly matched. A vector encoder can be used to encode the user question, and then the triples and paragraphs in the S&T knowledge graph are encoded separately. Similarity is calculated, and the triples, paragraphs, and their corresponding paragraphs that exceed a predetermined threshold are retained.

[0042] A large language model is then used to generate an answer with a chain of thought for each retrieved triple and its corresponding paragraph or directly matched paragraph. This large language model can be a Deepseek-R1 large model or a GPT-4 large model. Based on the standard answers in the training data, the generated answers are determined to be correct and incorrect, and the correct and incorrect answers are labeled as accept and reject, respectively. The user question input and the labeled answers are then used as input for model training, thereby improving the accuracy of the large language model's answer generation based on the reference content. Reinforcement learning training using the Direct Preference Optimization (DPO) algorithm can be used to further improve the accuracy of the large language model's answer generation based on the reference content and reduce large language model hallucinations.

[0043] In this embodiment, assigning acceptance and rejection labels to correct and incorrect answers, respectively, may also include scoring the correct answers. Specifically, the scores include a first score and a second score. The first score is obtained by extracting triplets from the correct answer, matching these triplets with triplets in the graph database, and obtaining the number of matching triplets. Each matching triplet is scored as 1, with a maximum score of 10. The second score is obtained by inputting the user question and the generated answer into a large language model, and using the large language model to score the relevance between the question and the answer, with a score range of 0-10. The first and second scores are then added together to obtain a total score. The correct answer with the highest total score is selected and labeled as acceptance, while a randomly selected incorrect answer is labeled as rejection.

[0044] S3, the question-answering step based on the knowledge graph, extracts triplets and paragraphs related to the user input text from the scientific and technological knowledge graph, and constructs a large language model for prompt word input to generate answers.

[0045] Specifically, search-enhanced generative technology can be used to search the scientific knowledge graph, extracting triplets and paragraphs related to the user's input text from the knowledge graph. The specific method is the same as the search and extraction method in the large language model training step mentioned above. Prompt words can be dynamically constructed to further improve the quality of answers.

[0046] In summary, it can be seen that this is different from the traditional use of retrieval enhancement generation technology to conduct scientific and technological document question and answer. In the stage of constructing the knowledge graph, the above-mentioned technical solution provided in the embodiment of the present application uses a large language model to extract triples from each paragraph of each document, and integrates them into a schema-free open knowledge graph. Compared with other traditional fields that define the schema first and then store the data, the schema-free database is more open, can include more scientific and technological knowledge, and saves labor costs, but the knowledge of different data sources may be expressed in different ways, and knowledge fusion is required. In the embodiment of the present application, by adding synonym edges between phrase pairs, synonyms between different paragraphs or chapters are linked, thereby promoting the integration of new and old knowledge in the learning process. Finally, the phrase-based open knowledge graph is combined with the original paragraph, and the paragraph edges are added so that the generated scientific and technological knowledge graph can contain entity, relationship and paragraph information at the same time. During the model training and data retrieval stages, user questions and inputs are linked to relevant triples and paragraphs, and triples and paragraphs are used as model inputs at the same time. This allows the fine-grained knowledge in the knowledge graph to be combined with the coarse-grained knowledge in the document fragments, providing the model with more reference knowledge, further improving the accuracy of the model in generating answers based on reference content, and reducing the illusion of large language models.

[0047] Reference Figure 2 , the embodiment of the present application further provides a scientific document question-and-answer device, comprising:

[0048] The knowledge graph generation module is used to extract triples from scientific documents based on the open knowledge graph and generate a scientific knowledge graph containing entity, relationship and paragraph information;

[0049] The large language model training module is used to retrieve user questions from the training data in the scientific and technological knowledge graph and generate answers with thought chains. The generated answers are judged correct or incorrect based on the standard answers in the training data and correct and incorrect answers are labeled respectively. The user questions and labeled answers are used as input to train the large language model.

[0050] The knowledge graph-based question-answering module is used to extract triplets and paragraphs related to the user input text from the scientific and technological knowledge graph, and build a large language model trained on prompt word input to generate answers.

[0051] An embodiment of the present application further provides a storage medium having a computer program stored thereon, which implements the aforementioned method when executed by a processor.

[0052] An embodiment of the present application further provides an electronic device, the electronic device comprising a processor and a memory;

[0053] The memory is used to store computer programs;

[0054] The processor is connected to the memory and is used to execute the computer program stored in the memory. When the computer program is executed by the processor, the above method is implemented.

[0055] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and replacements can be made without departing from the technical principles of the present application. These improvements and replacements should also be regarded as the scope of protection of the present application.

Claims

1. A scientific document question-answering method, characterized in that: The following steps are involved: The knowledge graph generation step extracts triples from each paragraph of the scientific document based on the open knowledge graph, integrates the extracted triples into the open knowledge graph, identifies the similarity of phrase pairs in the open knowledge graph, adds synonym edges to phrase pairs whose similarity exceeds a predetermined threshold, and adds the corresponding paragraph edges in the open knowledge graph in combination with the paragraphs of the scientific document to generate a scientific knowledge graph containing entity, relationship and paragraph information, wherein the paragraph information includes at least the paragraph and its number; the open knowledge graph is a schema-free open knowledge graph; The large language model training step involves searching the scientific and technological knowledge graph for user questions in the training data to obtain relevant triples and their corresponding paragraphs. If there are no matching triples, the paragraphs are directly matched. The large language model is used to generate an answer with a thought chain for each retrieved triple and its corresponding paragraph or directly matched paragraph. The correctness of the generated answer is judged based on the standard answer in the training data, and the correct and incorrect answers are labeled respectively. The user question and the labeled answer are used as input to train the large language model. Labeling the correct answers and the wrong answers separately includes: scoring the correct answers, selecting the correct answer with the highest score, and randomly selecting a wrong answer to label; The score is the sum of a first score and a second score, where the first score is the score calculated by matching the triples in the correct answer with the triples in the scientific and technological knowledge graph, and the second score is the score calculated by inputting the user question and the generated answer into the large language model and using the large language model to score the relevance between the question and the answer; In the question-answering step based on the knowledge graph, triples and paragraphs related to the user input text are extracted from the scientific and technological knowledge graph, and a large language model trained after prompt word input is constructed to generate answers.

2. The scientific document question-answering method according to claim 1, characterized in that: In the question-answering step based on the knowledge graph, prompt words are dynamically constructed.

3. A scientific document question-answering device, characterized in that: include: A knowledge graph generation module is configured to extract triples from each paragraph of a scientific document based on an open knowledge graph, integrate the extracted triples into the open knowledge graph, identify the similarity of phrase pairs in the open knowledge graph, add synonym edges to phrase pairs whose similarity exceeds a predetermined threshold, and, in combination with paragraphs in the scientific document, add corresponding paragraph edges to the open knowledge graph to generate a scientific knowledge graph containing entity, relationship, and paragraph information, wherein the paragraph information includes at least the paragraph and its number; the open knowledge graph is a schema-free open knowledge graph; The large language model training module is used to retrieve user questions from the training data in the scientific and technological knowledge graph, obtain relevant triples and their corresponding paragraphs, and directly match the paragraphs if there are no matching triples. The large language model is used to generate an answer with a thought chain for each retrieved triple and its corresponding paragraph or directly matched paragraph. The correctness of the generated answer is judged based on the standard answer in the training data, and the correct and incorrect answers are labeled respectively. The user question and the labeled answer are used as input to train the large language model; Labeling the correct answers and the wrong answers separately includes: scoring the correct answers, selecting the correct answer with the highest score, and randomly selecting a wrong answer to label; The score is the sum of a first score and a second score, where the first score is the score calculated by matching the triples in the correct answer with the triples in the scientific and technological knowledge graph, and the second score is the score calculated by inputting the user question and the generated answer into the large language model and using the large language model to score the relevance between the question and the answer; The knowledge graph-based question-answering module is used to extract triplets and paragraphs related to the user input text from the scientific and technological knowledge graph, and build a large language model trained on prompt word input to generate answers.

4. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 2.

5. An electronic device, characterized in that: The electronic device includes a processor and a memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute a computer program stored in the memory. When the computer program is executed by the processor, the method according to any one of claims 1 to 2 is implemented.

Images