Scientific and technological document question and answer method and device, storage medium and electronic equipment

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a science and technology document question answering method and device, a storage medium and electronic equipment. The method comprises a knowledge graph generation step, a large language model training step and a knowledge graph-based question and answer step. According to the technical scheme provided by the invention, document question answering in the science and technology field based on the large language model can be more accurately and efficiently realized, questions can be quickly, accurately and professionally answered, and a new possibility and direction are provided for further development of the artificial intelligence field in theory and practice.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and particularly relates to a method, device, storage medium, and electronic device for answering questions about scientific and technological documents. Background Art

[0002] With the rapid development of the fields of artificial intelligence and machine learning, language models have evolved from simple Bag-of-Words models and N-gram models to more complex and powerful neural network models. In this process, large language models (LLMs) have attracted particular attention. They not only perform well in natural language processing (NLP) tasks but also demonstrate amazing potential in various cross-domain applications. From generating text and dialogue systems to more complex tasks such as text summarization, machine translation, and sentiment analysis, large language models are gradually changing the way we interact with the digital world. Artificial intelligence technology based on large language models can help people complete specific tasks more quickly and accurately, significantly reducing labor costs and improving work efficiency.

[0003] However, there are still the following problems in using general large language models for answering questions about scientific and technological documents: First, although general large language models learn a wide range of knowledge, they often lack depth, and may lack professionalism when answering questions in the scientific and technological field; second, the knowledge update speed of general large language models is slow, and they cannot master newly proposed concepts in the latest published literature; in addition, general large language models may produce hallucinations and fabricate content that does not conform to facts, resulting in incorrect answers.

[0004] Retrieval-augmented generation technology can help large language models master the latest knowledge of scientific and technological literature, but it poses high requirements for the relevance of document retrieval. Irrelevant document content may mislead large language models. Retrieval-augmented generation technology based on graph structures can alleviate the above problems to a certain extent. This technology can better capture and utilize the complex relationships between information fragments. Compared with the flat document structure, the graph structure supports the system to reason along the relationship chain, enabling more complex and in-depth logical analysis, and can more naturally represent hierarchical and non-hierarchical relationships, which is closer to the knowledge organization method of the real world. However, the graph database storing the knowledge graph often only stores entities and their relationships, lacking context information, resulting in relatively short answers to questions from large language models without containing relevant information. In addition, although fine-grained information is provided, large language models may still give incorrect answers, and the reasoning ability of large language models also needs to be further strengthened. Summary of the Invention

[0005] One of the purposes of this application is to provide a method, device, storage medium, and electronic device for answering questions about scientific and technological documents to more accurately and efficiently implement question answering for scientific and technological documents based on large language models.

[0006] For achieving the above object and other related objects, in a first aspect, an embodiment of the present application provides a method for answering questions about scientific and technological documents, including the following steps: A knowledge graph generation step, based on an open knowledge graph, extracts triples from scientific and technological documents to generate a scientific and technological knowledge graph containing entity, relationship, and paragraph information; A large language model training step, retrieves user questions in the training data in the scientific and technological knowledge graph and generates answers with a chain of thought, determines the correctness of the generated answers according to the standard answers in the training data, and tags the correct answers and wrong answers respectively, uses the user questions and the tagged answers as inputs to train the large language model; A question answering step based on the knowledge graph, extracts triples and paragraphs related to the user input text from the scientific and technological knowledge graph, constructs a prompt word, and inputs it into the trained large language model to generate an answer.

[0007] In some embodiments, the knowledge graph generation step includes: Extract triples from each paragraph of the scientific and technological document, and integrate the extracted triples into the open knowledge graph; Identify the similarity of phrase pairs in the open knowledge graph, and add synonym edges to the phrase pairs with similarity higher than a predetermined threshold; Combine with the paragraphs of the scientific and technological document, and add belonging paragraph edges in the open knowledge graph.

[0008] In some embodiments, the open knowledge graph is a schema-free open knowledge graph.

[0009] In some embodiments, in the large language model training step, retrieve user questions in the training data in the scientific and technological knowledge graph, obtain relevant triples and their corresponding paragraphs, and directly match paragraphs if there are no matching triples; use the large language model to generate an answer with a chain of thought for each retrieved triple and its corresponding paragraph or directly matched paragraph.

[0010] In some embodiments, in the large language model training step, tagging the correct answers and wrong answers respectively includes: scoring the correct answers, and tagging the correct answer with the highest score and a randomly selected wrong answer.

[0011] In some embodiments, the score is the sum of a first score and a second score. The first score is the score calculated according to the number of matching triples by matching the triples in the correct answer with the triples in the scientific and technological knowledge graph. The second score is the score calculated by inputting the user question and the generated answer into the large language model and using the large language model to score the relevance of the question and the answer.

[0012] In some implementations, in the question-answering step based on the knowledge graph, prompt words are dynamically constructed.

[0013] In a second aspect, an embodiment of the present application provides a scientific document question-and-answer device, comprising: The knowledge graph generation module is used to extract triples from scientific and technological documents based on the open knowledge graph and generate a scientific and technological knowledge graph containing entity, relationship and paragraph information; The large language model training module is used to retrieve user questions in the training data in the scientific and technological knowledge graph and generate answers with thought chains. The correctness of the generated answers is judged according to the standard answers in the training data and the correct and incorrect answers are labeled respectively. The user questions and the labeled answers are used as input to train the large language model. The question-answering module based on the knowledge graph is used to extract triples and paragraphs related to the user input text from the scientific and technological knowledge graph, and to build a large language model trained with prompt word input to generate answers.

[0014] In a third aspect, an embodiment of the present application provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any of the aforementioned methods is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, and the computer program implements any of the aforementioned methods when executed by the processor.

[0016] The above technical solutions provided in the embodiments of the present application can bring about the following technical effects: quickly, accurately and professionally answer questions in the field of science and technology, and provide new possibilities and directions for the further development of the field of artificial intelligence in theory and practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 The present invention is a flowchart of the steps of a scientific document question and answer method according to an embodiment of the present application.

[0019] Figure 2 It is a structural principle diagram of a scientific and technological document question and answer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following describes the implementation manners of the present application through specific specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in the present application can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0021] The present application is at least proposed to solve the deficiencies of general large language models in slow information update, unprofessional and inaccurate question answering in scientific and technological field documents, and proposes a brand-new scientific and technological document question answering method, which can more accurately and efficiently implement question answering for scientific and technological field documents based on large language models, quickly, accurately and professionally answer questions, and provide new possibilities and directions for the further development of the artificial intelligence field in theory and practice.

[0022] Refer to Figure 1 , in one embodiment, the present application proposes a scientific and technological document question answering method, including the following steps: S1, a knowledge graph generation step, based on an open knowledge graph, extract triples from scientific and technological documents to generate a scientific and technological knowledge graph including entity, relationship and paragraph information. The paragraph information includes at least paragraphs and their numbers.

[0023] In one example, the knowledge graph generation step may specifically include the following processing flow: S11, extract triples from each paragraph of the scientific and technological document and integrate the extracted triples into a schema-free open knowledge graph.

[0024] Specifically, the extraction of triples can be carried out using a large language model, for example, extract entity-relationship-entity / attribute value triples such as (laser, type, semiconductor laser). Among them, schema is a structured definition of data structure, entity type, relationship and constraint conditions in the fields of knowledge graph, database, data modeling, etc., used to standardize the organization and representation of data, and ensure the consistency, understandability and interoperability of data. A schema-free open knowledge graph is a knowledge graph that does not depend on predefined data structures or entity relationship templates, etc., and allows dynamic expansion of entity types, attributes and relationships during data collection and storage.

[0025] S12, identify the similarity of phrase pairs in the open knowledge graph, and add synonym edges to the phrase pairs with similarity higher than a predetermined threshold.

[0026] Specifically, a vector encoder and cosine similarity can be used to calculate the similarity of phrase pairs in an open knowledge graph to identify synonyms, and phrase pairs with vector similarity higher than a predetermined threshold are detected, and a synonym edge is added between these phrase pairs. The synonym edge can be represented as (entity 1, synonym, entity 2), for example, (tomato, synonym, tomato).

[0027] S13. Combine the paragraphs of the scientific and technological document and add the paragraph belonging edge in the open knowledge graph.

[0028] For example, associate the triple (laser, type, semiconductor laser) with the paragraph "P001", indicating that this fact comes from this paragraph. Thus, the generated scientific and technological knowledge graph contains entity, relationship, and paragraph information at the same time.

[0029] S2. Steps for large language model training. Retrieve the user's question in the scientific and technological knowledge graph and generate an answer with a chain of thought. Judge the correctness of the generated answer according to the standard answer in the training data, and label the correct answers and wrong answers respectively. Use the user's question and the labeled answers as inputs for training.

[0030] Specifically, the retrieval of the user's question in the scientific and technological knowledge graph can be carried out using retrieval-augmented generation technology to obtain relevant triples and their corresponding paragraphs. If there are no matching triples, directly match the paragraphs. The user's question can be encoded using a vector encoder, and then the triples and paragraphs in the scientific and technological knowledge graph are encoded respectively, the similarity is calculated, and the triples, paragraphs, and the paragraphs corresponding to the triples with similarity greater than a predetermined threshold are retained.

[0031] Then, a large language model can be used to generate an answer with a chain of thought for each retrieved triple and its corresponding paragraph or directly for the matched paragraph. The large language model can be the Deepseek-R1 large model, GPT-4 large model, etc. Judge which generated answers are correct answers and which are wrong answers according to the standard answer in the training data, and label the correct answers and wrong answers with an acceptance label and a rejection label respectively. Subsequently, use the user's question input and the labeled answers as inputs for model training, so as to improve the accuracy of the large language model in generating answers according to the reference content. The direct preference optimization algorithm (Direct Preference Optimization, abbreviated as DPO) can be used for reinforcement learning training to further improve the accuracy of the large language model in generating answers according to the reference content and reduce the hallucination of the large language model.

[0032] In this embodiment, adding acceptance tags and rejection tags to the correct answers and wrong answers respectively may further include scoring the correct answers. Specifically, the scores include a first score and a second score, where: the first score is to extract the triples in the correct answer, match these triples with the triples in the graph database, obtain the number of matching triples, and each matching triple is recorded as 1 point, with the highest score being 10 points; the second score is to input the user's question and the generated answer into the large language model, and use the large language model to score the relevance between the question and the answer, with the scoring range being 0 - 10 points. Then, add the first score and the second score to get the total score, select the correct answer with the highest total score and add an acceptance tag, and at the same time randomly select a wrong answer and add a rejection tag.

[0033] S3, the question - answering step based on the knowledge graph, extracts triples and paragraphs related to the user - input text from the scientific and technological knowledge graph, constructs a prompt and inputs it into the large language model to generate an answer.

[0034] Specifically, the retrieval - enhanced generation technology can be used to retrieve the scientific and technological knowledge graph, and extract triples and paragraphs related to the user - input text from the knowledge graph. The specific method can be the same as the retrieval and extraction method in the aforementioned large language model training step. And the prompt can be constructed dynamically to further improve the answer quality.

[0035] In summary, it is different from the traditional use of retrieval - enhanced generation technology for scientific and technological document question - answering. In the knowledge graph construction stage of the above - mentioned technical solution provided by the embodiment of the present application, the large language model is used to extract triples from each paragraph of each document and integrate them into an open knowledge graph without a schema. Compared with the traditional method in other fields of first defining a schema and then storing data in the database, the database without a schema is more open, can include more scientific and technological knowledge, and saves labor costs. However, the knowledge of different data sources may have different expression forms, and knowledge fusion is required. In the embodiment of the present application, by adding synonym edges between phrase pairs to link synonyms between different paragraphs or chapters, the integration of new and old knowledge in the learning process is promoted. Finally, combining the phrase - based open knowledge graph with the original paragraphs, adding the belonging - paragraph edges, enables the generated scientific and technological knowledge graph to contain entity, relationship, and paragraph information at the same time. In the model training stage and the data retrieval stage, linking the user's question and input to relevant triples and paragraphs, and using the triples and paragraphs as the input of the model at the same time, can combine the fine - grained knowledge in the knowledge graph with the coarse - grained knowledge in the document fragments, provide more reference knowledge for the model, further improve the accuracy of the model to generate answers according to the reference content, and reduce the hallucination of the large language model.

[0036] Refer to Figure 2 , the embodiment of the present application also provides a scientific and technological document question - answering device, including: A knowledge graph generation module, which is used to extract triples from scientific and technological documents based on an open knowledge graph and generate a scientific and technological knowledge graph containing entity, relationship, and paragraph information; A large language model training module, which is used to retrieve user questions in training data in the scientific and technological knowledge graph and generate answers with a chain of thought, judge the correctness of the generated answers according to the standard answers in the training data, label the correct answers and wrong answers respectively, and use the user questions and the labeled answers as inputs to train the large language model; A question and answer module based on a knowledge graph, which is used to extract triples and paragraphs related to the user input text from the scientific and technological knowledge graph and construct a prompt to input the trained large language model to generate an answer.

[0037] An embodiment of the present application also provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the foregoing method is implemented.

[0038] An embodiment of the present application also provides an electronic device, which includes a processor and a memory; The memory is used to store a computer program; The processor is connected to the memory and is used to execute the computer program stored in the memory. When the computer program is executed by the processor, the foregoing method is implemented.

[0039] The foregoing is only a preferred embodiment of the present application. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present application, several improvements and replacements can still be made, and these improvements and replacements should also be regarded as the protection scope of the present application.

Claims

1. A method for answering questions in scientific and technological documents, characterized in that, It includes the following steps: A knowledge graph generation step, based on an open knowledge graph, extracts triples from scientific and technological documents to generate a scientific and technological knowledge graph containing entity, relationship, and paragraph information; A large language model training step, retrieves user questions in the training data in the scientific and technological knowledge graph and generates answers with a chain of thought, determines the correctness of the generated answers according to the standard answers in the training data, and labels the correct and incorrect answers respectively, and uses the user questions and the labeled answers as inputs to train the large language model; A question and answer step based on the knowledge graph, extracts triples and paragraphs related to the user input text from the scientific and technological knowledge graph, and constructs prompt words to input into the trained large language model to generate answers.

2. The method for answering questions in a scientific and technological document according to claim 1, wherein The knowledge graph generation step includes: Extract triples from each paragraph of the scientific and technological document and integrate the extracted triples into the open knowledge graph; Identify the similarity of phrase pairs in the open knowledge graph and add synonym edges to the phrase pairs with similarity higher than a predetermined threshold; Combine the paragraphs of the scientific and technological document and add belonging paragraph edges in the open knowledge graph.

3. The method for answering questions in a scientific and technological document according to claim 1, wherein The open knowledge graph is a schema-free open knowledge graph.

4. The method for answering questions in a scientific and technological document according to claim 1, characterized in that In the large language model training step, retrieve the user questions in the training data in the scientific and technological knowledge graph to obtain relevant triples and their corresponding paragraphs. If there are no matching triples, directly match the paragraphs; use the large language model to generate an answer with a chain of thought for each retrieved triple and its corresponding paragraph or directly matched paragraph.

5. The scientific and technological document question and answer method according to claim 1, wherein In the large language model training step, labeling the correct and incorrect answers respectively includes: scoring the correct answers, and selecting the correct answer with the highest score and randomly selecting an incorrect answer to label.

6. The method for answering questions in a scientific and technological document according to claim 5, characterized in that, The score is the sum of a first score and a second score. The first score is the score calculated according to the number of matching triples by matching the triples in the correct answer with the triples in the scientific and technological knowledge graph. The second score is the score calculated by inputting the user question and the generated answer into the large language model and using the large language model to score the relevance of the question and the answer.

7. The method for answering questions about scientific and technological documents according to claim 1, characterized in that, In the question and answer step based on the knowledge graph, construct prompt words dynamically.

8. A scientific and technological document Q&A device, characterized in that It includes: A knowledge graph generation module, which is used to extract triples from scientific and technological documents based on an open knowledge graph and generate a scientific and technological knowledge graph containing entity, relationship, and paragraph information; A large language model training module, which is used to retrieve user questions in the training data in the scientific and technological knowledge graph and generate answers with a chain of thought, determine the correctness of the generated answers according to the standard answers in the training data, and label the correct and incorrect answers respectively, and use the user questions and the labeled answers as inputs to train the large language model; A question and answer module based on the knowledge graph, which is used to extract triples and paragraphs related to the user input text from the scientific and technological knowledge graph, and construct prompt words to input into the trained large language model to generate answers.

9. A storage medium, characterized in that: The computer program is stored on the storage medium, and when the computer program is executed by the processor, it implements the method according to any one of claims 1-7.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory; The memory is used to store a computer program; The processor is connected to the memory and is used to execute the computer program stored in the memory. When the computer program is executed by the processor, the method according to any one of claims 1-7 is implemented.

Images