Water conservancy project knowledge question and answer construction method and system based on large model

Abstract

The invention relates to the technical field of water conservancy projects, in particular to a water conservancy project knowledge question and answer construction method and system based on a large model, and the method comprises the following steps: knowledge preparation, continuous evaluation optimization, user query processing, answer retrieval recall, and content generation. The hydraulic engineering knowledge question and answer construction method and system based on the large model have the beneficial effects that the large model and traditional knowledge of the hydraulic engineering industry are fused, so that the generated content is higher in quality and safer, and the problems of illusion and safety of LLM generation content of the large model are relieved; and a continuous evaluation index of the water conservancy field knowledge document system is provided, so that the system can answer questions with high quality.

Description

Technical Field

[0001] The present invention relates to the technical field of water conservancy engineering, and in particular to a method and system for constructing water conservancy engineering knowledge questions and answers based on a large model. Background Art

[0002] Intelligent question-and-answer technology for water conservancy engineering knowledge is an important technology involving the construction and operation and maintenance of water conservancy projects. It aims to solve problems in the field of water conservancy projects through a variety of knowledge access methods, access various general, authorized, and private domain knowledge, and build a knowledge base security management system based on authority control. Through RAG retrieval enhancement, knowledge graphs, large models and other technologies, users can automatically and accurately generate matching knowledge topics through question-and-answer interaction.

[0003] Traditional knowledge bases in the field of water conservancy engineering mainly rely on structured databases and search algorithms, and provide information retrieval services through keyword matching or semantic analysis. Their technical architecture is relatively fixed, and their intelligence is relatively low. They rely on the user's query ability and familiarity with the knowledge base to obtain the required information. The underlying principle of the general LLM large model is to make predictions based on mathematical probability, and its model output is essentially a result of probabilistic prediction. Therefore, the LLM large model sometimes speaks nonsense or generates some seemingly plausible answers. In areas where the large model is not good at, such as water conservancy engineering, the problem of hallucinations will be more serious.

[0004] In addition, consider data security: Data security is crucial for enterprises. General LLM training is a vertical field LLM, which requires users to upload their own private data to a third-party platform for training, which poses a risk of data leakage. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for constructing water conservancy engineering knowledge question and answer based on a large model to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for constructing a water conservancy engineering knowledge question and answer based on a large model, the method comprising the following steps:

[0007] Knowledge preparation: Load water conservancy project data from multiple formats and data sources, process the data to form a unified paradigm, and then perform text segmentation. According to the semantic integrity of knowledge and the token restrictions of the embedding model, select the segmentation method of "question-answer pair", "natural paragraph" or "token processing window". Then use the open source embedding model to convert the text into vector form, and use the Milvus vector library to store and manage the document vectorized data;

[0008] Continuous evaluation and optimization: The question-answering system is evaluated using three types of evaluation optimization formulas: accuracy evaluation, coverage evaluation, and efficiency evaluation. The system is optimized and adjusted based on the evaluation results.

[0009] User query processing: converting user query questions into vector form;

[0010] Answer retrieval and recall: Match in the vector database and find the closest answer in the semantic vector dimension for retrieval and recall;

[0011] Content generation: The retrieved answers are screened and provided to the large model for summarization and final generation of answers.

[0012] Preferably, in knowledge preparation, data loading and processing include multi-format data loading, acquisition of different data sources, data filtering, compression, formatting, and metadata acquisition.

[0013] Preferably, in the continuous evaluation and optimization, the accuracy evaluation measures the accuracy of the retrieval results by calculating the ratio of the number of relevant documents retrieved to the total number of retrieved documents; the coverage evaluation measures the recall ability of the retrieval system by calculating the ratio of the number of relevant documents retrieved to the number of all relevant documents in the database; the effectiveness evaluation is the harmonic average of accuracy and coverage, which is used to comprehensively evaluate the performance of the question-answering system.

[0014] Preferably, in user query processing, the method of converting the user's query question into a vector form includes using the same open source Embedding model as document vectorization.

[0015] Preferably, in content generation, the method of screening the retrieved answers includes selecting the first N closest answers, where N is a positive integer, preferably 10, and then providing them to the large model for summarization and finally generating the answer.

[0016] A water conservancy engineering knowledge question and answer construction system based on a large model is applied to a water conservancy engineering knowledge question and answer construction method based on a large model, and the system includes:

[0017] The knowledge preparation module is used to vectorize the private domain data of water conservancy projects, build indexes and store them in the database;

[0018] Continuous evaluation and optimization module, used to evaluate the question-answering system and make optimization adjustments based on the evaluation results;

[0019] The question and answer application module is used to receive user queries and generate answers.

[0020] Preferably, the knowledge preparation module specifically includes:

[0021] The knowledge access unit is used to load water conservancy project data from multiple formats and multiple data sources, and to process the data and obtain metadata to process the data into a unified paradigm;

[0022] The document segmentation unit segments the text according to the knowledge semantic integrity and the token restrictions of the embedding model, including QA question-answer pair segmentation, natural paragraph segmentation, and token processing window segmentation.

[0023] The document vectorization unit uses the open source Embedding model to convert text into vector form, and uses the Milvus vector library to store and manage document vectorization data.

[0024] Preferably, the continuous evaluation and optimization module specifically includes:

[0025] The accuracy evaluation unit is used to calculate the ratio of the number of relevant documents retrieved to the total number of documents retrieved, and measure the accuracy of the retrieval results;

[0026] The coverage evaluation unit is used to calculate the ratio of the number of retrieved relevant documents to the number of all relevant documents in the database, measuring the recall ability of the retrieval system;

[0027] The effectiveness evaluation unit, as the harmonic average of accuracy and coverage, is used to comprehensively evaluate the performance of the question answering system.

[0028] Preferably, the question-and-answer application module specifically includes:

[0029] User Query Vectorization Unit, used to convert user query questions into vector form;

[0030] The answer retrieval and recall unit matches the vector database and finds the closest answer in the semantic vector dimension for retrieval and recall;

[0031] The content generation unit screens the retrieved answers and provides them to the large model for summarization and final generation of answers.

[0032] Preferably, when performing Token processing window segmentation, the document segmentation unit segments the text into fixed-length tokens according to the token length limit of the embedding model, and mitigates the loss of semantic information by adding a certain amount of redundancy at the head and tail;

[0033] When generating an answer, the content generation unit screens the first N closest answers retrieved, where N is a positive integer and is preferably 10, and then provides them to the large model for induction and summary, and finally generates the answer.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The method and system for constructing water conservancy engineering knowledge question and answer based on a big model proposed in the present invention integrate the big model with the traditional knowledge of the water conservancy engineering industry, so that the content generated by it is of higher quality and safer, and the problems of "illusion" and "security" of the content generated by the big model LLM are alleviated; and continuous evaluation indicators of the knowledge document system in the water conservancy field are proposed to ensure that the system can answer questions with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the method of the present invention;

[0037] Figure 2 This is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] For example, see Figure 1 The present invention provides a technical solution: a method for constructing a water conservancy engineering knowledge question and answer based on a large model, the method comprising the following steps:

[0040] Knowledge preparation: Load water conservancy project data from multiple formats and multiple data sources, process the data to form a unified paradigm, and then perform text segmentation. According to the semantic integrity of the knowledge and the Tokens restrictions of the embedding model, select the "question-answer pair", "natural paragraph" or "Token processing window" segmentation method, and then use the open source Embedding model to convert the text into vector form, and use the Milvus vector library to store and manage document vectorized data; data loading and processing include multi-format data loading, acquisition of different data sources, data filtering, compression, formatting and metadata acquisition.

[0041] Continuous evaluation and optimization: The question-answering system is evaluated using three evaluation optimization formulas: accuracy evaluation, coverage evaluation, and effectiveness evaluation. The system is optimized and adjusted based on the evaluation results. Accuracy evaluation measures the accuracy of retrieval results by calculating the ratio of the number of retrieved relevant documents to the total number of retrieved documents. Coverage evaluation measures the recall ability of the retrieval system by calculating the ratio of the number of retrieved relevant documents to the number of all relevant documents in the database. Effectiveness evaluation, as the harmonic average of accuracy and coverage, is used to comprehensively evaluate the performance of the question-answering system.

[0042] User query processing: converting the user's query questions into vector form; the method of converting the user's query questions into vector form includes using the same open source Embedding model as document vectorization.

[0043] Answer retrieval and recall: Match in the vector database and find the closest answer in the semantic vector dimension for retrieval and recall;

[0044] Content generation: The retrieved answers are screened and provided to the large model for summarization and finally the answers are generated; the method for screening the retrieved answers includes selecting the first N closest answers, where N is a positive integer, preferably 10, and then providing them to the large model for summarization and finally the answers are generated.

[0045] Embodiment 2, refer to the attached Figure 2 Based on the first embodiment, a water conservancy engineering knowledge question and answer construction system based on a large model is proposed, which is applied to the water conservancy engineering knowledge question and answer construction method based on a large model. The system includes:

[0046] (1) Knowledge preparation module:

[0047] Knowledge preparation is generally an offline process, which mainly involves vectorizing the private domain data of water conservancy projects, building an index and storing it in the database. It mainly includes: data extraction, text segmentation, vectorization, data storage and other links.

[0048] 1) Knowledge access

[0049] Data loading: including loading data in multiple formats and acquiring data from different data sources. Data can be processed into the same format based on the data’s own conditions.

[0050] Data processing: including data filtering, compression, formatting, etc.

[0051] Metadata acquisition: extract key information from the data, such as file name, title, time, etc.

[0052] 2) Document fragmentation:

[0053] Text segmentation mainly considers two factors: 1) the impact of knowledge semantic integrity on retrieval results; 2) the token restrictions of the selected embedding model;

[0054] This patent adopts the following text segmentation method:

[0055] QA question-answer pair segmentation: segmentation is performed at the granularity of "question-answer pair" to retain the complete semantics of a question-answer pair.

[0056] Natural paragraph segmentation: segment at the granularity of "paragraph" to retain the complete semantics of the entire paragraph.

[0057] Token processing window segmentation: According to the token length limit of the embedding model, the text is segmented into fixed lengths (for example, 256 / 512 tokens). This segmentation method will lose a lot of semantic information, which is generally alleviated by adding a certain amount of redundancy at the beginning and end.

[0058] 3) Document vectorization (embedding):

[0059] It is to convert text into vector form, that is, to represent text information as a vector that can express the semantics of the text, to use a numerical vector to represent the semantics of the text, and to store the conversion results into a vector database.

[0060] This patent uses an open source Embedding model to achieve vector conversion of water conservancy knowledge documents.

[0061] This patent uses Milvus vector library to realize the storage and management of document vectorized data. Milvus follows the separation of data flow and control flow, and is divided into four layers as a whole, namely access layer, coordinator service, worker node and storage layer. Each layer is independent of each other, and can be expanded and disaster-tolerant independently.

[0062] (2) Continuous evaluation and optimization module:

[0063] This patent discloses three types of evaluation optimization formulas, which are distributed as accuracy evaluation, coverage evaluation and efficiency evaluation.

[0064] 1) Precision

[0065] Metric explanation: Measures how relevant the retrieved results are.

[0066] Judgment method and calculation:

[0067] Automated calculations:

[0068] Precision = (number of relevant documents retrieved) / (total number of documents retrieved);

[0069] Tools / Models: You can use a test set (where you know which documents are relevant) to automatically calculate accuracy.

[0070] According to experience, if the accuracy rate (Precision)>80%, it is considered high accuracy, otherwise re-optimization and adjustment will be performed;

[0071] 2) Coverage

[0072] Metric explanation: Measures the proportion of all relevant documents that the retrieval system can find.

[0073] Judgment method and calculation:

[0074] Automated calculations:

[0075] Coverage = (number of relevant documents retrieved) / (number of all relevant documents in the database);

[0076] Tools / Models: A labeled dataset is required to evaluate recall.

[0077] According to experience, if the recall rate is > 95%, it is considered a high recall rate. Otherwise, re-optimization and adjustment are required.

[0078] 3) Effective

[0079] Indicator explanation: The harmonic mean of precision and recall is a comprehensive indicator.

[0080] Judgment method and calculation:

[0081] Automated calculations:

[0082] Effective=2*(Precision*Recall) / (Precision+Recall);

[0083] Tools / models: also rely on correct annotation of the test set;

[0084] According to experience, if the efficiency is greater than 85%, it is considered a high recall rate, otherwise re-optimization and adjustment will be performed;

[0085] (3) Question and answer application stage

[0086] 1) User Query Vectorization

[0087] In the RAG system, the user's query question will be converted into a vector, and then matched in the vector database to find the answer that is most similar in the semantic vector dimension for retrieval and recall. Common data retrieval methods include similarity retrieval, full-text retrieval, etc. According to the retrieval effect, you can generally choose to combine multiple retrieval methods to improve the recall rate.

[0088] 2) Content Generation

[0089] Because the data recall phase only involves vector recall, this step can filter out only the top 10 pieces of knowledge returned by the previous recall phase, and then provide them to the large model for summarization and ultimately generate the answer.

[0090] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a water conservancy engineering knowledge question and answer based on a large model, characterized by: The method comprises the following steps: Knowledge preparation: Load water conservancy project data from multiple formats and data sources, process the data to form a unified paradigm, and then perform text segmentation. According to the semantic integrity of knowledge and the token restrictions of the embedding model, select the segmentation method of "question-answer pair", "natural paragraph" or "token processing window". Then use the open source embedding model to convert the text into vector form, and use the Milvus vector library to store and manage the document vectorized data; Continuous evaluation and optimization: The question-answering system is evaluated using three types of evaluation optimization formulas: accuracy evaluation, coverage evaluation, and efficiency evaluation. The system is optimized and adjusted based on the evaluation results. User query processing: converting user query questions into vector form; Answer retrieval and recall: Match in the vector database and find the closest answer in the semantic vector dimension for retrieval and recall; Content generation: The retrieved answers are screened and provided to the large model for summarization and final generation of answers.

2. A method for constructing a water conservancy engineering knowledge question and answer based on a large model according to claim 1, characterized in that: In knowledge preparation, data loading and processing include multi-format data loading, acquisition of different data sources, data filtering, compression, formatting, and metadata acquisition.

3. The method for constructing a water conservancy engineering knowledge question and answer based on a large model according to claim 1, characterized in that: In the continuous evaluation and optimization, the precision evaluation measures the accuracy of the retrieval results by calculating the ratio of the number of retrieved relevant documents to the total number of retrieved documents; the coverage evaluation measures the recall ability of the retrieval system by calculating the ratio of the number of retrieved relevant documents to the number of all relevant documents in the database; the effectiveness evaluation is the harmonic average of the accuracy and coverage, which is used to comprehensively evaluate the performance of the question-answering system.

4. The method for constructing a water conservancy engineering knowledge question and answer based on a large model according to claim 1, characterized in that: In user query processing, the method of converting the user's query question into a vector form includes using the same open source Embedding model as document vectorization.

5. The method for constructing a water conservancy engineering knowledge question and answer based on a large model according to claim 1, characterized in that: In content generation, the method of screening the retrieved answers includes selecting the first N closest answers, where N is a positive integer, preferably 10, and then providing them to the large model for summarization and finally generating the answer.

6. A water conservancy engineering knowledge question-answering construction system based on a large model, applied to the water conservancy engineering knowledge question-answering construction method based on a large model according to any one of claims 1 to 5, characterized in that: The system comprises: The knowledge preparation module is used to vectorize the private domain data of water conservancy projects, build indexes and store them in the database; Continuous evaluation and optimization module, used to evaluate the question-answering system and make optimization adjustments based on the evaluation results; The question and answer application module is used to receive user queries and generate answers.

7. A large-model-based water conservancy engineering knowledge question-answering system according to claim 6, characterized in that: The knowledge preparation modules include: The knowledge access unit is used to load water conservancy project data from multiple formats and multiple data sources, and to process the data and obtain metadata to process the data into a unified paradigm; The document segmentation unit segments the text according to the knowledge semantic integrity and the token restrictions of the embedding model, including QA question-answer pair segmentation, natural paragraph segmentation, and token processing window segmentation. The document vectorization unit uses the open source Embedding model to convert text into vector form, and uses the Milvus vector library to store and manage document vectorization data.

8. A large-model-based water conservancy engineering knowledge question-answering system according to claim 6, characterized in that: The continuous evaluation and optimization module specifically includes: The accuracy evaluation unit is used to calculate the ratio of the number of relevant documents retrieved to the total number of documents retrieved, and measure the accuracy of the retrieval results; The coverage evaluation unit is used to calculate the ratio of the number of retrieved relevant documents to the number of all relevant documents in the database, measuring the recall ability of the retrieval system; The effectiveness evaluation unit, as the harmonic average of accuracy and coverage, is used to comprehensively evaluate the performance of the question answering system.

9. A large-model-based water conservancy engineering knowledge question-answering system according to claim 6, characterized in that: The question and answer application module specifically includes: User Query Vectorization Unit, used to convert user query questions into vector form; The answer retrieval and recall unit matches the vector database and finds the closest answer in the semantic vector dimension for retrieval and recall; The content generation unit screens the retrieved answers and provides them to the large model for summarization and final generation of answers.

10. A large-scale model-based water conservancy engineering knowledge question-answering system according to claim 7, characterized in that: When the document segmentation unit performs token processing window segmentation, it segments the text into fixed-length tokens according to the token length limit of the embedding model, and mitigates the loss of semantic information by adding a certain amount of redundancy at the beginning and end. When generating an answer, the content generation unit screens the first N closest answers retrieved, where N is a positive integer and is preferably 10, and then provides them to the large model for induction and summary, and finally generates the answer.

Images