Water conservancy knowledge base construction method and system based on large model

Abstract

The invention discloses a water conservancy knowledge base construction method and system based on a large model, and relates to the technical field of intelligent water conservancy. Water conservancy related knowledge is integrated and managed, water conservancy knowledge dispersed in different regions and different departments is organically integrated through a large model, an information island is broken, and knowledge sharing and collaboration are achieved. Through data cleaning, standardization processing and other technologies, the large model can convert disordered data into standard and ordered knowledge, and more accurate and reliable information services are provided; the water conservancy knowledge base is constructed based on the knowledge graph, structured, unstructured and semi-structured water conservancy data can be flexibly and conveniently accessed, and multiple data types are compatible; massive water conservancy data and information are efficiently integrated and analyzed by using a large model, valuable information and knowledge are effectively extracted through deep mining of multi-source heterogeneous data, and water conservancy related technicians are effectively assisted in managing water conservancy projects.

Description

Technical Field

[0001] The present invention relates to the field of smart water conservancy technology, and in particular to a method and system for constructing a water conservancy knowledge base based on a large model. Background Art

[0002] At present, the construction of water conservancy knowledge platform is mainly based on converting various business rules and experiences into knowledge base. Under the urgent need for intelligent transformation and upgrading of water conservancy, how to use large models to efficiently use this knowledge, and how to let knowledge intelligence drive models to carry out business calculations are the directions that urgently need technical research.

[0003] The water conservancy industry is in urgent need of efficient, accurate and intelligent management solutions. The application of large model technology is expected to achieve breakthroughs in the fields of hydrological automation, basin water simulation and prediction, water project safety analysis and scientific scheduling. However, the application of general large language models in the field of water conservancy has semantic illusions and irrelevant answers. Therefore, the primary goal of building these professional field large models is to collect and establish the industry's knowledge base, and use the knowledge base as a basis to deal with specific scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for constructing a water conservancy knowledge base based on a large model to improve the above technical problems.

[0005] In order to achieve the above-mentioned object of the invention, the embodiment of the present invention provides the following technical solutions:

[0006] A method for constructing a water conservancy knowledge base based on a large model includes:

[0007] S1. Acquire water conservancy data of various data types and preprocess them to obtain preprocessed water conservancy data; the data types include PDF format, web page format and structured format;

[0008] S2, performing data cleaning on the pre-processed water conservancy data to obtain first water conservancy data;

[0009] S3. Use the big model and the knowledge graph to annotate the first water conservancy data and construct the corresponding first water conservancy knowledge graph;

[0010] S4. Based on the first water conservancy knowledge graph and knowledge base framework, construct a corresponding water conservancy knowledge base;

[0011] S5, collect the latest water conservancy data in real time, and process it using the same method as S1 to S3 to obtain corresponding second water conservancy data and second water conservancy knowledge graph;

[0012] S6. Update the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph.

[0013] Furthermore, the water conservancy data includes basic water conservancy information, hydrological data, water conservancy project data, water resource management data, water ecological environment data, water conservancy historical data, water conservancy business rule data, and water conservancy expert experience data.

[0014] Furthermore, the preprocessing includes:

[0015] For water conservancy data in PDF format: use OCR method and recognition tools to extract, identify and write the water conservancy data to obtain the corresponding pre-processed water conservancy data;

[0016] For water conservancy data in web page format: use crawler software to collect water conservancy data to obtain collected water conservancy data; use large model tools to analyze the collected water conservancy data to obtain corresponding pre-processed water conservancy data;

[0017] For water conservancy data with structured data type: use the big model to extract and analyze the water conservancy data to generate corresponding text data, that is, obtain the corresponding pre-processed water conservancy data.

[0018] Furthermore, the pre-processed water conservancy data is cleaned to obtain the first water conservancy data, including the following steps:

[0019] S2-1, converting the pre-processed water conservancy data corresponding to different data types into a unified format to obtain water conservancy data;

[0020] S2-2, using a regularized expression to match the water conservancy data in a uniform format to obtain a corresponding first matching result;

[0021] S2-3, based on the first matching result, removing irrelevant characters and sensitive characters in the water conservancy data with a unified format to obtain the water conservancy data after the removal;

[0022] S2-4, using the pinyin check library and the language big model to perform spelling correction on the water conservancy data after the removal, to obtain the corrected water conservancy data; the language big model includes the Chinese-Bert model and the SpellGCN model;

[0023] S2-5, deduplicating the corrected water conservancy data to obtain deduplicated water conservancy data;

[0024] S2-6. Classify the water conservancy data after deduplication to obtain first water conservancy data.

[0025] Furthermore, the use of the large model and the knowledge graph to annotate the first water conservancy data and construct the corresponding first water conservancy knowledge graph includes the following steps:

[0026] S3-1. Use the knowledge graph natural language processing tool to identify and extract the first water conservancy data to obtain corresponding entities and the relationships between them;

[0027] S3-2. Based on entities and their relationships, a corresponding water conservancy entity relationship recognition dataset is constructed;

[0028] S3-3, constructing an entity relationship recognition model in the water conservancy field; the entity relationship recognition model in the water conservancy field includes a DeepKE model;

[0029] S3-4, input the water conservancy field entity relationship recognition data set into the water conservancy field entity relationship recognition model, and output the corresponding knowledge;

[0030] S3-5. Based on the knowledge, construct the corresponding first water conservancy knowledge graph; the first water conservancy knowledge graph includes the basic water conservancy knowledge graph, the plan water conservancy knowledge graph, the historical water conservancy knowledge graph, the water conservancy business rules knowledge graph and the water conservancy expert experience knowledge graph.

[0031] Furthermore, the training process of the DeepKE model includes the following steps:

[0032] S3-3-1. Obtain first training water conservancy data and its label; the label is the annotation result corresponding to the first training water conservancy data, that is, the entity and the relationship between the entities;

[0033] S3-3-2, dividing the first training water conservancy data into a training set, a validation set and a test set;

[0034] S3-3-3. Set the initial model parameters of the DeepKE model and select the corresponding training strategy;

[0035] S3-3-4. Based on the initial parameters and training strategy, the training set is input into the DeepKE model for training and optimization, and the corresponding model parameters are adjusted to obtain the optimized DeepKE model.

[0036] S3-3-5. Input the validation set and test set into the optimized DeepKE model respectively to obtain the corresponding validation results and test results;

[0037] S3-3-6. Based on the verification results and the test results, calculate the corresponding verification performance indicators and test performance indicators;

[0038] S3-3-7. Determine whether both the verification performance index and the test performance index meet the performance conditions; if so, complete the training of the DeepKE model; otherwise, return to S3-3-3.

[0039] Furthermore, the knowledge base framework is a bottom-up five-layer architecture of "data layer - model layer - instance layer - storage layer - application layer";

[0040] The method of constructing a corresponding water conservancy knowledge base based on the first water conservancy knowledge graph and the knowledge base framework includes the following steps:

[0041] S4-1, constructing a water conservancy vector database; the water conservancy vector database is a Milvus vector database;

[0042] S4-2, processing the data in each first water conservancy knowledge graph to obtain processed data;

[0043] S4-3, storing the processed data into a water conservancy vector database;

[0044] S4-4. Based on the knowledge base framework and water conservancy vector database, construct the corresponding water conservancy knowledge base.

[0045] Furthermore, the latest water conservancy data are newly published water conservancy research, water conservancy books, and water conservancy projects;

[0046] The updating of the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph comprises the following steps:

[0047] S6-1, calculating a first similarity between the first water conservancy data and the second water conservancy data;

[0048] S6-2, calculating a second similarity between the first water conservancy knowledge graph and the second water conservancy knowledge graph;

[0049] S6-3. Update the water conservancy knowledge base based on the update rule, the first similarity and the second similarity.

[0050] A water conservancy knowledge base construction system based on a large model includes a water conservancy data acquisition module, a preprocessing module, a data cleaning module, a data annotation module, a water conservancy knowledge base construction module, a water conservancy knowledge base update module and a water conservancy knowledge base application module; wherein:

[0051] The water conservancy data acquisition module is used to obtain water conservancy data of various data types; collect the latest water conservancy data in real time;

[0052] A preprocessing module is used to preprocess the water conservancy data and the latest water conservancy data to obtain the preprocessed water conservancy data and the preprocessed latest water conservancy data;

[0053] A data cleaning module is used to clean the pre-processed water conservancy data and the pre-processed latest water conservancy data to obtain the corresponding first water conservancy data and second water conservancy data;

[0054] A data annotation module, used to annotate the first water conservancy data and the second water conservancy data using a large model and a knowledge graph, and construct a corresponding first water conservancy knowledge graph and a second water conservancy knowledge graph;

[0055] A water conservancy knowledge base construction module, used to construct a corresponding water conservancy knowledge base based on the first water conservancy knowledge graph and the knowledge base framework;

[0056] A water conservancy knowledge base updating module, used for updating the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph;

[0057] The water conservancy knowledge base application module is used to use the water conservancy knowledge base to conduct water conservancy intelligent knowledge question and answer and water conservancy intelligent query / retrieval.

[0058] Furthermore, the preprocessing module includes a parallel PDF preprocessing submodule, a web page preprocessing submodule and a structured preprocessing submodule; wherein:

[0059] The PDF preprocessing submodule is used to extract, identify and write water conservancy data in PDF format using the OCR method to obtain the corresponding preprocessed water conservancy data;

[0060] The webpage preprocessing submodule is used to collect water conservancy data in webpage format using crawler software to obtain collected water conservancy data; and to analyze the collected water conservancy data using a large model tool to obtain corresponding preprocessed water conservancy data;

[0061] The structured preprocessing submodule is used to extract and analyze water conservancy data with structured data types using a large model, generate corresponding text data, and obtain corresponding preprocessed water conservancy data.

[0062] The beneficial effects of the present invention are:

[0063] This method integrates and manages water conservancy related knowledge, and organically integrates water conservancy knowledge scattered in different regions and departments through a large model, breaking down information silos and realizing knowledge sharing and collaboration. Through data cleaning, standardization and other technologies, the large model can transform disorganized data into standardized and orderly knowledge, providing more accurate and reliable information services; building a water conservancy knowledge base based on the knowledge graph can flexibly and conveniently access structured, unstructured and semi-structured water conservancy data, and is compatible with multiple data types, making data management more flexible and convenient.

[0064] This system has powerful language understanding, knowledge processing and data analysis capabilities. It uses large models to efficiently integrate and analyze massive amounts of water conservancy data and information. Through deep mining of multi-source heterogeneous data, it effectively extracts valuable information and knowledge, provides the information services needed by water conservancy-related technical personnel, and effectively assists water conservancy-related technical personnel in managing water conservancy projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 A flow chart of a method in an embodiment of the present invention;

[0067] Figure 2 is an overall block diagram of a water conservancy knowledge base in an embodiment of the present invention;

[0068] Figure 3 A system block diagram in an embodiment of the present invention;

[0069] Figure 4 A schematic diagram of an intelligent application in an embodiment of the present invention;

[0070] Figure 5 4 is a structural block diagram of a preprocessing module in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0072] See also Figure 1 and Figure 2 This embodiment provides a method for constructing a water conservancy knowledge base based on a large model, which includes:

[0073] S1. Acquire water conservancy data of various data types and preprocess them to obtain preprocessed water conservancy data; the data types include pdf format, web page format and structured format; pdf format and web page format are unstructured; structured formats include doc format, xlsx format and csv format.

[0074] The water conservancy data includes basic water conservancy information, hydrological data, water conservancy project data, water resource management data, water ecological environment data, water conservancy historical data, water conservancy business rule data, and water conservancy expert experience data.

[0075] The pre-processing comprises:

[0076] For water conservancy data in PDF format: use OCR method and recognition tools to extract, identify and write the water conservancy data to obtain the corresponding pre-processed water conservancy data; OCR method is a technology that converts text in various types of documents into text corresponding to machine encoding. This embodiment uses Umi-OCR open source tool for recognition. Umi-OCR is a powerful open source optical character recognition tool with built-in deep learning model. Through advanced algorithms, it can realize high-precision and high-speed capture and conversion of text information in various complex images. For some books and literature materials, text data can be quickly extracted.

[0077] For water conservancy data in web page format: use crawler software to collect water conservancy data to obtain collected water conservancy data; use big model tools to parse the collected water conservancy data to obtain corresponding pre-processed water conservancy data; big model tools are intelligent data processing tools developed based on deep learning technology.

[0078] Acquire data in web page format through crawlers, extract web page structure with the help of open source crawler libraries and software, quickly acquire text content, and quickly build collection tasks with Octopus to quickly collect repeated subpages. Or directly parse web pages with the help of large model tools to directly sort out valid text data. Or use Reader and Firecrawl tools to achieve automated crawling. This can solve the problem of being restricted by anti-pickpocketing when crawling some websites and reduce resource search time.

[0079] For water conservancy data in structured format: use the big model to extract and analyze the water conservancy data to generate corresponding text data, such as [River name: Yellow River, River system: Yellow River, Origin: ...], that is, to obtain the corresponding pre-processed water conservancy data. Or ignore the processing of structured data, build an intelligent agent, and let the big model call the internal function to automatically filter and find the required content when querying is required.

[0080] Preprocessing also includes merging, and the preprocessed water conservancy data are merged according to the merging conditions. The merging conditions are: calculating the spatiotemporal correlation and semantic correlation of the preprocessed water conservancy data, fusing the preprocessed water conservancy data whose spatiotemporal correlation and semantic correlation both meet the threshold, and only deeply fusing the data with strong spatiotemporal-semantic correlation; the calculation method of spatiotemporal correlation and semantic correlation is an existing technology, so it will not be described in detail.

[0081] S2, performing data cleaning on the pre-processed water conservancy data to obtain first water conservancy data;

[0082] Since the pre-processed water conservancy data contains a lot of noise interference, such as garbled characters, typesetting errors, missing characters and other shortcomings, which seriously affect the quality of text data and cause confusion in subsequent applications, the pre-processed water conservancy data is cleaned to obtain the first water conservancy data, including the following steps:

[0083] S2-1. Perform format conversion on the pre-processed water conservancy data corresponding to different data types to unify the encoding format of the text, for example, converting traditional Chinese into simplified Chinese to obtain water conservancy data in a unified format;

[0084] S2-2, using a regularized expression to match the water conservancy data in a uniform format to obtain a corresponding first matching result;

[0085] S2-3. Based on the first matching result, irrelevant characters and sensitive characters are removed from the water conservancy data in a unified format, such as line breaks, spaces, headers, footers and other irrelevant information, such as personal privacy information (user ID number, contact information, etc.), infrastructure secrets (reservoir / hydropower station latitude and longitude coordinates, design drawing parameters, embankment engineering structure details, etc.), national security data (codes of confidential water conservancy projects, construction standards, etc.) and other sensitive characters to obtain the removed water conservancy data;

[0086] S2-4. Use the pinyin check library and the language big model to perform spelling correction on the removed water conservancy data to obtain the corrected water conservancy data; for English text data, the spelling check library (PySpellChecker) can be used for identification and correction. For Chinese text data, the errors are more complex and difficult to judge directly; because Chinese spelling errors are mainly concentrated in the two aspects of pinyin similarity and glyph similarity, part of it is because the OCR method causes certain Chinese spelling errors during recognition, so the language big model can be used for identification and error correction. Among them, the language big model includes the Chinese-Bert model and the SpellGCN model;

[0087] S2-5. De-duplicate the corrected water conservancy data, such as de-duplicating file names and data MD5 values, and clean up some easily distinguishable duplicate data to obtain de-duplicated water conservancy data, thereby further reducing the difficulty of subsequent processing, reducing the data processing time of this method, and saving a certain amount of workload.

[0088] S2-6. Classify the water conservancy data after deduplication to obtain first water conservancy data.

[0089] S3. Use the big model and the knowledge graph to annotate the first water conservancy data and construct the corresponding first water conservancy knowledge graph;

[0090] The method of using the large model and the knowledge graph to annotate the first water conservancy data and construct the corresponding first water conservancy knowledge graph includes the following steps:

[0091] S3-1. Use the knowledge graph natural language processing tool to identify and extract the first water conservancy data to obtain corresponding entities and the relationships between them;

[0092] S3-2. Based on entities and their relationships, a corresponding water conservancy entity relationship recognition dataset is constructed;

[0093] S3-3. Construct an entity relationship recognition model in the water conservancy field; the entity relationship recognition model in the water conservancy field includes the DeepKE model;

[0094] The training process of the DeepKE model includes the following steps:

[0095] S3-3-1. Obtain first training water conservancy data and its label; the label is the annotation result corresponding to the first training water conservancy data, that is, the entity and the relationship between the entities;

[0096] S3-3-2, dividing the first training water conservancy data into a training set, a validation set and a test set;

[0097] S3-3-3. Set the initial model parameters of the DeepKE model and select the corresponding training strategy;

[0098] S3-3-4. Based on the initial model parameters and training strategy, the training set is input into the DeepKE model for training and optimization, and the corresponding model parameters are adjusted to obtain the optimized DeepKE model;

[0099] S3-3-5. Input the validation set and test set into the optimized DeepKE model respectively to obtain the corresponding validation results and test results;

[0100] S3-3-6. Based on the verification results and the test results, calculate the corresponding verification performance indicators and test performance indicators;

[0101] S3-3-7. Determine whether both the verification performance index and the test performance index meet the performance conditions. If so, complete the training of the DeepKE model. Otherwise, return to S3-3-3. The performance conditions include the verification performance index threshold and the test performance index threshold. Set the verification performance index threshold and the test performance index threshold according to the actual situation.

[0102] S3-4, input the water conservancy field entity relationship recognition data set into the water conservancy field entity relationship recognition model, and output the corresponding knowledge;

[0103] S3-5. Based on the knowledge, construct the corresponding first water conservancy knowledge graph; the first water conservancy knowledge graph includes the basic water conservancy knowledge graph, the plan water conservancy knowledge graph, the historical water conservancy knowledge graph, the water conservancy business rules knowledge graph and the water conservancy expert experience knowledge graph.

[0104] S4. Based on the first water conservancy knowledge graph and knowledge base framework, construct a corresponding water conservancy knowledge base;

[0105] The knowledge base framework is a bottom-up five-layer architecture of "data layer-model layer-instance layer-storage layer-application layer"; among them, the storage layer includes a vector database and a graph database, the vector database adopts Milvus vector database, and the graph database is Neo4j graph database.

[0106] The method of constructing a corresponding water conservancy knowledge base based on the first water conservancy knowledge graph and the knowledge base framework includes the following steps:

[0107] S4-1. Construct a water conservancy vector database; the water conservancy vector database is the Milvus vector database; since more than 80% to 90% of the data on the Internet is unstructured information, and unstructured data does not have a standard row and column structure, it is completely different from the storage, analysis, mining and query of structured data, and it is difficult to store the content of unstructured data in a relational database. The vector database is the storage core of the large model knowledge base. In industry practice, retrieval enhancement generation (RAG) is usually closely integrated with the vector database, which relies on the vector database to efficiently retrieve relevant information to enhance the large model. By converting the query generated by the large model into a vector, the RAG system can quickly locate the corresponding knowledge entry in the vector database. This retrieval mechanism enables the large model to use the latest information stored in the vector database when facing specific problems, effectively solving the problems of knowledge update delays and illusions inherent in the large model, thereby selecting the Milvus vector database.

[0108] Milvus vector database is different from SQL database direct query. Vector database retrieval is mainly based on the similarity calculation method between vector data to retrieve the data most similar to the query vector. Generally, it does not enter the database itself for query, but converts it into a high-dimensional vector format through external tools (embedding model) for retrieval. The retrieval results will be arranged according to the similarity with the query vector, and the top k most similar text blocks will be retained.

[0109] S4-2, processing the data in each first water conservancy knowledge graph, that is, performing word segmentation, cleaning and vectorization on the data in each first water conservancy knowledge graph to obtain processed data; Figure 2 As shown in the figure, the specific process of vectorization is: divide the corresponding text data to obtain the divided text data; input the divided text data into the embedding model for vectorization, generate a fixed high-dimensional vector (1024 dimensions), and obtain the corresponding high-dimensional vector data. The principle is to map the unstructured data into a vector space with a fixed dimension, and similar text segments have a closer distance in the space; store the high-dimensional vector data and the corresponding text in the water conservancy vector database. The embedding model can adopt the embedding model and the bge-base model.

[0110] S4-3, storing the processed data into a water conservancy vector database;

[0111] S4-4. Based on the knowledge base framework and water conservancy vector database, construct the corresponding water conservancy knowledge base.

[0112] S5, collect the latest water conservancy data in real time, and process it using the same method as S1 to S3 to obtain corresponding second water conservancy data and second water conservancy knowledge graph;

[0113] S6. Update the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph.

[0114] The latest water conservancy data mentioned above are newly published water conservancy research, water conservancy books, and water conservancy projects;

[0115] The updating of the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph comprises the following steps:

[0116] S6-1, calculating a first similarity between the first water conservancy data and the second water conservancy data;

[0117] S6-2, calculating a second similarity between the first water conservancy knowledge graph and the second water conservancy knowledge graph;

[0118] S6-3. Update the water conservancy knowledge base based on the update rule, the first similarity and the second similarity.

[0119] Adopt update rules that combine dynamic update, traceability, knowledge association, and automation and manual review, namely:

[0120] (1) Similarity calculation methods: text similarity, graph structure similarity, semantic similarity. Combine multiple indicators to calculate and obtain the corresponding average similarity.

[0121] (2) Specific conditions of the update rules: When the average similarity exceeds 90%, the first water conservancy data is covered by the second water conservancy data; when the average similarity is in the range of 50%-90%, the first water conservancy data and the second water conservancy data are merged; when the average similarity is lower than 50%, the second water conservancy data is used as new data and stored in the water conservancy knowledge base.

[0122] (3) Processing of raw data: retain historical data for traceability.

[0123] (4) Knowledge association methods: linking new and old knowledge through entity linking, attribute inheritance, time-space association, etc.

[0124] (5) Conflict resolution mechanism: When new and old data conflict, manual review or priority determination is performed.

[0125] In summary, this method integrates and manages water conservancy related knowledge, and organically integrates water conservancy knowledge scattered in different regions and departments through a large model, breaking down information silos and realizing knowledge sharing and collaboration. Through data cleaning, standardization and other technologies, the large model can transform disorganized data into standardized and orderly knowledge, providing more accurate and reliable information services; building a water conservancy knowledge base based on knowledge graphs can flexibly and conveniently access structured, unstructured and semi-structured water conservancy data, and is compatible with a variety of data types, making data management more flexible and convenient.

[0126] like Figure 3 As shown, a water conservancy knowledge base construction system based on a large model is provided, which includes a water conservancy data acquisition module, a preprocessing module, a data cleaning module, a data annotation module, a water conservancy knowledge base construction module, a water conservancy knowledge base update module and a water conservancy knowledge base application module; wherein:

[0127] The water conservancy data acquisition module is used to obtain water conservancy data of various data types; collect the latest water conservancy data in real time;

[0128] A preprocessing module is used to preprocess the water conservancy data and the latest water conservancy data to obtain the preprocessed water conservancy data and the preprocessed latest water conservancy data;

[0129] A data cleaning module is used to clean the pre-processed water conservancy data and the pre-processed latest water conservancy data to obtain the corresponding first water conservancy data and second water conservancy data;

[0130] A data annotation module, used to annotate the first water conservancy data and the second water conservancy data using a large model and a knowledge graph, and construct a corresponding first water conservancy knowledge graph and a second water conservancy knowledge graph;

[0131] A water conservancy knowledge base construction module is used to construct a corresponding water conservancy knowledge base based on the first water conservancy knowledge graph and the knowledge base framework; the water conservancy knowledge base includes a basic water conservancy knowledge base, a plan water conservancy knowledge base, a historical water conservancy knowledge base, a water conservancy business rule knowledge base and a water conservancy expert experience knowledge base;

[0132] A water conservancy knowledge base updating module, used for updating the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph;

[0133] The water conservancy knowledge base application module is used to use the water conservancy knowledge base to conduct water conservancy intelligent knowledge question and answer and water conservancy intelligent query / retrieval.

[0134] When performing intelligent queries, Figure 4As shown in the figure, external resources (book encyclopedias, historical archives, journals, laws and regulations, project materials, expert experience, standards and specifications, etc.) and data bases (basic data, business data, monitoring data, etc.) are classified in the data layer of the water conservancy knowledge base to determine whether the data is in the fields of data twin water conservancy, flood control and drought relief, water resources scheduling, water conservancy project construction, etc., and then further classified into unstructured data and structured data. For structured data, data extraction and sorting are performed on different types of files. For unstructured data, identification, format conversion, and data cleaning are performed in turn; the text files of unstructured data are divided into fixed blocks and semantics to obtain the corresponding text blocks, and the text blocks are vectorized and stored in the vector database in the storage layer of the water conservancy knowledge base. In the model layer of the water conservancy knowledge base, based on the concept and relationship classification system of the water conservancy field, the corresponding entities and relationships are extracted from the processed structured data, the processed unstructured data, and the text blocks, and the entities and relationships are classified to construct the corresponding entity model. The classified entities and relations are labeled and then input into the entity recognition model and the relationship recognition model respectively to obtain the corresponding entity recognition results and relationship recognition results. The entity recognition results and the relationship recognition results are manually corrected. Based on the manually corrected entity recognition results and the manually corrected relationship recognition results, the corresponding entity-relationship is constructed. In the instance layer of the water conservancy knowledge base, the corresponding links are constructed based on the entity-relationship. Based on the links, the heterogeneous entities are fused to construct the corresponding knowledge graph. The text block is used as the original corpus and input into the large language model for training and fine-tuning to obtain a trained large language model. In the storage layer of the water conservancy knowledge base, the knowledge graph is used as a graph database and connected to the trained large language model with the vector database. In the application layer of the water conservancy knowledge base, when the client performs a document query, the trained large language model is run, and the graph database and the vector database are searched. The corresponding search results are fed back to the trained large language model for processing and then fed back to the client.

[0135] When performing intelligent retrieval, for the water conservancy knowledge base (basic water conservancy knowledge base, emergency water conservancy knowledge base, historical water conservancy knowledge base, water conservancy business rules knowledge base, and water conservancy expert experience knowledge base), enter the entity name keyword in the search box, and the entity list will list the entities related to the keyword in real time. In the fuzzy search list, select one of them as the search entity, and the search results of the corresponding knowledge base will be displayed after entering the search. For each water conservancy object, multi-dimensional information can be obtained. The user's search keywords are semantically expanded to return richer and more comprehensive information. For example, searching for the Xiaolangdi Reservoir will not only return the reservoir working information, but also return flood control semantic related information such as dispatching rules, downstream protection objects, responsible persons, building information models (BIM), and video surveillance.

[0136] When conducting intelligent question-answering, the deep fusion of large language models and knowledge graphs can achieve in-depth understanding and accurate answers to water conservancy knowledge. Intelligent question-answering can obtain water conservancy document data from multiple trusted data sources, and provide users with the best answer content through data processing and question-answering models. It can answer basic knowledge about water conservancy, such as the basic situation of Xiaolangdi Reservoir, the total storage capacity of Xiaolangdi Reservoir, etc., and can also answer prediction questions such as the expected water inflow of Xiaolangdi Reservoir.

[0137] like Figure 5 As shown, the preprocessing module includes a parallel PDF preprocessing submodule, a web page preprocessing submodule and a structured preprocessing submodule; wherein:

[0138] The PDF preprocessing submodule is used to extract, identify and write water conservancy data in PDF format using the OCR method;

[0139] The webpage preprocessing submodule is used to collect water conservancy data in webpage format using crawler software to obtain collected water conservancy data; and to analyze the collected water conservancy data using a large model tool;

[0140] The structured preprocessing submodule is used to extract and analyze water conservancy data in a structured format using a large model to generate corresponding text data.

[0141] In summary, this system has powerful language understanding, knowledge processing and data analysis capabilities. It uses large models to efficiently integrate and analyze massive amounts of water conservancy data and information. Through deep mining of multi-source heterogeneous data, it effectively extracts valuable information and knowledge, provides the information services needed by water conservancy-related technical personnel, and effectively assists water conservancy-related technical personnel in managing water conservancy projects.

[0142] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for constructing a water conservancy knowledge base based on a large model, characterized in that: include: S1. Acquire water conservancy data of various data types and preprocess them to obtain preprocessed water conservancy data; The data types include pdf format, web page format and structured format; S2, performing data cleaning on the pre-processed water conservancy data to obtain first water conservancy data; S3. Use the big model and the knowledge graph to annotate the first water conservancy data and construct the corresponding first water conservancy knowledge graph; S4. Based on the first water conservancy knowledge graph and knowledge base framework, construct a corresponding water conservancy knowledge base; S5, collect the latest water conservancy data in real time, and process it using the same method as S1 to S3 to obtain corresponding second water conservancy data and second water conservancy knowledge graph; S6. Update the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph.

2. The method for constructing a water conservancy knowledge base based on a large model according to claim 1 is characterized in that: The water conservancy data includes basic water conservancy information, hydrological data, water conservancy project data, water resource management data, water ecological environment data, water conservancy historical data, water conservancy business rule data, and water conservancy expert experience data.

3. The method for constructing a water conservancy knowledge base based on a large model according to claim 1 is characterized in that: The pre-processing comprises: For water conservancy data in PDF format: use OCR method and recognition tools to extract, identify and write the water conservancy data to obtain the corresponding pre-processed water conservancy data; For water conservancy data in web page format: use crawler software to collect water conservancy data to obtain collected water conservancy data; use large model tools to analyze the collected water conservancy data to obtain corresponding pre-processed water conservancy data; For water conservancy data with structured data type: use the big model to extract and analyze the water conservancy data to generate corresponding text data, that is, obtain the corresponding pre-processed water conservancy data.

4. The method for constructing a water conservancy knowledge base based on a large model according to claim 1 is characterized in that: The step of performing data cleaning on the pre-processed water conservancy data to obtain the first water conservancy data comprises the following steps: S2-1, converting the pre-processed water conservancy data corresponding to different data types into a unified format to obtain water conservancy data; S2-2, using a regularized expression to match the water conservancy data in a uniform format to obtain a corresponding first matching result; S2-3, based on the first matching result, removing irrelevant characters and sensitive characters in the water conservancy data with a unified format to obtain the water conservancy data after the removal; S2-4, using the pinyin check library and the language big model to perform spelling correction on the water conservancy data after the removal, to obtain the corrected water conservancy data; the language big model includes the Chinese-Bert model and the SpellGCN model; S2-5, deduplicating the corrected water conservancy data to obtain deduplicated water conservancy data; S2-6. Classify the water conservancy data after deduplication to obtain first water conservancy data.

5. The method for constructing a water conservancy knowledge base based on a large model according to claim 1 is characterized in that: The method of using the large model and the knowledge graph to annotate the first water conservancy data and construct the corresponding first water conservancy knowledge graph includes the following steps: S3-1. Use the knowledge graph natural language processing tool to identify and extract the first water conservancy data to obtain corresponding entities and the relationships between them; S3-2. Based on entities and their relationships, a corresponding water conservancy entity relationship recognition dataset is constructed; S3-3, constructing an entity relationship recognition model in the water conservancy field; the entity relationship recognition model in the water conservancy field includes a DeepKE model; S3-4, input the water conservancy field entity relationship recognition data set into the water conservancy field entity relationship recognition model, and output the corresponding knowledge; S3-5. Based on the knowledge, construct the corresponding first water conservancy knowledge graph; the first water conservancy knowledge graph includes the basic water conservancy knowledge graph, the plan water conservancy knowledge graph, the historical water conservancy knowledge graph, the water conservancy business rules knowledge graph and the water conservancy expert experience knowledge graph.

6. The method for constructing a water conservancy knowledge base based on a large model according to claim 5 is characterized in that: The training process of the DeepKE model includes the following steps: S3-3-1. Obtain first training water conservancy data and its label; the label is the annotation result corresponding to the first training water conservancy data, that is, the entity and the relationship between the entities; S3-3-2, dividing the first training water conservancy data into a training set, a validation set and a test set; S3-3-3. Set the initial model parameters of the DeepKE model and select the corresponding training strategy; S3-3-4. Based on the initial parameters and training strategy, the training set is input into the DeepKE model for training and optimization, and the corresponding model parameters are adjusted to obtain the optimized DeepKE model. S3-3-5. Input the validation set and test set into the optimized DeepKE model respectively to obtain the corresponding validation results and test results; S3-3-6. Based on the verification results and the test results, calculate the corresponding verification performance indicators and test performance indicators; S3-3-7. Determine whether both the verification performance index and the test performance index meet the performance conditions; if so, complete the training of the DeepKE model; otherwise, return to S3-3-3.

7. The method for constructing a water conservancy knowledge base based on a large model according to claim 1 is characterized in that: The knowledge base framework is a bottom-up five-layer architecture of "data layer-model layer-instance layer-storage layer-application layer"; The method of constructing a corresponding water conservancy knowledge base based on the first water conservancy knowledge graph and the knowledge base framework includes the following steps: S4-1, constructing a water conservancy vector database; the water conservancy vector database is a Milvus vector database; S4-2, processing the data in each first water conservancy knowledge graph to obtain processed data; S4-3, storing the processed data into a water conservancy vector database; S4-4. Based on the knowledge base framework and water conservancy vector database, construct the corresponding water conservancy knowledge base.

8. The method for constructing a water conservancy knowledge base based on a large model according to claim 1 is characterized in that: The latest water conservancy data mentioned above are newly published water conservancy research, water conservancy books, and water conservancy projects; The updating of the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph comprises the following steps: S6-1, calculating a first similarity between the first water conservancy data and the second water conservancy data; S6-2, calculating a second similarity between the first water conservancy knowledge graph and the second water conservancy knowledge graph; S6-3. Update the water conservancy knowledge base based on the update rule, the first similarity and the second similarity.

9. A water conservancy knowledge base construction system based on a large model, used to implement the water conservancy knowledge base construction method based on a large model according to any one of claims 1 to 8, characterized in that: It includes water conservancy data acquisition module, preprocessing module, data cleaning module, data annotation module, water conservancy knowledge base construction module, water conservancy knowledge base update module and water conservancy knowledge base application module; among which: The water conservancy data acquisition module is used to obtain water conservancy data of various data types; collect the latest water conservancy data in real time; A preprocessing module is used to preprocess the water conservancy data and the latest water conservancy data to obtain the preprocessed water conservancy data and the preprocessed latest water conservancy data; A data cleaning module is used to clean the pre-processed water conservancy data and the pre-processed latest water conservancy data to obtain the corresponding first water conservancy data and second water conservancy data; A data annotation module, used to annotate the first water conservancy data and the second water conservancy data using a large model and a knowledge graph, and construct a corresponding first water conservancy knowledge graph and a second water conservancy knowledge graph; A water conservancy knowledge base construction module, used to construct a corresponding water conservancy knowledge base based on the first water conservancy knowledge graph and the knowledge base framework; A water conservancy knowledge base updating module, used for updating the water conservancy knowledge base based on the second water conservancy data and the second water conservancy knowledge graph; The water conservancy knowledge base application module is used to use the water conservancy knowledge base to conduct water conservancy intelligent knowledge question and answer and water conservancy intelligent query / retrieval.

10. The water conservancy knowledge base construction system based on a large model according to claim 9 is characterized in that: The preprocessing module includes a parallel PDF preprocessing submodule, a web page preprocessing submodule and a structured preprocessing submodule; wherein: The PDF preprocessing submodule is used to extract, identify and write water conservancy data in PDF format using the OCR method to obtain the corresponding preprocessed water conservancy data; The webpage preprocessing submodule is used to collect water conservancy data in webpage format using crawler software to obtain collected water conservancy data; and to analyze the collected water conservancy data using a large model tool to obtain corresponding preprocessed water conservancy data; The structured preprocessing submodule is used to extract and analyze water conservancy data with structured data types using a large model, generate corresponding text data, and obtain corresponding preprocessed water conservancy data.

Images