Multi-modal large model construction method, system, equipment and medium applied to water conservancy field

Abstract

A multi-modal large model construction method, system, device and medium applied to the field of water conservancy comprises the steps of firstly obtaining and preprocessing text data in the field of water conservancy, constructing a local knowledge base in combination with a vector database, designing cue words for the preprocessed text data in the field of water conservancy, and constructing a multi-modal large model according to the cue words. Calling a large language model and manual annotation to generate a question and answer form data set; carrying out word list expansion on the local large language model, and carrying out secondary pre-training on the local large language model after word list expansion based on the preprocessed water conservancy field text data to obtain a water conservancy field large model; based on the question and answer form data set, a Lora algorithm is used to carry out instruction fine tuning on the water conservancy field large model, and a water conservancy field intelligent chat large model is generated; constructing an evaluation data set to evaluate the performance of the intelligent chat large model in the water conservancy field; the method comprises the following steps: constructing a WaterAgent special for the water conservancy field based on an intelligent chat large model and a local knowledge base in the water conservancy field, and evaluating and optimizing based on an evaluation data set; the method has the advantages of high accuracy and efficiency and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method, system, device and medium for constructing a multimodal large model applied to the field of water conservancy. Background Art

[0002] In the field of natural language processing (NLP), large language models (LLMs) such as ChatGPT and GPT-4 have demonstrated their excellent ability to handle complex language tasks, especially in few-shot and zero-shot learning, which makes it possible to apply them in multiple professional fields. However, when these general models are applied to fields that require highly specialized knowledge, such as medicine, law, and water conservancy, the results are often unsatisfactory. Especially in the field of water conservancy, despite the demand for highly specialized large language models, there are still few specific applications in this field. Water conservancy includes complex contents such as water conservancy engineering applications, water conservancy safety management, regulations and history, requiring the model to not only accurately understand the engineering details, but also master the relevant laws and management regulations. At present, there are two methods for constructing large language models in professional fields. One method is to train the large language model twice, and the other method is to build a vector library to store professional water conservancy knowledge and plug it into the large language model.

[0003] The patent application document with publication number CN117171331A discloses a professional field information interaction method, device and equipment based on a large language model, and its implementation scheme includes: retrieving similar information in the vector database based on the user's first content information, and generating input information to input into the language model to obtain an output result; if the user's second content information is received, the first output result is stored in the historical interaction data, and the historical data part is input into the language model to obtain a new output result, and the target input variable is generated and input into the vector database, and the interaction result is obtained after obtaining similar information and output to the user. This method can provide more accurate and professional answers; but because it needs to interact with the vector database and language model multiple times, it may result in a long processing time, affecting the user experience.

[0004] In the patent application with publication number CN117851558A, a medical question-and-answer system based on a large language model is disclosed. Its implementation scheme includes: input processing module, language understanding module, database query module, answer generation module, interaction module, data privacy and security module, user interface module, data update and continuous optimization module, professionalism and accuracy module. The system receives and understands the user's medical questions, queries relevant information from the medical knowledge base and literature database, and generates accurate and professional answers. However, due to the complexity of dialogue state tracking and context understanding, the system may have problems with lack of coherence when processing continuous dialogues, affecting the user experience. In addition, the system needs to be continuously optimized and updated to cope with the rapid development of medical knowledge and ensure the accuracy and timeliness of answers.

[0005] The shortcomings of the existing technology mainly include three aspects: 1. Large models in the patent field often use the form of multiple calls to the knowledge base, which requires high data quality and long response time. 2. When it is actually applied to actual scenarios, it often requires the input of multi-modal (image, text, audio) information. 3. The multi-round dialogue capability is weak. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a multimodal large model construction method, system, device and medium applied to the field of water conservancy, by constructing a comprehensive and detailed professional knowledge base in the field of water conservancy, that is, a local knowledge base D dedicated to the field of water conservancy, and creating a question-answering dataset Q preprocess and the evaluation dataset EvalWater, as well as the use of innovative technologies such as the Water_Agent intelligent architecture, aim to solve the problems of complex information and insufficient intelligence in the water conservancy field. The present invention can effectively improve the accurate understanding and execution efficiency of water conservancy tasks, enrich the functions and evaluation standards of the intelligent dialogue system, enhance the flexibility of task processing and the overall performance of the system, thereby providing more professional, real-time and intelligent support for the water conservancy field.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A multi-modal large model construction method applied to the water conservancy field comprises the following steps:

[0009] Step 1: Obtain text data W in the water conservancy field raw ;

[0010] Step 2: the water conservancy field text data W obtained in step 1 raw Perform preprocessing to obtain the preprocessed water conservancy field text data W preprocess ;

[0011] Step 3: Based on the vector database Chroma and the water conservancy field text data W preprocessed in step 2 preprocess , build a local knowledge base D dedicated to the field of water conservancy;

[0012] Step 4: the water conservancy field text data W preprocessed in step 2 preprocess Design different prompt words, call the large language model API and manually annotate to generate question-answering datasets in different formats for the water conservancy field. preprocess , including knowledge-based datasets, task-based datasets, multi-round question-answering and negative sample datasets;

[0013] Step 5: Expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and use the water conservancy field text data W preprocessed in step 2 preproccess , perform secondary pre-training on the local large language model Chatglm3 or the local large language model qwen after the vocabulary expansion to obtain the water conservancy field large model waterGPT_base with a large amount of water conservancy field knowledge;

[0014] Step 6: Based on the question-answering dataset Q generated in step 4 preproccess , use the Lora algorithm to fine-tune the instructions of the large water conservancy model waterGPT_base obtained in step 5 to generate the large water conservancy intelligent chat model waterGPT;

[0015] Step 7: The water conservancy field text data W preprocessed in step 2 is preproccess , design different prompt words, use large model generation and manual annotation to generate several multiple-choice questions, build the evaluation data set EvalWater, and evaluate the performance of the large model WaterGPT in the water conservancy field on the evaluation data set EvalWater;

[0016] Step 8: Based on the water conservancy intelligent chat large model waterGPT obtained in step 6 and the local knowledge base D constructed in step 3, construct Water_Agent dedicated to the water conservancy field, which is used to intelligently call various tools to complete various water conservancy tasks;

[0017] Step 9: Based on the evaluation data set EvalWater obtained in step 7, the Water_Agent constructed in step 8 specifically for the water conservancy field is evaluated and optimized.

[0018] In step 1, the water conservancy field text data W raw Including but not limited to laws and regulations in the field of water conservancy, professional vocabulary and industry terminology, water conservancy news and test questions, specifications, minutes of important meetings, announcements, work progress and major engineering events.

[0019] The step 2 preprocessing includes data cleaning and data classification;

[0020] Step 2.1, data cleaning: The water conservancy field text data W obtained in step 1 raw Delete duplicate characters and filter messy characters;

[0021] Step 2.2, data classification is as follows: the water conservancy field text data W after the data cleaning in step 2.1 raw Divided into multiple water conservancy sub-fields; the water conservancy sub-fields include but are not limited to reservoir and water conservancy knowledge, water conservancy safety, hydropower and hydroelectricity, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, geotechnical mechanics, engineering structure and materials, water conservancy history, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economy, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage.

[0022] The large language model API in step 4 includes but is not limited to Chatgpt, gpt4, DeepSeek-V2 or Wenxinyiyan.

[0023] The evaluation data set EvalWater in step 7 includes reservoir and water conservancy knowledge, water conservancy safety, hydropower and hydroelectricity, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, geotechnical mechanics, engineering structure and materials, water conservancy history, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economy, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage.

[0024] The tools in step 8 include but are not limited to external text-based graph models, voice systems or code executors; water conservancy tasks include but are not limited to water body extraction, water body area statistics, water body segmentation, intelligent question and answer of water conservancy knowledge or generation of water conservancy effect images.

[0025] The present invention also provides a multi-modal large model construction system applied to the water conservancy field, comprising:

[0026] Data acquisition and data preprocessing module, used to obtain text data W in the water conservancy field raw , and the water conservancy field text data W raw Perform preprocessing to obtain the preprocessed water conservancy field text data W preprocess ;

[0027] The knowledge base construction module is used to implement the knowledge base based on the vector database Chroma and the pre-processed water conservancy field text data W preprocess , build a local knowledge base D dedicated to the field of water conservancy;

[0028] The training data and evaluation data generation module is used to generate the pre-processed water conservancy field text data W preprocess Design different prompt words, call the large language model API and manually annotate to generate question-answering datasets in different formats for the water conservancy field. preprocess , including knowledge-based datasets, task-based datasets, multi-round question-answering and negative sample datasets; and using large model generation and manual annotation to generate several multiple-choice questions, construct the evaluation dataset EvalWater, and evaluate the performance of the large model WaterGPT in the water conservancy field on the evaluation dataset EvalWater;

[0029] Model training module, used to expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and based on the pre-processed water conservancy field text data W preproccess , perform secondary pre-training on the local large language model Chatglm3 or the local large language model qwen after the vocabulary expansion to obtain the water conservancy field large model waterGPT_base with a large amount of water conservancy field knowledge; based on the question-answering dataset Q preproccess , use the Lora algorithm to fine-tune the instructions of the large model waterGPT_base in the water conservancy field, and generate the large model waterGPT of intelligent chat in the water conservancy field;

[0030] Intelligent agent module, used to build WaterAgent dedicated to the water conservancy field based on the waterGPT intelligent chat model and local knowledge base D;

[0031] The effect evaluation and optimization module is used to evaluate and optimize the WaterAgent dedicated to the water conservancy field based on the evaluation data set EvalWater.

[0032] The present invention also provides a multi-modal large model construction device applied to the water conservancy field, comprising:

[0033] Memory: a computer program storing the above-mentioned multi-modal large model construction method applied to the water conservancy field, which is a computer-readable device;

[0034] Processor: used to implement the multi-modal large model construction method applied to the water conservancy field when executing the computer program.

[0035] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the multi-modal large model construction method applied to the water conservancy field.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] First, step 3 of the present invention constructs the first comprehensive, detailed and diverse local knowledge base D dedicated to the water conservancy field through technologies such as web crawlers, which solves the problem of lack of high-quality knowledge base data in the water conservancy field. The local knowledge base D dedicated to the water conservancy field can provide real-time, comprehensive professional information and data support, thereby helping researchers and engineers to quickly acquire relevant knowledge in the design, management and decision-making process.

[0038] Second, steps 5 and 6 of the present invention are performed by applying the model to the text data W in the water conservancy field. preprocess And the question-answering dataset Q preproccess After two rounds of pre-training and instruction fine-tuning, the trained large model of intelligent chat in the water conservancy field, waterGPT model, has high accuracy and efficiency in the precise understanding and execution of water conservancy tasks, which significantly improves the professionalism of applications in the water conservancy field.

[0039] Third, steps 4 and 7 of the present invention create the first question-answering dataset Q preproccess The evaluation dataset EvalWater solves the problem of poor multi-round dialogue capabilities in professional fields. At the same time, it greatly enriches the intelligent dialogue system and evaluation standards in the water conservancy field, and provides a more sophisticated and comprehensive evaluation method.

[0040] Fourth, step 8 of the present invention adopts the Water_Agent architecture in the water conservancy field to effectively decompose the water conservancy tasks and complete them collaboratively by different role agents, thereby enhancing the flexibility of task processing and the overall performance of the system, and being able to understand multimodal information such as graphics and text, and can be better applied to the water conservancy field.

[0041] In summary, the present invention constructs a comprehensive and detailed professional knowledge base and proprietary data in the field of water conservancy, creates a question-answering dataset Q preprocess and the evaluation dataset EvalWater, as well as the Water_Agent architecture in the water conservancy field, have the advantages of improving the accuracy and efficiency of precise understanding and execution of water conservancy tasks, enriching the intelligent dialogue system and evaluation standards, and enhancing the flexibility of task processing and the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the evaluation result of each model based on the classification of the evaluation dataset EvalWater.

[0043] Figure 2 It is the overall evaluation result of each model based on the evaluation dataset EvalWater.

[0044] Figure 3 It is the detailed classification and proportion of the evaluation data set EvalWater.

[0045] Figure 4 It is the overall flow chart of Water_Agent runtime which is specially used in water conservancy field.

[0046] Figure 5 It is the startup interface of WaterGPT, a large intelligent chat model in the water conservancy field.

[0047] Figure 6 It is the operating interface of WaterGPT, a large intelligent chat model in the water conservancy field.

[0048] Figure 7 It is a flow chart of the implementation method of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] The purpose of the present invention is to address the limitations of existing technologies in specific field applications, especially the insufficient application of large language models in the field of water conservancy. To this end, a multimodal large model construction method for application in the field of water conservancy is proposed. This technology enables the large language model to accurately understand the application details and relevant regulations of water conservancy projects by integrating water conservancy expertise and terminology. Through highly customized training and fine-tuning, the present invention can provide professional intelligent support in water conservancy safety management, project planning, laws and regulations, etc., thereby significantly improving the decision-making quality and operational efficiency of the water conservancy industry.

[0051] See also Figure 7 , a multi-modal large model construction method applied to the water conservancy field, comprising the following steps:

[0052] Step 1: Build a local knowledge base D and a question-answering dataset Q dedicated to the water conservancy field preprocess , implemented as follows:

[0053] Step 1.1, obtain water conservancy field text data W raw ;

[0054] Through crawling and other technologies, we crawled various water conservancy official websites, such as the Ministry of Water Resources of the People's Republic of China, Shaanxi Water Resources Department, National Water Conservancy Magazine and other platforms to obtain text data including laws and regulations in the field of water conservancy, professional vocabulary and terminology, water conservancy news and test questions, specifications, important meeting records, announcements, government work progress and major engineering events, etc., to obtain text data in the field of water conservancy. raw ;

[0055] Step 1.2: the water conservancy field text data W obtained in step 1.1 raw Perform preprocessing, including data cleaning and data classification, to obtain the preprocessed water conservancy field text data W preproccess ;

[0056] Step 1.2.1, data cleaning: The water conservancy field text data W obtained in step 1.1 raw Delete duplicate characters and filter messy characters;

[0057] Step 1.2.2, data classification is as follows: the water conservancy field text data W after the data cleaning in step 1.2.1 raw It is divided into 21 water conservancy sub-fields; the water conservancy sub-fields include but are not limited to reservoir and water conservancy knowledge, water conservancy safety, hydropower and hydropower, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, geotechnical mechanics, engineering structure and materials, water conservancy history, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economy, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage.

[0058] Step 1.3, based on the vector database Chroma and the water conservancy field text data W preprocessed in step 1.2 preproccess , build a local knowledge base D dedicated to the field of water conservancy;

[0059] Step 1.4: the water conservancy field text data W preprocessed in step 1.2 preproccess Design different prompt words, call Chatgpt and gpt4 and other large language model APIs and manually annotate to generate special water conservancy field question and answer datasets Qpreprocess in different formats, including knowledge-based datasets, task-based datasets, multi-round question and answer, and negative sample datasets;

[0060] Step 2: Based on the Lora algorithm, fine-tune the local large language model Chatglm3 or the local large language model qwen to generate the waterGPT large model of intelligent chat in the water conservancy field, which is implemented as follows:

[0061] Step 2.1, expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and use the water conservancy field text data W preprocessed in step 1.2 preproccess , perform secondary pre-training on the local large language model Chatglm3 or the local large language model qwen after the vocabulary expansion to obtain the water conservancy field large model waterGPT with a large amount of water conservancy field knowledge _ base;

[0062] Step 2.2: Based on the question-answering dataset Q generated in step 1.4 preproccess , use the Lora algorithm to fine-tune the instructions of the large water conservancy model waterGPT_base obtained in step 2.1 to generate the large water conservancy intelligent chat model waterGPT;

[0063] Step 3: The water conservancy field text data W preprocessed by step 1.2 preproccess , design different prompt words, use large model generation and manual annotation to generate about 10,000 multiple-choice questions, which are used to evaluate the waterGPT intelligent chat large model in the water conservancy field obtained in step 2.2. The evaluation data set EvalWater includes reservoir and water conservancy knowledge, water conservancy safety, hydraulic power, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, geotechnical mechanics, engineering structure and materials, water conservancy history, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economy, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage 21 categories. At the same time, the performance of the water conservancy field intelligent chat large model WaterGPT was evaluated on the evaluation data set EvalWater. The water conservancy field intelligent chat large model WaterGPT achieved the highest accuracy of 70.17%, which is about 5 percentage points higher than GPT4.

[0064] Step 4: Based on the water conservancy intelligent chat model WaterGPT obtained in step 2.2 and the local knowledge base D constructed in step 1.3, Water_Agent dedicated to the water conservancy field is constructed through prompt engineering, which is used to intelligently call various tools to complete various water conservancy tasks; the implementation is as follows:

[0065] 4.1) In the multi-agent framework, water conservancy worker agents determine the task type and task solution through simulated communication and discussion;

[0066] 4.2) If it is determined that the existing tools cannot solve the problem, programming will be used to solve the problem. The water conservancy worker agent will submit a detailed task description and discussion results to the programmer agent;

[0067] 4.3) The programmer agent generates preliminary code after reflection, thinking and reasoning, and submits the code to the error detector agent;

[0068] 4.4) The error detector agent executes the code and collects terminal feedback information. After careful consideration, it makes code optimization suggestions and feeds them back to the programmer agent;

[0069] 4.5) The programmer agent corrects the code according to the suggestions of the error checker agent, and after multiple rounds of iterations, both parties reach a consensus that the code is correct;

[0070] 4.6) Finally, the approved code, task details and execution results are reported to the execution brain agent;

[0071] 4.7) The execution brain agent encapsulates the code into tools, conducts in-depth analysis on the results, and compiles them into a system execution report to complete the execution and feedback process of the entire task.

[0072] Step 5: Based on the evaluation data set EvalWater obtained in step 3, the Water_Agent constructed in step 4 and dedicated to the water conservancy field is evaluated and optimized.

[0073] Specifically, we first record the scores of each category of Water_Agent in the evaluation dataset EvalWater. Then, we collect data from categories with low scores. Finally, we conduct secondary training based on the dataset of this category to optimize the performance of Water_Agent in this category.

[0074] The present invention is designed for water conservancy tasks, and improves the automation and intelligent processing capabilities in the water conservancy field by integrating multiple data sources with advanced machine learning algorithms. The present invention can be widely used in water conservancy related tasks such as water body extraction and area statistics, as well as enhancing decision support and information management in the water conservancy field through intelligent question-answering systems.

[0075] The present invention divides the training data of the supervised fine-tuning stage in detail and develops a multi-agent framework called Water_Agent, which can handle complex multimodal tasks and data, including text and images, so as to deeply understand and analyze complex hydrological environments and realize water body extraction and area calculation. This method significantly improves the accuracy of the model in the field of water conservancy. On the EvalWater dataset, WaterGPT achieved a maximum accuracy of 70.17%, which is about 5 percentage points higher than GPT4.

[0076] Experimental analysis

[0077] 1. Analysis of different training methods and results of WaterGPT, a large intelligent chat model in the water conservancy field

[0078] Based on Qwen-7B-Chat as the benchmark model, the present invention explores the impact of different training methods on model performance, and the corresponding comparison results are summarized in Table 1 below.

[0079] Table 1. Comparison of results of different training methods

[0080]

[0081]

[0082] It can be clearly seen from the results in the above table that both the Dora and Lora+ training methods outperform the LoRA training method in the water conservancy field. However, it is worth noting that after adopting the Lora+ training method, the performance on the ceval evaluation data set has dropped significantly. In contrast, the model obtained by the Dora training method has surpassed the model obtained by the LoRA training method in both the water conservancy field and the general field evaluation. Based on these findings, the present invention selects the model trained by the Dora method as the final model of the present invention, and compares it with the performance of each benchmark model. The results are shown in the following Figure 1 and Figure 2 middle.

[0083] Depend on Figure 1 It can be seen that the trained water conservancy intelligent chat big model WaterGPT has achieved the highest score in each category. Among them, the water conservancy intelligent chat big model WaterGPT has the highest score in the "Reservoir Water Conservancy Knowledge" domain, which is 78.28 points. This is mainly due to the comprehensiveness of the data set collected in this field. However, in the field of "Local Water Conservancy", the score is only 48.15, and in the fields of "Water Resources and Project Management" and "Water Conservancy History", the scores are also relatively low, at 59.96 and 62.16 respectively. Such a result is mainly that these subdomains are complex and difficult to understand, and require more extensive and high-quality data sets for training.

[0084] Depend on Figure 2 It can be seen that the trained intelligent chat model WaterGPT in the water conservancy field achieved the highest overall score on the evaluation dataset EvalWater, 5 points higher than GPT4.

[0085] like Figure 3As shown in the figure, the evaluation dataset EvalWater is divided into 21 categories. The name and proportion of each category are shown in the figure. Specifically, among all categories, reservoir and water conservancy knowledge accounts for 1.94%, water conservancy safety accounts for 0.90%, hydropower accounts for 0.25%, water law accounts for 6.41%, water resources and project management accounts for 4.82%, local water conservancy accounts for 0.53%, urban and rural water supply and drainage accounts for 5.36%, engineering mechanics, geotechnical mechanics, engineering structure and materials account for 7.65%, and water conservancy history accounts for 3. 99%, water conservancy engineering survey accounts for 9.18%, water conservancy engineering construction accounts for 8.08%, water conservancy scientific research, education, information publishing, and academic groups account for 2.60%, water conservancy management accounts for 4.76%, water conservancy economy accounts for 2.73%, water conservancy planning accounts for 2.46%, hydropower accounts for 6.36%, hydraulics, river and coastal dynamics accounts for 7.48%, soil and water conservation accounts for 3.78%, hydraulic structures account for 9.31%, hydrology and water resources account for 4.21%, and irrigation and drainage account for 7.36%. It can be seen that the evaluation data set EvalWater has sufficient category division and uniform data distribution, which provides a good foundation for the performance evaluation of the model in the field of water conservancy.

[0086] like Figure 4 As shown in the figure, the actual operation process of Water_Agent dedicated to the water conservancy field is demonstrated. It can be seen that Water_Agent dedicated to the water conservancy field generates professional responses through complex interactive processes and intelligent calls to external applications. For example, the question input by the user is "What is hydrological and water resources investigation and evaluation?" The Water_Agent dedicated to the water conservancy field replies "Hydrological and water resources investigation and evaluation refers to the analysis and evaluation of the quantity, quality, spatiotemporal distribution characteristics, and development and utilization conditions of water resources."

[0087] like Figure 5 and Figure 6 As shown in the figure, the actual operation interface of the WaterGPT intelligent chat model in the water conservancy field is shown. It can be seen that the WaterGPT intelligent chat model in the water conservancy field already has a mature web page interface and can answer user questions well.

[0088] like Figure 7 As shown in the figure, the complete process of building a multimodal large model system for water conservancy is presented. First, a professional corpus is collected to train the intelligent chat large model WaterGPT in the water conservancy field. Then, by plugging in the local knowledge base, an intelligent question-answering and voice interaction system is built, and the model plug-in capabilities and multimodal technology are further developed, and finally an intelligent agent system dedicated to the water conservancy field - Water_Agent is built.

[0089] The present invention also provides a multi-modal large model construction system applied to the water conservancy field, comprising:

[0090] Data acquisition and data preprocessing module, used to obtain text data W in the water conservancy field raw , ensuring that it covers information such as professional knowledge, regulations, and engineering events in the field of water conservancy; and the acquired text data in the field of water conservancy W raw Perform preprocessing to obtain the preprocessed water conservancy field text data W preprocess ;

[0091] The knowledge base construction module is used to implement the knowledge base based on the vector database Chroma and the pre-processed water conservancy field text data W preprocess , build a local knowledge base D dedicated to the water conservancy field, so that the water conservancy field intelligent chat big model WaterGPT can access and call relevant water conservancy knowledge, and can update its own data in real time to make the reply more accurate;

[0092] The training data and evaluation data generation module is used to generate the pre-processed water conservancy field text data W preprocess Design different prompt words, call the large language model API and manually annotate to generate question-answering datasets in different formats for the water conservancy field. preprocess , including knowledge-based datasets, task-based datasets, multi-round question-answering and negative sample datasets; and using large model generation and manual annotation to generate several multiple-choice questions, construct the evaluation dataset EvalWater, and evaluate the performance of the large model WaterGPT in the water conservancy field on the evaluation dataset EvalWater;

[0093] Model training module, used to expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and based on the pre-processed water conservancy field text data W preproccess , perform secondary pre-training on the local large language model Chatglm3 or the local large language model qwen after the vocabulary expansion to obtain the water conservancy field large model waterGPT_base with a large amount of water conservancy field knowledge; based on the question-answering dataset Q preproccess , use the Lora algorithm to fine-tune the instructions of the large model waterGPT_base in the water conservancy field, and generate the large model waterGPT of intelligent chat in the water conservancy field;

[0094] The intelligent agent module is used to build Water_Agent dedicated to the water conservancy field based on the waterGPT intelligent chat model and the local knowledge base D. It supports the efficient execution of water conservancy tasks through prompt engineering and multi-agent collaboration framework, including the whole process of task allocation, programming agent, code error detection, task feedback, etc. This enables the model to handle complex tasks such as multi-modality and better help workers in the water conservancy field complete their tasks;

[0095] The effect evaluation and optimization module is used to evaluate and optimize the Water_Agent dedicated to the water conservancy field based on the evaluation data set EvalWater.

[0096] The present invention also provides a multi-modal large model construction device applied to the water conservancy field, comprising:

[0097] Memory: a computer program storing the above-mentioned multi-modal large model construction method applied to the water conservancy field, which is a computer-readable device;

[0098] Processor: used to implement the multi-modal large model construction method applied to the water conservancy field when executing the computer program.

[0099] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the multi-modal large model construction method applied to the water conservancy field.

Claims

1. A multi-modal large model construction method applied to the field of water conservancy, characterized in that: The following steps are involved: Step 1: Obtain text data W in the water conservancy field raw ; Step 2: the water conservancy field text data W obtained in step 1 raw Perform preprocessing to obtain the preprocessed water conservancy field text data W preprocess ; Step 3: Based on the vector database Chroma and the water conservancy field text data W preprocessed in step 2 preprocess , build a local knowledge base D dedicated to the field of water conservancy; Step 4: the water conservancy field text data W preprocessed in step 2 preprocess Design different prompt words, call the large language model API and manually annotate to generate question-answering datasets in different formats for the water conservancy field. preprocess , including knowledge-based datasets, task-based datasets, multi-round question-answering and negative sample datasets; Step 5: Expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and use the water conservancy field text data W preprocessed in step 2 preproccess , perform secondary pre-training on the local large language model Chatglm3 or the local large language model qwen after the vocabulary expansion to obtain the water conservancy field large model waterGPT_base with a large amount of water conservancy field knowledge; Step 6: Based on the question-answering dataset Q generated in step 4 preproccess , use the Lora algorithm to fine-tune the instructions of the large water conservancy model waterGPT_base obtained in step 5 to generate the large water conservancy intelligent chat model waterGPT; Step 7: The water conservancy field text data W preprocessed in step 2 is preproccess , design different prompt words, use large model generation and manual annotation to generate several multiple-choice questions, build the evaluation data set EvalWater, and evaluate the performance of the large model WaterGPT in the water conservancy field on the evaluation data set EvalWater; Step 8: Based on the water conservancy intelligent chat large model waterGPT obtained in step 6 and the local knowledge base D constructed in step 3, construct Water_Agent dedicated to the water conservancy field, which is used to intelligently call various tools to complete various water conservancy tasks; Step 9: Based on the evaluation data set EvalWater obtained in step 7, the Water_Agent constructed in step 8 specifically for the water conservancy field is evaluated and optimized.

2. The method for constructing a multimodal large model applied in the field of water conservancy according to claim 1, characterized in that: In step 1, the water conservancy field text data W raw Including but not limited to laws and regulations in the field of water conservancy, professional vocabulary and industry terminology, water conservancy news and test questions, specifications, minutes of important meetings, announcements, work progress and major engineering events.

3. The method for constructing a multimodal large model applied in the field of water conservancy according to claim 1, characterized in that: The step 2 preprocessing includes data cleaning and data classification; Step 2.1, data cleaning: The water conservancy field text data W obtained in step 1 raw Delete duplicate characters and filter messy characters; Step 2.2, data classification is as follows: the water conservancy field text data W after the data cleaning in step 2.1 raw Divided into multiple water conservancy sub-fields; the water conservancy sub-fields include but are not limited to reservoir and water conservancy knowledge, water conservancy safety, hydropower and hydroelectricity, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, geotechnical mechanics, engineering structure and materials, water conservancy history, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economy, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage.

4. The method for constructing a multimodal large model applied in the field of water conservancy according to claim 1, characterized in that: The large language model API in step 4 includes but is not limited to Chatgpt, gpt4, DeepSeek-V2 or Wenxinyiyan.

5. The method for constructing a multimodal large model applied in the field of water conservancy according to claim 1, characterized in that: The evaluation data set EvalWater in step 7 includes reservoir and water conservancy knowledge, water conservancy safety, hydropower and hydroelectricity, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, geotechnical mechanics, engineering structure and materials, water conservancy history, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economy, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage.

6. The method for constructing a multi-modal large model applied in the field of water conservancy according to claim 1, characterized in that: The tools in step 8 include but are not limited to external text-based graph models, voice systems or code executors; water conservancy tasks include but are not limited to water body extraction, water body area statistics, water body segmentation, intelligent question and answer of water conservancy knowledge or generation of water conservancy effect images.

7. A multi-modal large model construction system applied in the field of water conservancy, characterized in that: include: Data acquisition and data preprocessing module, used to obtain text data W in the water conservancy field raw , and the water conservancy field text data W raw Perform preprocessing to obtain the preprocessed water conservancy field text data W preprocess ; The knowledge base construction module is used to implement the knowledge base based on the vector database Chroma and the pre-processed water conservancy field text data W preprocess , build a local knowledge base D dedicated to the field of water conservancy; The training data and evaluation data generation module is used to generate the pre-processed water conservancy field text data W preprocess Design different prompt words, call the large language model API and manually annotate to generate question-answering datasets in different formats for the water conservancy field. preprocess , including knowledge-based datasets, task-based datasets, multi-round question-answering and negative sample datasets; and using large model generation and manual annotation to generate several multiple-choice questions, construct the evaluation dataset EvalWater, and evaluate the performance of the large model WaterGPT in the water conservancy field on the evaluation dataset EvalWater; Model training module, used to expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and based on the pre-processed water conservancy field text data W preprocess , perform secondary pre-training on the local large language model Chatglm3 or the local large language model qwen after the vocabulary expansion to obtain the water conservancy field large model waterGPT_base with a large amount of water conservancy field knowledge; based on the question-answering dataset Q preproccess , use the Lora algorithm to fine-tune the instructions of the large model waterGPT_base in the water conservancy field, and generate the large model waterGPT of intelligent chat in the water conservancy field; Intelligent agent module, used to build Water_Agent dedicated to the water conservancy field based on the waterGPT intelligent chat model and local knowledge base D; The effect evaluation and optimization module is used to evaluate and optimize the Water_Agent dedicated to the water conservancy field based on the evaluation data set EvalWater.

8. A multi-modal large model construction device applied in the field of water conservancy, characterized in that: include: Memory: a computer program storing a method for constructing a multimodal large model applied to the water conservancy field as described in any one of claims 1 to 6, which is a computer-readable device; Processor: used to implement a multimodal large model construction method applied to the water conservancy field as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement a multimodal large model construction method applied to the water conservancy field as described in any one of claims 1-6.

Images