A multi-modal large model construction method, system, device and medium applied to the water conservancy field

Abstract

A multi-modal large model construction method, system, device and medium applied to the water conservancy field, first acquire water conservancy field text data and preprocess, and combine a vector database to construct a local knowledge base, then design prompt words for the preprocessed water conservancy field text data, call a large language model and artificial labeling to generate a question and answer form dataset; the local large language model vocabulary is expanded, and based on the preprocessed water conservancy field text data, the local large language model vocabulary after expansion is secondly pre-trained to obtain a water conservancy field large model; then based on the question and answer form dataset, the Lora algorithm is used to finely adjust the water conservancy field large model, and a water conservancy field intelligent chat large model is generated; a performance evaluation dataset is constructed to evaluate the performance of the water conservancy field intelligent chat large model; based on the water conservancy field intelligent chat large model and the local knowledge base, a Water_Agent special for the water conservancy field is constructed, and based on the evaluation dataset, evaluation and optimization are carried out; the application has the advantages of accuracy, efficiency and the like.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and specifically relates to a method, system, device and medium for constructing multimodal large models in the field of water conservancy. Background Technology

[0002] In the field of Natural Language Processing (NLP), Large Language Models (LLMs) such as ChatGPT and GPT-4 have demonstrated exceptional capabilities in handling complex language tasks, particularly in few-shot and zero-shot learning, enabling their application in various specialized fields. However, when these general-purpose models are applied to fields requiring highly specialized knowledge, such as medicine, law, and water resources, their performance is often unsatisfactory. Particularly in the field of water resources, despite the demand for highly specialized large language models, specific applications in this area remain limited. Water resources science encompasses complex content such as water conservancy engineering applications, water safety management, regulations, and history, requiring models to not only accurately understand engineering details but also master relevant laws and regulations. Currently, the construction of specialized large language models employs two main methods: one is to train the large language model twice, and the other is to construct a vector library to store specialized water resources knowledge and attach it to the large language model.

[0003] Patent application CN117171331A discloses a method, apparatus, and device for information interaction in a specialized field based on a large-scale language model. The implementation includes: retrieving similar information from a vector database based on the user's initial content information, generating input information, inputting it into the language model, and obtaining an output result; if a second piece of content information is received from the user, storing the first output result in historical interaction data, inputting a portion of the historical data into the language model to obtain a new output result, generating a target input variable, inputting it into the vector database, obtaining similar information, obtaining the interaction result, and outputting it to the user. This method can provide more accurate and professional answers; however, because it requires multiple interactions with the vector database and language model, the processing time may be long, affecting the user experience.

[0004] Patent application CN117851558A discloses a medical question-answering system based on a large language model. Its implementation includes: an input processing module, a language understanding module, a database query module, an answer generation module, an interaction module, a data privacy and security module, a user interface module, a data update and continuous optimization module, and a professionalism and accuracy module. The system receives and understands the user's medical questions, queries relevant information from a medical knowledge base and literature database, and generates accurate and professional answers. However, due to the complexity of dialogue state tracking and contextual understanding, the system may experience insufficient coherence when handling continuous dialogues, affecting the user experience. Furthermore, the system requires continuous optimization and updates to cope with the rapid development of medical knowledge and ensure the accuracy and timeliness of the answers.

[0005] The existing technology has three main drawbacks: 1. Large models in the patent field often involve multiple calls to the knowledge base, requiring high-quality data and resulting in long response times. 2. It ignores the fact that real-world applications often require multimodal (image, text, audio) input. 3. It has weak multi-turn dialogue capabilities. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device, and medium for constructing multimodal large models in the field of water conservancy. This is achieved by constructing a comprehensive and detailed professional knowledge base for the water conservancy field, namely a local knowledge base D specifically for the water conservancy field, and creating a question-and-answer format dataset Q. preprocess This invention, using the EvalWater dataset and innovative technologies such as the Water_Agent intelligent architecture, aims to address the problems of complex information and insufficient intelligence in the water conservancy field. It can effectively improve the accurate understanding and execution efficiency of water conservancy tasks, enrich the functions and evaluation standards of intelligent dialogue systems, enhance the flexibility of task processing and the overall performance of the system, thereby providing more professional, real-time and intelligent support for the water conservancy field.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for constructing a multimodal large model for water conservancy applications includes the following steps:

[0009] Step 1: Obtain text data W from the water resources sector. raw ;

[0010] Step 2, process the text data W in the water resources field obtained in Step 1. raw Preprocessing is performed to obtain preprocessed text data W in the water resources field. preprocess ;

[0011] Step 3, based on the vector database Chroma and the preprocessed water conservancy text data W from Step 2. preproccess Build a local knowledge base D specifically for the water conservancy field;

[0012] Step 4: Process the preprocessed water conservancy text data W from Step 2. preproccess Different prompt words were designed, and datasets in various question-and-answer formats for the water conservancy field were generated by calling large language model APIs and manually annotating the data. preproccess This includes knowledge-based datasets, task-based datasets, multi-turn question answering datasets, and negative sample datasets.

[0013] Step 5: Expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and based on the water conservancy field text data W preprocessed in Step 2. preproccess The local large language model Chatglm3 or the local large language model qwen with expanded vocabulary is pre-trained twice to obtain a large water conservancy model waterGPT_base with a large amount of water conservancy knowledge.

[0014] Step 6, based on the question-and-answer format dataset Q generated in Step 4 preproccess The LoRa algorithm is used to fine-tune the water conservancy big model waterGPT_base obtained in step 5, and the water conservancy intelligent chat big model waterGPT is generated.

[0015] Step 7, using the preprocessed water conservancy field text data W from Step 2 preproccess Different prompts were designed, and several multiple-choice questions were generated using a combination of large-scale model generation and manual annotation. An evaluation dataset, EvalWater, was constructed, and the performance of the intelligent chatbot WaterGPT model in the water conservancy field was evaluated on the EvalWater dataset.

[0016] Step 8: Based on the intelligent chat model waterGPT obtained in Step 6 and the local knowledge base D built in Step 3, construct Water_Agent specifically for the water conservancy field, which is used to intelligently call various tools to complete various water conservancy tasks.

[0017] Step 9: Based on the evaluation dataset EvalWater obtained in Step 7, evaluate and optimize the Water_Agent specifically for the water conservancy field constructed in Step 8.

[0018] The text data W in the water conservancy field in step 1 raw This includes, but is not limited to, laws and regulations, professional terminology and industry terminology in the water conservancy field, water conservancy news and test questions, standards, important meeting minutes, announcements and bulletins, work progress, and major engineering events.

[0019] Step 2, preprocessing, includes data cleaning and data classification.

[0020] Step 2.1, Data Cleaning: Cleaning the text data W in the water resources field obtained in Step 1. raw Remove duplicate characters and filter out random characters;

[0021] Step 2.2, data classification: The text data W from the water resources field after data cleaning in Step 2.1 is classified. raw It is divided into several sub-fields of water conservancy; the sub-fields of water conservancy include, but are not limited to, reservoirs and water conservancy knowledge, water conservancy safety, hydraulics and hydropower, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, rock and soil mechanics, engineering structures and materials, history of water conservancy, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economics, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage.

[0022] The large language model APIs mentioned in step 4 include, but are not limited to, Chatgpt, gpt4, DeepSeek-V2, or Wenxin Yiyan.

[0023] The evaluation dataset EvalWater in step 7 includes reservoir and water conservancy knowledge, water safety, hydraulics and hydropower, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, rock and soil mechanics, engineering structures and materials, history of water conservancy, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economics, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage.

[0024] The tools in step 8 include, but are not limited to, external text-based image models, voice systems, or code executors; the water conservancy tasks include, but are not limited to, water body extraction, water body area statistics, water body segmentation, intelligent question-and-answer of water conservancy knowledge, or generation of water conservancy effect images.

[0025] This invention also provides a multimodal large model construction system for the field of water conservancy, comprising:

[0026] The data acquisition and data preprocessing module is used to acquire text data W in the water conservancy field. raw And the obtained text data W in the field of water conservancy raw Preprocessing is performed to obtain preprocessed text data W in the water resources field. preprocess ;

[0027] The knowledge base construction module is used to implement the knowledge base construction based on the vector database Chroma and preprocessed text data W in the water conservancy field. preproccess Build a local knowledge base D specifically for the water conservancy field;

[0028] The training and evaluation data generation modules are used to process preprocessed text data W from the water resources field. preproccess Different prompt words were designed, and datasets in various question-and-answer formats for the water conservancy field were generated by calling large language model APIs and manually annotating the data. preproccess The dataset includes knowledge-based datasets, task-based datasets, multi-turn question answering datasets, and negative sample datasets; and several multiple-choice questions generated using a large model and manual annotation to construct the evaluation dataset EvalWater. The performance of the intelligent chat model WaterGPT in the field of water conservancy is evaluated on the evaluation dataset EvalWater.

[0029] The model training module is used to expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and to train it based on preprocessed water conservancy-related text data W. preproccess The local large language model Chatglm3 or the local large language model qwen, after vocabulary expansion, is pre-trained a second time to obtain a large water conservancy model waterGPT_base with a large amount of knowledge in the water conservancy field; based on the question-answering dataset Q... preproccess The LoRa algorithm was used to fine-tune the large-scale water conservancy model waterGPT_base, generating the intelligent chat large-scale water conservancy model waterGPT.

[0030] The intelligent agent module is used to build Water_Agent, which is specifically designed for the water conservancy field, based on the intelligent chat model waterGPT in the water conservancy field and the local knowledge base D.

[0031] The performance evaluation and optimization module is used to evaluate and optimize the Water_Agent, which is specifically designed for the water conservancy field, based on the evaluation dataset EvalWater.

[0032] This invention also provides a multimodal large model construction device for the field of water conservancy, comprising:

[0033] Memory: A computer program that stores the above-mentioned method for constructing a multimodal large model applied in the field of water conservancy, and is a computer-readable device;

[0034] Processor: Used to implement the multimodal large model construction method applied in the field of water conservancy when executing the computer program.

[0035] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the aforementioned method for constructing a multimodal large model in the field of water conservancy.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] First, step 3 of this invention constructs the first comprehensive, detailed, and diverse local knowledge base D specifically for the water conservancy field using technologies such as web crawlers. This solves the problem of a lack of high-quality knowledge base data in the water conservancy field. The local knowledge base D specifically for the water conservancy field can provide real-time and comprehensive professional information and data support, thereby helping researchers and engineers to quickly acquire relevant knowledge in the design, management, and decision-making processes.

[0038] Second, steps 5 and 6 of this invention involve processing text data W from the model in the field of water conservancy. preproccess And question-and-answer format dataset Q preproccess After two rounds of pre-training and instruction fine-tuning, the waterGPT model, a large-scale intelligent chatbot for the water conservancy field, exhibits high accuracy and efficiency in the precise understanding and execution of water conservancy tasks, significantly improving the professionalism of water conservancy applications.

[0039] Third, steps 4 and 7 of this invention create the first question-and-answer format dataset Q in the field of water conservancy. preproccess The EvalWater evaluation dataset addresses the issue of poor multi-turn dialogue capabilities in professional fields, while also greatly enriching intelligent dialogue systems and evaluation standards in the water conservancy sector, providing a more refined and comprehensive evaluation method.

[0040] Fourth, step 8 of this invention effectively decomposes water conservancy tasks and allows them to be completed collaboratively by intelligent agents with different roles by adopting the Water_Agent architecture in the field of water conservancy. This enhances the flexibility of task processing and the overall performance of the system, and it can understand multimodal information such as graphics and text, making it better applicable to the field of water conservancy.

[0041] In summary, this invention constructs a comprehensive and detailed professional knowledge base and proprietary data in the field of water conservancy, and creates a question-and-answer format dataset Q. preprocess The evaluation dataset EvalWater, along with the Water_Agent architecture adopted from the water conservancy field, has the advantages of improving the accuracy and efficiency of precise understanding and execution of water conservancy tasks, enriching intelligent dialogue systems and evaluation standards, and enhancing task processing flexibility and overall system performance. Attached Figure Description

[0042] Figure 1 These are the evaluation results of each model based on the classification of the EvalWater dataset.

[0043] Figure 2 It represents the overall evaluation results of each model based on the EvalWater evaluation dataset.

[0044] Figure 3 This provides a detailed breakdown and percentage of the EvalWater dataset used for evaluation.

[0045] Figure 4 This is the overall flowchart of the Water_Agent runtime, which is specifically designed for the water conservancy field.

[0046] Figure 5 This is the startup interface of WaterGPT, a large-scale intelligent chat model for the water conservancy field.

[0047] Figure 6 This is a flowchart of the implementation method of the present invention. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] The purpose of this invention is to address the limitations of existing technologies in specific fields, particularly the insufficient application of large language models in the water conservancy sector. To this end, a method for constructing multimodal large models for the water conservancy field is proposed. This technology integrates water conservancy expertise and terminology, enabling the large language model to accurately understand the application details of water conservancy projects and related regulations. Through highly customized training and fine-tuning, this invention can provide professional intelligent support in areas such as water conservancy safety management, engineering planning, and laws and regulations, thereby significantly improving the decision-making quality and operational efficiency of the water conservancy industry.

[0050] See Figure 6 A method for constructing a multimodal large model for water conservancy applications includes the following steps:

[0051] Step 1: Construct a local knowledge base D and a question-answering dataset Q specifically for the water conservancy field. preprocess The implementation is as follows:

[0052] Step 1.1: Obtain text data W from the water resources sector. raw ;

[0053] Using web scraping and other technologies, text data in the water resources field was collected from various official websites, such as the website of the Ministry of Water Resources of China, the website of the Shaanxi Provincial Department of Water Resources, and the National Water Resources Magazine. This data included laws and regulations, professional terms and industry information, water resources news and test questions, standards, important meeting minutes, announcements and bulletins, government department work progress reports, and major engineering events. raw ;

[0054] Step 1.2, process the text data W in the water resources field obtained in Step 1.1.raw Preprocessing, including data cleaning and data classification, yields preprocessed textual data W for the water resources field. preproccess ;

[0055] Step 1.2.1, Data Cleaning: Cleaning the text data W in the water resources field obtained in Step 1.1. raw Remove duplicate characters and filter out random characters;

[0056] Step 1.2.2, data classification: The text data W from the water resources field after data cleaning in Step 1.2.1 is classified. raw It is divided into 21 sub-fields of water conservancy; the sub-fields of water conservancy include, but are not limited to, reservoirs and water conservancy knowledge, water conservancy safety, hydraulics and hydropower, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, rock and soil mechanics, engineering structures and materials, history of water conservancy, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic groups, water conservancy management, water conservancy economics, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, irrigation and drainage.

[0057] Step 1.3, based on the vector database Chroma and the preprocessed water conservancy text data W from Step 1.2. preproccess Build a local knowledge base D specifically for the water conservancy field;

[0058] Step 1.4, process the preprocessed water conservancy text data W from Step 1.2. preproccess Different prompt words were designed, and large language model APIs such as Chatgpt and gpt4 were called, along with manual annotation, to generate question-answering datasets in different formats specifically for the water conservancy field. preproccess This includes knowledge-based datasets, task-based datasets, multi-turn question answering datasets, and negative sample datasets.

[0059] Step 2: Based on the LoRa algorithm, fine-tune the local large language model Chatglm3 or the local large language model qwen to generate the intelligent chat large model waterGPT for the water conservancy field, as follows:

[0060] Step 2.1: Expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and base it on the water conservancy field text data W preprocessed in Step 1.2. preproccess The local large language model Chatglm3 or the local large language model qwen with expanded vocabulary is pre-trained twice to obtain a large water conservancy model waterGPT_base with a large amount of water conservancy knowledge.

[0061] Step 2.2, based on the question-and-answer format dataset Q generated in step 1.4preproccess The LoRa algorithm is used to fine-tune the water conservancy big model waterGPT_base obtained in step 2.1 to generate the water conservancy intelligent chat big model waterGPT.

[0062] Step 3, using the preprocessed water conservancy text data W from Step 1.2 preproccess Different prompts were designed, and approximately 10,000 multiple-choice questions were generated using a combination of large-scale model generation and manual annotation. These questions were then used to evaluate the waterGPT intelligent chatbot model obtained in step 2.2 on the EvalWater dataset. The EvalWater dataset includes 21 categories: reservoirs and water conservancy knowledge, water safety, hydraulics and hydropower, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, rock and soil mechanics, engineering structures and materials, history of water conservancy, water conservancy engineering surveying, water conservancy engineering construction, water conservancy research, education, information publishing, academic organizations, water conservancy management, water conservancy economics, water conservancy planning, hydropower generation, hydraulics, river and coastal dynamics, soil and water conservation, hydraulic structures, hydrology and water resources, and irrigation and drainage. The performance of the waterGPT intelligent chatbot model was evaluated on the EvalWater dataset. The WaterGPT model achieved a maximum accuracy of 70.17%, approximately 5 percentage points higher than GPT4.

[0063] Step 4: Based on the WaterGPT intelligent chat model for the water conservancy field obtained in Step 2.2 and the local knowledge base D constructed in Step 1.3, Water_Agent, specifically designed for the water conservancy field, is built through a prompting project. This agent is used to intelligently invoke various tools to complete various water conservancy tasks. The implementation is as follows:

[0064] 4.1) In the multi-agent framework, water conservancy workers determine the types of tasks and task solutions through simulated communication and discussion processes;

[0065] 4.2) If it is determined that the existing tools cannot solve the problem, a programming approach will be adopted to solve the problem. The water conservancy worker agent will submit a detailed task description and discussion results to the programmer agent.

[0066] 4.3) After reflection, thinking, and reasoning, the programmer agent generates preliminary code and submits it to the error checker agent;

[0067] 4.4) The error checker executes the code and collects terminal feedback information. After careful consideration, they propose code optimization suggestions and provide feedback to the programmer agent.

[0068] 4.5) The programmer agent corrects the code based on the suggestions of the error checker agent, and iterates through multiple rounds until both parties agree that the code is error-free;

[0069] 4.6) Finally, the verified code, task details, and execution results are reported to the execution brain agent;

[0070] 4.7) The execution brain agent encapsulates the code into a tool, performs in-depth analysis of the results, and compiles a system execution report to complete the entire task execution and feedback process.

[0071] Step 5: Based on the evaluation dataset EvalWater obtained in Step 3, evaluate and optimize the Water_Agent specifically for the water conservancy field constructed in Step 4.

[0072] Specifically, the process involves: first, recording the scores of the Water_Agent, specifically designed for the water conservancy field, across various categories in the EvalWater evaluation dataset; then, extensively collecting data for the categories with lower scores; and finally, performing secondary training based on the dataset for those categories to optimize the performance of the Water_Agent specifically designed for the water conservancy field in those categories.

[0073] This invention is specifically designed for water conservancy tasks. By integrating multiple data sources and advanced machine learning algorithms, it enhances the automation and intelligent processing capabilities in the water conservancy field. This invention can be widely applied to water-related tasks such as water body extraction and area statistics, and can also enhance decision support and information management in the water conservancy field through an intelligent question-and-answer system.

[0074] This invention, through meticulously dividing the training data during the supervised fine-tuning phase and developing a multi-agent framework called Water_Agent, enables the processing of complex multimodal tasks and data, including text and images, thereby achieving a deeper understanding and analysis of complex hydrological environments and realizing water body extraction and area calculation. This method significantly improves the model's accuracy in the field of water resources; on the EvalWater dataset, WaterGPT achieves a peak accuracy of 70.17%, approximately 5 percentage points higher than GPT4.

[0075] Experimental Analysis

[0076] 1. Analysis of Different Training Methods and Results of WaterGPT, a Large-Scale Intelligent Chat Model in the Water Conservancy Field

[0077] This invention uses Qwen-7B-Chat as a benchmark model to explore the impact of different training methods on model performance. The corresponding comparison results are summarized in Table 1 below.

[0078] Table 1. Comparison of results from different training methods

[0079]

[0080]

[0081] The results in the table above clearly show that both Dora and Lora+ training methods outperform the LoRA training method in the field of water conservancy. However, it is particularly noteworthy that the performance on the ceval evaluation dataset significantly decreased after using the Lora+ training method. In contrast, the model trained using the Dora method outperformed the model trained using the LoRA method in both water conservancy and general domain evaluations. Based on these findings, this invention selected the model trained using the Dora method as the final model and compared its performance with various benchmark models, the results of which are presented in the following sections. Figure 1 and Figure 2 middle.

[0082] Depend on Figure 1 As can be seen, the trained intelligent chatbot model WaterGPT achieved the highest scores across all categories. Specifically, WaterGPT scored highest in the "Reservoir and Water Conservancy Knowledge" domain, with a score of 78.28. This is mainly due to the comprehensiveness of the dataset collected for this domain. However, in the "Local Water Conservancy" domain, the score was only 48.15, and the scores in "Water Resources and Project Management" and "Water Conservancy History" were also relatively low, at 59.96 and 62.16 respectively. These results are primarily because these subdomains are complex and difficult to understand, requiring training on broader and higher-quality datasets.

[0083] Depend on Figure 2 It can be seen that the trained intelligent chat model WaterGPT in the field of water conservancy achieved the highest overall score on the evaluation dataset EvalWater, which is 5 points higher than GPT4.

[0084] like Figure 3As shown, the evaluation dataset EvalWater is divided into 21 categories. The figure shows the names of each category and their proportions, specifically: Reservoir and water conservancy knowledge accounts for 1.94%, water conservancy safety accounts for 0.90%, hydropower accounts for 0.25%, water law accounts for 6.41%, water resources and project management accounts for 4.82%, local water conservancy accounts for 0.53%, urban and rural water supply and drainage accounts for 5.36%, engineering mechanics, rock and soil mechanics, engineering structures and materials accounts for 7.65%, and water conservancy history accounts for 3%. The evaluation dataset EvalWater comprises 99% water conservancy engineering surveying, 9.18% water conservancy engineering construction, 8.08% water conservancy research, education, information publishing, and academic organizations, 2.60% water conservancy management, 4.76% water conservancy economics, 2.73% water conservancy planning, 2.46% hydropower generation, 6.36% hydraulics, river and coastal dynamics, 7.48% soil and water conservation, 3.78% hydraulic structures, 9.31% hydrology and water resources, and 4.21% irrigation and drainage. As can be seen, the EvalWater dataset has sufficient category divisions and even data distribution, providing a good foundation for evaluating the model's performance in the water conservancy field.

[0085] like Figure 4 The image shows the actual operation flow of Water_Agent, a software specifically designed for the water conservancy sector. It demonstrates that Water_Agent, through a complex interactive process and intelligent invocation of external applications, ultimately generates a professional response.

[0086] like Figure 5 The actual operating interface of WaterGPT, a large-scale intelligent chat model for the water resources field, is showcased. It can be seen that WaterGPT has a mature web interface and can answer user questions effectively.

[0087] like Figure 6 The diagram illustrates the complete process of building a multimodal large-scale model system for the water conservancy field. First, a specialized corpus is collected to train the intelligent chatbot model WaterGPT for water conservancy. Then, by adding a local knowledge base, an intelligent question-answering and voice interaction system is built. Further development of the model's external plugin capabilities and multimodal technology ultimately results in the construction of an intelligent agent system specifically for the water conservancy field—Water_Agent.

[0088] This invention also provides a multimodal large model construction system for the field of water conservancy, comprising:

[0089] The data acquisition and data preprocessing module is used to acquire text data W in the water conservancy field. rawThis ensures the coverage of professional knowledge, regulations, and engineering events in the water resources field; and it also ensures the accuracy of the acquired textual data in the water resources field. raw Preprocessing is performed to obtain preprocessed text data W in the water resources field. preprocess ;

[0090] The knowledge base construction module is used to implement the knowledge base construction based on the vector database Chroma and preprocessed text data W in the water conservancy field. preproccess We will build a local knowledge base D specifically for the water conservancy field so that the intelligent chat model WaterGPT in the water conservancy field can access and call relevant water conservancy knowledge and update its own data in real time, making the replies more accurate.

[0091] The training and evaluation data generation modules are used to process preprocessed text data W from the water resources field. preprocess Different prompt words were designed, and datasets in various question-and-answer formats for the water conservancy field were generated by calling large language model APIs and manually annotating the data. preprocess The dataset includes knowledge-based datasets, task-based datasets, multi-turn question answering datasets, and negative sample datasets; and several multiple-choice questions generated using a large model and manual annotation to construct the evaluation dataset EvalWater. The performance of the intelligent chat model WaterGPT in the field of water conservancy is evaluated on the evaluation dataset EvalWater.

[0092] The model training module is used to expand the vocabulary of the local large language model Chatglm3 or the local large language model qwen, and to train it based on preprocessed water conservancy-related text data W. preproccess The local large language model Chatglm3 or the local large language model qwen, after vocabulary expansion, is pre-trained a second time to obtain a large water conservancy model waterGPT_base with a large amount of knowledge in the water conservancy field; based on the question-answering dataset Q... preproccess The LoRa algorithm was used to fine-tune the large-scale water conservancy model waterGPT_base, generating the intelligent chat large-scale water conservancy model waterGPT.

[0093] The intelligent agent module, based on the large-scale intelligent chat model waterGPT in the water conservancy field and the local knowledge base D, constructs a Water_Agent specifically for the water conservancy field. Through prompting engineering and a multi-agent collaboration framework, it supports the efficient execution of water conservancy tasks, including the entire process of task allocation, programming agent, code error detection, and task feedback. This enables the model to handle complex tasks such as multimodal tasks, better assisting water conservancy workers in completing their tasks.

[0094] The performance evaluation and optimization module is used to evaluate and optimize the Water_Agent, which is specifically designed for the water conservancy field, based on the evaluation dataset EvalWater.

[0095] This invention also provides a multimodal large model construction device for the field of water conservancy, comprising:

[0096] Memory: A computer program that stores the above-mentioned method for constructing a multimodal large model applied in the field of water conservancy, and is a computer-readable device;

[0097] Processor: Used to implement the multimodal large model construction method applied in the field of water conservancy when executing the computer program.

[0098] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the aforementioned method for constructing a multimodal large model in the field of water conservancy.

Claims

1. A method for constructing a multi-modal large model applied to the field of water conservancy, characterized in that, Comprising the following steps: Step 1, obtaining water conservancy field text data W raw ; Step 2, preprocessing the water conservancy field text data W obtained in step 1 raw to obtain the preprocessed water conservancy field text data W preprocess ; The step 2 preprocessing includes data cleaning and data classification; Step 2.1, data cleaning: on the water conservancy field text data W obtained in step 1 raw Delete duplicate characters, filter messy characters; Step 2.2, data classification: the water conservancy field text data W after data cleaning in step 2.1 raw is divided into multiple water conservancy sub-fields; the water conservancy sub-fields include but are not limited to reservoir and water conservancy knowledge, water conservancy safety, water power, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, rock and soil mechanics, engineering structure and materials, water conservancy history, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic community, water conservancy management, water conservancy economy, water conservancy planning, water power generation, hydraulics, river and coastal dynamics, soil and water conservation, water conservancy structures, hydrology and water resources, irrigation and drainage; Step 3, based on the vector database Chroma and the water conservancy field text data W preprocessed in step 2 preprocess , a local knowledge base D specific to the water conservancy field is constructed; Step 4, the water conservancy field text data W after pretreatment in step 2 preprocess Different prompt words are designed to call large language model api and artificial annotation to generate different formats of question and answer data sets Q in the special water conservancy field preprocess , including knowledge type data set, task type data set, multi-turn question and answer and negative sample data set; Step 5, the local large language model Chatglm3 or the local large language model qwen is word table expanded, and the water conservancy field text data W preprocessed based on step 2 preproccess The local large language model Chatglm3 or the local large language model qwen after word table expansion is second pre-trained to obtain a water conservancy field large model waterGPT_base with a large amount of water conservancy field knowledge. Step 6: Generating the question and answer form dataset Q based on the dataset generated in step 4 preproccess Step 5: Using the Lora algorithm to fine-tune the water conservancy field large model waterGPT_base obtained in step 5, and generating a water conservancy field intelligent chat large model waterGPT. Step 7, the water conservancy field text data W after pretreatment in step 2 preproccess , design different prompt words, generate several multiple-choice questions in the way of large model generation and manual annotation, construct the evaluation dataset EvalWater, and evaluate the performance of the water conservancy field intelligent chat large model WaterGPT on the evaluation dataset EvalWater. The evaluation data set EvalWater in step 7 includes reservoirs and water conservancy knowledge, water conservancy safety, water and electricity, water law, water resources and project management, local water conservancy, urban and rural water supply and drainage, engineering mechanics, rock and soil mechanics, engineering structure and materials, water conservancy history, water conservancy engineering survey, water conservancy engineering construction, water conservancy scientific research, education, information publishing, academic community, water conservancy management, water conservancy economy, water conservancy planning, hydropower, hydraulics, river and coastal dynamics, water and soil conservation, water conservancy structures, hydrology and water resources, irrigation and drainage; Step 8, based on the water conservancy field intelligent chat large model waterGPT obtained in step 6 and the local knowledge base D constructed in step 3, a Water_Agent specialized for the water conservancy field is constructed through prompting engineering, which is used to intelligently call various tools to complete various water conservancy tasks; 4.1) In the multi-agent framework, water conservancy worker agents determine the task type and task solution by simulating the process of communication and discussion; 4.2) If it is judged that existing tools cannot solve the problem, the idea of programming is adopted to solve the problem, and the water conservancy worker agent submits the detailed task description and discussion results to the programmer agent; 4.3) The programmer agent generates preliminary code after reflection, thinking and reasoning, and submits the code to the bug checker agent; 4.4) The bug checker agent executes the code and collects terminal feedback information, and after careful consideration, proposes code optimization suggestions and feeds back to the programmer agent; 4.5) The programmer agent modifies the code according to the suggestions of the bug checker agent, and iterates until both parties agree that the code is correct; 4.6) Finally, the code that has been approved, the task details and the execution results are reported to the execution brain agent; 4.7) The execution brain agent encapsulates the code into a tool, analyzes the results in depth, and compiles a systematic execution report to complete the execution and feedback process of the entire task; Step 9, based on the evaluation data set EvalWater obtained in step 7, the Water_Agent specialized for the water conservancy field constructed in step 8 is evaluated and optimized.

2. The multi-modal large model construction method applied to the field of water conservancy according to claim 1, characterized in that, The water conservancy field text data W in step 1 raw Including but not limited to water conservancy laws and regulations, professional vocabulary and terminology industry, water conservancy news and test questions, specifications, important meeting records, announcements, work progress and major engineering events.

3. The multi-modal large model construction method applied to the field of water conservancy according to claim 1, characterized in that, The large language model api in step 4 includes but is not limited to Chatgpt, gpt4, DeepSeek-V2 or Wenxin Yiyang.

4. The multi-modal large model construction method applied to the field of water conservancy according to claim 1, characterized in that, The tools in step 8 include but are not limited to external text generation models, voice systems or code executors; water conservancy tasks include but are not limited to water body extraction, water body area statistics, water body segmentation, water conservancy knowledge intelligent question answering or water conservancy effect image generation.

5. A multi-modal large model construction system for use in the field of water conservancy based on the method of claim 1, characterized in that, Comprising: A data acquisition and data preprocessing module is configured to acquire water conservancy field text data W raw , and preprocess the acquired water conservancy field text data W raw to obtain preprocessed water conservancy field text data W preprocess . The knowledge base construction module is used for realizing the construction of the local knowledge base D dedicated to the water conservancy field based on the vector database Chroma and the preprocessed water conservancy field text data W preprocess . The training data and evaluation data generation module is used for pre-processing the water conservancy field text data W preprocess Different prompt words are designed, and a special water conservancy field data set Q in different formats is generated in the form of question and answer by calling a large language model API and manual annotation. preprocess , including knowledge data set, task data set, multi-turn question and answer and negative sample data set; and a number of selection questions are generated by using a large model generation and manual annotation method to build an evaluation data set EvalWater, and the performance of the water conservancy field intelligent chat large model WaterGPT is evaluated on the evaluation data set EvalWater. The model training module is configured to perform vocabulary expansion on the local large language model Chatglm3 or the local large language model qwen, and perform secondary pre-training on the local large language model Chatglm3 or the local large language model qwen based on the preprocessed water conservancy field text data W preproccess , to obtain a water conservancy field large model waterGPT_base with a large amount of water conservancy field knowledge; based on a question and answer form data set Q preproccess , the water conservancy field large model waterGPT_base is fine-tuned using the Lora algorithm to generate a water conservancy field intelligent chat large model waterGPT. An intelligent agent module for constructing a Water_Agent specialized for the water conservancy field based on the water conservancy field intelligent chat large model waterGPT and the local knowledge base D; An effect evaluation and optimization module for evaluating and optimizing the Water_Agent specialized for the water conservancy field based on the evaluation data set EvalWater.

6. A multi-modal large model construction device applied to the field of water conservancy, characterized in that, Comprising: Memory: store the computer program of claim 1-4 any one of the application of multi-modal large model construction method in the field of water conservancy, for computer readable equipment; Processor: for executing the computer program to realize the application of multi-modal large model construction method in the field of water conservancy of any one of claims 1-4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the application of multi-modal large model construction method in the field of water conservancy of any one of claims 1-4.

Images