A multi-modal large model construction method for water conservancy scenarios

Abstract

The application discloses a kind of multi-modal large model construction methods for water conservancy scene, comprising the following steps: data acquisition;Data preprocessing;Build water conservancy multi-modal dataset;Training benchmark multi-modal large model;Model weight is saved using BF16 data format, and attention mechanism key tensor uses 4-bit k quantization technology.The application proposes that the weight of quantization is decomposed low rank adapter and is supervised fine-tuning, effectively improve the performance of benchmark multi-modal large model in water conservancy field specific task, by the parameter efficient fine-tuning strategy, reduce the calculation cost of training, provide convenience for subsequent iteration optimization.

Description

Technical Field

[0001] This invention relates to the field of water conservancy, and in particular to a method for constructing a multimodal large model for water conservancy scenarios. Background Technology

[0002] Traditional decision-making models for water conservancy information systems face bottlenecks in data acquisition, analysis, and risk assessment. On the one hand, experts need to spend a significant amount of time extracting key information from heterogeneous data from multiple sources, including historical data, meteorological data, hydrological data, and engineering structure information. This process is cumbersome and inefficient. On the other hand, expert experience is easily influenced by subjective factors, which may lead to biases in risk assessment results and make it difficult to guarantee the objectivity and accuracy of decision-making.

[0003] Traditional decision-making models are relatively slow to respond to sudden water safety issues. In the face of extreme weather events or sudden engineering accidents, expert consultations and plan formulation take time, and delayed decisions may lead to increased disaster losses. With the rapid development of information technology, the water conservancy sector has accumulated massive amounts of multi-source heterogeneous data; however, traditional decision-making models face bottlenecks in processing and analyzing this data, making it difficult to fully explore and utilize the information value contained within it. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a simple and practical method for constructing multimodal large models for water conservancy scenarios.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is: a method for constructing a multimodal large model for water conservancy scenarios, comprising the following steps:

[0006] Step S1, Data Acquisition: Use a drone equipped with a camera to collect raw data and obtain image data of the entire shooting area;

[0007] Step S2, Data Preprocessing: After downsampling the acquired image data, select images related to water conservancy;

[0008] Step S3, constructing a water conservancy multimodal dataset: using a general multimodal large model as the benchmark multimodal large model, and utilizing the zero-shot learning capability of the benchmark multimodal large model, perform topic classification on the preprocessed image dataset, and finally construct a water conservancy multimodal dataset;

[0009] Step S4: Train the baseline multimodal large model; train the baseline multimodal large model in the field of water conservancy to obtain the multimodal large model HydroMLLM for water conservancy scenarios.

[0010] Step S5: Save the model weights of HydroMLLM using the BF16 data format. Use 4-bit k-quantization technology for the attention mechanism key tensor in HydroMLLM to obtain the final multimodal large model for water conservancy scenarios.

[0011] The specific process of step S2 in the above-mentioned multimodal large model construction method for water conservancy scenarios is as follows:

[0012] S21: Image downsampling;

[0013] All original images were downsampled to a standard size of 600×400 pixels.

[0014] S22: Raw image filtering;

[0015] The original images were filtered to select those related to water conservancy.

[0016] The above-mentioned method for constructing a multimodal large model for water conservancy scenarios, in step S3, involves the following process for constructing the water conservancy multimodal dataset:

[0017] S31: Zero-shot image classification; The preprocessed image data is input into the benchmark multimodal large model, and combined with the designed prompt words, the benchmark multimodal large model is guided to analyze and recognize the image content;

[0018] S32: Classification Correction; Image pre-classification is performed using the zero-shot reasoning capability of the benchmark multimodal large model to obtain the corrected category labels;

[0019] S33: Set a subject discrimination mechanism; Considering that there may be cases where the content of the actual image is blurry, contains multiple subjects, or does not match any of the preset categories, a discrimination mechanism is set: if the benchmark multimodal large model determines that there is no subject in the image that matches the preset category, then the label "no obvious image subject" is output.

[0020] S34: Manual correction and verification; organize a verification team of at least two professionals in the field of water conservancy to conduct sampling verification of the category labels after classification correction and the "no obvious image theme" label, with a sampling ratio of not less than 30% of the total data volume; the verification content includes the matching degree between the label and the actual content of the image, and the accuracy of the category determination; if misjudged labels are found, correct the labels and record the misjudgment type; for categories with a misjudgment rate of more than 5%, conduct full manual verification of all images under the category to ensure the accuracy of the labels;

[0021] S35: Combine the manually corrected and verified category labels with the image file names to form a "image file name – category" tuple, thus creating a water conservancy multimodal dataset.

[0022] The above-mentioned method for constructing a multimodal large model for water conservancy scenarios, specifically step S4, involves the following process:

[0023] S41, during training, supervised fine-tuning training is performed using quantized weight decomposition of the low-rank adapter QDoRA, and the model weights are updated by quantizing the model parameters and using the low-rank matrix.

[0024] S42, Construct a water conservancy preference dataset for direct preference optimization alignment training, i.e. DPO alignment training: Design a data engine to construct a water conservancy preference dataset for DPO alignment training, and then use the data engine in conjunction with DPO alignment training to perform water conservancy field alignment training on the benchmark multimodal large model to obtain the multimodal large model HydroMLLM for water conservancy scenarios.

[0025] S43, Design the Water-CoT hint method for the water conservancy thinking chain to improve the performance of the HydroMLLM inference stage in the water conservancy field.

[0026] The above-mentioned method for constructing a multimodal large model for water conservancy scenarios, specifically step S41, is as follows:

[0027] Original weights First, it is quantized to k bits, and the original weights after quantization are... During forward computation, the weights are dynamically dequantized into approximate full-precision weights. The original weights after dequantization;

[0028] In QDoRA's overall weight update, the weights have been fine-tuned. The original weights after dequantization Add the incremental matrix What was obtained The specific expression is:

[0029]

[0030] in, ; To solve the quantization function;

[0031] It is decomposed into an amplitude vector and a direction component calculated from two low-rank fitness matrices. The calculation method is as follows:

[0032]

[0033] in, Indicates to To formalize; and Both are low-rank adaptation matrices introduced by QDoRA; Denotes the Frobenius norm; It is a learnable magnitude vector. , Represents the real number field. Indicates the output dimension of the weight matrix; This represents element-wise multiplication;

[0034] For any quantizable parameter Its expression is:

[0035]

[0036] in, The quantization scaling factor calculated for the parameter block; The zero point is calculated for the parameter block; P is the original floating-point parameter, and round is the rounding function;

[0037] During the calculation, the quantized parameters are dequantized on the fly, and the expression is:

[0038]

[0039] for The result after dequantization;

[0040] The forward propagation process of the fine-tuned baseline multimodal large model is represented as follows:

[0041]

[0042] Activation of given input in each layer of the baseline multimodal large model during fine-tuning process and output activation In reverse propagation, only And the quantization parameters receive gradients and are updated, while Then remain frozen.

[0043] In the above-mentioned method for constructing a multimodal large model for water conservancy scenarios, in step S42, the representative errors generated by the benchmark multimodal large model when answering water conservancy-related questions are used as negative samples in the water conservancy preference dataset, and the negative samples represent the answers that are not expected to be generated.

[0044] For representative errors, accurate corrected versions are provided. The corrected versions are considered as corresponding positive samples, and positive samples represent the expected responses.

[0045] The data engine supports multiple iterations, gradually accumulating preference pairs containing diverse error patterns and their corresponding correct answers through a multi-round cycle of model output-correction-data collection-DPO alignment training, ultimately constructing a water conservancy preference dataset.

[0046] In the above-mentioned method for constructing a multimodal large model for water conservancy scenarios, step S42 involves a data engine that is a closed-loop training mechanism based on iterative optimization of a water conservancy preference dataset. The core process is as follows:

[0047] ① Train the benchmark multimodal large model to convergence using the existing water conservancy preference dataset;

[0048] ② Use the trained benchmark multimodal large model for inference: on the one hand, collect representative errors in the model's answers to water conservancy questions as the first type of negative samples; on the other hand, by comparing the model output with the original dataset labels, identify cases where the labels do not match the actual content of the images, and use these as the second type of negative samples, i.e., dataset label misjudgment.

[0049] ③ Manually correct the two types of negative samples simultaneously: provide accurate answers for incorrect model responses and correct misjudged labels in the dataset; pair "negative samples - positive samples" and merge them into the water conservancy preference dataset, while updating the erroneous labels of the original water conservancy multimodal dataset;

[0050] ④ Use the water resources preference dataset to perform another round of DPO alignment training on the benchmark multimodal large model trained by QDoRA to obtain an updated benchmark multimodal large model;

[0051] ⑤ Return to step ② and continue the loop until the performance of the benchmark multimodal large model no longer improves and the label accuracy of the original water conservancy multimodal dataset is ≥99%.

[0052] In the above-mentioned method for constructing a multimodal large model for water conservancy scenarios, the Water-CoT prompting method in step S43 is as follows: Before the question-and-answer interaction process of the benchmark multimodal large model, a key intelligent cropping and focusing stage of the region of interest is introduced in advance. This aims to guide the benchmark multimodal large model to first focus on and fully understand the core target region in the image. The specific process is as follows:

[0053] 1) Use a prompt that combines knowledge of the water conservancy field to identify the target area of ​​interest for the user based on the user's original question;

[0054] 2) Based on the location coordinate information of the target object recognition fed back by the benchmark multimodal large model in the first stage, the original image is cropped for the target region to generate one or more region of interest image slices containing only clear target objects;

[0055] 3) In the subsequent second round of dialogue or analysis tasks, the image slices of the region of interest containing the target details, along with the specific questions or analysis instructions raised by the user, are submitted to the benchmark multimodal large model for in-depth analysis, attribute recognition and intelligent solution.

[0056] The above-mentioned method for constructing a multimodal large model for water conservancy scenarios, in step S5, the process of saving the model weights using the BF16 data format is as follows:

[0057] BF16 retains the exponent part completely and only compresses the mantissa part, with the mathematical form as follows:

[0058]

[0059] in, Represents the raw, full-precision floating-point number input; This represents a floating-point number after BF16 data format conversion.

[0060] In the above-mentioned method for constructing a multimodal large model for water conservancy scenarios, in step S5, the attention mechanism key tensor in HydroMLLM uses 4-bit k-quantization. The 4-bit k-quantization technique employs a weight organization method based on a superblock structure. In the superblock structure, the weights are organized hierarchically: each superblock contains 8 blocks, and each block contains 32 weight elements. The formula for the 4-bit k-quantization weight recovery process is as follows:

[0061]

[0062] in, This represents the original weight value after restoration. It is a 4-bit k-quantized integer value. It is a block scale factor. It is the minimum value in the block;

[0063] The quantization process first divides the weight matrix into multiple blocks, then calculates the statistical properties of each block, and finally performs linear quantization on each weight. The linear quantization formula is as follows:

[0064]

[0065] The floating-point weights are mapped to a 4-bit integer space, and the block scale factor and block minimum are also quantized to further reduce memory usage. Finally, the quantized values ​​and the original data are packaged into a compact memory layout to complete the entire quantization process.

[0066] The beneficial effects of this invention are as follows:

[0067] 1. This invention proposes a quantized weight decomposition low-rank adapter (QDoRA) for supervised fine-tuning, which effectively improves the performance of multimodal large models on specific tasks in the field of water conservancy. Through a parameter-efficient fine-tuning strategy, the computational cost of training is reduced, which facilitates subsequent iterative optimization.

[0068] 2. This invention utilizes a water conservancy preference data engine combined with a direct preference optimization (DPO) preference training strategy to perform alignment training on a multimodal large model in the water conservancy field. This makes the multimodal large model more consistent with human cognition and water conservancy industry standards when generating responses, thereby improving the practicality and reliability of the multimodal large model.

[0069] 3. This invention designs the Water-CoT prompting method for water conservancy thinking chain, which effectively solves the challenges of high complexity and strong professionalism in water conservancy problems by guiding multimodal large models to conduct deeper logical reasoning, and improves the performance of multimodal large models in complex water conservancy question-and-answer scenarios.

[0070] 4. This invention uses BF16 data type for inference and applies 4-bit k-quantization technology to successfully achieve low-computational-power inference for multimodal large models. While ensuring the performance of multimodal large models, it reduces the computing resources and memory usage required for inference. Attached Figure Description

[0071] Figure 1 This is the overall flowchart of the present invention.

[0072] Figure 2 A flowchart for zero-shot image classification in a multimodal large model.

[0073] Figure 3 This is a flowchart of the water conservancy preference data engine of the present invention.

[0074] Figure 4 This is a flowchart of the Water-CoT suggestion method of the present invention. Detailed Implementation

[0075] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0076] like Figure 1 As shown, a method for constructing a multimodal large model for water conservancy scenarios includes the following steps:

[0077] Step S1, Data Acquisition: In the water conservancy project, DJI series high-performance drones are used to conduct systematic drone aerial photography of key water conservancy areas to acquire 5,000 high-resolution color orthophoto images (6000×4000 pixels) covering the entire study area.

[0078] Step S2, Data Preprocessing: After downsampling the acquired image data, select images related to water conservancy.

[0079] The specific process of step S2 is as follows:

[0080] S21: Image downsampling;

[0081] All original image pixels are downsampled to a standard size of 600×400 pixels; this size is chosen to balance computational feasibility with the preservation of image details so that subsequent algorithms can process them effectively.

[0082] S22: Original image filtering;

[0083] To ensure the validity and relevance of the dataset, the original images were screened, selecting 2000 images related to water conservancy. An image was considered related to water conservancy if it contained any of the following: pond, ship, lock, breakwater, riverbank road, river surface, garbage, dock pillars, stone riverbank, water pit, dredger, or floodgate. This process eliminated a large number of redundant or irrelevant background images, allowing subsequent model training or data analysis to focus more on the core research content.

[0084] Step S3: Construct a water conservancy multimodal dataset: Using the general multimodal large model Qwen2.5-VL as the benchmark multimodal large model, and leveraging the zero-shot learning capability of Qwen2.5-VL, perform topic classification on the preprocessed image dataset to finally construct a water conservancy multimodal dataset.

[0085] The process of constructing the multimodal dataset for water conservancy is as follows:

[0086] S31: Zero-shot image classification; e.g. Figure 2 As shown, the preprocessed image data is input into Qwen2.5-VL, and combined with the designed prompts, Qwen2.5-VL, based on its extensive global knowledge, limits the most relevant water conservancy topics to twelve preset specific topic categories, including: ponds, ships, locks, breakwaters, riverbank roads, river surfaces, garbage, wharf pillars, stone riverbanks, water pits, dredgers, and floodgates, to analyze and identify the image content.

[0087] S32: Classification Correction; Using Qwen2.5-VL's zero-shot reasoning capability, image pre-classification is performed to obtain the corrected category labels;

[0088] S33: Set a subject discrimination mechanism; Considering that there may be cases where the content of the actual image is blurry, contains multiple subjects, or does not match any of the preset categories, a discrimination mechanism is set: If Qwen2.5-VL determines that there is no subject in the image that matches the preset category, then output the label "No obvious image subject" to ensure the accuracy and coverage of the classification results;

[0089] S34: Manual correction and verification; organize a verification team of at least two professionals in the field of water conservancy to conduct sampling verification of the category labels after classification correction and the "no obvious image theme" label, with a sampling ratio of not less than 30% of the total data volume; the verification content includes the matching degree between the label and the actual content of the image, and the accuracy of the category determination; if misjudged labels are found, correct the labels and record the misjudgment type; for categories with a misjudgment rate of more than 5%, conduct full manual verification of all images under the category to ensure the accuracy of the labels;

[0090] S35: The manually corrected and verified category labels are combined with the image file names to form an "image file name - category" tuple, thus forming a water conservancy multimodal dataset. The water conservancy multimodal dataset contains 1,000 labeled question-answer pairs, and the image content and corresponding question-answer formats both exhibit high diversity.

[0091] The core function of the water conservancy multimodal dataset is to provide basic cognitive samples of "visual + category" in water conservancy scenarios for the benchmark multimodal large model, and to provide a data foundation for supervised fine-tuning of the QDoRA low-rank adapter through tuples.

[0092] Step S4: Train the baseline multimodal large model; train Qwen2.5-VL in the field of water conservancy to obtain the multimodal large model HydroMLLM for water conservancy scenarios.

[0093] The specific process of step S4 is as follows:

[0094] S41. During training, the QDoRA low-rank adapter with quantized weight decomposition is used to perform supervised fine-tuning training on the Qwen2.5-VL multimodal large model. The model parameters are quantized and the model weights are updated using the low-rank matrix, which optimizes the parameters efficiently.

[0095] Original weights First, it is quantized to k bits, and the original weights after quantization are... During forward computation, the weights are dynamically dequantized into approximate full-precision weights. The original weights after dequantization;

[0096] In QDoRA's overall weight update, the weights have been fine-tuned. The original weights after dequantization Add the incremental matrix What was obtained The specific expression is:

[0097]

[0098] in, ; To solve the quantization function; this formula reflects that the optimization focus of QDoRA lies in how to construct it efficiently and effectively. .

[0099] It is decomposed into an amplitude vector and a direction component calculated from two low-rank fitness matrices. The calculation method is as follows:

[0100]

[0101] in, Indicates to To formalize; and Both are low-rank fitness matrices introduced by QDoRA. and It is stored in 4-bit quantized form, but during calculation, it is dynamically dequantized to 16-bit floating-point BF16 precision to ensure calculation accuracy. Denotes the Frobenius norm, used to denote... Normalized to directional components to ensure that they represent only directional information; It is a learnable magnitude vector. , Represents the real number field. Indicates the output dimension of the weight matrix; The overall size of the update volume was controlled; This indicates element-wise multiplication, used to combine the magnitude component and the direction component;

[0102] For any quantizable parameter Its expression is:

[0103]

[0104] in, The quantization scaling factor calculated for the parameter block; Zeros calculated for the parameter block;

[0105] During the calculation, the quantized parameters are dequantized on the fly, and the expression is:

[0106]

[0107] for The result after dequantization;

[0108] The forward propagation process of fine-tuning Qwen2.5-VL is represented as follows:

[0109]

[0110] Activation given input in each layer of Qwen2.5-VL and output activation In reverse propagation, only And the quantization parameters (if learnable) receive gradients and are updated, while Then remain frozen.

[0111] S42, Construct a water resources preference dataset for Direct Preference Optimization (DPO) alignment training: Design a data engine to construct a water resources preference dataset for DPO alignment training. Then, utilize the data engine in conjunction with DPO alignment training to perform water resources-related alignment training on a benchmark multimodal large model, resulting in the multimodal large model HydroMLLM oriented towards water resources scenarios. The data engine process is as follows: Figure 3 As shown.

[0112] Using the representative errors generated by Qwen2.5-VL when answering water conservancy-related questions, the representative errors are used as negative samples in the preference dataset, where negative samples represent "unexpected answers";

[0113] For representative errors, accurate corrected versions are provided. The corrected versions are regarded as the corresponding positive samples, and the positive samples represent the "expected response".

[0114] Considering that the number of incorrect answers that can serve as effective negative samples in a single model output may be limited, the data engine supports multiple iterations. Through multiple rounds of "model output-correction-data collection-DPO training", a sufficient number of preference pairs containing diverse error patterns and their corresponding correct answers are gradually accumulated, and finally the water conservancy preference dataset is constructed.

[0115] The core function of the water conservancy preference dataset is to serve as a "precise alignment dataset" for optimizing the professional capabilities of the model. On the one hand, it locates the cognitive biases of the model in the water conservancy field by correcting the version of representative errors. On the other hand, it guides the model to output results that conform to the water conservancy industry standards by combining DPO training. At the same time, it corrects the labeling errors of the water conservancy multimodal dataset, thus achieving dual optimization of "model performance + basic data quality".

[0116] The data engine is a closed-loop training mechanism based on iterative optimization of a water resources preference dataset. Its core process is as follows:

[0117] ① Train Qwen2.5-VL to convergence using the existing water conservancy preference dataset;

[0118] ② Use the trained Qwen2.5-VL for inference: On the one hand, collect representative errors in the model's answers to water conservancy questions as the first type of negative samples; on the other hand, by comparing the model output with the original dataset labels, identify cases where the labels do not match the actual content of the images, and use these as the second type of negative samples, i.e., dataset label misjudgment.

[0119] ③ Manually correct the two types of negative samples simultaneously: provide accurate answers for incorrect model responses and correct misjudged labels in the dataset; pair "negative samples - positive samples" and merge them into the water conservancy preference dataset, while updating the erroneous labels of the original water conservancy multimodal dataset;

[0120] ④ Use the water resources preference dataset to perform another round of DPO alignment training on the Qwen2.5-VL trained by QDoRA to obtain the updated Qwen2.5-VL;

[0121] ⑤ Return to step ② and continue the loop until the performance of Qwen2.5-VL no longer improves and the label accuracy of the original water conservancy multimodal dataset is ≥99%, thus obtaining HydroMLLM.

[0122] The water conservancy multimodal dataset and the water conservancy preference dataset form a collaborative logic of "basic cognition → professional optimization → quality closed loop": the water conservancy multimodal dataset is responsible for enabling the model to "understand water conservancy objects", and the water conservancy preference dataset is responsible for enabling the model to "correctly answer water conservancy questions", together supporting HydroMLLM to transform from a general model into a high-precision model specifically for water conservancy scenarios.

[0123] S43, Design the Water-CoT hint method for the water conservancy thinking chain to improve the performance of the HydroMLLM inference stage in the water conservancy field.

[0124] The Water-CoT hinting method in the Water Resources Thinking Chain involves introducing a key region of interest (ROI) intelligent cropping and focusing stage before the standard multimodal question-and-answer interaction process. This aims to guide HydroMLLM to first focus on and fully understand the core target region in the image. The Water-CoT process is as follows: Figure 4 As shown, the specific process is as follows:

[0125] 1) Use a prompt that combines knowledge of the water conservancy field to identify the target area of ​​interest for the user based on the user's original question;

[0126] 2) Based on the location coordinate information of the target object recognition fed back by HydroMLLM in the first stage, the original image is cropped for the target region to generate one or more region of interest image slices containing only clear target objects;

[0127] 3) In the subsequent second round of dialogue or analysis tasks, the image slices of the region of interest containing the target details, along with the specific questions or analysis instructions raised by the user, are submitted to HydroMLLM for in-depth analysis, attribute recognition, and intelligent solutions.

[0128] Step S5: Save the model weights of HydroMLLM using the BF16 data format. Use 4-bit k-quantization technology for the attention mechanism key tensor in HydroMLLM to reduce the computing power required by the deployed terminal machine and obtain the final multimodal large model for water conservancy scenarios.

[0129] The process of saving model weights using the BF16 data format is as follows:

[0130] BF16 retains the exponent part completely and only compresses the mantissa. While maintaining a large numerical range, BFloat16 reduces computational and storage overhead by sacrificing some decimal precision. Its mathematical form is:

[0131]

[0132] in, Represents the raw, full-precision floating-point number input; This represents a floating-point number after BF16 data format conversion.

[0133] The attention mechanism key tensor in HydroMLLM uses 4-bit k-quantization, which employs a weight organization method based on a superblock structure. In this superblock structure, the weights are organized hierarchically: each superblock contains 8 blocks, and each block contains 32 weight elements. The formula for the weight recovery process using 4-bit k-quantization is as follows:

[0134]

[0135] in, This represents the original weight value after restoration. It is a 4-bit k-quantized integer value. It is a block scale factor. It is the minimum value in the block;

[0136] The quantization process first divides the weight matrix into multiple blocks, then calculates the statistical properties of each block, and finally performs linear quantization on each weight. The linear quantization formula is as follows:

[0137]

[0138] The floating-point weights are mapped to a 4-bit integer space, and the block's scale factor and minimum value are also quantized to further reduce memory usage. Finally, the quantized values ​​and metadata are packaged into a compact memory layout to complete the entire quantization process.

Claims

1. A method for constructing a multimodal large model for water conservancy scenarios, characterized in that, Includes the following steps: Step S1, Data Acquisition: Use a drone equipped with a camera to collect raw data and obtain image data of the entire shooting area; Step S2, Data Preprocessing: After downsampling the acquired image data, select images related to water conservancy; Step S3, constructing a water conservancy multimodal dataset: using a general multimodal large model as the benchmark multimodal large model, and utilizing the zero-shot learning capability of the benchmark multimodal large model, perform topic classification on the preprocessed image dataset, and finally construct a water conservancy multimodal dataset; Step S4: Train the benchmark multimodal large model; Training the baseline multimodal large model in the field of water conservancy yields the multimodal large model HydroMLLM, which is designed for water conservancy scenarios. The specific process of step S4 is as follows: S41, during training, supervised fine-tuning training is performed using quantized weight decomposition of the low-rank adapter QDoRA, and the model weights are updated by quantizing the model parameters and using the low-rank matrix. The specific process of step S41 is as follows: Original weights First, it is quantized to k bits, and the original weights after quantization are... During forward computation, the weights are dynamically dequantized into approximate full-precision weights. The original weights after dequantization; In QDoRA's overall weight update, the weights have been fine-tuned. The original weights after dequantization Add the incremental matrix What was obtained The specific expression is: ； in, ; To solve the quantization function; It is decomposed into an amplitude vector and a direction component calculated from two low-rank fitness matrices. The calculation method is as follows: ； in, Indicates to To formalize; and Both are low-rank adaptation matrices introduced by QDoRA; Denotes the Frobenius norm; It is a learnable magnitude vector. , Represents the real number field. Indicates the output dimension of the weight matrix; This represents element-wise multiplication; For any quantizable parameter Its expression is: ； in, The quantization scaling factor calculated for the parameter block; The zero point is calculated for the parameter block; P is the original floating-point parameter, and round is the rounding function; During the calculation, the quantized parameters are dequantized on the fly, and the expression is: ； for The result after dequantization; The forward propagation process of the fine-tuned baseline multimodal large model is represented as follows: ； Activation of each layer of the baseline multimodal large model given input during fine-tuning process and output activation In reverse propagation, only And the quantization parameters receive gradients and are updated, while Then keep frozen; S42, Construct a water conservancy preference dataset for direct preference optimization alignment training, i.e. DPO alignment training: Design a data engine to construct a water conservancy preference dataset for DPO alignment training, and then use the data engine in conjunction with DPO alignment training to perform water conservancy field alignment training on the benchmark multimodal large model to obtain the multimodal large model HydroMLLM for water conservancy scenarios. S43, Design the Water-CoT hint method for the water conservancy thinking chain to improve the performance of the HydroMLLM inference stage in the water conservancy field; Step S5: Save the model weights of HydroMLLM using the BF16 data format. Use 4-bit k-quantization technology for the attention mechanism key tensor in HydroMLLM to obtain the final multimodal large model for water conservancy scenarios.

2. The method for constructing a multimodal large model for water conservancy scenarios according to claim 1, characterized in that, The specific process of step S2 is as follows: S21: Image downsampling; All original images were downsampled to a standard size of 600×400 pixels. S22: Original image filtering; The original images were filtered to select those related to water conservancy.

3. The method for constructing a multimodal large model for water conservancy scenarios according to claim 1, characterized in that, In step S3, the process of constructing the water conservancy multimodal dataset is as follows: S31: Zero-shot image classification; The preprocessed image data is input into the benchmark multimodal large model, and combined with the designed prompt words, the benchmark multimodal large model is guided to analyze and recognize the image content; S32: Categorical correction; Image pre-classification is performed using the zero-shot inference capability of a benchmark multimodal large model to obtain the class labels after classification correction; S33: Set a subject discrimination mechanism; Considering that there may be cases in actual images where the content is blurry, contains multiple subjects, or does not match any of the preset categories, a discrimination mechanism is set: if the benchmark multimodal large model determines that there is no subject in the image that matches the preset category, then the label "no obvious image subject" is output. S34: Manual correction and verification; Organize a verification team of at least two professionals in the field of water conservancy to conduct sampling verification of the category labels after classification correction and the "no obvious image theme" label, with a sampling ratio of no less than 30% of the total data volume; the verification content includes the matching degree between the label and the actual content of the image, and the accuracy of the category determination. If a misclassified label is found, the label is corrected and the type of misclassification is recorded; for categories with a misclassification rate exceeding 5%, all images in that category are manually verified to ensure label accuracy. S35: Combine the manually corrected and verified category labels with the image file names to form a "image file name – category" tuple, thus creating a water conservancy multimodal dataset.

4. The method for constructing a multimodal large model for water conservancy scenarios according to claim 3, characterized in that, In step S42, the representative errors generated by the benchmark multimodal large model when answering water conservancy-related questions are used as negative samples in the water conservancy preference dataset. The negative samples represent the answers that are not expected to be generated. For representative errors, accurate corrected versions are provided. The corrected versions are considered as corresponding positive samples, and positive samples represent the expected responses. The data engine supports multiple iterations, gradually accumulating preference pairs containing diverse error patterns and their corresponding correct answers through multiple rounds of model output-correction-data collection-DPO alignment training, ultimately constructing a water conservancy preference dataset.

5. The method for constructing a multimodal large model for water conservancy scenarios according to claim 4, characterized in that, In step S42, the data engine is a training closed loop based on iterative optimization of the water conservancy preference dataset. The core process is as follows: ① Train the benchmark multimodal large model to convergence using the existing water conservancy preference dataset; ② Use the trained benchmark multimodal large model for inference: on the one hand, collect representative errors in the model's answers to water conservancy questions as the first type of negative samples; on the other hand, by comparing the model output with the original dataset labels, identify cases where the labels do not match the actual content of the images, and use these as the second type of negative samples, i.e., dataset label misjudgment. ③ Manually correct two types of negative samples simultaneously: provide accurate answers for incorrect model responses and correct misjudged labels in the dataset; The "negative sample - positive sample" pairings were incorporated into the water conservancy preference dataset, while the mislabels of the original water conservancy multimodal dataset were updated. ④ Use the water resources preference dataset to perform another round of DPO alignment training on the benchmark multimodal large model trained by QDoRA to obtain an updated benchmark multimodal large model; ⑤ Return to step ② and continue the loop until the performance of the benchmark multimodal large model no longer improves and the label accuracy of the original water conservancy multimodal dataset is ≥99%.

6. The method for constructing a multimodal large model for water conservancy scenarios according to claim 4, characterized in that, In step S43, the Water-CoT suggestion method of the water conservancy thinking chain is as follows: Before the benchmark multimodal large model question-and-answer interaction process, a key intelligent cropping and focusing stage of the region of interest is introduced in advance, which aims to guide the benchmark multimodal large model to first focus on and fully understand the core target region in the image. The specific process is as follows: 1) Use a prompt that combines knowledge of the water conservancy field to identify the target area of ​​interest for the user based on the user's original question; 2) Based on the location coordinate information of the target object recognition fed back by the benchmark multimodal large model in the first stage, the original image is cropped for the target region to generate one or more region of interest image slices containing only clear target objects; 3) In the subsequent second round of dialogue or analysis tasks, the image slices of the region of interest containing the target details, along with the specific questions or analysis instructions raised by the user, are submitted to the benchmark multimodal large model for in-depth analysis, attribute recognition and intelligent solution.

7. The method for constructing a multimodal large model for water conservancy scenarios according to claim 4, characterized in that, In step S5, the process of saving the model weights using the BF16 data format is as follows: BF16 retains the exponent part completely and only compresses the mantissa part, with the mathematical form as follows: ； in, Represents the raw, full-precision floating-point number input; This represents a floating-point number after BF16 data format conversion.

8. The method for constructing a multimodal large model for water conservancy scenarios according to claim 4, characterized in that, In step S5, the attention mechanism key tensor in HydroMLLM uses 4-bit k-quantization. 4-bit k-quantization employs a weight organization method based on a superblock structure. In this superblock structure, weights are organized hierarchically: each superblock contains 8 blocks, and each block contains 32 weight elements. The formula for the 4-bit k-quantized weight recovery process is as follows: ； in, This represents the original weight value after restoration. It is a 4-bit k-quantized integer value. It is a block scale factor. It is the minimum value in the block; The quantization process first divides the weight matrix into multiple blocks, then calculates the statistical properties of each block, and finally performs linear quantization on each weight. The linear quantization formula is as follows: ； The floating-point weights are mapped to a 4-bit integer space, and the block scale factor and block minimum are also quantized to further reduce memory usage. Finally, the quantized values ​​and the original data are packaged into a compact memory layout to complete the entire quantization process.

Images