A workflow-based multimodal water conservancy large model decision support method

Abstract

The present invention discloses a workflow-based multimodal water conservancy large-scale model decision support method, comprising: S1. pre-processing multimodal input data based on a local model and converting the data format; S2. constructing a modular workflow that supports parallel processing by specifying a DSL framework; S3. constructing a feature model adapted to the water conservancy field by synchronously collecting and processing multimodal data, and performing domain adaptation and fine-tuning on the feature model; S4. constructing a specified workflow based on the domain-adapted and fine-tuned feature model to dynamically schedule multimodal task resources; S5. using a dynamic fusion algorithm to combine a local knowledge base with networked data to achieve dual-channel knowledge fusion for parallel execution of the multimodal tasks in S4. The present invention can achieve real-time collection and parallel processing of multimodal data, and through a dual-channel knowledge fusion mechanism, dynamically balance the stability of the local knowledge base with the real-time performance of the networked data.

Description

Technical Field

[0001] The present invention relates to the technical field of water conservancy data processing, and in particular to a workflow-based multimodal water conservancy large model decision support method, system, device and storage medium. Background Art

[0002] Water conservancy projects play a vital role in the safety monitoring and emergency management of key infrastructure such as reservoirs and dams. Existing water conservancy monitoring systems mainly rely on traditional single data collection and static data analysis methods, and mostly use offline knowledge bases and single-modal data processing. They are unable to cope with the challenges brought by data diversity, real-time nature and complex working conditions in actual applications. Especially under extreme climatic conditions (such as heavy rainfall and rainstorms), the dynamic changes in multi-source data and complex scenarios faced by water conservancy facilities urgently require more efficient and accurate monitoring and decision-making methods.

[0003] The following technical bottlenecks exist in the existing technology:

[0004] (1) Insufficient multimodal data processing capabilities: Existing systems mostly focus on a single type of data (such as water level sensor values) and lack the ability to jointly analyze multimodal data such as text (such as emergency reports), images (such as dam inspection photos), and voice (such as on-site personnel instructions), resulting in low information utilization;

[0005] (2) Conflict between knowledge base updating and stability: Local knowledge bases (such as flood control plans and historical cases) are unable to cope with real-time flood conditions due to delayed updates, while those that rely entirely on networked data (such as weather warnings) face the risk of network fluctuations. A coordination mechanism between the two is lacking;

[0006] (3) Poor model adaptability and scalability: General artificial intelligence models lack terminology and scenario adaptation in the water conservancy field, and it is difficult for models (such as image recognition and text generation) to collaborate in parallel, which limits the efficiency of complex task processing;

[0007] (4) Fragmentation of decision-making processes: The links from data input and analysis to decision output rely on manual connection and lack modular workflow support, resulting in slow response and poor traceability.

[0008] To this end, this application specifically proposes a workflow-based multimodal water conservancy large model decision support method to solve the above technical problems, which can realize the real-time collection and parallel processing of multimodal data (text, image, voice, etc.), and through the dual-channel fusion of offline knowledge base and networked real-time data, build an efficient, real-time and intelligent "question-analysis-decision" one-stop service platform. Summary of the Invention

[0009] The main purpose of the present invention is to provide a workflow-based multimodal water conservancy large model decision support method, which supports multimodal data collection and processing (including text, images, voice, etc.), and realizes the dual-channel fusion of local knowledge base and networked real-time knowledge, thereby constructing a one-stop "question-analysis-decision" intelligent service platform that can not only ensure the stability of offline data but also realize real-time updating of network information. At the same time, by integrating embedded models, language models, rearrangement models and voice models, multi-task parallel processing and precise data analysis are realized, providing efficient and accurate decision support for water conservancy project management, flood warning, water resources scheduling and other fields, so as to solve the technical problems raised in the background technology.

[0010] The present invention adopts the following technical solutions to solve the above technical problems:

[0011] A workflow-based multimodal water conservancy large model decision support method includes:

[0012] S1. Preprocess multimodal input data based on the local model and convert the file, image, and voice data formats into JSON format;

[0013] S2. By specifying a DSL framework, a modular workflow that supports parallel processing is built to ensure that different types of data after format conversion can be collected and processed simultaneously.

[0014] S3. Build a feature model adapted to the water conservancy sector using synchronously collected and processed multimodal data. Establish data dependencies based on the DSL framework, adapt and fine-tune the feature model to the domain, and parameterize the water conservancy mechanism through a three-layer architecture: implicit rules, explicit constraints, and rule activation, before embedding it into the model fine-tuning process.

[0015] S4. Based on the domain-adapted and fine-tuned feature model, a specified workflow is constructed to dynamically schedule multimodal task resources and implement a dynamic processing flow from data input, analysis to decision output.

[0016] S5. Use a dynamic fusion algorithm to combine the local knowledge base and network data to achieve dual-channel knowledge fusion, which is used to parallelly execute multimodal tasks in S4 to ensure the stability of the offline knowledge base and the real-time update of network data.

[0017] Preferably, the specific operation process of the pretreatment in step S1 includes:

[0018] S11. Based on the obtained multimodal input data, perform preliminary classification according to the specified type standard to ensure that they enter the corresponding preprocessing process according to type;

[0019] S12. Use specific models for preprocessing different types of multimodal data:

[0020] For text data, semantic features are extracted through embedding models, and then language models are used for noise removal, format unification, word segmentation, and part-of-speech tagging.

[0021] For image data, the embedding model is used to extract image features, perform denoising, resolution adjustment, and feature point extraction;

[0022] For speech data, a speech model is used for speech recognition and processing, including denoising, speech-to-text conversion, and speech feature extraction.

[0023] Preferably, the specific construction process of the modular workflow in step S2 includes:

[0024] S21. Define different data processing modules within the specified DSL framework and employ a parallel processing architecture to ensure that data processing of the specified data types is performed simultaneously within different processing channels.

[0025] S22. Based on the specified DSL framework, optimize and coordinate the parallel execution of multiple tasks by defining task dependencies and execution order.

[0026] Preferably, the specific operation process of step S3 includes:

[0027] S31. Use a feature extraction model adapted to the water conservancy field to extract specified features from different data types of multimodal data in the water conservancy field to obtain a feature model;

[0028] S32. Build pre-training data based on the existing water conservancy knowledge base and use the MLM task to perform domain adaptation on the feature model.

[0029] S33. Perform LoRA lightweight adjustment operations on the feature model, freeze the parameters of the low-rank matrix of the pre-trained data model, only adjust the newly added LoRA adaptation layer, and use the water conservancy field labeled dataset for supervised adjustment. The specified control indicators of the water conservancy project are incorporated into the model adjustment process as explicit constraints to build a physical consistency verification mechanism. It is used to check whether the output results violate the laws of hydrological physics during each model update, and a penalty term mechanism is used to correct predictions that violate physical laws to ensure the physical rationality of the model output.

[0030] Preferably, after the step S3 is executed, model verification and dynamic optimization operations are performed, specifically including:

[0031] Construct a test set in the water conservancy field to conduct domain accuracy evaluation, and use F1-score and BLEU indicators to verify model performance;

[0032] Construct a set of cross-modal consistency samples for testing to ensure the logical consistency of different modal outputs;

[0033] The deployment model is executed by integrating the A / B testing module and adopting a grayscale release strategy. User feedback data is used to trigger incremental model training to perform online dynamic optimization and closed-loop improvement.

[0034] Preferably, a cross-modal alignment algorithm is used during the verification of the cross-modal consistency sample to ensure semantic consistency of data in different modalities, wherein the method of using the cross-modal alignment algorithm includes:

[0035] L1. Perform knowledge graph embedding pre-training in the form of TransE for all triples Minimize, with knowledge graph embedding pre-training loss The calculation formula is:

[0036]

[0037] in, The margin hyperparameter of the model is used to ensure the effective distinction between positive and negative samples in the knowledge graph embedding pre-training; after training, all and ; 、 、 are the vectors of the head entity, relationship, and tail entity in the graph embedding space, and represent the starting entity of the relationship, the semantic relationship between entities, and the target entity of the relationship in the knowledge graph, respectively. is the L2 norm, is the set of all triples in the knowledge graph;

[0038] L2. For each positive sample ,from Sampling several negative entities composition , perform cross-modal feature mapping and align the loss, with:

[0039]

[0040] in, It is a set of negative samples or a space containing negative samples. Specifically, it is the source from which negative entities are sampled and used to optimize the comparison with positive samples during training. Represents a mapping network for visual features With text features Mapped to an embedding space with the same dimensions as the entity, The cross-modal alignment loss includes the hinge loss of visual alignment and text alignment. Represents the positive entity The sampled negative entity set, The margin hyperparameter representing the alignment loss, Represents the similarity function between A and B;

[0041] L3. Construct training objectives, including:

[0042]

[0043] in, , used to represent the weights and bias sets of the vision-text mapping network, is the weight of the visual mapping network, is the bias of the visual mapping network, is the weight of the text mapping network, is the bias of the text mapping network, is a regularization term whose value is the sum of the squares of all trainable parameters, is the L2 regularization weight.

[0044] Preferably, the S32 step further introduces an adversarial training mechanism, introduces a domain feature adversarial network, uses a domain classifier to distinguish general data from water conservancy data, so as to enhance the sensitivity of the feature model to professional terminology, enable the feature model to implicitly learn the physical constraints of water conservancy projects, and set a specified pre-training task to allow the feature model to predict results that conform to hydrological physical laws under given boundary conditions, so as to realize the implicit encoding and storage of mechanism knowledge in the model weights. The adversarial training mechanism specifically includes the following adversarial training improvement method operation steps:

[0045] P1. Divide the water conservancy dataset into source domain and target domain, and eliminate data distribution differences through standardization;

[0046] P2. Generate an adversarial network feature extraction module. This module combines a deep residual shrinkage network to compress redundant feature dimensions and introduces a long short-term memory network to capture temporal dependencies. This forms the model architecture. A domain discriminator is added to the model architecture as the core of adversarial training. A gradient reversal layer is used to force the feature encoder to generate domain-invariant features.

[0047] P3. Based on the model architecture constructed in step P2, perform adversarial training operations, including:

[0048] Stage 1: Pre-training feature encoders on source domain data to initially extract key features related to water conservancy parameters;

[0049] Stage 2: Fix the feature encoder and train the domain discriminator to distinguish the source domain and target domain features. By minimizing the domain discriminator loss function, the feature encoder is optimized to confuse the discriminator and achieve domain feature alignment.

[0050] Phase 3: Based on adversarial training, the model is fine-tuned using labeled data from the source domain and a small amount of labeled data from the target domain. A weighted cross-entropy loss function is used to balance the class imbalance problem.

[0051] P4. Embed a dynamic threshold judgment module in the output layer of the model to adaptively adjust the classification threshold based on the statistical characteristics of historical data of water conservancy projects.

[0052] Preferably, the specific operation process of step S4 includes:

[0053] S41. Based on the fine-tuned feature model, a set of "question-analysis-decision" workflows is constructed, defining the task flow relationships between each sub-module. This ensures that each module can initiate the corresponding analysis and processing tasks based on user input and smoothly advance to decision output according to the predetermined logic.

[0054] S42. Using a preset task priority scheduling strategy for dynamic resource scheduling to optimize task execution efficiency in the workflow, the scheduling judgment logic of the task priority scheduling strategy includes:

[0055] Dynamically adjust the baseline priority of various tasks according to the water conservancy operation status:

[0056] During flood season, real-time sensor data tasks are automatically elevated to the highest priority;

[0057] For flood control instructions and routine monitoring tasks, during the non-flood season, the normal priority setting will be restored;

[0058] The scheduler uses preemptive scheduling. When a new task arrives, it compares the priority of the new task with that of the currently executing task. If the former has a higher priority, the new task is immediately interrupted and switched; otherwise, the task continues to execute.

[0059] For the same priority, the CPU is allocated on a first-come-first-served basis or in a round-robin manner.

[0060] Preferably, the specific operation process of executing the dynamic fusion algorithm to combine data in step S5 includes:

[0061] S51. Obtain offline knowledge base and real-time data through the local knowledge base and network data acquisition interface, and perform independent preprocessing operations respectively;

[0062] S52. Combine the local knowledge base and networked data. The offline knowledge base provides stable information, and the networked data is updated in real time and provides the latest decision support. A dynamic fusion algorithm is used to balance the stability of the local knowledge base and the real-time nature of the networked data.

[0063] Preferably, the method of using the dynamic fusion algorithm includes:

[0064] (1) Calculate the time-effectiveness trust value. The calculation formula is:

[0065]

[0066]

[0067] in, and Represent the timeliness trust value of the local knowledge base and network data respectively. The longer the time since the local knowledge base was updated, the The smaller the value, the shorter the time since the network data was updated. The bigger;

[0068] Represents the static credibility benchmark of the local knowledge base, Represents a static credibility benchmark for web search results. Represents the time interval since the last full synchronization of the local knowledge base. Represents the average delay in crawling network data updates or the time since article publication. represents the normalization factor that controls the attenuation of the credibility of the local knowledge base, Represents the normalization factor that controls the attenuation of network data credibility;

[0069] (2) Perform weight normalization operation, there are:

[0070]

[0071]

[0072] in, is the normalized weight of the local knowledge base, is the normalized weight of network data, Represents network connectivity indication, Indicates online availability. Indicates offline switching to pure local mode;

[0073] (3) Perform retrieval result score fusion and ,have:

[0074]

[0075] in, For each candidate The score fusion result of and For each candidate The local knowledge base retrieval score and network data retrieval score;

[0076] (4) Finally Output the fusion result list in descending order.

[0077] On the other hand, the present invention also discloses a workflow-based multimodal water conservancy large model decision support system, which is used to construct a corresponding model to execute the workflow-based multimodal water conservancy large model decision support method, including:

[0078] Data input layer, used for data collection, input and preprocessing;

[0079] Model processing layer, used to build feature models based on preprocessed data;

[0080] The water conservancy domain adaptation layer is used to perform domain adaptation and fine-tuning operations on the feature model;

[0081] The application service layer is used to perform corresponding application service operations based on the fine-tuned model.

[0082] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0083] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0084] As can be seen from the above technical solution, the present invention provides a workflow-based multimodal water conservancy large-scale model decision support method. Compared with the existing technology, the present invention has the following advantages:

[0085] 1. The present invention sets a modular parallel processing architecture in the workflow, which can synchronously process text, image and voice data, realize deep feature extraction and multi-task collaborative execution operations, and shorten the water conservancy emergency response time.

[0086] 2. The present invention constructs a dual-channel knowledge fusion mechanism, combines a large-scale local knowledge base with a real-time networked knowledge base, and adopts a data fusion algorithm to effectively and comprehensively process information, thereby achieving the stability of the offline knowledge base and the real-time updating of network knowledge. At the same time, by setting a spatiotemporal exponential decay fusion algorithm in the knowledge management module, it can dynamically balance the stability of the local knowledge base and the real-time nature of the networked data, achieve a smooth switching effect between offline / online modes, and thus shorten the decision-making response time during the flood season.

[0087] 3. By setting up adversarial training and LoRA lightweight modules in the model fine-tuning stage, the present invention can enhance the ability to recognize water conservancy professional terms, thereby further improving the domain adaptation accuracy and reducing the prediction error of hydrological parameters.

[0088] 4. By setting a cross-modal alignment algorithm based on the knowledge graph at the output layer, the present invention can verify the logical consistency between image recognition and text reporting, thereby reducing the multimodal error rate and avoiding major misjudgments.

[0089] 5. By setting a dynamic priority strategy for flood season in the task scheduler, the present invention can give priority to processing real-time sensor data during heavy rainfall to improve system throughput and facilitate automatic switching of local libraries to ensure service continuity when the network is interrupted.

[0090] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easy to understand through the following description. Of course, it is not necessary to achieve all of the above-mentioned advantages simultaneously in order to implement any product of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0092] Figure 1 Schematic diagram of the overall framework of the decision support method of the present invention;

[0093] Figure 2 Schematic diagram of the model data processing flow in the model processing layer of the present invention;

[0094] Figure 3 This is a schematic diagram of the iterative process of model data adaptation in the adaptation layer in the water conservancy field of the present invention. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. In the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0096] In the embodiment, see Figures 1 to 3 .

[0097] The embodiments of the present invention provide a workflow-based multimodal water conservancy large model decision support system and method.

[0098] Among them Figure 1 As shown, the multimodal water conservancy large model decision support system is used to construct a corresponding model to execute the multimodal water conservancy large model decision support method. At this time, the multimodal water conservancy large model decision support system includes:

[0099] Data input layer, used for data collection, input and preprocessing;

[0100] Model processing layer, used to build feature models based on preprocessed data;

[0101] The water conservancy domain adaptation layer is used to perform domain adaptation and fine-tuning operations on the feature model;

[0102] The application service layer is used to perform corresponding application service operations based on the fine-tuned model.

[0103] At this time, the system execution method process is as follows Figure 2 As shown in the figure, it includes: first, realizing the real-time input of text, image and voice data through the multimodal data acquisition interface, and adopting a unified preprocessing method to ensure the stability and accuracy of the data; then, using the local embedding model bge-m3, language model, voice model and rearrangement model deployed on the server to perform deep feature extraction and semantic analysis on the preprocessed data, supporting multi-task parallel processing such as image recognition and text generation; constructing a dual-channel knowledge fusion mechanism based on the large-scale local knowledge base and the real-time networked knowledge base, and using the data fusion algorithm to comprehensively process the information, so as to realize the organic connection between the stability of the offline knowledge base and the real-time update of the network knowledge; further, according to the needs of the water conservancy field, the collected data is professionally labeled and preprocessed, and the model data is fine-tuned in the field to improve the system's ability to accurately identify and process hydrological information; finally, by generating a workflow to automatically schedule each processing module, develop a dedicated API interface and user interface, and realize a one-stop intelligent service from "questioning" to "analysis" to "decision-making".

[0104] At this time, the local model data deployed in the server is combined with the full-link automation of "questioning-analyzing-decision-making" in the water conservancy scenario, which can realize the real-time collection and parallel processing of multimodal data (text, images, voice, etc.). Through the dual-channel knowledge fusion of offline knowledge base and networked real-time data and multimodal task parallel technology, an efficient, real-time and intelligent "questioning-analyzing-decision-making" one-stop service platform is constructed. This platform not only breaks through the limitations of traditional systems in data processing and real-time updates, but also realizes intelligent processing and decision-making support for water conservancy information under complex working conditions through multi-task parallelism and modular design, thereby significantly improving the accuracy and response speed of monitoring and early warning, and can further significantly improve the stability and real-time response capabilities of the offline environment, providing an integrated solution for water conservancy project monitoring, emergency command and knowledge services.

[0105] Therefore, the system can perform deep feature extraction and semantic analysis of data, support multi-task parallel processing such as image recognition and text generation, and greatly improve the system processing efficiency and accuracy.

[0106] Therefore, further reference Figure 1 The specific implementation steps of the multimodal water conservancy model decision support method include:

[0107] S1. Preprocess multimodal input data (including text, images, and speech) in the water conservancy field through a local model, and convert files, images, and speech in various formats into JSON format using labels and specific Python code.

[0108] The specific operation process of preprocessing at this time includes:

[0109] S11. Acquire multimodal input data in the water conservancy field in the system, including text, images, and voice, and perform preliminary classification based on the acquired multimodal input data according to specified type standards to ensure that they enter the corresponding preprocessing process according to type;

[0110] S12. Use specific models for preprocessing different types of multimodal data:

[0111] For text data, we first use an embedding model to extract semantic features, then use a language model to remove noise, unify the format, perform word segmentation, and tag parts of speech to ensure the quality and processability of the text data, thereby ensuring its semantic stability and accuracy.

[0112] For image data, we use the embedding model to extract image features, perform denoising, resolution adjustment, and feature point extraction to ensure the stability and accuracy of image data.

[0113] For voice data, a voice model is used for voice recognition and processing, including denoising, voice-to-text conversion, and voice feature extraction, to ensure the quality of the voice data for subsequent processing and analysis.

[0114] S2. Using the Dify DSL framework, we build modular workflows that support parallel processing, ensuring that different types of data, such as text, images, and voice, can be collected and processed synchronously after format conversion.

[0115] The specific construction process of the modular workflow includes:

[0116] S21. Within the Dify DSL framework, modularize the workflow and define different data processing modules to ensure that each data type (e.g., text, image, and voice) has an independent processing channel. A parallel processing architecture is also used to ensure that data processing for a specific data type is performed simultaneously within different processing channels. The data processing modules include data acquisition, preprocessing, feature extraction, feature understanding, and template conversion.

[0117] S22. Based on the Dify DSL framework, a parallel processing architecture is employed, enabling data types such as text, images, and voice to be executed simultaneously on multiple processing units. By defining task dependencies and execution order, this optimizes and coordinates multi-task parallel execution, ensuring coordination and synchronization between modules, avoiding data processing bottlenecks, and improving overall efficiency.

[0118] At this time, Dify DSL's dynamic resource scheduling and cross-modal synchronization mechanism are used to optimize and coordinate the parallel execution of multiple tasks. Dynamic resource allocation strategies (GPU prioritization for image processing and CPU for text processing) are configured in modular workflows, and lightweight containerization technology is used to isolate task resources. This ensures efficient resource utilization and enhances system robustness.

[0119] S3. Build a feature model adapted to the water conservancy field by synchronously collecting and processing multimodal data, build data dependencies based on the DSL framework, and perform domain adaptation and fine-tuning on the feature model to enhance the system's ability to accurately identify and process hydrological data, engineering parameters, and other aspects, thereby meeting the high-precision requirements in the water conservancy decision-making process. The water conservancy mechanism is parameterized and embedded into the model fine-tuning process through the three-layer architecture of implicit rules, explicit constraints, and rule activation. Specifically, by inputting early warning and prediction rules and historical hydrological data into the water conservancy feature model, a deep integration of mechanism knowledge and data-driven models is achieved, enabling the water conservancy feature model to understand and apply hydrological physical laws for reasoning.

[0120] refer to Figure 3 The specific operation process includes:

[0121] S31. Multimodal Data Analysis and Feature Extraction: First, we conduct in-depth analysis and feature extraction of multimodal data (including text, images, and speech) in the water conservancy field. Using a feature extraction model adapted for the water conservancy field, we extract useful features such as hydrological data and engineering parameters from different data types, ensuring the data's domain relevance and high-precision representation. These features will provide a foundation for subsequent domain adaptation and model fine-tuning. We also construct a parameterized encoder for the water conservancy mechanism model, converting hydrological physical laws (such as the Saint-Venant equation, the Manning formula, and the flood evolution equation) into a vector representation that can be understood by the water conservancy feature model. Through adversarial training, the water conservancy feature model implicitly learns the physical constraints of water conservancy projects. We design specialized pre-training tasks to enable the model to predict results that conform to hydrological physical laws under given boundary conditions, thus achieving implicit encoding and storage of mechanism knowledge in the model weights.

[0122] S32. Domain Adaptation of Model Data: To meet the needs of the water conservancy sector, we first adapt the model data to the domain by integrating knowledge and data from the water conservancy sector. This allows the model to better understand and process water conservancy-specific data, such as hydroclimatic data and engineering design parameters. To this end, we construct pre-training data based on a water conservancy domain knowledge base and employ the Masked Language Model (MLM) task to domain-adapt the embedding model. Furthermore, we introduce an adversarial training mechanism, using a domain classifier to distinguish between general data and water conservancy data, enhancing the model's sensitivity to specialized terminology.

[0123] S33. Perform LoRA lightweight fine-tuning on the feature model, freeze the parameters of the low-rank matrix of the pre-trained data model, fine-tune only the newly added LoRA adaptation layer, and use a labeled dataset from the water conservancy field for supervised adjustment. This further reduces video memory usage and improves inference speed, ensuring the feature model's accurate processing capabilities in the water conservancy field while optimizing resource consumption and processing efficiency.

[0124] Key control indicators of water conservancy projects (such as flood control limit water levels, minimum ecological flow rates, and structural safety thresholds) are incorporated as explicit constraints into the model optimization and fine-tuning process. A physical consistency verification mechanism is established to verify whether the output violates hydrophysical laws during each model update. A penalty mechanism is used to correct predictions that violate physical laws to ensure the physical rationality of the model output. Applicable water conservancy project rules and mechanism models are automatically identified based on the input multimodal data features. A dynamic rule weight allocation algorithm is designed to dynamically invoke relevant physical laws and early warning rules during model inference based on current hydrological conditions, project status, and task requirements. This implements an intelligent rule application process of "data features → rule matching → weight allocation → inference activation," enabling the model to adaptively apply the most relevant water conservancy expertise in different engineering scenarios.

[0125] Specifically, through the innovative implicit rule-explicit constraint-rule activation three-layer architecture, the hydraulic mechanism model is parameterized and embedded into the large model fine-tuning process, achieving a deep integration of mechanism knowledge and data-driven model. The three-layer architecture specifically includes the following steps:

[0126] T1. Implicit rule encoding layer: This layer parameterizes hydraulic mechanism models (such as the Saint-Venant equation and hydrological models) into vector representations. This layer then uses specially designed pre-training tasks to enable the large model to implicitly learn the physical constraints of hydraulic engineering. These pre-training tasks include:

[0127] Predict physically correct hydraulic results based on given hydrological boundary conditions;

[0128] Establish a mapping relationship between numerical simulation and measured data so that the model can learn the inherent laws of the physical model;

[0129] Through adversarial training, the model's implicit encoding of physical laws is strengthened, enabling it to develop capabilities similar to "physical intuition";

[0130] T2. Explicit Constraint Layer: Design a multi-objective constraint loss function based on water conservancy project early warning standards and safety regulations. Key control indicators (such as flood control limit water level, minimum ecological flow, and structural safety threshold) are incorporated into the model optimization process as explicit constraints. This layer specifically includes:

[0131] Build a physical consistency verification mechanism to check whether the output results violate the laws of hydrological physics every time the model is updated;

[0132] Design a penalty mechanism to automatically correct predictions that violate physical laws;

[0133] The statistical characteristics of historical hydrological data are introduced as prior knowledge to guide the model to generate prediction results within a physically reasonable range;

[0134] T3. Rule Activation Mechanism Layer: Establish an intelligent water conservancy project rule activation system based on the attention mechanism to implement the intelligent rule application process of "data features → rule matching → weight allocation → inference activation". This layer specifically includes:

[0135] Automatically identify applicable water conservancy engineering rules and mechanism models based on input multimodal data features;

[0136] Design a dynamic rule weight allocation algorithm to dynamically call relevant rules based on current hydrological conditions, project status, and task requirements;

[0137] Construct an association mapping between the warning rule base and the mechanism model so that the model can call the corresponding warning threshold when predicting hydrological changes;

[0138] Implement a scenario-adaptive mechanism to intelligently call the most relevant professional rules in different engineering scenarios (such as flood control, water supply, and ecological scheduling);

[0139] T4. Collaborative training of the three-layer architecture: The three-layer architecture is trained end-to-end, and gradient backpropagation is used to coordinate the layers. Specifically, the following steps are performed:

[0140] Use distillation technology to use the prediction results of the mechanism model as soft labels to guide the learning of the large model;

[0141] Design a multi-task learning framework to simultaneously optimize prediction accuracy and physical plausibility;

[0142] A circular verification mechanism is used to provide complementary verification between the model prediction results and the mechanism model calculation results;

[0143] The knowledge graph in the field of water conservancy projects is introduced as an auxiliary information source to enhance the model's understanding of the conceptual relationship of water conservancy projects.

[0144] Through the above three-layer architecture, this method achieves a deep integration of water conservancy mechanism knowledge and deep learning models, enabling large models to understand and apply hydrophysical laws for reasoning, providing more accurate and reliable technical support for water conservancy project risk prediction and decision-making.

[0145] S4. Perform model validation and dynamic optimization operations, including:

[0146] After the feature model was adapted and fine-tuned for the domain, a test set (including text, image, and voice samples) was constructed for the water conservancy field to conduct domain accuracy assessment. Model performance was verified using metrics such as F1-score and BLEU to ensure that the term recognition accuracy reached above 97%.

[0147] Construct a set of cross-modal consistency samples for verification to ensure the logical consistency of different modal outputs. For example, check whether the "crack width" identified by image recognition matches the "osmotic pressure level" described in text, thereby improving the credibility of multimodal outputs.

[0148] The model is deployed by integrating the A / B testing module and adopting a grayscale release strategy. User feedback data (such as expert-corrected reports) automatically triggers incremental training of the model to perform online dynamic optimization and closed-loop improvement.

[0149] In summary, through these verification and optimization processes, the model will be continuously fine-tuned and optimized to ensure the accuracy of data processing and the reliability of decision support in the water conservancy field, and ultimately meet the needs of high-precision identification and processing in the water conservancy decision-making process.

[0150] Furthermore, a cross-modal alignment algorithm is used during the cross-modal consistency sample verification process to ensure the semantic consistency of data in different modalities, thereby reducing the error rate after consistency verification and providing experimental data. The use of the cross-modal alignment algorithm includes:

[0151] L1. Perform knowledge graph embedding pre-training in the form of TransE for all triples Minimize, with knowledge graph embedding pre-training loss The calculation formula is:

[0152]

[0153] in, The margin hyperparameter of the model is used to ensure the effective distinction between positive and negative samples in the knowledge graph embedding pre-training; after training, all and ; 、 、 are the vectors of the head entity, relationship, and tail entity in the graph embedding space, and represent the starting entity of the relationship, the semantic relationship between entities, and the target entity of the relationship in the knowledge graph, respectively. is the L2 norm, is the set of all triples in the knowledge graph;

[0154] L2. For each positive sample ,from Sampling several negative entities composition , perform cross-modal feature mapping and align the loss, with:

[0155]

[0156] in, It is a set of negative samples or a space containing negative samples. Specifically, it is the source from which negative entities are sampled and used to optimize the comparison with positive samples during training. Represents a mapping network for visual features With text features Mapped to an embedding space with the same dimensions as the entity, The cross-modal alignment loss includes the hinge loss of visual alignment and text alignment. Represents the positive entity The sampled negative entity set, The margin hyperparameter representing the alignment loss, Represents the similarity function between A and B;

[0157] L3. Construct training objectives, including:

[0158]

[0159] in, , used to represent the weights and bias sets of the vision-text mapping network, is the weight of the visual mapping network, is the bias of the visual mapping network, is the weight of the text mapping network, is the bias of the text mapping network, is a regularization term whose value is the sum of the squares of all trainable parameters, is the L2 regularization weight.

[0160] This cross-modal alignment algorithm first pre-trains TransE embeddings on entities and relationships in the water conservancy engineering knowledge graph to obtain a unified vector space representation. It then uses two linear mapping networks—one to map image features into this embedding space, and the other to map text features into the same space—to enable direct comparison of both visual and textual representations with knowledge graph entity vectors. By constructing a hinge-like alignment loss for positive and negative samples, it ensures that the "crack width" image feature is aligned closely with its correct "crack width segment" entity vector, and that the text description is aligned with its "permeability level" entity vector, while simultaneously distancing erroneous entities and fine-tuning network parameters. During inference, the most similar entities in the knowledge graph entity space for both image and text features are selected and checked for match based on predefined "semantic consistency" relationships in the knowledge graph, completing a cross-modal consistency check. This method significantly improves algorithm alignment accuracy and reduces consistency errors in multimodal water conservancy inspection scenarios.

[0161] In summary, adversarial training, LoRA fine-tuning and dynamic incremental optimization are combined to solve the problem of professional terminology recognition in the water conservancy field using general models.

[0162] It should also be noted that in step S32, an adversarial training mechanism is introduced, a domain feature adversarial network is introduced, and the performance differences of the general model are compared. A domain classifier is used to distinguish general data from water conservancy data to improve the feature model's sensitivity to professional terminology. The adversarial training mechanism specifically includes the following adversarial training improvement method operation steps:

[0163] P1. Divide the water conservancy dataset into a source domain (labeled data, such as typical watershed monitoring data) and a target domain (unlabeled or lightly labeled data, such as real-time data from newly built facilities). Standardize the data to eliminate differences in distribution.

[0164] P2. Build a feature extraction module based on an improved conditional generative adversarial network. Combined with a deep residual shrinkage network to compress redundant feature dimensions, a long short-term memory network is introduced to capture temporal dependencies to form a model architecture. A domain discriminator is added to the model architecture as the core of adversarial training. A gradient reversal layer is used to force the feature encoder to generate domain-invariant features.

[0165] P3. Based on the model architecture built in step P2, perform adversarial training operations, including:

[0166] Phase 1: Pre-training feature encoders on source domain data to initially extract key features related to hydraulic parameters (such as flow, pressure, and vibration frequency);

[0167] Stage 2: Fix the feature encoder and train the domain discriminator to distinguish the source domain and target domain features. By minimizing the domain discriminator loss function, the feature encoder is optimized to confuse the discriminator and achieve domain feature alignment.

[0168] Phase 3: Based on adversarial training, the model is fine-tuned using labeled data from the source domain and a small amount of labeled data from the target domain. A weighted cross-entropy loss function is used to balance the class imbalance problem.

[0169] P4. Embed a dynamic threshold judgment module in the model output layer to adaptively adjust the classification threshold based on the statistical characteristics of historical water conservancy project data (such as the range of flow fluctuations under extreme weather conditions) to improve model robustness.

[0170] In a specific embodiment, experimental results show that the proposed adversarial fine-tuning model (DFAN) significantly outperforms both general and conventional fine-tuning models on target domain water conservancy project data: DFAN's root mean square error (RMSE) is reduced by 13.4% (1.032 to 0.894), and its MAE (mean absolute error) is reduced by 14.9% (0.824 to 0.701). Furthermore, its coefficient of determination (R²) is improved to 0.837. Notably, DFAN effectively eliminates the data distribution differences between the general and target domains during adversarial training by introducing a domain feature adversarial network. This mechanism not only improves prediction accuracy (R² approaches the ideal value of 1) but also significantly enhances model robustness. The extraction of cross-domain invariant features during adversarial training enables the model to maintain stable output even in the face of target domain data fluctuations or local outliers (such as sudden sensor noise interference in water conservancy projects). In contrast, conventional fine-tuning models, which fail to address domain shift, are limited in both accuracy and stability.

[0171] S4. Based on the domain-adapted and fine-tuned feature model, a "question-analysis-decision" workflow is constructed to automatically and dynamically schedule the multimodal task resources of each sub-module, realizing a fully automated dynamic processing flow from data input and analysis to decision output.

[0172] The specific operation process includes:

[0173] S41. Based on the fine-tuned feature model, a set of "question-analysis-decision" workflows is constructed, clarifying each link from data input, analysis process, and decision output. Through the automated scheduling system, the task flow relationship between each sub-module is defined, including data collection, model inference, and result analysis. Ensure that each module can automatically initiate the corresponding analysis and processing tasks based on user input and smoothly advance to decision output according to the predetermined logic;

[0174] S42. Implementation and optimization of fully automated processes through workflow automation systems: Establish a fully automated process from data input to decision output, ensuring that the system can automatically handle various needs in the water conservancy field based on real-time data and preset rules, ensuring that the system can perform various tasks efficiently and stably, and providing intelligent decision-making support.

[0175] At this time, we designed modular workflows for water conservancy scenarios (such as the full automation of the "question-analysis-decision" process), adopted preset water conservancy-specific task priority scheduling strategies (such as prioritizing real-time sensor data processing during flood season), and verified their improvement in system throughput. We also described the unique fault-tolerance mechanisms in the workflow (such as automatically switching to a local knowledge base when a network interruption occurs) and compared the stability differences with traditional systems. This was used for dynamic resource scheduling to optimize the efficiency of task execution in the workflow, including:

[0176] A. Network status monitoring:

[0177] (a) The system periodically (e.g., every 5 seconds) initiates a lightweight heartbeat check on the cloud retrieval interface or gateway in the background;

[0178] (b) If the heartbeat detection fails three times in a row or the request times out, the network is deemed unavailable.

[0179] B. Automatic switching:

[0180] (a) Trigger condition: triggered immediately after detecting network unavailability;

[0181] (b) Switching Logic: All tasks in the "questioning" phase that originally required external retrieval or online model prediction are now invoked using a local offline knowledge base and local model. Simultaneously, user requests and system feedback are written to a local persistent queue (to ensure they are not lost).

[0182] (c) Fallback recovery: After the network is restored, the local queue is immediately cleared, and all requests accumulated during the offline period are resent to the cloud service in the original order and synchronized with the latest index or model update. After the local queue is cleared, the system returns to online mode.

[0183] A. Subtask retry and backoff:

[0184] (a) For critical sub-processes (such as real-time sensor pull and image recognition), a maximum of three retries are performed in the event of a single failure, with an exponential backoff interval (2s, 4s, and 8s) between each retry;

[0185] (b) If all retries fail, the system immediately switches to local mode and stops blocking the main process.

[0186] Furthermore, in the scheduling judgment logic of the task priority scheduling strategy, the baseline priority of each task is dynamically adjusted according to the water conservancy operation status (such as flood season and non-flood season):

[0187] During flood season, real-time sensor data tasks are automatically elevated to the highest priority;

[0188] For flood control instructions and routine monitoring tasks, during the non-flood season, the normal priority setting will be restored;

[0189] The scheduler uses preemptive scheduling. When a new task arrives, it compares the priority of the new task with that of the currently executing task. If the former has a higher priority, the new task is immediately interrupted and switched; otherwise, the task continues to execute.

[0190] For the same priority, the CPU is allocated on a first-come-first-served basis or in a round-robin manner.

[0191] Through simulation experiments, compared with static priority and no-priority strategies, this solution significantly improves system throughput in flood season scenarios, verifying its effectiveness in ensuring real-time data processing capabilities at critical moments.

[0192] S5. Use a dynamic fusion algorithm to combine the local knowledge base and network data to achieve dual-channel knowledge fusion, which is used to parallelly execute multimodal tasks in S4 to ensure the stability of the offline knowledge base and the real-time update of network data. At this time, through dual-channel knowledge fusion and multimodal task parallel technology, the stability and real-time response capabilities of the offline environment are significantly improved, providing an integrated solution for water conservancy project monitoring, emergency command and knowledge services, and realizing accurate and efficient intelligent decision-making support.

[0193] The specific operation process of executing the dynamic fusion algorithm to combine data includes:

[0194] S51. The offline knowledge base and real-time data are acquired through the local knowledge base and networked data collection interfaces, respectively, and independent preprocessing is performed on each to ensure uniform data format and quality assurance. For networked real-time data, timely format conversion, noise removal, and verification are required to ensure its timeliness and accuracy. The local knowledge base focuses on the stability and consistency of offline data, ensuring the provision of high-quality reference information even when there is no network connection.

[0195] S52. Utilizing a fusion strategy, the local knowledge base and online real-time data are effectively combined, dynamically integrating the strengths of both types of data. This ensures that, in different application scenarios, the offline knowledge base provides stable information, while the online data is updated in real time to provide the latest decision support. This fusion process ensures that the system can balance stability and real-time performance in various working environments, providing accurate and comprehensive data support.

[0196] At this time, a new fusion algorithm is proposed based on the spatiotemporal weight allocation strategy of water conservancy scenarios. Through the dynamic fusion algorithm, the stability of the local knowledge base and the real-time nature of the networked data are balanced, which can shorten the decision response time after fusion by 30%, and is used to solve the contradiction between the traditional system in offline environment and real-time update.

[0197] The dynamic fusion algorithm uses a dynamic fusion formula of spatiotemporal exponential decay + normalization, which can smoothly transition between offline and online modes and effectively shorten the response time of post-fusion decision-making in water conservancy scenarios. Its specific application method adopts a multi-dimensional credibility assessment algorithm based on the unique time series characteristics of water conservancy projects. This algorithm comprehensively considers factors such as data timeliness, hydrological periodicity, and the impact of emergencies to construct a dynamic weight adaptive integration mechanism, including:

[0198] (1) Calculate the time-effectiveness trust value. The calculation formula is:

[0199]

[0200]

[0201] in, and Represent the timeliness trust value of the local knowledge base and network data respectively. The longer the time since the local knowledge base was updated, the The smaller the value, the shorter the time since the network data was updated. The bigger;

[0202] Represents the static credibility benchmark of the local knowledge base, Represents a static credibility benchmark for web search results. Represents the time interval since the last full synchronization of the local knowledge base. Represents the average delay in crawling network data updates or the time since article publication. represents the normalization factor that controls the decay (“time half-life”) of the credibility of the local knowledge base, Represents the normalization factor that controls the decay of the credibility of network data ("time half-life");

[0203] (2) Perform weight normalization operation, there are:

[0204]

[0205]

[0206] in, is the normalized weight of the local knowledge base, is the normalized weight of network data, Represents network connectivity indication, Indicates online availability. Indicates offline switching to pure local mode;

[0207] (3) Perform retrieval result score fusion and ,have:

[0208]

[0209] in, For each candidate The score fusion result of and For each candidate The local knowledge base retrieval score and network data retrieval score;

[0210] (4) Finally Output the fusion result list in descending order.

[0211] This spatiotemporal exponential decay + normalization fusion algorithm first converts the two data sources into decaying trust scores based on the update time delay and spatial distance between the local knowledge base and the networked data, using adjustable spatiotemporal sensitivity and normalization factors. These scores are then normalized so that their sum equals one, automatically falling back to a purely local mode when offline or local data is outdated, while rapidly favoring online results when the network is unobstructed and the data is fresh. Finally, these normalized weights are used to perform a weighted fusion of the local and network decision vectors to output the final decision. This method not only achieves smooth offline / online switching, but also significantly shortens response time and improves spatiotemporal accuracy, meeting the dual requirements of real-time performance and stability in water conservancy scenarios.

[0212] In a further embodiment, the method also constructs a dual-channel knowledge fusion mechanism: first, a dual-channel knowledge fusion mechanism is constructed, combining the local offline knowledge base and the real-time networked knowledge base, and using a data fusion algorithm to effectively and comprehensively process information, which can achieve the stability of the offline knowledge base and the real-time update of network knowledge, so as to achieve an organic connection between the two.

[0213] The dual-channel knowledge fusion mechanism also professionally labels and pre-processes the collected data to meet the special needs of the water conservancy field, and fine-tunes the feature model in the field, thereby significantly improving the system's ability to accurately identify and process hydrological information.

[0214] The dual-channel knowledge fusion mechanism also integrates the local knowledge base and networked data through data fusion algorithms, ensuring that the system can provide stable and reliable knowledge support in an offline environment, while ensuring real-time updating of networked data and improving the system's adaptability in a changing environment.

[0215] In a further embodiment, the method also optimizes multimodal task parallel processing technology: Based on dual-channel knowledge fusion, this technology is optimized to ensure that tasks such as image recognition, text generation, and speech recognition can be executed simultaneously without interfering with each other. Through efficient task scheduling and resource management, the execution efficiency of each task is improved, and the system can achieve seamless collaborative processing between different tasks, thereby accelerating the response speed of decision support.

[0216] In further embodiments, this method can also implement an integrated intelligent decision support system solution: based on dual-channel knowledge fusion and multimodal task parallel processing technology, it provides an integrated intelligent decision support solution for applications in water conservancy project monitoring, emergency command, and knowledge services. By integrating different data sources and processing modules, the system ensures accurate and efficient decision support in various environments, providing comprehensive intelligent services for the water conservancy industry.

[0217] In a further embodiment, this method automatically schedules each processing module by generating a workflow and developing a dedicated API and user interface, ensuring a one-stop intelligent service from "questioning" to "analysis" to "decision-making." This system, for the first time, achieves full-chain automation in water conservancy scenarios by combining local model data deployed on servers. This significantly improves stability and real-time responsiveness in offline environments, providing an integrated solution for water conservancy project monitoring, emergency command, and knowledge services.

[0218] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0219] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0220] In another embodiment provided by the present application, a computer program product containing instructions is also provided, which, when executed on a computer, enables the computer to execute any of the workflow-based multi-modal water conservancy large model decision support methods in the above embodiments.

[0221] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above method.

[0222] The embodiment of the present application further provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0223] Memory for storing computer programs;

[0224] The processor is used to implement the above-mentioned workflow-based multi-modal water conservancy large model decision support method when executing the program stored in the memory.

[0225] The communication bus mentioned in the above electronic device can be a peripheral component interconnect standard bus or an extended industry standard architecture bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0226] The communication interface is used for communication between the above electronic device and other devices.

[0227] The memory may include a random access memory, or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0228] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0229] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part.

[0230] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0231] In addition, it should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0232] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or schemes in which A and B are satisfied at the same time. In addition, in the embodiments of the present invention, "multiple" refers to more than two. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A workflow-based multimodal water conservancy large model decision support method, characterized by: include: S1. Preprocess multimodal input data based on the local model and convert the file, image, and voice data formats into JSON format; S2. By specifying a DSL framework, a modular workflow that supports parallel processing is built to ensure that different types of data after format conversion can be collected and processed simultaneously. S3. Build a feature model adapted to the water conservancy sector using synchronously collected and processed multimodal data. Establish data dependencies based on the DSL framework, adapt and fine-tune the feature model to the domain, and parameterize the water conservancy mechanism through a three-layer architecture: implicit rules, explicit constraints, and rule activation, before embedding it into the model fine-tuning process. S4. Based on the domain-adapted and fine-tuned feature model, a specified workflow is constructed to dynamically schedule multimodal task resources and implement a dynamic processing flow from data input, analysis to decision output. S5. Use a dynamic fusion algorithm to combine the local knowledge base and network data to achieve dual-channel knowledge fusion, which is used to parallelize the multimodal tasks in S4 to ensure the stability of the offline knowledge base and the real-time update of network data. The specific operation process of the S3 step includes: S31. Use a feature extraction model adapted to the water conservancy field to extract specified features from different data types of multimodal data in the water conservancy field to obtain a feature model; S32. Build pre-training data based on the existing water conservancy knowledge base and use the MLM task to perform domain adaptation on the feature model. S33. Perform LoRA lightweight adjustments on the feature model, freeze the parameters of the low-rank matrix of the pre-trained data model, adjust only the newly added LoRA adaptation layer, and use a labeled dataset from the water conservancy field for supervised adjustment. The specified control indicators of the water conservancy project are incorporated into the model adjustment process as explicit constraints to establish a physical consistency verification mechanism. This is used to verify whether the output results violate hydrophysical laws during each model update. A penalty term mechanism is used to correct predictions that violate physical laws to ensure the physical rationality of the model output. The specific operation process of the S4 step includes: S41. Based on the fine-tuned feature model, a set of "question-analysis-decision" workflows is constructed, defining the task flow relationships between each sub-module. This ensures that each module can initiate the corresponding analysis and processing tasks based on user input and smoothly advance to decision output according to the predetermined logic. S42. Use a preset task priority scheduling strategy for dynamic resource scheduling to optimize task execution efficiency in the workflow.

2. The workflow-based multimodal water conservancy large model decision support method according to claim 1, characterized in that: The specific operation process of the pretreatment in step S1 includes: S11. Based on the obtained multimodal input data, perform preliminary classification according to the specified type standard to ensure that they enter the corresponding preprocessing process according to type; S12. Use specific models for preprocessing different types of multimodal data: For text data, semantic features are extracted through embedding models, and then language models are used for noise removal, format unification, word segmentation, and part-of-speech tagging. For image data, the embedding model is used to extract image features, perform denoising, resolution adjustment, and feature point extraction; For speech data, a speech model is used for speech recognition and processing, including denoising, speech-to-text conversion, and speech feature extraction.

3. The workflow-based multimodal water conservancy large model decision support method according to claim 1, characterized in that: The specific construction process of the modular workflow in step S2 includes: S21. Define different data processing modules within the specified DSL framework and employ a parallel processing architecture to ensure that data processing of the specified data types is performed simultaneously within different processing channels. S22. Based on the specified DSL framework, optimize and coordinate the parallel execution of multiple tasks by defining task dependencies and execution order.

4. The workflow-based multimodal water conservancy large model decision support method according to claim 1, characterized in that: After the S3 step is executed, model validation and dynamic optimization operations are performed, specifically including: Construct a test set in the water conservancy field to conduct domain accuracy evaluation, and use F1-score and BLEU indicators to verify model performance; Construct a set of cross-modal consistency samples for testing to ensure the logical consistency of different modal outputs; The deployment model is executed by integrating the A / B testing module and adopting a grayscale release strategy. User feedback data is used to trigger incremental model training to perform online dynamic optimization and closed-loop improvement.

5. The workflow-based multimodal water conservancy large model decision support method according to claim 4, characterized in that: During the cross-modal consistency sample verification process, a cross-modal alignment algorithm is used to ensure the semantic consistency of data in different modalities. The method of using the cross-modal alignment algorithm includes: L1. Perform knowledge graph embedding pre-training in the form of TransE for all triples Minimize, with knowledge graph embedding pre-training loss The calculation formula is: in, The margin hyperparameter of the model is used to ensure the effective distinction between positive and negative samples in the knowledge graph embedding pre-training; after training, all and ; 、 、 are the vectors of the head entity, relationship, and tail entity in the graph embedding space, and represent the starting entity of the relationship, the semantic relationship between entities, and the target entity of the relationship in the knowledge graph, respectively. is the L2 norm, is the set of all triples in the knowledge graph; L2. For each positive sample ,from Sampling several negative entities composition , perform cross-modal feature mapping and align the loss, with: in, It is a set of negative samples or a space containing negative samples. Specifically, it is the source from which negative entities are sampled and used to optimize the comparison with positive samples during training. Represents a mapping network for visual features With text features Mapped to an embedding space with the same dimensions as the entity, The cross-modal alignment loss includes the hinge loss of visual alignment and text alignment. Represents the positive entity The sampled negative entity set, The margin hyperparameter representing the alignment loss, Represents the similarity function between A and B; L3. Construct training objectives, including: in, , used to represent the weights and bias sets of the vision-text mapping network, is the weight of the visual mapping network, is the bias of the visual mapping network, is the weight of the text mapping network, is the bias of the text mapping network, is a regularization term whose value is the sum of the squares of all trainable parameters, is the L2 regularization weight.

6. The workflow-based multimodal water conservancy large model decision support method according to claim 1, characterized in that: The S32 step also introduces an adversarial training mechanism, introduces a domain feature adversarial network, uses a domain classifier to distinguish general data from water conservancy data, so as to enhance the sensitivity of the feature model to professional terminology, enable the feature model to implicitly learn the physical constraints of water conservancy projects, and set a specified pre-training task to allow the feature model to predict results that conform to hydrological physical laws under given boundary conditions, so as to realize the implicit encoding and storage of mechanism knowledge in the model weights. The adversarial training mechanism specifically includes the following adversarial training improvement method operation steps: P1. Divide the water conservancy dataset into source domain and target domain, and eliminate data distribution differences through standardization; P2. Generate an adversarial network feature extraction module. This module combines a deep residual shrinkage network to compress redundant feature dimensions and introduces a long short-term memory network to capture temporal dependencies. This forms the model architecture. A domain discriminator is added to the model architecture as the core of adversarial training. A gradient reversal layer is used to force the feature encoder to generate domain-invariant features. P3. Based on the model architecture constructed in step P2, perform adversarial training operations, including: Phase 1: Pre-training feature encoders on source domain data to initially extract key features related to water conservancy parameters; Stage 2: Fix the feature encoder and train the domain discriminator to distinguish the source domain and target domain features. By minimizing the domain discriminator loss function, the feature encoder is optimized to confuse the discriminator and achieve domain feature alignment. Phase 3: Based on adversarial training, the model is fine-tuned using labeled data from the source domain and a small amount of labeled data from the target domain. A weighted cross-entropy loss function is used to balance the class imbalance problem. P4. Embed a dynamic threshold judgment module in the output layer of the model to adaptively adjust the classification threshold based on the statistical characteristics of historical data of water conservancy projects.

7. The workflow-based multimodal water conservancy large model decision support method according to claim 1, characterized in that: The scheduling judgment logic of the task priority scheduling strategy in step S4 includes: Dynamically adjust the baseline priority of various tasks according to the water conservancy operation status: During flood season, real-time sensor data tasks are automatically elevated to the highest priority; For flood control instructions and routine monitoring tasks, during the non-flood season, the normal priority setting will be restored; The scheduler uses preemptive scheduling. When a new task arrives, it compares the priority of the new task with that of the currently executing task. If the former has a higher priority, the new task is immediately interrupted and switched; otherwise, the task continues to execute. For the same priority, the CPU is allocated on a first-come-first-served basis or in a round-robin manner.

8. The workflow-based multimodal water conservancy large model decision support method according to claim 1, characterized in that: The specific operation process of executing the dynamic fusion algorithm to combine data in step S5 includes: S51. Obtain offline knowledge base and real-time data through the local knowledge base and network data acquisition interface, and perform independent preprocessing operations respectively; S52. Combine the local knowledge base and networked data. The offline knowledge base provides stable information, and the networked data is updated in real time and provides the latest decision support. A dynamic fusion algorithm is used to balance the stability of the local knowledge base and the real-time nature of the networked data.

9. The workflow-based multimodal water conservancy large model decision support method according to claim 8, characterized in that: The method of using the dynamic fusion algorithm includes: (1) Calculate the time-effectiveness trust value. The calculation formula is: in, and Represent the timeliness trust value of the local knowledge base and network data respectively. The longer the time since the local knowledge base was updated, the The smaller the value, the shorter the time since the network data was updated. The bigger; Represents the static credibility benchmark of the local knowledge base, Represents a static credibility benchmark for web search results. Represents the time interval since the last full synchronization of the local knowledge base. Represents the average delay in crawling network data updates or the time since article publication. represents the normalization factor that controls the attenuation of the credibility of the local knowledge base, Represents the normalization factor that controls the attenuation of network data credibility; (2) Perform weight normalization operation, there are: in, is the normalized weight of the local knowledge base, is the normalized weight of network data, Represents network connectivity indication, Indicates online availability. Indicates offline switching to pure local mode; (3) Perform retrieval result score fusion and ,have: in, For each candidate The score fusion result of and For each candidate The local knowledge base retrieval score and network data retrieval score; (4) Finally Output the fusion result list in descending order.

Images