A large-scale multi-sub-basin hydrological model deep learning rate setting method
By employing GPU parallel computing and a parallel gradient optimization mechanism with multiple initializations, the problems of low computational efficiency and insufficient automated processing in large-scale watershed hydrological models are solved, achieving efficient and accurate parameter calibration and meeting real-time business requirements.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAN UNIV OF TECH
- Filing Date
- 2026-02-10
- Publication Date
- 2026-06-05
AI Technical Summary
Existing large-scale watershed hydrological models are computationally inefficient, making it difficult to balance real-time performance and accuracy. Furthermore, they lack a fully automated processing mechanism, resulting in parameter updates lagging far behind business needs in emergency scenarios.
The high concurrency characteristics of graphics processing units (GPUs) are used for global collaborative matrix parallel computation. Combined with a parallel gradient optimization mechanism with multiple initializations and a unified batch processing architecture, a fully automatic closed loop from data input to parameter output is achieved.
It significantly improved computing throughput, met the timeliness requirements of business needs such as flood forecasting, enhanced the robustness and accuracy of parameter calibration, and reduced operation and maintenance costs and operational complexity.
Smart Images
Figure CN122153298A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of hydrological forecasting, specifically to a deep learning calibration method for large-scale multi-sub-basin hydrological models. Background Technology
[0002] Accurate runoff forecasting is fundamental to watershed flood control, water resource allocation, and water ecosystem protection. In large-scale watershed management (such as the Yangtze and Yellow River systems or trans-regional water conservancy projects), the entire watershed is typically divided into hundreds or thousands of sub-basin units to improve simulation accuracy. With climate change leading to more frequent extreme rainfall events and the need for digital twin watershed construction, operational systems not only require high-precision models but also place extremely high demands on real-time performance—that is, they need to use the latest measured data to quickly correct the model parameters of all sub-basins across the entire region to ensure that the forecast results are consistent with the actual hydrological conditions.
[0003] Hydrological models are the core tool for runoff prediction. They contain a large number of physical parameters that cannot be directly measured (such as soil water storage capacity and groundwater recession coefficient), which must be determined through "parameter calibration" using historical observation data. Although calibration techniques for single watersheds are relatively mature, when dealing with large systems including oceanic watersheds, and requiring real-time calibration that is coordinated across the entire region and highly efficient and accurate, existing technologies face the following three main problems: First, existing computing architectures mostly employ a serial computing model, resulting in low computational efficiency and difficulty in meeting the demands of real-time applications. Traditional hydrological model programs are largely based on single-machine serial logic designs. When processing large-scale watersheds, computers must process sub-watersheds sequentially. This serial model causes computation time to increase linearly or even exponentially with the number of sub-watersheds. Faced with hundreds of sub-watersheds and long-sequence historical data, a single calibration often takes hours to days. However, in emergency scenarios such as flood forecasting, updates to parameters across the entire region need to be completed within minutes, and the computational speed of existing technologies lags far behind business requirements.
[0004] Second, existing parameter optimization methods struggle to balance computational speed and optimization accuracy (global convergence). Currently, mainstream parameter calibration methods fall into two main categories: one is global search algorithms (such as genetic algorithms and particle swarm optimization), which have strong optimization capabilities but require tens of thousands of model iterations, resulting in enormous computational overhead and making them unsuitable for online real-time computation; the other is gradient-based local search algorithms, which are computationally fast, but traditional hydrological models contain numerous discontinuous hard threshold judgments (such as runoff thresholds), leading to difficulties in gradient calculation or non-differentiability, and single-gradient searches are prone to getting trapped in local optima. Existing technologies lack a mechanism that can leverage the computational advantages of gradients while possessing global search capabilities in a very short time, resulting in unstable or insufficiently accurate online computation results.
[0005] Third, existing processing workflows lack unified data management and automated closed-loop systems, resulting in low data I / O and post-processing efficiency. In current technologies, hydrological and meteorological data from different sub-basins are often stored in a fragmented manner as massive amounts of small files, leading to extremely high I / O latency during retrieval. Furthermore, model execution and result analysis are typically separate steps, often requiring manual intervention or external scripts to sift through the vast amounts of data after calculation. This non-automated process not only increases operational complexity but also makes it difficult to achieve a fully automated closed loop from data input to parameter output, limiting its large-scale application in digital twins and real-time operational systems.
[0006] In summary, existing technologies for parameter calibration of large-scale sub-basin groups suffer from problems such as low computational efficiency, difficulty in balancing speed and accuracy, and lack of automated batch processing. Therefore, there is an urgent need for a new method that can utilize parallel computing technology to perform collaborative, rapid, and high-precision calibration of the entire hydrological model. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention provides a deep learning calibration method for large-scale multi-sub-basin hydrological models. This method solves the technical problems existing in current large-scale watershed hydrological model parameter calibration techniques, such as inefficient serial computation, difficulty in balancing speed and accuracy in optimization algorithms, and lack of a fully automated batch processing mechanism. It provides a parallel, fast, and accurate calibration method for hydrological models that is designed for large-scale watershed-wide collaborative operation.
[0008] Specifically, this invention aims to achieve the following objectives: Breaking through the bottleneck of serial computing and meeting the requirements of real-time response, this invention addresses the problem of low efficiency in existing technologies that rely on "queuing up one by one" for computation. It utilizes the high concurrency characteristics of modern graphics processing units (GPUs) to reconstruct the physical processes of hundreds or thousands of sub-basins within a large-scale watershed into a globally collaborative matrix parallel operation, eliminating linear computation time lag and providing computational support for digital twin watersheds and real-time rolling forecasts that meet the "minute-level" timeliness requirements.
[0009] By combining the advantages of speed and accuracy, and ensuring robustness of online calibration, this invention addresses the problem that existing optimization methods struggle to balance convergence speed and global optimality within a limited time. It constructs a parallel gradient optimization mechanism based on multiple initializations, which searches concurrently from multiple starting points in the solution space at the same time. This retains the efficient convergence capability of the gradient algorithm while avoiding local optimum traps by utilizing wide-area coverage, ensuring high-precision and stable physical parameters in fast online computing scenarios.
[0010] By constructing a unified batch processing architecture to achieve end-to-end fully automated closed loop, this invention addresses the problems of fragmented processes and reliance on manual intervention in existing processes. It establishes a tensor-based unified batch processing and automated optimization mechanism based on full-domain data. By connecting the entire chain of data input, parallel model computation, and automatic result selection, it eliminates cumbersome intermediate processing steps and achieves a "one-click" fully automated closed loop from massive multi-source data to high-precision parameter output, significantly reducing the threshold for business applications and operation and maintenance costs.
[0011] To achieve the above objectives, the present invention is implemented through the following technical solution:
[0012] Preferably, the aforementioned.
[0013] This invention provides a deep learning calibration method for large-scale multi-sub-basin hydrological models. It offers the following advantages: 1. This invention breaks through the efficiency bottleneck of serial computing in large-scale watersheds, achieving efficient concurrent calibration of massive watershed parameters. Addressing the problem of linear efficiency decay with the number of watersheds caused by serial cyclic computing in existing technologies, this invention reconstructs hydrophysical processes into high-dimensional matrix operations based on tensors, fully utilizing the single instruction multiple data (SID) characteristics of graphics processors. The system integrates the independent computing tasks of hundreds or thousands of watersheds into a single batch processing task, enabling the concurrent hardware triggering of rainfall and runoff simulations for all watersheds within the same clock cycle. This architecture avoids the queuing time in traditional methods, significantly compressing the large-scale regional calibration tasks that originally consumed a long time on the CPU, significantly improving computing throughput, and meeting the time-sensitive business needs of flood forecasting, reservoir group scheduling, and other applications.
[0014] 2. This invention solves the problem of single initial value gradient optimization easily getting trapped in local optima, balancing convergence speed and global optimization accuracy. Addressing the shortcomings of existing technologies such as high computational cost of evolutionary algorithms and the tendency of traditional gradient descent algorithms to get trapped in local minima, this invention constructs a dual parallel initialization mechanism of "watershed-parameter set". By expanding the search space in the tensor dimension, the system can launch thousands of independent model evolution tasks simultaneously, using parallel computing capabilities to achieve wide coverage of the solution space. Combined with the precise gradient guidance provided by automatic differentiation technology, this method retains the advantage of rapid convergence of the gradient descent algorithm while effectively avoiding local traps under non-convex objective functions through a multi-path concurrent exploration mechanism. This enables the model to obtain a global (or near-global) optimal solution in a shorter time, enhancing the robustness of parameter calibration results under complex hydrogeographic conditions.
[0015] 3. This invention constructs an end-to-end automated batch processing and optimization mechanism based on a unified tensor architecture, reducing the complexity of engineering applications. Addressing the issues of fragmented data processing and reliance on manual intervention for result optimization in existing technologies, this invention establishes a standardized data flow and automated decision-making system. Through a unified index mapping table, data broadcast alignment, and parallel optimization logic, the system achieves a closed-loop process from multi-source heterogeneous data input and massive parameter parallel evolution to automatic export of optimal results. The system can automatically traverse massive parallel results, automatically filter and output the best parameter scheme for each watershed based on preset physical and statistical indicators, eliminating the need for tedious manual data splicing and post-processing comparisons. This end-to-end automated workflow reduces labor costs and operational barriers, eliminates errors caused by inconsistent manual operation standards, and standardizes the construction of large-scale hydrological model systems. Attached Figure Description
[0016] Figure 1 This is a system block diagram of the present invention; Figure 2 This is a diagram illustrating the batch data mapping and broadcasting mechanism of the present invention; Figure 3 This is a calculation diagram of the discretized physical process of the present invention; Figure 4 This is a diagram of parallel loss aggregation and gradient backpropagation in this invention. Detailed Implementation
[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] Please see the appendix Figure 1-4 This invention provides a deep learning calibration method for a large-scale multi-sub-basin hydrological model.
[0019] The overall system architecture and workflow of this invention includes three main stages: data construction and initialization stage, GPU parallel training loop stage, and output stage. The entire process is divided into seven core steps (S1 to S7). Each step is coupled with data structures such as parameter tensors and index mapping tables to form a fully automated process from data input to parameter output.
[0020] Step S1: Data Integration and Index Building First, a large-scale data warehouse supporting high-concurrency reads is established. The collected multi-source heterogeneous data is cleaned and standardized. The hydrological and meteorological data (including driving data such as rainfall, evaporation, and temperature, as well as measured runoff data) of multiple watersheds to be calibrated are uniformly processed. The method is to assign a globally unique digital identifier (UID) to each watershed and construct a structured index map. This map records the physical storage address, data length, and start and end time of each watershed's data. Through this indexing mechanism, data can be read by directly locating memory addresses or file fragments during the calculation process, avoiding I / O latency caused by traversing the original file and ensuring that the data flow meets the high throughput requirements of the computing core.
[0021] Step S2: Model Reconstruction and Hardware Adaptation To adapt to the Single Instruction Multiple Data (SIMD) characteristics of GPUs, the continuous differential equations describing hydrological processes are transformed into a computer-executable discrete form and then reconstructed using vectorization. In specific implementation, the computational timing of physical processes (such as rainfall infiltration, evaporation, soil moisture exchange, and runoff generation and confluence) is defined, and the above logical operations are reconstructed into matrix operation forms to replace traditional scalar logic judgments. In addition, a smoothing function is introduced to approximate discontinuous boundaries in physical processes (such as threshold judgments) to ensure the differentiability of the computation graph throughout the process, laying the foundation for gradient-based automatic parameter optimization.
[0022] Step S3: Initialize the parallel parameter set A data organization structure with dual parallelism is constructed. In computational memory, the data is organized into a dual parallel structure of number of watersheds × number of initialization sets. For each individual watershed, multiple sets of independent random initial physical parameters are generated simultaneously on the GPU. Each watershed is configured with [number] watersheds. The initial parameters will be used to execute the system concurrently on the GPU. This is an independent model evolution task. This step increases the probability of finding the globally optimal parameters by using computational resources to improve the coverage of the search space and start searching for the optimal solution from multiple different starting points at the same time.
[0023] Step S4: Batch Mapping and Broadcasting After entering the training loop, the dimensions of the execution data and parameters are aligned. The data loader extracts the watershed data of the current training batch according to the batch index and extracts the corresponding parameter blocks from the parameter tensor. Using tensor broadcasting technology, the meteorological driving data of each watershed is virtually expanded in the dimension of the parameter group so that it is completely matched with the dimension of multiple sets of parallel parameters. This step ensures that each set of parameters has an independent input data context, and achieves large-scale parallel forward computation without increasing the actual physical memory usage.
[0024] Step S5: Compilation Optimization and Forward Computation The just-in-time (JIT) compilation technology of the deep learning framework is used to perform static analysis and optimization on the reconstructed hydrological model code. Operator fusion is used to merge multiple consecutive mathematical operations into a single computing kernel, reducing the number of GPU memory read and write operations. Under hardware drive, the computing core processes the rainfall-runoff process of all watershed-parameter combinations in parallel, updates state variables such as soil moisture content and groundwater status in time sequence, and finally outputs the full simulated runoff sequence.
[0025] Step S6: Loss Aggregation and Gradient Optimization Parameter updates are performed using gradient-based deterministic optimization algorithms (such as the Adam optimizer). After the forward computation is completed, the simulated runoff and the measured runoff are compared point by point to calculate the error index. Then, the errors of all watersheds and all parameter groups are aggregated to generate scalar loss values. The reverse-mode automatic differentiation mechanism is used to traverse the computation graph in reverse to accurately calculate the gradient of the loss function with respect to each physical parameter. Based on the differential chain rule, the gradient updates of each watershed and each parameter group are independent and do not affect each other.
[0026] Step S7: Result Optimization and Export After completing a specified number of iterations of training, an automated optimization logic is executed. For each watershed, a horizontal comparison is made between the results of the multiple sets of initialization parameters mentioned above in parallel. Based on the performance evaluation metrics on the training set (such as Nash efficiency coefficient NSE or Kling-Gupta efficiency coefficient KGE), the optimal set of parameters is automatically selected. The selected parameter set is then applied to the validation set to verify the correctness of the calibration results. Finally, the optimal parameter set is extracted using tensor indexing technology and serialized and exported.
[0027] The specific execution process of step S1 is as follows: this step supports high-concurrency reading of large-scale data by building a structured index mechanism.
[0028] S1.1 Multi-source heterogeneous data cleaning and standardization processing The collected long-term meteorological driving data (including rainfall, temperature, radiation, etc.) and measured runoff data of the watershed were preprocessed. For missing values and outliers in the original data, linear interpolation or statistical methods were used to fill and correct them. At the same time, data from different sources were resampled to a unified time step.
[0029] To eliminate the influence of different physical dimensions on the stability of subsequent numerical calculations, a standardization operation is performed on all continuous time series data. The standardization calculation formula is as follows:
[0030] in, Indicates the first The river basin The original observation value at time; and These are the mean and standard deviation of the corresponding variable for this watershed over a long time series; It is the numerical stability constant; The standardized dimensionless input data is processed in this way to ensure that the mean of all input features is 0 and the variance is 1, which lays the foundation for the smooth convergence of the gradient of the subsequent model parameters.
[0031] S1.2 Construction of Globally Unique Identifier and Index Mapping Table After data cleaning, the system assigns a globally unique numerical identifier (ID) to each independent watershed. Based on this, an index map is constructed. This map records the correspondence between physical storage locations and logical data attributes. The map is designed as a metadata structure, with each row corresponding to the data index information of a watershed. Its structure is defined as follows:
[0032] in, For the first Index vectors of watersheds; A unique integer identifier for the watershed; A handle or path to the physical file that stores the watershed data; This is the starting byte offset of the watershed data in the file; The length of the data block in bytes; and The start and end times of valid data are identified separately. By constructing this static index table, unstructured file storage is transformed into structured metadata that can be directly queried.
[0033] S1.3 Direct addressing based on file pointers During the data loading phase of the training iteration, a direct addressing mechanism based on file pointers is adopted, and the data loader reads the index mapping table. The physical address of the target data block is located directly by moving the file pointer. The specific data acquisition logic is as follows:
[0034] in, This refers to the set of watersheds included in the current training batch; For file pointer movement operations, it is possible to locate the file pointer in O(1) time complexity. Location; This is a data read operation, used to read data of length. The mechanism uses contiguous blocks of memory, which reduces the reading of irrelevant data and disk seek time, enabling the supply of data to the computing core at a high throughput rate.
[0035] The specific execution process of step S2 is as follows: this step transforms the continuous hydrophysical process into a discrete form adapted to GPU tensor operations and constructs a computational graph that is differentiable throughout the process.
[0036] S2.1 Time Discretization and State-Space Reconstruction of Continuous Physical Processes Using the explicit Euler method or the second-order Runge-Kutta method, the set of ordinary differential equations controlling the hydrological process is transformed into a set of difference equations. In this embodiment, a state-space model of the hydrological cycle is constructed, where the reservoir state at each time step is determined by the residual amount at the previous time step and the input and output fluxes at the current time step. The discretization update logic of the state variables is as follows:
[0037] in, represent The watershed state vector at time (including components such as soil moisture content and groundwater storage). and These represent the current rainfall input and potential evapotranspiration input, respectively. A set of physical parameters; a function and They respectively characterize inflow processes (such as infiltration and recharge) and outflow processes (such as evaporation and runoff generation). This step represents the time step of the discrete simulation, and it realizes the mapping of physical conservation laws in the discrete time dimension.
[0038] S2.2 Tensorization and High-Dimensional Parallelism Adaptation for Logical Operations To adapt to the Single Instruction Multiple Data (SIMD) characteristics of graphics processing units (GPUs) and eliminate scalar loops and conditional judgments for individual watersheds, this embodiment employs tensor quantization to reconstruct the physical calculation logic. Specifically, all watersheds to be simulated and their corresponding multiple parameter sets are stacked into a batch tensor, and the scalar operators are expanded into matrix operations using a broadcast mechanism. The calculation formula is as follows:
[0039] in, For state tensors, For the batch size of the watershed, The number of independent parallel parameter groups set for each watershed. For state variables; For weather-driven tensors; This is the corresponding parameter matrix; This represents the vectorized flow generation operator. Through this architecture, the hardware can execute physical operators in parallel on batches of stacked flow domain-parameter combinations, avoiding the latency caused by serial computation.
[0040] S2.3 Smoothing Approximation and Operator Fusion of Discontinuous Thresholds To address the gradient discontinuity problem caused by hard threshold logic (such as soil water full-storage runoff determination) in traditional hydrological models, a smoothing function is introduced to approximate discontinuous boundaries. For example, a Sigmoid weighted average or Softplus function is used to replace the hard truncation determination. The runoff calculation after smoothing is as follows:
[0041] in, The flow rate is the superpermeability production rate; This represents the maximum water storage capacity of the soil. For smooth activation functions; To control the hyperparameters of smoothing steepness, this operation eliminates non-differentiable points, ensuring that the computation graph meets the requirements of automatic differentiation. On this basis, operator fusion technology is applied to merge multiple mathematical operations such as smoothing calculation and state update into a single computation kernel, reducing the read and write usage of GPU memory.
[0042] The specific execution process of step S3 is as follows: This step constructs a high-dimensional tensor for storing model parameters and performs multi-path parallel initialization to provide data structure support for subsequent gradient-based parallel optimization.
[0043] S3.1 Memory Allocation and Structure Definition of High-Dimensional Parametric Tensors To support parallel computing across multiple watersheds and parameter sets, the system allocates contiguous storage space in the GPU memory and constructs a three-dimensional parameter tensor as the input container for the computational core. The tensor's dimensional design includes both watershed and parameter set dimensions, and its mathematical structure is defined as follows:
[0044] in, This represents the total number of watersheds to be trained; The number of independent parallel parameter groups set for each watershed; To address the number of physical parameters (such as saturated hydraulic conductivity and drainage coefficient) that need to be calibrated in the hydrological model, this tensor-based memory layout unifies the parameter optimization task for multiple watersheds into matrix operations on a single data structure. This allows the optimizer to adjust parameters simultaneously through a single gradient update operation. A single, independent combination of parameters can improve computational throughput.
[0045] S3.2 Multi-way random initialization under physical constraints After the tensor is constructed, its elements are initialized. To avoid optimization getting trapped in local minima and to ensure that the parameters have physical meaning, a constrained uniform distribution or Latin hypercube sampling (LHS) strategy is adopted to generate diverse initial values within the physically feasible region of the parameters. Taking the constrained uniform distribution as an example, the initialization calculation logic is as follows:
[0046] in, Indicates the first The first watershed, the first In the parallel group, the th Initial values for each parameter; and The first The theoretical lower and upper bounds of a physical parameter; To obtain from a standard uniform distribution The random variables sampled in the middle ensure that the initial parameter group has broad coverage in the physical space, providing a diverse search starting point for the subsequent gradient descent algorithm, thereby increasing the possibility of obtaining the global optimum.
[0047] The specific execution process of step S4 is as follows. This step is executed in the training loop. Through dynamic index extraction and dimension broadcasting technology, the dimension matching problem between multiple parallel parameters and single meteorological driving data is solved, and the alignment of input data and parameter tensors is achieved.
[0048] S4.1 Index-based dynamic parameter block extraction At the beginning of each iteration of the training loop, the data loader retrieves the set of watershed indices for the current training batch. The system uses this set of indexes to retrieve global parameter tensors located in GPU memory. The process performs slicing operations to dynamically extract the parameter sub-tensors required for the current batch. This process uses a vectorized indexing mechanism, which can efficiently retrieve data from non-contiguous memory addresses. The calculation logic for parameter extraction is as follows:
[0049] in, This is a full parameter tensor for storing all watershed parameters; This is a tensor aggregation operator used to collect data along a specified dimension based on an index. This indicates that the operation is performed at the watershed level. This step, which extracts local parameter blocks, enables local computation tasks to load global parameters on demand, ensuring that each forward computation only processes parameter data for the relevant watershed, thus reducing memory bandwidth usage.
[0050] S4.2 Driven Data Dimension Broadcasting and Context Alignment Because each watershed has The system consists of independent parallel parameters, and the corresponding meteorological driving data (such as rainfall and temperature) are unique within the same time period. Therefore, dimension alignment is required. Tensor broadcasting technology is used to virtually expand the meteorological driving data in the parallel group dimensions without increasing the actual physical memory footprint. The expanded data dimensions are from... Become Thus, with parameter tensors The dimensions are fully aligned, and its mathematical expression is as follows:
[0051] in, This refers to the original batch of meteorological data; For view expansion operations; For the expanded data tensor, this step establishes a one-to-one mapping between meteorological data and multiple sets of parameters, ensuring that each set of parameters has corresponding driving data inputs to support subsequent operations. Parallel forward propagation of a hydrological model instance.
[0052] The specific execution process of step S5 is as follows: This step constructs a static computation graph through just-in-time compilation technology and executes the forward propagation process of the physical model in parallel on the hardware.
[0053] S5.1 Static Computation Graph Construction and Operator Fusion To overcome the efficiency bottlenecks caused by the interpreted execution of dynamic programming languages (such as the parallel limitations caused by the Global Interpreter Lock (GIL)), the system introduces Just-In-Time (JIT) compilation technology. The system performs execution tracing on the dynamically defined hydrological model code, converting it into a static computation graph. During this process, the compiler performs layer-level optimization on the computation graph, executing operator fusion. This operation merges multiple previously independent consecutive mathematical operations (including addition, multiplication, and activation function calculations) into a single fused kernel. This process reduces the number of GPU memory read / write operations and kernel startup overhead. The formal expression of its optimization logic is as follows:
[0054] in, This is the original physical model code; This is the optimized sequence of machine instructions; This represents the fused high-performance computing core. Through this step, the forward propagation logic of the model is transformed into an intermediate representation (IR), which then directly calls the underlying hardware instruction set for execution.
[0055] S5.2 Parallel State Evolution and Flux Calculation After compilation and optimization, the aligned meteorological driving tensor and parameter tensor from step S4 are input into the optimized computational kernel. A time-step-based cyclic forward propagation is executed. The computational kernel, based on hydrophysical equations, updates the internal state variables (such as soil moisture content and groundwater storage) of all watershed-parameter combinations in parallel, and calculates the water flux at each stage. The state update and runoff output process follows the batch-based state transition equations defined below:
[0056] in, for A slice of weather data at any given moment; This is the global state tensor at the current moment; To simulate runoff output tensors; As the parameter tensor of the current batch, thanks to the prior data alignment and operator optimization, this calculation process is characterized by high-concurrency matrix multiplication and element-wise operations in hardware, which can efficiently complete the simulation of long sequences and large-scale batches of hydrological processes and generate full simulated runoff sequences.
[0057] The specific execution process of step S6 is as follows: This step is executed after the forward propagation is completed. The parameter gradient is calculated using automatic differentiation technology, and the physical parameters are updated through deterministic optimization algorithm.
[0058] S6.1 Multidimensional Error Assessment and Loss Scalarization After forward propagation, the system obtains a shape of To construct the optimization objective, the difference between the simulated and measured values of the simulated runoff tensor is first calculated. The system uses the Nash efficiency coefficient (NSE) or mean square error (MSE) as the difference measurement function to perform independent accuracy evaluation for each watershed-parameter group unit.
[0059] Subsequently, the system performs loss scalarization, aggregating the high-dimensional error tensor into a global loss value through a weighted summation or averaging strategy. The calculation logic is as follows:
[0060] in, The global scalar loss after aggregation; A difference measure function for a single sequence; For the first The first watershed Simulated runoff sequences generated by group parameters; For the corresponding measured runoff sequence, this step transforms the errors of the multi-path parallel physical simulation into a single global optimization objective function.
[0061] S6.2 Gradient Backpropagation Based on Automatic Differentiation Obtain scalar loss Then, the system triggers the reverse mode automatic differentiation mechanism. This mechanism utilizes the chain rule of differentiation to traverse the computation graph constructed in the forward computation phase from top to bottom. The system automatically calculates the partial derivatives (i.e., gradients) of the loss function with respect to each physical parameter, although... It is the aggregate value of all errors, but according to the linear property of partial derivatives, a specific parameter only affects the calculation path it is in. Therefore, this process can accurately analyze the contribution of the parameter to the error. The gradient calculation logic is as follows:
[0062] in, Let the gradient matrix of the parameter tensor have the same dimensions as the original parameter tensor. Completely identical. This step utilizes a computational graph framework to obtain high-dimensional gradient vectors for parameter tuning.
[0063] S6.3 Optimizer Iteration and Parameter Adjustment Using the calculated gradient information, a preset optimization algorithm (such as Adam or SGD) is invoked to update the physical parameters. The optimizer calculates the parameter correction step size based on the current gradient direction and the preset learning rate, and applies it to the parameter tensor. The parameter update rule follows the form below (taking the Adam optimizer as an example):
[0064] in, The parameter value at the current moment; The learning rate; and These are the first-order moment estimate and the second-order moment estimate of the gradient, respectively; To prevent numerical stability constants with a denominator of zero; The updated parameter values are as follows. The specific implementation details of the optimization algorithm are well-known in this field and will not be elaborated here. Through multiple rounds of forward computation-error evaluation-backward propagation-parameter update loop, the physical parameters gradually converge to the global or local optimal solution.
[0065] The specific execution process of step S7 is as follows: This step is executed after the training iteration terminates or the gradient converges. The optimal parameter set for each watershed is selected by traversing the parameter results of multiple parallel paths and then exported.
[0066] S7.1 Constructing a Multidimensional Performance Evaluation Matrix Once the training meets the preset termination conditions (such as reaching the maximum number of iterations or the change in the loss function being less than a threshold), the system enters the verification and evaluation phase. Based on independent training set data, the system performs performance calculations for each watershed-parameter group unit. The computational core executes simulation calculations in parallel and constructs a performance evaluation matrix based on hydrological accuracy indicators such as the Nash efficiency coefficient (NSE) or Kling-Gupta efficiency coefficient (KGE). This matrix stores the final simulation accuracy of each watershed under different random initialization paths, and its elements are calculated as follows:
[0067] in, For the first The first watershed Evaluation scores of group parameters; This is a function for evaluating the accuracy of hydrological data. Forward computation operators of hydrological models; For the first Training-driven data for each watershed; At the end of training The first watershed Group parameter status; For the corresponding measured runoff data, this step generates a shape of The two-dimensional evaluation matrix.
[0068] S7.2 Index Positioning Based on Optimization Strategy After obtaining the evaluation matrix, automatic optimization logic is executed for each watershed. A maximum value search (Argmax) operation is performed on the parameter group dimension to identify the index position corresponding to the parameter group with the highest evaluation score. The logic for locating the optimal index is as follows:
[0069] in, For the first The index of the optimal parameter set for each watershed; through this operation, the index value of the optimal parameter combination for each watershed in the multi-path parallel search is determined.
[0070] S7.3 Extraction and Persistence of Optimal Parameter Set The system calculates the optimal index vector. The final calibration result is extracted from a high-dimensional parameter tensor using tensor indexing techniques. This operation is based on the index. Slice and extract data along the parameter group dimension, reducing the data dimension from... Become The parameter extraction process is as follows:
[0071] in, That is, the first output of the system The final physical parameter vectors of each watershed are extracted, and the parameter set is first substituted into the model and run on independent validation set data to evaluate its generalization performance and verify the correctness of the calibration results. After successful verification, the parameter set is stored in the medium as the final physical parameter scheme for the watershed calibration.
[0072] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A deep learning calibration method for a large-scale multi-sub-basin hydrological model, characterized in that, Includes the following steps: Step S1: Data preparation and index building: Establish a data warehouse that supports high-concurrency reading, clean and standardize multi-source heterogeneous data, assign a globally unique digital identifier to each independent watershed, and build a structured index mapping table that records the correspondence between physical storage location and logical data attributes; Step S2: Model Reconstruction and Computational Graph Construction: The continuous differential equations describing the hydrological process are transformed into discrete form and reconstructed using vectorization to build a fully differentiable computational graph. Step S3: Parameter Tensor Initialization: Construct a high-dimensional parameter tensor with a dual parallel structure in the computation memory, and generate multiple sets of independent random initial physical parameters on the graphics processor simultaneously for each individual watershed. Step S4: Data Loading and Dimension Alignment: Enter the training loop, extract the parameter blocks of the current training batch based on the structured index mapping table constructed in step S1, and use tensor broadcasting technology to expand the meteorological driving data on the parallel parameter group dimension of the high-dimensional parameter tensor to achieve dimension alignment between the input data and the parameter tensor. Step S5: Forward propagation calculation: Using just-in-time compilation technology, the reconstructed hydrological model code is converted into a static calculation graph. The aligned data and parameters from step S4 are input into the fully differentiable calculation graph constructed in step S2. The forward propagation calculation of the physical model is executed in parallel, and the full simulated runoff sequence is output. Step S6: Gradient calculation and parameter update: Calculate the error between the simulated runoff sequence and the measured runoff output in step S5 and aggregate them into a global scalar loss value. Calculate the gradient of the physical parameters using the inverse mode automatic differentiation mechanism and update the high-dimensional parameter tensor in step S3 using a deterministic optimization algorithm. Step S7: Result Optimization and Derivation: After training, a multidimensional performance evaluation matrix is constructed based on the training set data. The optimal parameter set for each watershed is selected from the high-dimensional parameter tensor updated in step S6. The optimal parameter set is then applied to the validation set to verify the correctness of the model parameter calibration results. Finally, the optimal parameter set is derived.
2. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 1, characterized in that, In step S1, the structured index mapping table is a metadata structure. Each row in the structured index mapping table corresponds to the data index information of a watershed. The data index information specifically includes: a unique integer number of the watershed, a physical file path pointing to the storage of the watershed data, the starting byte offset of the watershed data in the file, the byte length of the data block, and the start and end times of the valid data.
3. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 2, characterized in that, In step S1, the structured index mapping table is used for a direct addressing mechanism based on file pointers: During the data loading phase, the data loader reads the structured index mapping table, uses file pointer movement operations to directly locate the position of the starting byte offset, and reads the continuous memory block corresponding to the byte length, thereby obtaining the data of the watershed set included in the current training batch.
4. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 1, characterized in that, In step S2, the construction of a fully differentiable computational graph specifically includes: A smoothing function is introduced to approximate the discontinuous boundaries in the physical process, and a weighted function is used to replace the hard cutoff judgment, eliminating non-differentiable gradient breakpoints; By applying operator fusion technology, multiple consecutive mathematical operations of smoothing calculation and state update are merged into a single computational kernel; the state update follows a state space model, in which the reservoir state at each time step is determined by the residual amount at the previous time step and the input flux and output flux at the current time step.
5. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 1, characterized in that, In step S3, the construction of a high-dimensional parametric tensor with a dual parallel structure specifically includes: A contiguous storage space is allocated in the video memory to construct a three-dimensional parameter tensor. The dimension design of the three-dimensional parameter tensor includes the watershed dimension, the dimension of the number of independent parallel parameter groups set for each watershed, and the dimension of the number of physical parameters that need to be calibrated in the hydrological model. A constrained uniform distribution strategy or a Latin hypercube sampling strategy is used to generate diverse initial values within the physically feasible domain of the parameters to initialize the three-dimensional parameter tensor.
6. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 5, characterized in that, In step S4, the step of aligning the dimensions of the input data with the parameter tensor using tensor broadcasting specifically includes: Local parameter blocks for the current training batch are extracted from the full parameter tensor using a vectorized indexing mechanism. Using tensor broadcasting technology, without increasing the actual physical memory usage, the meteorological driving data of the current training batch is expanded in terms of the number of independent parallel parameter groups, generating an expanded data tensor so that each parameter group has corresponding driving data input.
7. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 1, characterized in that, In step S5, the conversion of the reconstructed hydrological model code into a static computation graph using just-in-time (JIT) compilation technology specifically includes: Execution tracing is performed on dynamically defined hydrological model code, and the hydrological model code is converted into an intermediate representation; The compiler performs layer-level optimization and operator fusion operations on the static computation graph, merging the originally independent addition, multiplication and activation function calculations into a single computation kernel, and calling the underlying hardware instruction set to process the rainfall and runoff processes of all watersheds and parameter combinations in parallel.
8. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 1, characterized in that, In step S6, the calculation of the gradient of the physical parameters using the inverse mode automatic differentiation mechanism specifically includes: Each combination of watershed and parameter group is independently evaluated for accuracy. The high-dimensional error tensor is aggregated into the global scalar loss value through a summation strategy. Using the differential chain rule, the computation graph is traversed from top to bottom in reverse order. The partial derivative of the global scalar loss value with respect to each physical parameter is calculated as the gradient. The specific parameter only affects the computation path it is in, and the gradient updates are independent of each other.
9. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 5, characterized in that, In step S7, constructing a multidimensional performance evaluation matrix based on the training set data specifically includes: Based on the training set data, simulation operations are performed in parallel for each combination unit of watershed and parameter group; Based on the hydrological accuracy index, the evaluation score of each watershed under different random initialization paths is calculated, and a two-dimensional evaluation matrix with the shape of watershed batch size multiplied by the number of independent parallel parameter groups is generated.
10. The deep learning calibration method for a large-scale multi-sub-basin hydrological model according to claim 9, characterized in that, In step S7, the process of filtering and exporting the optimal parameter set for each watershed specifically includes: For each watershed, a maximum value search operation is performed on the dimension of the number of independent parallel parameter groups to identify the index position corresponding to the parameter group with the highest evaluation score on the training set data. Using tensor indexing technology, the final physical parameter vector is extracted from the three-dimensional parameter tensor by slicing according to the index position and then serialized and exported.