A Lightweight Runoff Forecasting Method Based on Adaptive Knowledge Distillation

Abstract

This invention discloses a lightweight runoff forecasting method based on adaptive knowledge distillation. The method constructs a dual-teacher model consisting of an LSTM-CAM-Transformer deep model and an AR-modified Xin'anjiang model, and builds a lightweight dual-student network. Utilizing the adaptive knowledge transfer mechanism, knowledge is transferred from the dual-teacher model library to the lightweight dual-student network model, and multi-objective collaborative training is performed to jointly optimize the parameters of the two student networks and the gated fusion unit. The output of the lightweight dual-student network model is dynamically weighted and fused through the gated fusion unit to obtain the final runoff forecast result. This method effectively integrates data features and conceptual hydrological mechanisms, significantly reducing the number of model parameters and computational costs while maintaining forecast accuracy. It is suitable for real-time flood forecasting of watersheds and deployment of edge devices.

Description

Technical Field

[0001] This invention relates to the field of runoff forecasting technology, specifically a lightweight runoff forecasting method based on adaptive knowledge distillation. Background Technology

[0002] Runoff forecasting is a core component of non-engineering flood control and disaster reduction measures. Its forecast accuracy and ease of deployment directly affect the flood control safety of the basin, the rational utilization of water resources, and the scientific and timely nature of disaster prevention and mitigation decisions. It is of great significance for reducing flood disaster losses and protecting people's lives and property.

[0003] With the rapid development of computer technology, deep learning has been applied to the field of runoff forecasting. Deep learning models (such as Long Short-Term Memory networks LSTM) have shown excellent fitting ability in processing long-sequence hydrological data, but they are often regarded as "black box" models, lacking hydrophysical interpretability. At the same time, in pursuit of high forecast accuracy, the structure of deep learning models has become increasingly complex (with many layers and a large number of parameters), resulting in extremely high computational resource consumption and making it difficult to deploy on edge terminal devices with limited computing power (such as hydrological monitoring stations and drones).

[0004] To address the issues of high computational costs and difficulty in deploying deep learning models for runoff prediction on edge devices, knowledge distillation technology has been introduced. As a model compression method, knowledge distillation aims to transfer knowledge from complex, parameter-intensive, and computationally resource-intensive models (i.e., "teacher models") to simpler, less computationally burdensome, lightweight models (i.e., "student models").

[0005] However, existing knowledge distillation techniques have significant limitations when applied to runoff forecasting. First, most employ a "single teacher-single student" architecture, relying solely on a data-driven teacher model for guidance, failing to simultaneously address both in-depth data feature extraction and the transmission of hydrophysical laws. Second, the core of runoff forecasting model training lies in parameter calibration and error calculation (such as the NSE objective function), while traditional distillation methods typically assign equal weights to samples across all time steps when calculating errors. This approach neglects "flood peaks" or flow "abrupt changes" in the runoff hydrograph, which are of great value for disaster prevention and mitigation, resulting in severely insufficient accuracy in flood peak forecasting under extreme conditions for the lightweight student model. Summary of the Invention

[0006] The purpose of this invention is to provide a lightweight runoff forecasting method based on adaptive knowledge distillation. By constructing a dual-teacher model library of "data-driven + hydrological mechanism", the training of a lightweight dual-student network is guided. Furthermore, by utilizing an adaptive weighting mechanism, the limitations of equal sample weights in traditional distillation are overcome, and the learning of difficult samples such as flood peaks is enhanced. Finally, through the fusion output of the gating unit, a high-precision, low-computing-power-consumption runoff forecast with hydrological mechanism interpretability is achieved.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: The lightweight runoff forecasting method based on adaptive knowledge distillation includes the following steps: Acquire and preprocess watershed hydrological and meteorological data; Based on the preprocessed hydrological and meteorological data, a dual-teacher model library and a dual-student lightweight network model are constructed. Using an adaptive knowledge transfer mechanism, knowledge is transferred from the dual-teacher model library to the dual-student lightweight network model, and multi-objective collaborative training is performed to jointly optimize the parameters of the two student networks and the gated fusion unit. The output of the dual-student lightweight network model is dynamically weighted and fused by a gated fusion unit to obtain the final runoff forecast result.

[0008] According to the above technical solution, the dual-teacher model library includes a first teacher model and a second teacher model; The first teacher model is an LSTM-CAM-Transformer deep learning model, used to extract the nonlinear length-short dependency features of watershed hydrological data, serving as an expert in the data-driven field. The second teacher model is the Xin'anjiang hydrophysical model combined with AR correction, used to simulate the physical processes of runoff generation and confluence in the basin and to ensure water balance, as an expert in the field of physics.

[0009] According to the above technical solution, the dual-student lightweight network model includes a first student model and a second student model; The first student model is a lightweight LSTM model, with fewer LSTM layers than the first teacher model. For example, the first teacher model contains a 5-layer LSTM stacked structure, while the first student model is configured as a 3-layer LSTM structure. The number of network nodes in each layer is compressed to 50% of that in the teacher model through channel pruning to significantly reduce computational complexity.

[0010] The second student model is a recurrent neural network constrained by hydrological mechanisms, used to learn the generalized state variables of the second teacher model in order to inherit the hydrological evolution law.

[0011] According to the above technical solution, the feature map and soft label of the first teacher model are used as reference targets in the training process of the first student model. Based on the feature map and predicted value generated by the first student model, the training loss function of the first student model is calculated, and the weight parameters of the first student model are updated through backpropagation to realize knowledge transfer. The physical state sequence output by the second teacher model is used as the supervision target. The output value of the fully connected mapping layer of the second student model is required to approximate and align with the physical state sequence. During the approximation and alignment process, the error value between the output value of the fully connected mapping layer of the second student model and the physical state sequence is calculated through the physical constraint loss function, thereby completing the transfer of physical knowledge from the second teacher model to the second student model.

[0012] The adaptive knowledge transfer mechanism addresses the shortcoming of traditional distillation methods, which assign equal weights to all samples during error calculation (ignoring peak flow or sudden flow mutations). It calculates the characteristic differences between the teacher and student models along the runoff process and dynamically generates adaptive weights for the samples. This mechanism employs an adaptive weighting strategy based on sample difficulty. When the prediction difference is significant (e.g., peak flow or sudden flow mutations), the weight of that sample in the loss function is automatically increased, forcing the student model to focus on and fit the more difficult samples.

[0013] According to the above technical solution, the training loss function of the first student model It consists of hard loss, soft loss, and feature matching loss: ; in, Indicates hard loss. This represents soft loss, and y represents the measured flow rate. This represents the output logical value of the first teacher model. This represents the output logical value of the first student model (referring to the most primitive numerical value of the model output). , and Represented as the balance coefficient, This represents the predicted flow value of the first student model (referring to the final predicted result with actual physical meaning output by the model). This represents the feature matching loss.

[0014] According to the above technical solution, the feature matching loss : ; in, This represents a 1×1 convolutional layer. This represents the j-th eigenvalue of the feature map of the first teacher model. This represents the j-th feature value of the feature map of the first student model. This indicates the dimension of the feature map. Convolutional layers are used to increase the number of feature channels in the student model to match that of the teacher model, thus addressing the channel number mismatch issue in lightweight models.

[0015] Based on the above technical solution, the weights of the first student model : ; in, Let represent the prediction value of the first teacher model at time i. Let β represent the prediction value of the first student model at time i, and β be the sensitivity hyperparameter.

[0016] When the predictions of the teacher model and the student model differ significantly (i.e., runoff peak or abrupt change point), the weights... Automatically increasing the size forces the student model to more closely follow the teacher model on difficult samples, thereby achieving precise control over the learning direction.

[0017] According to the above technical solution, the knowledge transfer from the second teacher model to the second student model adopts the "hydrological state alignment" method; the output of the second teacher model (Xin'anjiang model) includes a set of physical state sequences. Where W is the water storage capacity of the basin, and S free R represents free water storage, and R represents runoff; the hidden layer state vector of the second student model. Through a fully connected mapping layer Mapping to the physical state space, and constructing the physical constraint loss function based on it. : The physical constraint loss function : ; In the formula, The intermediate conceptual hydrological state variables represent the second teacher model; This represents the hidden layer state vector of the second student model; This represents a fully connected mapping layer used to map the hidden state vectors of the second student model. Mapped to the physical state space.

[0018] According to the above technical solution, the gated fusion unit receives the predicted flow of the first student model and the predicted flow of the second student model in the dual-student lightweight network model, and generates dynamic fusion coefficients using a lightweight attention mechanism. The predicted flow of the first student model and the predicted flow of the second student model are dynamically weighted and fused according to the dynamic fusion coefficients to obtain the final runoff forecast.

[0019] Wherein, the dynamic fusion coefficient g: ; Final forecast flow : ; in, It is the Sigmoid activation function. bf are learnable parameters. This represents the predicted flow of the first student model. This represents the predicted flow of the second student model. This mechanism ensures that data-driven results are emphasized under normal flow conditions, while results are constrained by hydrological mechanisms under extreme conditions.

[0020] Compared with the prior art, the beneficial effects achieved by the present invention are: 1. Lightweight, efficient and easy to deploy: This invention introduces knowledge distillation and channel pruning techniques to compress the large and complex deep learning model into a lightweight student model, which greatly reduces the number of parameters and computational complexity, thus greatly meeting the low computing power deployment requirements of hydrological edge computing devices (such as monitoring stations and drones).

[0021] 2. Breaking the black box and possessing interpretability of hydrological mechanisms: Unlike purely data-driven networks, this invention innovatively proposes a "hydrological state alignment" mechanism, which injects intermediate variables of the Xin'anjiang conceptual hydrological model (such as water storage and runoff) into the hidden layer of the student network, giving the neural network a clear hydrological and physical meaning.

[0022] 3. Accurately captures flood peaks and exhibits strong robustness across all operating conditions: It overcomes the shortcomings of traditional distillation methods in the unreasonable allocation of sample weights. The designed adaptive weighting mechanism forces the model to tilt towards difficult samples with large errors, such as flood peaks, during training, which significantly improves the forecast accuracy of extreme runoff. At the same time, the gated fusion unit dynamically coordinates the advantages of data-driven nonlinear fitting with the conservation advantages of hydrological models, ensuring robustness across all operating conditions during dry and runoff periods. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the steps of the lightweight runoff forecasting method based on adaptive knowledge distillation of the present invention; Figure 2 The computational logic diagram for the gated fusion unit; Figure 3 The graph shows the dynamic response and data-model trust level changes of the adaptive gating fusion unit. Figure 4 Comparison diagram of runoff process lines and extreme flood peak abrupt change zones for different models. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Taking the actual flood season hydrological and meteorological data (including hourly precipitation P, evaporation E, and flow Q) from the Lingkou hydrological station in the Yellow River-Yiluo River basin as an example, this invention applies the lightweight runoff forecasting method based on adaptive knowledge distillation to predict runoff. Specific steps ( Figure 1 )include: S1. Environment and Data Preprocessing Stage: Accurate flood season hydrological and meteorological data (including hourly precipitation P, evaporation E, and flow rate Q) from the Lingkou hydrological station in the Yellow River-Yilo River basin were acquired and preprocessed. The data was divided into training and test sets according to chronological order (e.g., 80% as training and 20% as test). To prevent future information leakage, only the training set was used to fit a normalization function (MinMaxScaler) to scale the input features to the [0-1] interval. Subsequently, a sliding window technique (e.g., setting the window length to 24 hours) was employed to segment the long sequence into segments containing historical features and the forecast target for the next hour.

[0026] S2. Construct a dual-teacher model library (including a first teacher model and a second teacher model): The first teacher model (data-driven expert) adopts an LSTM-CAM-Transformer architecture. Specifically, it includes a 5-layer LSTM network with 128 hidden dimensions for extracting Long Short-Term Memory features; followed by a Channel Attention (CAM) mechanism to enhance feature channels useful for flood peaks; finally, a 2-layer Transformer Encoder with 8 attention heads is connected to extract global dependencies. It is fully pre-trained using a training set until the Nash efficiency coefficient (NSE) reaches above 0.85. After training, all weight parameters of the model are frozen (putting it in eval mode), so that it only performs forward inference in subsequent distillation. The input data is pre-processed real flood season hydrological and meteorological data (including hourly precipitation P, evaporation E, and flow rate Q) from the Lingkou hydrological station in the Yellow River-Yiluo River basin. The output is a predicted logistic value. and feature map .

[0027] The second teacher model (hydrological mechanism expert): This model runs the Xin'anjiang hydrological model based on autoregressive (AR) corrections, inputting basin-wide rainfall and evaporation to obtain the actual intermediate physical state variable sequence within its operation. In this embodiment, three core states are selected: basin water storage W, free water storage S. free Runoff volume R constitutes a set of physical states. , and its final forecast flow T2(x).

[0028] S3, a lightweight dual-student network model (including a first-student model and a second-student model): The first student model (data student) adopts a lightweight LSTM architecture, reducing the number of layers from 5 in the teacher model to 3. The hidden layer dimensions are compressed to 50% of the teacher model's (i.e., 64 dimensions) through channel pruning. To address the mismatch between the subsequent 128-dimensional features and the teacher model, an additional 1×1 convolutional layer is cascaded after the hidden layers in this model (i.e., ...). As a feature adapter, its dimension is increased to 128, and the output is a fitted feature map. .

[0029] The second student model (mechanism student) employs a hydrological mechanism-constrained recurrent neural network (PIRNN). In this embodiment, a single-layer GRU network with 32 hidden dimensions is selected. Its core feature lies in setting a fully connected mapping layer (fmap) after the GRU hidden layer, forcibly mapping the 32-dimensional abstract hidden state vector to a 3-dimensional physical space vector, corresponding to the Xin'anjiang model. .

[0030] S4. Using the adaptive knowledge transfer mechanism, knowledge is transferred from the dual-teacher model library to the dual-student lightweight network model. Specific steps include: The input to the first student model is the original hydrological and meteorological sequence data, and the feature map and soft label (predicted logistic value) output by the first teacher model are used as reference targets during the training process of the student model. The first student model generates its own feature map and predicted value based on the same input, and then the training loss function of the first student model is calculated, and the weight parameters of the first student model are updated through backpropagation, thereby realizing knowledge transfer.

[0031] The weights of the first student model are used for weighted loss calculation and gradient allocation during backpropagation. During the model training loop, the difficulty of the samples is dynamically sensed; for the i-th sample in each batch, the predicted value of the first teacher model is obtained. And the predicted values ​​of the first student model Calculate the weights of the first student model. : ; The sensitivity hyperparameter β was set to 2.0. When encountering highly valuable "peak" mutation samples, the student model often produces significant errors due to insufficient fitting ability (i.e.,...). (A dramatic increase), the tanh function rapidly approaches saturation, causing the weights to... Automatically scaled up to close to 2.0. This mechanism forces the backpropagation algorithm to allocate double the gradient update intensity on difficult samples without requiring manual labeling of flood peaks.

[0032] The training loss function of the first student model consists of hard loss, soft loss, and feature matching loss: ; in, Indicates hard loss. This represents soft loss, and y represents the measured flow rate. This represents the output logical value of the first teacher model. This represents the output logical value of the first student model. , and Represented as the balance coefficient, This represents the predicted flow value of the first student model. This represents the feature matching loss.

[0033] Hard loss The predicted flow value for the first student model Mean square error between measured flow rate y and actual flow rate y; soft loss The output logic value of the first student model The output logic value of the first teacher model Mean squared error; feature matching loss To adapt the feature map Feature map of the first teacher model The mean square error. After weighting, according to the formula... Perform fusion computing.

[0034] The input to the second student model is the raw hydrological data, and the physical state sequence output by the second teacher model is used as the supervision target. The second student model's fully connected mapping layer output value is required to approximate and align with the physical state sequence. Specifically: The predicted hydrological state sequence output from the fully connected mapping layer of the second student model is extracted and directly compared with the physical state sequence calculated by the second teacher model (Xin'anjiang model). Calculate the global mean square error and construct the physical constraint loss function. By minimizing This forces the neurons of the GRU to evolve the ability to represent water balance according to physical laws, addressing the pain point of deep learning models lacking hydrophysical interpretability. It also facilitates the transfer of physical knowledge from the second teacher model to the second student model.

[0035] In the process of knowledge transfer using the adaptive knowledge transfer mechanism described above, this invention performs multi-objective collaborative training, jointly optimizing the parameters of the two student models and the gating fusion unit.

[0036] Specifically, during the training phase, it is necessary to minimize not only the distillation loss between the student model and the teacher model output (to learn the teacher model's hidden knowledge), but also the prediction loss between the fused runoff forecast and the actual observations. A multi-objective loss function is constructed, and the backpropagation algorithm is used for end-to-end joint training of the entire network. In each iteration, the weight parameters of the first student model, the second student model, and the gated fusion unit are updated synchronously based on the total loss. This multi-objective collaborative training and joint optimization approach ensures that the two lightweight student models, while extracting features from their respective domains, achieve optimal fit with the gated fusion unit, thereby maximizing the final runoff forecast accuracy while reducing model complexity.

[0037] S5. Design a lightweight gating fusion unit to convert the predicted flow of the first student model. (i.e., the one mentioned above) ) and the predicted flow of the second student model (i.e., the runoff R in the physical state vector output by the second student model) is concatenated along the feature dimension. After processing through a fully connected layer and a sigmoid activation function, dynamic fusion coefficients g∈(0,1) are generated, and then the final predicted flow is calculated: The steps are as follows: Figure 2 As shown.

[0038] Will The mean square error obtained from the actual flow rate calculation Ultimately, , and The sums are used to obtain the total loss function, which is then used to update the parameters through backpropagation using the Adam optimizer.

[0039] And plot the dynamic response of the adaptive gating fusion unit and the change in data-model confidence, as shown in the figure. Figure 3 ,Depend on Figure 3It can be seen that the gating unit adaptively learns to prioritize high-precision data-driven results under normal flow conditions (larger g), and to combine robust results from the conceptual hydrological model under extreme flow conditions (smaller g). During deployment, only the trained first student model, second student model, and gating fusion unit need to be retained, eliminating the large teacher model. Testing shows that the total number of parameters in the dual-student fusion model is reduced by approximately 90% compared to the first teacher model, while its Nash efficiency coefficient (NSE) on the test set remains above 0.82. In particular, the mean absolute percentage error (MAPE) for peak flow is reduced by 15% compared to the traditional single-distillation model, fully meeting the real-time flood forecasting requirements of low-power edge devices.

[0040] Based on the same scenario, the predicted runoff process lines of the first-teacher model, the traditional single-distillation lightweight model, and the dual-teacher fusion lightweight model of this invention are compared, with a focus on highlighting the differences in fitting performance among the models in areas of extreme flood peak abrupt change, such as... Figure 4 As shown.

[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0042] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight runoff forecasting method based on adaptive knowledge distillation, characterized in that, The steps include: Acquire and preprocess watershed hydrological and meteorological data; Based on the preprocessed hydrological and meteorological data, a dual-teacher model library and a dual-student lightweight network model are constructed and pre-trained; the dual-teacher model library includes a first teacher model and a second teacher model; the dual-student lightweight network model includes a first student model and a second student model; An adaptive knowledge transfer mechanism is used to transfer knowledge from the dual-teacher model library to the dual-student lightweight network model. Specifically, this includes: using the feature maps and soft labels of the first teacher model as reference targets during the training process of the first student model; calculating the training loss function of the first student model based on the feature maps and predicted values ​​generated by the first student model; and updating the weight parameters of the first student model through backpropagation to achieve knowledge transfer. The weights of the first student model... : ； in, Let represent the prediction value of the first teacher model at time i. Indicates the first student model in the th... Predicted value at time, This is a sensitivity hyperparameter; The physical state sequence output by the second teacher model is used as the supervision target. The output value of the fully connected mapping layer of the second student model is required to approximate and align with the physical state sequence. During the approximation and alignment process, the error value between the output value of the fully connected mapping layer of the second student model and the physical state sequence is calculated through the physical constraint loss function. Based on the error value, the backpropagation algorithm is used to iteratively adjust and optimize the second student model until the error value converges to a preset threshold, thus completing the transfer of physical knowledge from the second teacher model to the second student model. The output of the dual-student lightweight network model is dynamically weighted and fused by a gated fusion unit to obtain the final runoff forecast result.

2. The lightweight runoff forecasting method based on adaptive knowledge distillation according to claim 1, characterized in that, The first teacher model is an LSTM-CAM-Transformer deep learning model, used to extract the nonlinear length-short dependency features of watershed hydrological data; The second teacher model is the Xin'anjiang hydrophysical model combined with AR correction, used to simulate the physical processes of runoff generation and confluence in the basin and to ensure water balance.

3. The lightweight runoff forecasting method based on adaptive knowledge distillation according to claim 1, characterized in that, The first student model is a lightweight LSTM model, and the number of LSTM layers in the first student model is less than the number of LSTM layers in the first teacher model. The second student model is a recurrent neural network constrained by hydrological mechanisms, used to learn the generalized state variables of the second teacher model.

4. The lightweight runoff forecasting method based on adaptive knowledge distillation according to claim 1, characterized in that, The training loss function of the first student model It consists of hard loss, soft loss, and feature matching loss: ； in, Indicates hard loss. Indicates soft loss. This represents the measured flow rate. This represents the output logical value of the first teacher model. This represents the output logical value of the first student model. , and Represented as the balance coefficient, This represents the predicted flow value of the first student model. This represents the feature matching loss.

5. The lightweight runoff forecasting method based on adaptive knowledge distillation according to claim 4, characterized in that, The feature matching loss : ； in, This represents a 1×1 convolutional layer. This represents the j-th eigenvalue of the feature map of the first teacher model. This represents the j-th feature value of the feature map of the first student model. Indicates the dimension of the feature map.

6. The lightweight runoff forecasting method based on adaptive knowledge distillation according to claim 1, characterized in that, The physical constraint loss function : ； In the formula, Represents the sequence of physical states of the second teacher model; This represents the hidden layer state vector of the second student model; This represents a fully connected mapping layer used to map the hidden state vectors of the second student model. Mapped to the physical state space.

7. The lightweight runoff forecasting method based on adaptive knowledge distillation according to claim 1, characterized in that, The gated fusion unit receives the predicted flow from the first student model and the predicted flow from the second student model in the dual-student lightweight network model, and generates dynamic fusion coefficients using a lightweight attention mechanism. The predicted flow from the first student model and the predicted flow from the second student model are dynamically weighted and fused according to the dynamic fusion coefficients to obtain the final runoff forecast.

Images