Data processing method and device for construction of hydrological runoff forecasting model, and storage medium

Abstract

The invention discloses a data processing method and device for constructing a hydrological runoff forecasting model and a storage medium, and the method comprises the steps: obtaining a heterogeneous hydrological data source for constructing the hydrological runoff forecasting model, carrying out the preprocessing of the heterogeneous hydrological data source, obtaining a plurality of different types of time series data, and carrying out the calculation of each type of time series data, identifying a data mode of the time series data by adopting adaptive LSTM and CNN models, generating a prediction interpolation according to the identified data mode and performing interpolation operation, and performing abnormal value detection and data cleaning on each type of time series data after interpolation by adopting an adaptive abnormal value detection strategy; the data interpolation and the data cleaning can adapt corresponding models and strategies based on different data types, so that complex and changeable hydrological data characteristics can be met, the integrity and the accuracy of the data for constructing the hydrological runoff forecasting model are ensured, and the precision of hydrological runoff forecasting is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method, device and storage medium for constructing a hydrological runoff forecast model. Background Art

[0002] In the field of hydrological monitoring and forecasting, the integrity and accuracy of data are crucial to the performance of the model. However, factors such as equipment failure and natural environmental conditions that cause missing values ​​in hydrological monitoring data will lead to missing values ​​in hydrological monitoring data. If these missing values ​​are not properly handled, they will directly affect the integrity of the model input data and thus affect the model's prediction results. In addition, the outliers that may be contained in the actual monitoring data will interfere with the training and prediction of the model and reduce the accuracy of the model. At the same time, the complex spatiotemporal correlation and seasonal fluctuations of hydrological data also bring challenges to data processing.

[0003] Existing solutions are not adequate in dealing with missing data and outliers. Missing data results in incomplete model input information, and outliers interfere with normal model learning. The two together lead to poor quality of model input data, which ultimately results in insufficient model accuracy and inability to provide accurate and reliable predictions for practical applications. In addition, traditional data interpolation and cleaning methods are mostly based on linear assumptions, which make it difficult to effectively deal with the nonlinear and non-stationary characteristics of hydrological time series data. Since hydrological data are affected by a variety of complex factors and show nonlinear and non-stationary changes, traditional methods cannot accurately capture their characteristics and trends, causing interpolation and cleaning results to deviate from the actual situation, seriously affecting the quality of model input data, thereby reducing prediction accuracy, and failing to meet the requirements of hydrological runoff forecasting for data processing. It is impossible to fully mine the effective information of the data, which limits the model's ability to capture complex patterns, thereby affecting prediction accuracy. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data processing method, device and storage medium for constructing a hydrological runoff forecast model, which can improve the accuracy of hydrological runoff prediction.

[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is: A data processing method for constructing a hydrological runoff forecast model comprises the following steps: Acquire a heterogeneous hydrological data source for constructing a hydrological runoff forecast model, and preprocess the heterogeneous hydrological data source to obtain a plurality of different types of time series data; For each type of time series data, an adapted LSTM and CNN model is used to identify the data pattern of the time series data, and a prediction interpolation is generated according to the identified data pattern and an interpolation operation is performed; For each type of time series data after interpolation, an adapted outlier detection strategy is used to detect outliers and perform data cleaning.

[0006] Furthermore, before performing data pattern recognition, the following steps are also included: Train LSTM and CNN models adapted to each type of time series data.

[0007] Furthermore, the training of LSTM and CNN models adapted to each type of time series data includes: Identify the data types of time series data used for training LSTM and CNN models; Adjust the parameters of the LSTM layer and the CNN layer in the LSTM and CNN models respectively according to the data type; The LSTM and CNN models are trained according to the adjusted parameters.

[0008] Further, the parameters of the LSTM layer include one or more of the number of hidden layers, the number of neurons, and the length of the time window; The parameters of the CNN layer include convolution kernel size and stride.

[0009] Furthermore, the adopting of the adapted LSTM and CNN models to identify the data pattern of the time series data includes: Adopting the LSTM layer in the adapted LSTM and CNN models to capture the long-term dependencies in the time series data; Extracting spatial patterns in the time series data using CNN layers in the adapted LSTM and CNN models; The corresponding data pattern is determined according to the long-term dependency and spatial pattern in the time series data.

[0010] Furthermore, the adopting an adapted outlier detection strategy to perform outlier detection on each type of interpolated time series data includes: Determine the data type corresponding to each type of time series data after interpolation, and determine its data distribution characteristics according to the data type; An adapted outlier detection strategy is used to perform outlier detection according to the data distribution characteristics.

[0011] Furthermore, the adopting an adapted outlier detection strategy to perform outlier detection according to the data distribution characteristics includes: Determining whether the interpolated time series data has seasonal fluctuations or long-term trend influences according to the data distribution characteristics; If yes, seasonal decomposition or trend fitting is performed on the interpolated time series data; Generate residuals or detrended series after seasonal decomposition or trend fitting; Solve for the mean and standard deviation of the residual or detrended series; Outlier detection is performed based on the mean and standard deviation.

[0012] Furthermore, the data cleaning includes: Determine the number and impact of outlier data; If the number of the abnormal value data is less than the first preset value, and the influence thereof is less than the second preset value, the abnormal value data is directly deleted; Otherwise, determining the distribution characteristics of the outlier data; If the outlier data described are univariate outliers, the mean, median, or mode will be used instead; If the outlier data has temporal or spatial continuity, linear interpolation or spline interpolation is used for replacement; If the outlier data is complex nonlinear data, it is replaced based on machine learning predictions.

[0013] In order to solve the above technical problems, another technical solution adopted by the present invention is: A data processing device for a hydrological runoff forecast model comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data processing method for the hydrological runoff forecast model are implemented.

[0014] A computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the steps of the data processing method for a hydrological runoff forecasting model.

[0015] The beneficial effects of the present invention are as follows: when performing interpolation and cleaning operations on heterogeneous hydrological data, different types of data are first determined, and then for each type of time series data, an adapted LSTM and CNN model is used to identify the data pattern of the time series data, and a prediction interpolation is generated according to the identified data pattern and an interpolation operation is performed; for each type of time series data after interpolation, an adapted outlier detection strategy is used to perform outlier detection and data cleaning; both data interpolation and data cleaning can adapt corresponding models and strategies based on different data types, can meet the complex and changeable characteristics of hydrological data, ensure the integrity and accuracy of data for constructing a hydrological runoff forecast model, avoid information loss and bias introduction, fully consider the complexity of hydrological data, effectively mine data information, can improve the accuracy of hydrological runoff prediction, and provide more reliable support for hydrological runoff forecasting. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flow chart of the steps of a data processing method for constructing a hydrological runoff forecast model according to an embodiment of the present invention; Figure 2 A data processing flow of a data processing method for constructing a hydrological runoff forecast model according to an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a data processing device for constructing a hydrological runoff forecast model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0018] The data processing method, device and computer-readable storage medium for constructing the hydrological runoff forecast model in the present application can be applied to data processing for constructing a hydrological runoff forecast model in various hydrological runoff forecasts, and the following is an explanation through specific implementation methods: like Figure 1 As shown, a data processing method for constructing a hydrological runoff forecast model comprises the following steps: S1. Acquire a heterogeneous hydrological data source for constructing a hydrological runoff forecast model, and preprocess the heterogeneous hydrological data source to obtain a plurality of different types of time series data; S2. For each type of time series data, use the adapted LSTM and CNN models to identify the data pattern of the time series data, generate a prediction interpolation according to the identified data pattern, and perform an interpolation operation; S3. Adopt an adapted outlier detection strategy to detect outliers and perform data cleaning on each type of time series data after interpolation.

[0019] in, Figure 2 The following is a specific data flow diagram, which includes the numerical interpolation process: [Start] -> Data input -> Data missing detection -> Feature extraction -> Model training (LSTM / CNN) -> Interpolation prediction -> [End]; And the data cleaning process: [Start] -> Outlier detection -> Outlier processing strategy selection -> Execute cleaning -> Verify cleaning results -> [End]; Data import is to integrate heterogeneous hydrological data sources (such as historical reservoir water levels, watershed characteristics, soil moisture, and reservoir inflow and outflow) into a central database using data integration technology. This process involves data synchronization, format conversion, and cleaning to ensure data consistency and accessibility. Feature extraction uses statistical analysis techniques, such as skewness and kurtosis measurement, to identify the characteristics of data distribution, extract data from heterogeneous hydrological data sources (such as historical reservoir water levels, watershed characteristics, soil moisture, and reservoir flow), and perform cleaning and format conversion to obtain various types of time series data to ensure data consistency and accessibility; In another optional embodiment, the method further includes the following steps before performing data pattern recognition: Train LSTM and CNN models adapted to each type of time series data.

[0020] The training of LSTM and CNN models adapted to each type of time series data includes: Identify the data types of time series data used for training LSTM and CNN models; Adjust the parameters of the LSTM layer and the CNN layer in the LSTM and CNN models respectively according to the data type; Training is performed according to the LSTM and CNN models after adjusting parameters; Specifically, the parameters of the LSTM layer include one or more of the number of hidden layers, the number of neurons, and the length of the time window; The parameters of the CNN layer include the convolution kernel size and stride.

[0021] In the specific implementation, during the model training phase, long short-term memory networks (LSTM) and convolutional neural networks (CNN) are used to learn complex patterns in time series data. Traditional LSTM and CNN model training methods do not fully consider the characteristics of hydrological data, resulting in limited ability of the model to capture complex patterns in hydrological data. In this implementation, the model structure is optimized based on the characteristics of hydrological data, and the parameters of the LSTM and CNN network layers are adjusted.

[0022] For example, for the LSTM layer, in order to optimize the number of hidden layers and neurons of the LSTM, adaptive adjustments need to be made according to the length and periodicity of the hydrological time series: For long time series data (such as multi-year precipitation), the number of hidden layers (3-4 layers) and the number of neurons (64-128) can be increased to capture long-term dependencies; For short time series (such as single-year or quarterly data), using 1-2 layers can avoid overfitting; If the data is periodic (such as flood season and dry season), adjust the time window (such as quarter or year) and increase the number of neurons to adapt to changes in different cycles; For emergencies, the number of neurons is increased to improve the adaptability to abnormal changes, thereby improving the model's ability to capture hydrological data and prediction accuracy. It is more suitable for capturing long-term dependencies in hydrological data. For the CNN layer, adjust the convolution kernel size and step size according to the spatial distribution characteristics of the hydrological data (such as watershed area, river direction, etc.): When the watershed is wide and the data resolution is low, large convolution kernels (7x7, 9x9) and large step sizes (3, 4) can expand the perception range of the convolutional neural network, accurately extract macro features, help predict runoff in large watersheds, and avoid local interference and overall trend misjudgment caused by small convolution kernels; For complex river network areas, small convolution kernels (3x3, 5x5) and small step sizes (1, 2) are good at capturing detailed associations. If a large size is used, key information may be missed, resulting in runoff simulation deviations. In the high mountain valley area, the complex terrain causes variable runoff. Using a small 3x3 convolution kernel and a step size of 1 on the steep slope can accurately capture the details of the flow velocity and direction caused by the sudden change of terrain, and grasp the key information for runoff prediction. The open valley flat principle uses a large 7x7 convolution kernel and a step size of 3, which can efficiently extract the macroscopic characteristics of water flow convergence and diffusion.

[0023] In this implementation, the convolution kernel and step size are adaptively adjusted according to the terrain, the spatial pattern of hydrological data is fully explored, the accuracy of hydrological simulation and runoff prediction in complex terrain is enhanced, so as to better extract the spatial pattern in the data and thus improve the accuracy of the prediction.

[0024] In another optional embodiment, the adopting of the adapted LSTM and CNN models to identify the data pattern of the time series data includes: Adopting the LSTM layer in the adapted LSTM and CNN models to capture the long-term dependencies in the time series data; Extracting spatial patterns in the time series data using CNN layers in the adapted LSTM and CNN models; The corresponding data pattern is determined according to the long-term dependency and spatial pattern in the time series data.

[0025] In this implementation, in the interpolation prediction link, pre-trained LSTM and CNN models are used to perform data interpolation to fill in missing data points. LSTM captures the long-term dependencies of time series with its gating mechanism, and CNN extracts spatial patterns with the help of convolutional layers, etc. Based on the learned data patterns, the two comprehensively consider various factors such as relevant data and spatial features in the time series to generate prediction interpolations, thereby ensuring the integrity of the time series data, providing complete and continuous data for the subsequent hydrological runoff forecast model, and improving the accuracy and reliability of the model prediction.

[0026] In another optional implementation, the step of performing outlier detection on each type of interpolated time series data using an adapted outlier detection strategy includes: Determine the data type corresponding to each type of time series data after interpolation, and determine its data distribution characteristics according to the data type; An adapted outlier detection strategy is used to perform outlier detection according to the data distribution characteristics.

[0027] Wherein, the adopting an adapted outlier detection strategy to perform outlier detection according to the data distribution characteristics includes: Determining whether the interpolated time series data has seasonal fluctuations or long-term trend influences according to the data distribution characteristics; If yes, seasonal decomposition or trend fitting is performed on the interpolated time series data; Generate residuals or detrended series after seasonal decomposition or trend fitting; Solve for the mean and standard deviation of the residual or detrended series; Outlier detection is performed based on the mean and standard deviation.

[0028] In specific implementation, statistical methods such as Z-Score are used to identify outliers in the data. Z-Score represents the number of standard deviations between a data point and the mean, and is an effective tool for identifying outliers that deviate from the normal data range.

[0029] In the field of hydrological monitoring data processing, the use of the Z-Score method to calculate outliers requires multiple adaptive adjustments. For data distribution characteristics, its skewness and multimodality should be considered, and it can be processed by data transformation, cluster analysis and other means before calculation; for time series characteristics, seasonal fluctuations and long-term trend effects need to be processed, such as calculating the Z-Score of the residual or detrended series after seasonal decomposition and trend fitting.

[0030] In one implementation, seasonal decomposition in hydrological data processing first uses a moving average method to smooth the data: Taking monthly hydrological data as an example, a 12-month window is selected to calculate the moving average sequence, so as to weaken the interference of short-term random fluctuations on data stability. This operation can effectively weaken its influence, preliminarily build a long-term trend framework of the data, and provide a basis for subsequent decomposition; then, the sequence obtained by subtracting the moving average from the original data is grouped according to the seasonal cycle, and the mean of each group is calculated to extract the seasonal component; As for monthly precipitation data, the data of the same month of each year are aggregated and averaged, and the seasonal factors obtained can accurately reflect the fluctuation characteristics and patterns of the monthly relative long-term mean; then the determined seasonal components are removed from the original data, and linear or polynomial fitting techniques are used to explore potential trends: for linear gradient data, linear equation fitting is used; for complex nonlinear data, high-order polynomial fitting is used; with the help of least squares method to optimize parameters, the fitting curve is made to closely fit the data trend line, so as to achieve accurate quantification of the trend direction, analyze the seasonal and trend elements of hydrological data, and support analysis and accurate prediction.

[0031] Generate residuals or detrended series as a follow-up step: The precisely fitted trend and seasonal components are subtracted from the original data in sequence to construct a residual sequence. After determining its mean and standard deviation, the Z-Score of each point is calculated. This value accurately measures the degree to which the data deviates from the residual mean, effectively identifies outliers, and helps analyze the local fluctuation characteristics and random interference conditions of the data, thereby ensuring the reliability of data quality.

[0032] The detrended sequence is formed by subtracting the fitted trend value from the original data, and its mean and standard deviation are solved. Based on this, the Z-Score of each data point is calculated. This indicator accurately captures the abnormal fluctuation of the data out of the trend line and accurately evaluates the trend stability and the degree of abnormal deviation, so that this method can more accurately identify outliers in hydrological monitoring data.

[0033] In another optional implementation, the data cleaning includes: Determine the number and impact of outlier data; If the number of the abnormal value data is less than the first preset value, and the influence thereof is less than the second preset value, the abnormal value data is directly deleted; Otherwise, determining the distribution characteristics of the outlier data; If the outlier data described are univariate outliers, the mean, median, or mode will be used instead; If the outlier data has temporal or spatial continuity, linear interpolation or spline interpolation is used for replacement; If the outlier data is complex nonlinear data, it is replaced based on machine learning predictions.

[0034] In this implementation, after an outlier is detected, a professional data cleaning strategy is adopted. When cleaning an outlier, it is necessary to comprehensively consider the outlier characteristics and data background: If the number of outliers is small and their impact is limited, they can be deleted directly; For univariate anomalies, the mean, median, or mode can be used instead; When the data has temporal or spatial continuity, linear interpolation, spline interpolation and other methods are used; For complex nonlinear data, prediction replacement based on machine learning is more suitable. For example, using LSTM and CNN to correct outliers can significantly improve data integrity. In addition, statistical thresholds (such as Z-Score or Pauta criterion) are combined to adjust outliers to a reasonable range. Specific strategies need to be selected based on data characteristics and business needs to ensure that cleaning does not introduce new biases or information loss.

[0035] The cleaned data is output and used to build a hydrological runoff forecast model. The output data needs to meet the input requirements of the model, including format, resolution and time synchronization, such as CSV or a specific database table structure, and the data field names and units must be consistent. Next, adjust the time and spatial resolution of the data to ensure that it matches the requirements of the model. The time resolution may need to be processed by aggregation or interpolation, and the spatial resolution can be generated by interpolation methods to generate the required grid data. Time synchronization is also important. It is necessary to align timestamps to ensure that data from different sources have the same time reference and smooth possible mutation data to avoid interference with the model. At the same time, check the integrity of the data to ensure that there are no missing values ​​or outliers. If there are, further clean and supplement them. In addition, normalization or standardization of the data is also very critical to ensure that it does not cause problems due to inconsistent numerical ranges when processed by the model.

[0036] In another optional embodiment, Figure 3 As shown, a data processing device for a hydrological runoff forecast model includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a data processing method for a hydrological runoff forecast model described in any one of the above-mentioned embodiments are implemented.

[0037] In another optional embodiment, a computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the steps of a data processing method for a hydrological runoff forecasting model described in any of the above embodiments.

[0038] In summary, the data processing method, device and computer-readable storage medium for constructing a hydrological runoff forecast model provided by the present invention can accurately fill in missing values ​​in terms of data integrity and accuracy. It can capture long-term dependencies and extract spatial patterns using LSTM based on the characteristics of hydrological data and CNN to provide a more complete data set for the model; at the same time, it combines statistical and machine learning cleaning techniques to accurately detect and effectively process outliers, comprehensively considers multiple factors to select appropriate strategies, and avoids information loss and bias introduction. In terms of model performance and generalization ability, the parameters of the LSTM and CNN network layers are optimized, and the parameters are adjusted according to the spatiotemporal characteristics of hydrological data, so that the model can accurately learn complex patterns and improve prediction accuracy; the automated preprocessing process reduces manual intervention, efficiently processes data, adapts to different data sets and environments, and enhances the generalization ability of the model. Overall, the present invention provides a comprehensive data processing solution, optimizes each link from data import to output, integrates heterogeneous data sources and ensures that the data meets the model requirements, and fully considers the complexity of hydrological data, effectively mines data information, and provides more reliable support for hydrological runoff forecasting.

[0039] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data processing method for constructing a hydrological runoff forecast model, characterized in that: Includes steps: Acquire a heterogeneous hydrological data source for constructing a hydrological runoff forecast model, and preprocess the heterogeneous hydrological data source to obtain a plurality of different types of time series data; For each type of time series data, an adapted LSTM and CNN model is used to identify the data pattern of the time series data, and a prediction interpolation is generated according to the identified data pattern and an interpolation operation is performed; For each type of time series data after interpolation, an adapted outlier detection strategy is used to detect outliers and perform data cleaning.

2. The data processing method for constructing a hydrological runoff forecast model according to claim 1, characterized in that: Before data pattern recognition, the following steps are also included: Train LSTM and CNN models adapted to each type of time series data.

3. The data processing method for constructing a hydrological runoff forecast model according to claim 2, characterized in that: The training of LSTM and CNN models adapted to each type of time series data includes: Identify the data types of time series data used for training LSTM and CNN models; Adjust the parameters of the LSTM layer and the CNN layer in the LSTM and CNN models respectively according to the data type; The LSTM and CNN models are trained according to the adjusted parameters.

4. The data processing method for constructing a hydrological runoff forecast model according to claim 3 is characterized in that: The parameters of the LSTM layer include one or more of the number of hidden layers, the number of neurons, and the length of the time window; The parameters of the CNN layer include the convolution kernel size and stride.

5. A data processing method for constructing a hydrological runoff forecast model according to any one of claims 1 to 4, characterized in that: The method of using the adapted LSTM and CNN models to identify the data pattern of the time series data includes: Adopting the LSTM layer in the adapted LSTM and CNN models to capture the long-term dependencies in the time series data; Extracting spatial patterns in the time series data using CNN layers in the adapted LSTM and CNN models; The corresponding data pattern is determined according to the long-term dependency and spatial pattern in the time series data.

6. A data processing method for constructing a hydrological runoff forecast model according to any one of claims 1 to 4, characterized in that: The method of using an adapted outlier detection strategy to perform outlier detection on each type of time series data after interpolation includes: Determine the data type corresponding to each type of time series data after interpolation, and determine its data distribution characteristics according to the data type; An adapted outlier detection strategy is used to perform outlier detection according to the data distribution characteristics.

7. The data processing method for constructing a hydrological runoff forecast model according to claim 6, characterized in that: The adopting an adapted outlier detection strategy to perform outlier detection according to the data distribution characteristics comprises: Determining whether the interpolated time series data has seasonal fluctuations or long-term trend influences according to the data distribution characteristics; If yes, seasonal decomposition or trend fitting is performed on the interpolated time series data; Generate residuals or detrended series after seasonal decomposition or trend fitting; Solve for the mean and standard deviation of the residual or detrended series; Outlier detection is performed based on the mean and standard deviation.

8. A data processing method for constructing a hydrological runoff forecast model according to any one of claims 1 to 4, characterized in that: The data cleaning comprises: Determine the number and impact of outlier data; If the number of the abnormal value data is less than the first preset value, and the influence thereof is less than the second preset value, the abnormal value data is directly deleted; Otherwise, determining the distribution characteristics of the outlier data; If the outlier data described are univariate outliers, the mean, median, or mode will be used instead; If the outlier data has temporal or spatial continuity, linear interpolation or spline interpolation is used for replacement; If the outlier data is complex nonlinear data, it is replaced based on machine learning predictions.

9. A data processing device for a hydrological runoff forecast model, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data processing method of a hydrological runoff forecast model as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of a data processing method for a hydrological runoff forecast model described in any one of claims 1 to 8 are implemented.

Images