Hydraulic Data Processing Method, Device and Storage Medium Based on Big Data Analysis

Abstract

The present invention relates to the technical field of data processing, and particularly to a water conservancy data processing method, device and storage medium based on big data analysis, including collecting water conservancy data and environmental data, and preprocessing the collected data to eliminate abnormal data; constructing data nodes, and assigning weights to the water conservancy data and environmental data to obtain water conservancy environment parameters, and analyzing water conservancy node characteristics in combination with the data nodes, water conservancy data and water conservancy environment parameters; storing the water conservancy data and environmental data, and analyzing the predicted values of the water conservancy data; analyzing and calculating the data residuals between the predicted values of the water conservancy data and the water conservancy data, and performing coupled analysis on the data residuals and the water conservancy node characteristics to judge the data state and give early warnings; collecting the early warning feedback after the early warning to trigger the feature optimization rules. The present invention realizes the precise monitoring and processing of water conservancy data by integrating multiple parameters in the water conservancy data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a water conservancy data processing method, device and storage medium based on big data analysis. Background Art

[0002] Water conservancy data processing mainly relies on modern information technologies such as big data, cloud computing, the Internet of Things and artificial intelligence. It can realize real-time collection, efficient storage, in-depth analysis and intelligent decision-making of massive water conservancy data, thereby optimizing water resources allocation and improving water conservancy management efficiency and decision-making accuracy.

[0003] Chinese Patent Publication No.: CN113626648A discloses a water conservancy data processing system, method and storage medium, including a water conservancy data acquisition platform, a water conservancy data sharing and exchange platform, a water conservancy data resource directory platform, a water conservancy data unified service platform, a water conservancy data resource operation and management platform, a water conservancy data intelligent platform, and a water conservancy data governance platform, wherein the water conservancy data acquisition platform obtains water conservancy data and aggregates the resource pool through the water conservancy data sharing and exchange platform, the water conservancy data resource directory platform catalogs and manages the resource pool data, and the water conservancy data unified service platform and the water conservancy data governance platform obtain the water conservancy data from the resource pool and obtain algorithms and models from the water conservancy data intelligent platform to process the water conservancy data and then provide the water conservancy data resource operation and management platform with the call. This invention realizes the unified management of data in the water conservancy platform, but does not realize the monitoring of high-sediment-content river data in combination with water conservancy data and environmental data, and has the problems of low efficiency in water conservancy data processing and inaccurate data anomaly monitoring. Summary of the invention

[0004] The object of the present invention is to provide a water conservancy data processing method, device and storage medium based on big data analysis to solve at least one of the problems existing in the prior art.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] Water conservancy data processing methods based on big data analysis include:

[0007] Collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data;

[0008] Construct data nodes and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environmental parameters;

[0009] Store water conservancy data and environmental data, and analyze water conservancy data prediction values;

[0010] Analyze and calculate the predicted value of water conservancy data and the data residuals of water conservancy data, and conduct a coupled analysis of the data residuals and water conservancy node characteristics to judge the data status and give early warnings;

[0011] Collect the warning feedback after the warning to trigger the feature optimization rule.

[0012] Furthermore, weight is assigned to the sediment content, soil displacement, and pump power according to the precipitation. If the precipitation is less than or equal to 8 mm, the expression for the water conservancy environment parameter is set as W(i)=α1×S(i)+α2×D(i)+α3×|ΔP(i) / T|; otherwise, the expression for the water conservancy environment parameter is set as W(i)=(α1 + sdp)×S(i)+(α2 - sdp)×D(i)+α3×|ΔP(i) / T|; where i represents the data node number, α1 represents the sediment content weight, S(i) represents the sediment content, α2 represents the soil displacement weight, D(i) represents the soil displacement, α3 represents the power change weight, ΔP(i) represents the change in pump power between the current analysis period and the previous analysis period, T represents the duration of the preset analysis period, and sdp represents the precipitation correction parameter.

[0013] Furthermore, construct water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environment parameter analysis;

[0014] Conduct a regression analysis on water conservancy data and environmental data to analyze the predicted value of water conservancy data.

[0015] Furthermore, take the difference between the water conservancy data collected in the current analysis period and the predicted value of the water conservancy data in the previous analysis period as the data residual;

[0016] Calculate the standard deviations of the water level residual, sediment content residual, and precipitation residual respectively. If the data residual in the current analysis period is greater than 2 times the standard deviation of its corresponding data, it is determined that the current analysis data is to be determined as abnormal; otherwise, it is determined that the current analysis data is to be determined as normal; and judge the data status according to the number of data that are to be determined as abnormal among the water level, sediment content, and precipitation. If the number of data that are to be determined as abnormal among the water level, sediment content, and precipitation in the current analysis period is greater than or equal to 2, it is determined that the data status in the current analysis period is abnormal and a warning is given; otherwise, it is determined that the data status in the current analysis period is normal.

[0017] Further, covariance analysis is separately performed on the water level residual, sediment concentration residual, and precipitation residual with the water conservancy node characteristics to obtain a covariance matrix. Set the covariance matrix of the water level residual and the water conservancy node characteristics as Cov(RL(i), F(i, θ(i))), set the covariance matrix of the sediment concentration residual and the water conservancy node characteristics as Cov(RS(i), F(i, θ(i))), and set the covariance matrix of the precipitation residual and the water conservancy node characteristics as Cov(RM(i), F(i, θ(i))); where RL(i) represents the water level residual, RS(i) represents the sediment concentration residual, and M(i) represents the precipitation residual.

[0018] Based on the covariance matrix, judge the feature correlation. If Cov(RL(i), F(i, θ(i))) is greater than or equal to the covariance threshold, it is determined that the water level feature correlation is strong; otherwise, it is determined that the water level feature correlation is weak; if Cov(RS(i), F(i, θ(i))) is greater than or equal to the covariance threshold, it is determined that the sediment concentration feature correlation is strong; otherwise, it is determined that the sediment concentration feature correlation is weak; if Cov(RM(i), F(i, θ(i))) is greater than or equal to the covariance threshold, it is determined that the precipitation feature correlation is strong; otherwise, it is determined that the precipitation feature correlation is weak.

[0019] Further, according to the feature correlation, update the judgment process of the data status. When the data status is abnormal, if all the data to be determined as abnormal have strong feature correlation, update the data status to severely abnormal; if only one kind of data among the data to be determined as abnormal has strong feature correlation, update the data status to slightly abnormal; if the number of data to be determined as abnormal is equal to 3 and the number of data with strong feature correlation among them is equal to 2, update the data status to moderately abnormal; when the data status is normal, if there is one kind of data to be determined as abnormal and all the data to be determined as normal have weak feature correlation, update the data status to slightly abnormal.

[0020] Further, based on the feature determination coefficient and the water conservancy environment determination coefficient, update the judgment process of the feature correlation. If the water conservancy environment determination coefficient is greater than or equal to the feature determination coefficient, adjust the covariance threshold to update the judgment process of the feature correlation; otherwise, do not update the judgment process of the feature correlation; the adjustment method of the covariance threshold is to increase the covariance threshold, and the increased amount is [(0.8 - covariance threshold) × (water conservancy environment determination coefficient - feature determination coefficient)].

[0021] Further, if there are three consecutive false alarms in the warning feedback, the feature optimization rule is triggered. The feature optimization rule is that if the water level is abnormally undetermined and the precipitation is normally undetermined, the weight of the pump power is increased by β; if the sediment concentration is abnormally undetermined, the weight of the sediment concentration is decreased by β; if the precipitation is abnormally undetermined and the water level is normally undetermined, the weight of the pump power is decreased by β; and the weight adjustment amount is evenly distributed to the weights of the other two types of data, where β represents a percentage parameter.

[0022] On the other hand, the present invention also provides a water conservancy data processing device based on big data analysis, including:

[0023] An acquisition and processing module for acquiring water conservancy data and environmental data, and preprocessing the acquired data to eliminate abnormal data;

[0024] A construction and analysis module for constructing data nodes, and allocating weights to water conservancy data and environmental data to obtain water conservancy environment parameters, and analyzing water conservancy node characteristics in combination with data nodes, water conservancy data, and water conservancy environment parameters;

[0025] A storage and analysis module for storing water conservancy data and environmental data, and analyzing the predicted value of water conservancy data;

[0026] A state judgment module for analyzing and calculating the data residual between the predicted value of water conservancy data and the water conservancy data, and performing coupled analysis on the data residual and water conservancy node characteristics to judge the data state and give a warning;

[0027] A feedback and optimization module for collecting the warning feedback after the warning to trigger the feature optimization rule.

[0028] On the other hand, the present invention also provides a storage medium storing instructions, which when run on a computer, cause the computer to execute the method described in any one of the above.

[0029] The beneficial effects of the present invention are as follows: Through the acquisition and analysis of water conservancy data and environmental data, the sediment concentration data is fused with dynamic weight allocation to achieve accurate abnormal monitoring of high sediment concentration rivers, and the covariance threshold and weight dynamic adjustment mechanism are used to improve the accuracy of abnormal detection in bad weather, and the hierarchical warning mechanism is used to shorten the emergency response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0031] Figure 1This is a flow chart of the water conservancy data processing method based on big data analysis in this embodiment.

[0032] Figure 2 This is a flow chart of the method for constructing water conservancy node features in this embodiment.

[0033] Figure 3 This is a flow chart of the method for predicting water conservancy data in this embodiment.

[0034] Figure 4 Flow chart of the data status determination method of this embodiment.

[0035] Figure 5 This is a schematic diagram of the structure of the water conservancy data processing device based on big data analysis in this embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0037] It should be understood that in the description of the embodiments of the present invention, the meaning of multiple (or multiple) is more than two, greater than, less than, and exceeding are understood to exclude the number itself, and above, below, and within are understood to include the number itself. If there is a description of "first", "second", etc., it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0038] See also Figure 1 As shown, it is a water conservancy data processing method based on big data analysis in this embodiment, including:

[0039] Step S1, collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data, the water conservancy data includes water level, flow velocity, sand content, gate opening and closing degree and pump power, the environmental data includes precipitation and soil displacement, the precipitation is the precipitation within one hour, the soil displacement is obtained by a Hall sensor array arranged at key points of the river bank to monitor the displacement change of the soil, the water conservancy data and environmental data can be obtained through sensors, the collection frequency of the water conservancy data and environmental data is once per preset analysis period, in this embodiment, the preset analysis period is set to 1 minute, and technical personnel in this field can freely set it, such as it can also be set to 0.5 minutes, 3 minutes and 5 minutes, etc., the timestamps of the water conservancy data and environmental data are the same.

[0040] Specifically, in step S1 of this embodiment, the dynamic interquartile range method is used to preprocess the sediment concentration in the water conservancy data. The upper quartile Q3 and the lower quartile Q1 of the sediment concentration in the previous hour of the current analysis period are calculated, and the standard value range is set as [Q1 - 1.5×(Q3 - Q1), Q3 + 1.5×(Q3 - Q1)]. The data outside the standard value range is marked as an outlier and removed. The calculation methods of the lower quartile and the upper quartile are to sort the sediment concentration within one hour from small to large, and take the data at the one-fourth position as the lower quartile and the data at the three-fourths position as the upper quartile.

[0041] Optionally, in this embodiment, the preprocessing method of the water conservancy data is not specifically limited, and those skilled in the art can freely set it. For example, the Z-score method and the DBSCAN clustering method can also be used to preprocess the water conservancy data, as long as the removal of outliers in the sediment concentration of the water conservancy data is satisfied.

[0042] Specifically, in step S1 of this embodiment, through the preprocessing of the water conservancy data, outliers in the water conservancy data are removed, the misjudgment rate is reduced, and false alarms caused by instantaneous interference are avoided.

[0043] Please continue to refer to Figure 1 As shown, the water conservancy data processing method based on big data analysis further includes:

[0044] Step S2, constructing data nodes, and assigning weights to the water conservancy data and the environmental data to obtain water conservancy environment parameters, and analyzing the water conservancy node characteristics in combination with the data nodes, the water conservancy data, and the water conservancy environment parameters.

[0045] Please refer to Figure 2 As shown, it is a construction method of water conservancy node characteristics, including:

[0046] Step S21, constructing data nodes based on the duration of the preset analysis period.

[0047] Specifically, in step S21 of this embodiment, a data node is set every preset analysis period duration, and the data nodes are numbered in chronological order to obtain node numbers, and the data nodes are used to represent the analysis periods corresponding to different acquisition times of the data. The data nodes correspond one-to-one with the timestamps of the acquired data.

[0048] Specifically, in step S21 of this embodiment, through the setting of the data nodes, time-serialized nodes are used to support time-series analysis, which is convenient for locating abnormal periods.

[0049] Please continue to refer to Figure 2 As shown, the construction method of the water conservancy node characteristics further includes:

[0050] Step S22: Assign weights to the data nodes, water conservancy data, and environmental data to obtain water conservancy environment parameters.

[0051] Specifically, in step S22 of this embodiment, weights are assigned to the sediment concentration, soil displacement, and pump power according to the precipitation. If the precipitation is less than or equal to 8 mm, the expression for the water conservancy environment parameter is set as W(i) = α1×S(i) + α2×D(i) + α3×|ΔP(i) / T|; otherwise, the expression for the water conservancy environment parameter is set as W(i) = (α1 + sdp)×S(i) + (α2 - sdp)×D(i) + α3×|ΔP(i) / T|; where i represents the data node number, α1 represents the sediment concentration weight, S(i) represents the sediment concentration, α2 represents the soil displacement weight, D(i) represents the soil displacement, α3 represents the power change weight, ΔP(i) represents the change in pump power between the current analysis period and the previous analysis period, T represents the duration of the preset analysis period, α1 + α2 + α3 = 1, α1 ≥ α2 > sdp, sdp represents the precipitation correction parameter, and 0.1 ≤ sdp ≤ 0.2.

[0052] Optionally, in this embodiment, the value of the precipitation correction parameter is not specifically limited, and those skilled in the art can freely set it as long as it satisfies the analysis of the water conservancy environment parameters. The optimal value of the precipitation correction parameter is sdp = 0.15; in this embodiment, the initial values of the sediment concentration weight, soil displacement weight, and power change weight are set as α1 = 0.5, α2 = 0.4, and α3 = 0.1.

[0053] Specifically, in step S22 of this embodiment, by dynamically adjusting the weights to accurately reflect the impact of the environment on the water conservancy parameters, a precipitation correction parameter is introduced to solve the problem of insufficient adaptability of the traditional fixed-weight model under extreme weather conditions.

[0054] Please continue to refer to Figure 2 As shown, the method for constructing the water conservancy node features further includes:

[0055] Step S23: Analyze and construct water conservancy node features by combining the data nodes, water conservancy data, and water conservancy environment parameters.

[0056] Specifically, in step S23 of this embodiment, water conservancy node features are analyzed and constructed by combining the data nodes, water conservancy data, and water conservancy environment parameters. The expression for the water conservancy node features is F(i,θ(i)) = L(i)×ln[V(i) + M(i)]×W(i); where θ(i) represents the gate opening degree, L(i) represents the water level, V(i) represents the flow velocity, and M(i) represents the precipitation.

[0057] Specifically, in step S23 of this embodiment, by fusing multi-parameters and dynamic weights to generate highly discriminative features, the logarithmic function is used to compress the flow rate dimension difference to avoid large flow rate values from masking the contributions of other parameters.

[0058] Please continue to refer to Figure 1 As shown, the water conservancy data processing method based on big data analysis further includes:

[0059] Step S3, storing water conservancy data and environmental data, and analyzing the predicted values of water conservancy data.

[0060] Specifically, in step S3 of this embodiment, the correlation between parameters is quantified through regression analysis to support the optimization of the prediction model. The binary regression equation is used to reveal the combined effects of water level and precipitation on sediment concentration, improving the prediction accuracy.

[0061] Please refer to Figure 3 As shown, it is a prediction method for water conservancy data, including:

[0062] Step S31, performing regression analysis on the water conservancy node features to obtain the feature determination coefficient.

[0063] Specifically, in step S31 of this embodiment, the gate opening degree θ(i) is used as the independent variable, and the water conservancy node feature F(i,θ(i)) is used as the dependent variable for regression analysis to analyze the unary regression equation of the water conservancy node feature with respect to the gate opening degree, and calculate the determination coefficient of the unary regression equation as the feature determination coefficient.

[0064] Step S32, performing regression analysis on the water level, sediment concentration, and precipitation to obtain the water conservancy environment determination coefficient and the water conservancy environment regression equation.

[0065] Specifically, in step S32 of this embodiment, the water level L(i) and precipitation M(i) are used as the independent variables, and the sediment concentration S(i) is used as the dependent variable for regression analysis to analyze the binary regression equation of the sediment concentration with respect to the water level and precipitation. The binary regression equation is used as the water conservancy environment regression equation, and the determination coefficient of the water conservancy environment regression equation is calculated as the water conservancy environment determination coefficient.

[0066] Step S33, based on the water level, sediment concentration, precipitation, and water conservancy environment regression equation in the current analysis period, analyze and calculate the predicted values of water conservancy data.

[0067] Specifically, in step S33 of this embodiment, the water level, sediment concentration, and precipitation in the current analysis period are combined pairwise and substituted into the water conservancy environment regression equation, and the calculated non-current substitution data is used as the predicted value of water conservancy data.

[0068] Specifically, in this embodiment, when calculating the predicted value of water conservancy data, the water level, sediment concentration, and precipitation are combined pairwise to obtain three groups of data, which are then substituted into the water conservancy environment regression equation to calculate the predicted value. For example, when substituting the water level and sediment concentration data, the calculated data is the predicted value of precipitation; when substituting the water level and precipitation data, the calculated data is the predicted value of sediment concentration; when substituting the sediment concentration and precipitation data, the calculated data is the predicted value of the water level.

[0069] Optionally, in this embodiment, the prediction method of water conservancy data is not specifically limited, and those skilled in the art can set it freely. For example, it can also be set to train an LSTM model using data from recent years, input the water level, sediment concentration, and precipitation as multi-dimensional features, and use the trained LSTM model to predict water conservancy data to obtain the predicted value of water conservancy data.

[0070] Step S4: Analyze and calculate the data residuals between the predicted value of water conservancy data and the water conservancy data, and conduct a coupling analysis on the data residuals and water conservancy node features to judge the data status and give an early warning.

[0071] Please refer to Figure 4 as shown, which is a method for judging the data status and includes:

[0072] Step S41: Calculate the data residuals between the predicted value of water conservancy data and the water conservancy data. The data residuals include water level residuals, sediment concentration residuals, and precipitation residuals.

[0073] Specifically, in this embodiment, in step S41, the difference between the water conservancy data collected in the current analysis period and the predicted value of water conservancy data in the previous analysis period is used as the data residual.

[0074] Please continue to refer to Figure 4 as shown, the method for judging the data status further includes:

[0075] Step S42: Conduct a covariance analysis on the data residuals and water conservancy node features to obtain a covariance matrix.

[0076] Specifically, in this embodiment, in step S42, a covariance analysis is respectively conducted on the water level residuals, sediment concentration residuals, and precipitation residuals and the water conservancy node features to obtain a covariance matrix. The covariance matrix between the water level residuals and the water conservancy node features is set as Cov(RL(i), F(i, θ(i))), the covariance matrix between the sediment concentration residuals and the water conservancy node features is set as Cov(RS(i), F(i, θ(i))), and the covariance matrix between the precipitation residuals and the water conservancy node features is set as Cov(RM(i), F(i, θ(i))); where RL(i) represents the water level residuals, RS(i) represents the sediment concentration residuals, and M(i) represents the precipitation residuals.

[0077] Specifically, in step S42 of this embodiment, covariance analysis is used to reveal the deep association between residuals and features, and to distinguish accidental noise from real anomalies.

[0078] Please continue to refer to Figure 4 As shown, the data status judgment method further includes:

[0079] Step S43, judging the data status according to the data residuals.

[0080] Specifically, in step S43 of this embodiment, the standard deviations of the water level residuals, sediment concentration residuals, and precipitation residuals are calculated respectively. If the data residuals in the current analysis period are greater than twice the standard deviation of their corresponding data, it is determined that the current analysis data is a pending anomaly; otherwise, it is determined that the current analysis data is pending normal; and the data status is judged according to the number of data that are pending anomalies among the water level, sediment concentration, and precipitation. If the number of data that are pending anomalies among the water level, sediment concentration, and precipitation in the current analysis period is greater than or equal to 2, it is determined that the data status in the current analysis period is abnormal and an alarm is given; otherwise, it is determined that the data status in the current analysis period is normal.

[0081] Please continue to refer to Figure 4 As shown, the data status judgment method further includes:

[0082] Step S44, judging the feature correlation based on the covariance matrix and updating the data status judgment process.

[0083] Specifically, in step S44 of this embodiment, the feature correlation is judged based on the covariance matrix. If Cov(RL(i), F(i, θ(i))) is greater than or equal to the covariance threshold, it is determined that the water level feature correlation is strong; otherwise, it is determined that the water level feature correlation is weak; if Cov(RS(i), F(i, θ(i))) is greater than or equal to the covariance threshold, it is determined that the sediment concentration feature correlation is strong; otherwise, it is determined that the sediment concentration feature correlation is weak; if Cov(RM(i), F(i, θ(i))) is greater than or equal to the covariance threshold, it is determined that the precipitation feature correlation is strong; otherwise, it is determined that the precipitation feature correlation is weak.

[0084] Optionally, the covariance threshold in this embodiment is set to 0.6. In this embodiment, the value of the covariance threshold is not specifically limited, and those skilled in the art can freely set it as long as it satisfies the judgment of the feature correlation. The setting of the covariance threshold should satisfy being greater than or equal to 0.5 and less than or equal to 0.8.

[0085] Specifically, in step S44 of this embodiment, in the judgment process of updating the data status according to the feature relevance, when the data status is abnormal, if all the data to be determined as abnormal have strong feature relevance, the data status is updated to severely abnormal; if only one kind of data among the data to be determined as abnormal has strong feature relevance, the data status is updated to slightly abnormal; if the number of data to be determined as abnormal is equal to 3 and the number of data with strong feature relevance among them is equal to 2, the data status is updated to moderately abnormal; when the data status is normal, if there is one kind of data to be determined as abnormal and all the data to be determined as normal have weak feature relevance, the data status is updated to slightly abnormal.

[0086] Specifically, in step S44 of this embodiment, a multi-level response mechanism is used to match the severity of the abnormality to optimize the resource allocation, and the judgment of the strength of the relevance is combined to improve the accuracy of the early warning and reduce the over-response.

[0087] Please continue to refer to Figure 4 As shown, the data status judgment method further includes:

[0088] Step S45, updating the judgment process of the feature relevance according to the feature decision coefficient and the water conservancy environment decision coefficient.

[0089] Specifically, in step S45 of this embodiment, in the judgment process of updating the feature relevance according to the feature decision coefficient and the water conservancy environment decision coefficient, if the water conservancy environment decision coefficient is greater than or equal to the feature decision coefficient, the covariance threshold is adjusted to update the judgment process of the feature relevance; otherwise, the judgment process of the feature relevance is not updated; the adjustment method of the covariance threshold is to increase the covariance threshold, and the increased amount is [(0.8 - covariance threshold) × (water conservancy environment decision coefficient - feature decision coefficient)].

[0090] Specifically, in step S45 of this embodiment, dynamic threshold optimization is used to reduce the misjudgment caused by environmental interference.

[0091] Step S5, collecting the early warning feedback after the early warning to trigger the feature optimization rule, where the early warning feedback includes valid and false alarms, and the collection method is user interaction input.

[0092] Specifically, in step S5 of this embodiment, if there are three consecutive false alarms in the warning feedback, the feature optimization rule is triggered. The feature optimization rule is that if the water level is abnormally undetermined and the precipitation is normally undetermined, the weight of the pump power is increased by β; if the sediment concentration is abnormally undetermined, the weight of the sediment concentration is decreased by β; if the precipitation is abnormally undetermined and the water level is normally undetermined, the weight of the pump power is decreased by β. And the weight adjustment amount is evenly distributed to the weights of the other two types of data. Here, β represents a percentage parameter, and 5% ≤ β ≤ 15%. When evenly distributing the adjustment amount, if the adjusted weight is increased by β, the weights of the other two types of data are decreased, and the decreased value of each weight is half of the increased amount. If the adjusted weight is decreased by β, the weights of the other two types of data are increased, and the increased value of each weight is half of the decreased amount.

[0093] Optionally, in this embodiment, the value of the percentage parameter is not specifically limited, and those skilled in the art can freely set it as long as the setting of the feature optimization rule is satisfied. The optimal value of the percentage parameter is β = 10%.

[0094] Specifically, in step S5 of this embodiment, through dynamic weight adjustment, the model drift problem is solved, and the feedback closed-loop optimization is used to improve the system adaptability.

[0095] Please refer to Figure 5 As shown, this embodiment also provides a water conservancy data processing device based on big data analysis, including:

[0096] An acquisition and processing module for acquiring water conservancy data and environmental data, and preprocessing the acquired data to eliminate abnormal data;

[0097] A construction and analysis module for constructing data nodes, and assigning weights to water conservancy data and environmental data to obtain water conservancy environment parameters, and analyzing water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environment parameters. The construction and analysis module is connected to the acquisition and processing module;

[0098] A storage and analysis module for storing water conservancy data and environmental data, and analyzing the predicted value of water conservancy data. The storage and analysis module is connected to the construction and analysis module;

[0099] A state judgment module for analyzing and calculating the data residual between the predicted value of water conservancy data and the water conservancy data, and performing coupled analysis on the data residual and water conservancy node characteristics to judge the data state and give warnings. The state judgment module is connected to the storage and analysis module;

[0100] A feedback optimization module for acquiring the warning feedback after warning to trigger the feature optimization rule. The feedback optimization module is connected to the state judgment module.

[0101] The embodiment of the present application also provides a computer-readable storage medium storing instructions, which, when running on a computer, cause the computer to execute the method described in the above method embodiment.

[0102] Those of ordinary skill in the art will appreciate that all or some of the steps and systems disclosed in the above methods may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable programs, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically includes computer-readable programs, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0103] The above is a specific description of the preferred embodiment of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A water conservancy data processing method based on big data analysis, characterized in that: include: Collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data; Construct data nodes and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environmental parameters; The weights of sediment content, soil displacement and pump power are allocated according to the precipitation. If the precipitation is less than or equal to 8 mm, the expression of the water conservancy environment parameter is set as W(i)=α1×S(i)+α2×D(i)+α3×|ΔP(i) / T|; Otherwise, the expression for setting the water conservancy environmental parameters is W(i)=(α1+sdp)×S(i)+(α2-sdp)×D(i)+α3×|ΔP(i) / T|; where i represents the data node number, α1 represents the sediment content weight, S(i) represents the sediment content, α2 represents the soil displacement weight, D(i) represents the soil displacement, α3 represents the power change weight, ΔP(i) represents the change in pump power between the current analysis period and the previous analysis period, T represents the duration of the preset analysis period, and sdp represents the precipitation correction parameter; The expression of the water conservancy node characteristics is F(i,θ(i))=L(i)×ln[V(i)+M(i)]×W(i); wherein θ(i) represents the gate opening degree, L(i) represents the water level, V(i) represents the flow velocity, and M(i) represents the precipitation; Store water conservancy data and environmental data, and analyze water conservancy data prediction values; Analyze and calculate the predicted value of water conservancy data and the data residual of water conservancy data, and conduct coupling analysis on the data residual and water conservancy node characteristics to determine the data status and issue early warnings; Collect warning feedback after warning to trigger feature optimization rules.

2. The water conservancy data processing method based on big data analysis according to claim 1 is characterized in that: Combine data nodes, water conservancy data and water conservancy environmental parameter analysis to construct water conservancy node characteristics; Regression analysis is performed on water conservancy data and environmental data to analyze the predicted value of water conservancy data.

3. The water conservancy data processing method based on big data analysis according to claim 2 is characterized in that: The difference between the water conservancy data collected in the current analysis period and the predicted value of the water conservancy data in the previous analysis period is taken as the data residual; Calculate the standard deviation of water level residual, sediment content residual and precipitation residual respectively. If the data residual of the current analysis period is greater than 2 times the standard deviation of the corresponding data, the current analysis data is judged to be an undetermined anomaly. Otherwise, the current analysis data is judged to be pending normal; and the data status is judged according to the number of data with pending abnormalities in water level, sediment content and precipitation. If the number of data with pending abnormalities in water level, sediment content and precipitation in the current analysis period is greater than or equal to 2, the data status of the current analysis period is judged to be abnormal, and an early warning is issued; Otherwise, the data status of the current analysis cycle is determined to be normal.

4. The water conservancy data processing method based on big data analysis according to claim 3 is characterized in that: Covariance analysis is performed on the water level residual, sediment content residual and precipitation residual with the water conservancy node characteristics to obtain the covariance matrix. The covariance matrix of the water level residual and the water conservancy node characteristics is set to Cov(RL(i), F(i, θ(i))), the covariance matrix of the sediment content residual and the water conservancy node characteristics is set to Cov(RS(i), F(i, θ(i))), and the covariance matrix of the precipitation residual and the water conservancy node characteristics is set to Cov(RM(i), F(i, θ(i))); where RL(i) represents the water level residual, RS(i) represents the sediment content residual, and M(i) represents the precipitation residual; The feature correlation is judged according to the covariance matrix. If Cov(RL(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the water level feature correlation is determined to be strong; otherwise, the water level feature correlation is determined to be weak; if Cov(RS(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the sediment content feature correlation is determined to be strong; otherwise, the sediment content feature correlation is determined to be weak; if Cov(RM(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the precipitation feature correlation is determined to be strong; otherwise, the precipitation feature correlation is determined to be weak.

5. The water conservancy data processing method based on big data analysis according to claim 4 is characterized in that: According to the judgment process of updating the data status based on the feature correlation, when the data status is abnormal, if all the data with pending abnormalities have strong feature correlation, the updated data status is severe abnormality; if only one type of data with strong feature correlation exists among the data with pending abnormalities, the updated data status is slight abnormality; if the number of data with pending abnormalities is equal to 3 and the number of data with strong feature correlation is equal to 2, the updated data status is moderate abnormality; when the data status is normal, if there is one type of data with pending abnormality and all the data with pending normality have weak feature correlation, the updated data status is slight abnormality.

6. The water conservancy data processing method based on big data analysis according to claim 5 is characterized in that: The judgment process of updating the feature correlation is based on the feature determination coefficient and the water environment determination coefficient. If the water environment determination coefficient is greater than or equal to the feature determination coefficient, the covariance threshold is adjusted to update the judgment process of the feature correlation; otherwise, the judgment process of the feature correlation is not updated; the adjustment method of the covariance threshold is to increase the covariance threshold, and the increase amount is (0.8-covariance threshold)×(water environment determination coefficient-feature determination coefficient).

7. The water conservancy data processing method based on big data analysis according to claim 6 is characterized in that: If there are three consecutive false alarms, the feature optimization rule is triggered. The feature optimization rule is that if the water level is a pending abnormality and the precipitation is a pending normal, the pump power weight is increased by β; if the sediment content is a pending abnormality, the sediment content weight is reduced by β; if the precipitation is a pending abnormality and the water level is a pending normal, the pump power weight is reduced by β; and the weight adjustment amount is evenly divided into the weights of the other two data, where β represents a percentage parameter.

8. A water conservancy data processing device based on big data analysis, applied to the method according to any one of claims 1 to 7, comprising: The collection and processing module is used to collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data; Construct an analysis module to construct data nodes and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environmental parameters; Storage and analysis module, used to store water conservancy data and environmental data, and analyze the predicted value of water conservancy data; The state judgment module is used to analyze and calculate the predicted value of water conservancy data and the data residual of water conservancy data, and conduct coupled analysis on the data residual and water conservancy node characteristics to judge the data state and issue early warnings; The feedback optimization module is used to collect warning feedback after warning to trigger feature optimization rules.

9. A storage medium, characterized in that: The device stores instructions which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 7.

Images