Water conservancy data processing method and device based on big data analysis and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a water conservancy data processing method and device based on big data analysis and a storage medium, and the method comprises the steps: collecting water conservancy data and environment data, and carrying out the preprocessing of the collected data, so as to remove abnormal data; constructing data nodes, performing weight distribution on the water conservancy data and the environment data to obtain water conservancy environment parameters, and analyzing water conservancy node features in combination with the data nodes, the water conservancy data and the water conservancy environment parameters; storing the water conservancy data and the environmental data, and analyzing a water conservancy data predicted value; analyzing and calculating a data residual error of the water conservancy data predicted value and the water conservancy data, and carrying out coupling analysis on the data residual error and the water conservancy node characteristics to judge a data state and carry out early warning; and collecting early warning feedback after early warning so as to trigger the feature optimization rule. According to the invention, accurate monitoring processing on the water conservancy data by integrating multiple parameters in the water conservancy data is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a water conservancy data processing method, device and storage medium based on big data analysis. Background Art

[0002] Water conservancy data processing mainly relies on modern information technologies such as big data, cloud computing, the Internet of Things and artificial intelligence. It can realize real-time collection, efficient storage, in-depth analysis and intelligent decision-making of massive water conservancy data, thereby optimizing water resources allocation and improving water conservancy management efficiency and decision-making accuracy.

[0003] Chinese Patent Publication No.: CN113626648A discloses a water conservancy data processing system, method and storage medium, including a water conservancy data acquisition platform, a water conservancy data sharing and exchange platform, a water conservancy data resource directory platform, a water conservancy data unified service platform, a water conservancy data resource operation and management platform, a water conservancy data intelligent platform, and a water conservancy data governance platform, wherein the water conservancy data acquisition platform obtains water conservancy data and aggregates the resource pool through the water conservancy data sharing and exchange platform, the water conservancy data resource directory platform catalogs and manages the resource pool data, and the water conservancy data unified service platform and the water conservancy data governance platform obtain the water conservancy data from the resource pool and obtain algorithms and models from the water conservancy data intelligent platform to process the water conservancy data and then provide the water conservancy data resource operation and management platform with the call. This invention realizes the unified management of data in the water conservancy platform, but does not realize the monitoring of high-sediment-content river data in combination with water conservancy data and environmental data, and has the problems of low efficiency in water conservancy data processing and inaccurate data anomaly monitoring. Summary of the invention

[0004] The object of the present invention is to provide a water conservancy data processing method, device and storage medium based on big data analysis to solve at least one of the problems existing in the prior art.

[0005] To achieve the above object, the present invention adopts the following technical solutions: Water conservancy data processing methods based on big data analysis include: Collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data; Construct data nodes and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environmental parameters; Store water conservancy data and environmental data, and analyze water conservancy data prediction values; Analyze and calculate the predicted value of water conservancy data and the data residual of water conservancy data, and conduct coupling analysis on the data residual and water conservancy node characteristics to determine the data status and issue early warnings; Collect warning feedback after warning to trigger feature optimization rules.

[0006] Furthermore, weights are allocated to sediment content, soil displacement and pump power according to precipitation. If the precipitation is less than or equal to 8 mm, the expression of the water conservancy environment parameter is set to W(i)=α1×S(i)+α2×D(i)+α3×|ΔP(i) / T|; otherwise, the expression of the water conservancy environment parameter is set to W(i)=(α1+sdp)×S(i)+(α2-sdp)×D(i)+α3×|ΔP(i) / T|; where i represents the data node number, α1 represents the sediment content weight, S(i) represents the sediment content, α2 represents the soil displacement weight, D(i) represents the soil displacement, α3 represents the power change weight, ΔP(i) represents the change in pump power between the current analysis period and the previous analysis period, T represents the duration of the preset analysis period, and sdp represents the precipitation correction parameter.

[0007] Furthermore, the water conservancy node characteristics are constructed by combining data nodes, water conservancy data and water conservancy environmental parameter analysis; Regression analysis is performed on water conservancy data and environmental data to analyze the predicted value of water conservancy data.

[0008] Furthermore, the difference between the water conservancy data collected in the current analysis period and the predicted value of the water conservancy data in the previous analysis period is taken as the data residual; The standard deviations of water level residuals, sediment content residuals and precipitation residuals are calculated respectively. If the data residuals of the current analysis period are greater than 2 times the standard deviation of the corresponding data, the current analysis data is judged to be pending abnormalities; otherwise, the current analysis data is judged to be pending normal; and the data status is judged according to the number of data with pending abnormalities in water level, sediment content and precipitation. If the number of data with pending abnormalities in water level, sediment content and precipitation in the current analysis period is greater than or equal to 2, the data status of the current analysis period is judged to be abnormal, and an early warning is issued; otherwise, the data status of the current analysis period is judged to be normal.

[0009] Furthermore, covariance analysis is performed on the water level residual, sediment content residual and precipitation residual with the water conservancy node characteristics to obtain the covariance matrix, and the covariance matrix of the water level residual and the water conservancy node characteristics is set to Cov(RL(i), F(i, θ(i))), the covariance matrix of the sediment content residual and the water conservancy node characteristics is set to Cov(RS(i), F(i, θ(i))), and the covariance matrix of the precipitation residual and the water conservancy node characteristics is set to Cov(RM(i), F(i, θ(i))); where RL(i) represents the water level residual, RS(i) represents the sediment content residual, and M(i) represents the precipitation residual; The feature correlation is judged according to the covariance matrix. If Cov(RL(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the water level feature correlation is determined to be strong; otherwise, the water level feature correlation is determined to be weak; if Cov(RS(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the sediment content feature correlation is determined to be strong; otherwise, the sediment content feature correlation is determined to be weak; if Cov(RM(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the precipitation feature correlation is determined to be strong; otherwise, the precipitation feature correlation is determined to be weak.

[0010] Furthermore, according to the judgment process of updating the data status based on the feature correlation, when the data status is abnormal, if all the data with pending abnormalities have strong feature correlation, the updated data status is severely abnormal; if only one type of data with strong feature correlation exists among the data with pending abnormalities, the updated data status is slightly abnormal; if the number of data with pending abnormalities is equal to 3 and the number of data with strong feature correlation is equal to 2, the updated data status is moderately abnormal; when the data status is normal, if there is one type of data with pending abnormalities and all the data with pending normalities have weak feature correlation, the updated data status is slightly abnormal.

[0011] Furthermore, the judgment process of the feature correlation is updated according to the feature determination coefficient and the water environment determination coefficient. If the water environment determination coefficient is greater than or equal to the feature determination coefficient, the covariance threshold is adjusted to update the judgment process of the feature correlation; otherwise, the judgment process of the feature correlation is not updated; the adjustment method of the covariance threshold is to increase the covariance threshold, and the increase amount is [(0.8-covariance threshold)×(water environment determination coefficient-feature determination coefficient)].

[0012] Furthermore, if there are three consecutive false alarms in the early warning feedback, the feature optimization rule is triggered. The feature optimization rule is that if the water level is a pending abnormality and the precipitation is a pending normal, the pump power weight is increased by β; if the sediment content is a pending abnormality, the sediment content weight is reduced by β; if the precipitation is a pending abnormality and the water level is a pending normal, the pump power weight is reduced by β; and the weight adjustment amount is evenly divided into the weights of the other two data, where β represents a percentage parameter.

[0013] On the other hand, the present invention also provides a water conservancy data processing device based on big data analysis, comprising: The collection and processing module is used to collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data; Construct an analysis module to construct data nodes and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environmental parameters; Storage and analysis module, used to store water conservancy data and environmental data, and analyze the predicted value of water conservancy data; The state judgment module is used to analyze and calculate the predicted value of water conservancy data and the data residual of water conservancy data, and conduct coupled analysis on the data residual and water conservancy node characteristics to judge the data state and issue early warnings; The feedback optimization module is used to collect warning feedback after warning to trigger feature optimization rules.

[0014] On the other hand, the present invention further provides a storage medium storing instructions, which, when executed on a computer, enables the computer to execute any of the above methods.

[0015] The beneficial effects of the present invention are as follows: by collecting and analyzing water conservancy data and environmental data, integrating sediment content data with dynamic weight allocation, accurate anomaly monitoring of rivers with high sediment content can be achieved, and a covariance threshold and weight dynamic adjustment mechanism is used to improve the accuracy of anomaly detection in severe weather, and a graded early warning mechanism is used to shorten emergency response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a flow chart of the water conservancy data processing method based on big data analysis in this embodiment.

[0018] Figure 2 This is a flow chart of the method for constructing water conservancy node features in this embodiment.

[0019] Figure 3 This is a flow chart of the water conservancy data prediction method of this embodiment.

[0020] Figure 4 Flow chart of the data status determination method of this embodiment.

[0021] Figure 5 This is a schematic diagram of the structure of the water conservancy data processing device based on big data analysis in this embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] It should be understood that in the description of the embodiments of the present invention, the meaning of multiple (or multiple) is more than two, greater than, less than, and exceeding are understood to exclude the number itself, and above, below, and within are understood to include the number itself. If there is a description of "first", "second", etc., it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0024] See also Figure 1 As shown, it is a water conservancy data processing method based on big data analysis in this embodiment, including: Step S1, collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data, the water conservancy data includes water level, flow velocity, sand content, gate opening and closing degree and pump power, the environmental data includes precipitation and soil displacement, the precipitation is the precipitation within one hour, the soil displacement is obtained by a Hall sensor array arranged at key points of the river bank to monitor the displacement change of the soil, the water conservancy data and environmental data can be obtained through sensors, the collection frequency of the water conservancy data and environmental data is once per preset analysis period, in this embodiment, the preset analysis period is set to 1 minute, and technical personnel in this field can freely set it, such as it can also be set to 0.5 minutes, 3 minutes and 5 minutes, etc., the timestamps of the water conservancy data and environmental data are the same.

[0025] Specifically, in step S1 described in this embodiment, the dynamic interquartile range method is used to pre-process the sediment content in the water conservancy data, and the third quartile Q3 and the first quartile Q1 of the sediment content within the hour before the current analysis period are calculated, and the standard value range is set to [Q1-1.5×(Q3-Q1), Q3+1.5×(Q3-Q1)], and the data that are not within the standard value range are marked as outliers and removed. The calculation method of the first quartile and the third quartile is to sort the sediment content within one hour from small to large, and use the data in the first quarter as the first quartile, and the data in the fourth third as the third quartile.

[0026] Optionally, the preprocessing method of water conservancy data is not specifically limited in this embodiment, and technical personnel in this field can freely set it. For example, the Z score method and DBSCAN clustering method can also be used to preprocess the water conservancy data, as long as the outliers in the sediment content of the water conservancy data are eliminated.

[0027] Specifically, in step S1 of this embodiment, the water conservancy data is preprocessed to remove abnormal values ​​in the water conservancy data, reduce the misjudgment rate, and avoid false alarms caused by instantaneous interference.

[0028] Please continue reading Figure 1As shown, the water conservancy data processing method based on big data analysis also includes: Step S2, construct data nodes, and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environmental parameters.

[0029] See also Figure 2 As shown in FIG. 1 , it is a method for constructing water conservancy node features, including: Step S21, constructing data nodes based on the duration of a preset analysis cycle.

[0030] Specifically, in step S21 of this embodiment, a data node is set for each preset analysis period, and the data nodes are numbered in chronological order to obtain node numbers, and the data nodes are used to represent the analysis period corresponding to the different collection times of each data. The data nodes correspond one-to-one to the timestamps of the collected data.

[0031] Specifically, in step S21 described in this embodiment, by setting the data nodes, time series analysis is supported by time series nodes, which facilitates the location of abnormal time periods.

[0032] Please continue reading Figure 2 As shown, the method for constructing the water conservancy node feature also includes: Step S22, weights are assigned to data nodes, water resources data and environmental data to obtain water resources environmental parameters.

[0033] Specifically, in step S22 described in this embodiment, the weights of the sand content, soil displacement and pump power are allocated according to the precipitation. If the precipitation is less than or equal to 8 mm, the expression of the water conservancy environment parameter is set to W(i)=α1×S(i)+α2×D(i)+α3×|ΔP(i) / T|; otherwise, the expression of the water conservancy environment parameter is set to W(i)=(α1+sdp)×S(i)+(α2-sdp)×D(i)+α3×|ΔP (i) / T|; where i represents the data node number, α1 represents the sediment content weight, S(i) represents the sediment content, α2 represents the soil displacement weight, D(i) represents the soil displacement, α3 represents the power change weight, ΔP(i) represents the change in pump power between the current analysis period and the previous analysis period, T represents the duration of the preset analysis period, α1+α2+α3=1, α1≥α2>sdp, sdp represents the precipitation correction parameter, 0.1≤sdp≤0.2.

[0034] Optionally, in this embodiment, the value of the precipitation correction parameter is not specifically limited, and technical personnel in this field can set it freely, as long as it meets the analysis of the water environment parameters. The optimal value of the precipitation correction parameter is sdp=0.15; in this embodiment, the initial values ​​of the sediment content weight, soil displacement weight and power change weight are set to α1=0.5, α2=0.4, and α3=0.1.

[0035] Specifically, in step S22 described in this embodiment, the weights are dynamically adjusted to accurately reflect the impact of the environment on water parameters, and precipitation correction parameters are introduced to solve the problem of insufficient adaptability of traditional fixed weight models under extreme weather conditions.

[0036] Please continue reading Figure 2 As shown, the method for constructing the water conservancy node feature also includes: Step S23, combining data nodes, water resources data and water resources environment parameter analysis to construct water resources node features.

[0037] Specifically, in step S23 described in this embodiment, the water conservancy node characteristics are constructed by combining data nodes, water conservancy data and water conservancy environmental parameter analysis, and the expression of the water conservancy node characteristics is F(i,θ(i))=L(i)×ln[V(i)+M(i)]×W(i); wherein θ(i) represents the gate opening degree, L(i) represents the water level, V(i) represents the flow velocity, and M(i) represents the precipitation.

[0038] Specifically, in step S23 described in this embodiment, multiple parameters and dynamic weights are fused to generate high-discrimination features, and the logarithmic function compresses the flow velocity dimension difference to avoid large flow velocity values ​​masking the contribution of other parameters.

[0039] Please continue reading Figure 1 As shown, the water conservancy data processing method based on big data analysis also includes: Step S3, storing water conservancy data and environmental data, and analyzing the predicted value of water conservancy data.

[0040] Specifically, in step S3 described in this embodiment, the correlation between quantitative parameters is analyzed through regression analysis to support the optimization of the prediction model, and the binary regression equation is used to reveal the combined impact of water level and precipitation on sediment content, thereby improving the prediction accuracy.

[0041] See also Figure 3 As shown in FIG. 1 , it is a prediction method for water conservancy data, including: Step S31, performing regression analysis on the water conservancy node characteristics to obtain the characteristic determination coefficient.

[0042] Specifically, in step S31 described in this embodiment, regression analysis is performed with the gate opening degree θ(i) as the independent variable and the water conservancy node feature F(i,θ(i)) as the dependent variable to analyze the univariate regression equation of the water conservancy node feature with respect to the gate opening degree, and the determination coefficient of the univariate regression equation is calculated as the characteristic determination coefficient.

[0043] Step S32, performing regression analysis on the water level, sediment content and precipitation to obtain the water environment determination coefficient and the water environment regression equation.

[0044] Specifically, in step S32 described in this embodiment, regression analysis is performed with water level L(i) and precipitation M(i) as independent variables and sediment content S(i) as dependent variable to analyze the binary regression equation of sediment content with respect to water level and precipitation, the binary regression equation is used as the water environment regression equation, and the determination coefficient of the water environment regression equation is calculated as the water environment determination coefficient.

[0045] Step S33, analyzing and calculating the predicted value of water resources data based on the water level, sediment content, precipitation and water resources environment regression equation of the current analysis period.

[0046] Specifically, in step S33 described in this embodiment, the water level, sediment content and precipitation of the current analysis period are combined in pairs and substituted into the water conservancy environment regression equation, and the calculated non-current substitution data is used as the water conservancy data prediction value.

[0047] Specifically, in this embodiment, when calculating the predicted value of water conservancy data, the water level, sediment content and precipitation are combined in pairs to obtain three sets of data, and the data are substituted into the water conservancy environment regression equation to calculate the predicted value. For example, when the water level and sediment content data are substituted, the calculated data is the predicted value of precipitation; when the water level and precipitation data are substituted, the calculated data is the predicted value of sediment content; when the sediment content and precipitation data are substituted, the calculated data is the predicted value of water level.

[0048] Optionally, the prediction method of water conservancy data is not specifically limited in this embodiment, and technical personnel in this field can set it freely. For example, it can be set to use data from recent years to train the LSTM model, input water level, sediment content and precipitation as multi-dimensional features, and use the trained LSTM model to predict water conservancy data to obtain the predicted value of water conservancy data.

[0049] Step S4, analyzing and calculating the predicted value of water conservancy data and the data residual of water conservancy data, and performing coupling analysis on the data residual and water conservancy node characteristics to determine the data status and issue an early warning; See also Figure 4 As shown, it is a data status judgment method, including: Step S41, calculating the predicted value of water conservancy data and the data residuals of the water conservancy data, wherein the data residuals include water level residuals, sediment content residuals and precipitation residuals.

[0050] Specifically, in step S41 described in this embodiment, the difference between the water conservancy data collected in the current analysis period and the predicted value of the water conservancy data in the previous analysis period is taken as the data residual.

[0051] Please continue reading Figure 4 As shown, the data status determination method further includes: Step S42, performing covariance analysis on the data residuals and water conservancy node characteristics to obtain a covariance matrix.

[0052] Specifically, in step S42 described in this embodiment, covariance analysis is performed on the water level residual, sediment content residual and precipitation residual and the water conservancy node characteristics respectively to obtain a covariance matrix, and the covariance matrix of the water level residual and the water conservancy node characteristics is set to Cov(RL(i), F(i, θ(i))), the covariance matrix of the sediment content residual and the water conservancy node characteristics is set to Cov(RS(i), F(i, θ(i))), and the covariance matrix of the precipitation residual and the water conservancy node characteristics is set to Cov(RM(i), F(i, θ(i))); wherein RL(i) represents the water level residual, RS(i) represents the sediment content residual, and M(i) represents the precipitation residual.

[0053] Specifically, in step S42 described in this embodiment, covariance analysis is used to reveal the deep correlation between residuals and features, and to distinguish occasional noise from real anomalies.

[0054] Please continue reading Figure 4 As shown, the data status determination method further includes: Step S43, determining the data status according to the data residual.

[0055] Specifically, in step S43 described in this embodiment, the standard deviations of water level residuals, sediment content residuals and precipitation residuals are calculated respectively. If the data residuals of the current analysis period are greater than 2 times the standard deviation of the corresponding data, the current analysis data is determined to be a pending abnormality; otherwise, the current analysis data is determined to be a pending normal; and the data status is judged according to the number of data with pending abnormalities in water level, sediment content and precipitation. If the number of data with pending abnormalities in water level, sediment content and precipitation in the current analysis period is greater than or equal to 2, the data status of the current analysis period is determined to be abnormal, and an early warning is issued; otherwise, the data status of the current analysis period is determined to be normal.

[0056] Please continue reading Figure 4 As shown, the data status determination method further includes: Step S44, determining the feature relevance based on the covariance matrix and updating the determination process of the data state.

[0057] Specifically, in step S44 described in this embodiment, the feature correlation is judged according to the covariance matrix. If Cov(RL(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the water level feature correlation is determined to be strong; otherwise, the water level feature correlation is determined to be weak; if Cov(RS(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the sediment content feature correlation is determined to be strong; otherwise, the sediment content feature correlation is determined to be weak; if Cov(RM(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the precipitation feature correlation is determined to be strong; otherwise, the precipitation feature correlation is determined to be weak.

[0058] Optionally, in this embodiment, the covariance threshold is set to 0.6. In this embodiment, the value of the covariance threshold is not specifically limited, and those skilled in the art can set it freely as long as it satisfies the judgment of feature correlation. The setting of the covariance threshold should be greater than or equal to 0.5 and less than or equal to 0.8.

[0059] Specifically, in step S44 described in this embodiment, according to the judgment process of updating the data status based on the feature correlation, when the data status is abnormal, if all the data with pending abnormalities have strong feature correlation, the updated data status is severely abnormal; if only one type of data with strong feature correlation exists among the data with pending abnormalities, the updated data status is slightly abnormal; if the number of data with pending abnormalities is equal to 3 and the number of data with strong feature correlation is equal to 2, the updated data status is moderately abnormal; when the data status is normal, if there is one type of data with pending abnormality and all the data with pending normality have weak feature correlation, the updated data status is slightly abnormal.

[0060] Specifically, in step S44 described in this embodiment, a multi-level response mechanism is used to match the severity of the abnormality to optimize resource allocation, and combined with the strength of the correlation judgment to improve the accuracy of the warning and reduce excessive response.

[0061] Please continue reading Figure 4 As shown, the data status determination method further includes: Step S45, updating the judgment process of feature relevance according to the feature determination coefficient and the water environment determination coefficient.

[0062] Specifically, in step S45 described in this embodiment, the judgment process of feature correlation is updated based on the feature determination coefficient and the water environment determination coefficient. If the water environment determination coefficient is greater than or equal to the feature determination coefficient, the covariance threshold is adjusted to update the judgment process of feature correlation; otherwise, the judgment process of feature correlation is not updated; the adjustment method of the covariance threshold is to increase the covariance threshold by [(0.8-covariance threshold)×(water environment determination coefficient-feature determination coefficient)].

[0063] Specifically, in step S45 described in this embodiment, dynamic threshold optimization is used to reduce misjudgment caused by environmental interference.

[0064] Step S5, collecting warning feedback after warning to trigger feature optimization rules, the warning feedback includes valid and false alarms, and the collection method is user interactive input.

[0065] Specifically, in step S5 described in this embodiment, if there are three consecutive warning feedbacks that are false alarms, the feature optimization rule is triggered. The feature optimization rule is that if the water level is a pending abnormality and the precipitation is a pending normal, the pump power weight is increased by β, if the sediment content is a pending abnormality, the sediment content weight is reduced by β, if the precipitation is a pending abnormality and the water level is a pending normal, the pump power weight is reduced by β; and the weight adjustment amount is evenly divided into the weights of the other two data, where β represents a percentage parameter, 5%≤β≤15%. When evenly dividing the adjustment amount, if the adjustment weight increases by β, the weights of the other two data are reduced, and the reduction value of each weight is half of the increase amount. If the adjustment weight decreases by β, the weights of the other two data are increased, and the increase value of each weight is half of the reduction amount.

[0066] Optionally, in this embodiment, the value of the percentage parameter is not specifically limited, and those skilled in the art can freely set it as long as the setting of the feature optimization rule is satisfied. The optimal value of the percentage parameter is β=10%.

[0067] Specifically, in step S5 described in this embodiment, dynamic weight adjustment is performed to solve the model drift problem, and feedback closed-loop optimization is used to improve the system's adaptive capability.

[0068] See also Figure 5 As shown, this embodiment also provides a water conservancy data processing device based on big data analysis, including: The collection and processing module is used to collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data; Construct an analysis module to construct data nodes, and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics in combination with data nodes, water conservancy data, and water conservancy environmental parameters, and construct an analysis module to connect with the acquisition and processing module; A storage and analysis module, used to store water conservancy data and environmental data and analyze the predicted value of water conservancy data, the storage and analysis module is connected to the construction and analysis module; A state judgment module is used to analyze and calculate the predicted value of water conservancy data and the data residual of water conservancy data, and to perform coupling analysis on the data residual and water conservancy node characteristics to judge the data state and issue an early warning. The state judgment module is connected to the storage and analysis module; The feedback optimization module is used to collect warning feedback after warning to trigger feature optimization rules. The feedback optimization module is connected to the state judgment module.

[0069] The embodiment of the present application also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute the method described in the above method embodiment.

[0070] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as a computer-readable program, a data structure, a program module or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally contain computer-readable programs, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0071] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A water conservancy data processing method based on big data analysis, characterized in that: include: Collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data; Construct data nodes and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environmental parameters; Store water conservancy data and environmental data, and analyze water conservancy data prediction values; Analyze and calculate the predicted value of water conservancy data and the data residual of water conservancy data, and conduct coupling analysis on the data residual and water conservancy node characteristics to determine the data status and issue early warnings; Collect warning feedback after warning to trigger feature optimization rules.

2. The water conservancy data processing method based on big data analysis according to claim 1 is characterized in that: The weights of sediment content, soil displacement and pump power are allocated according to the precipitation. If the precipitation is less than or equal to 8 mm, the expression of the water conservancy environment parameter is set as W(i)=α1×S(i)+α2×D(i)+α3×|ΔP(i) / T|; Otherwise, the expression for setting the water conservancy environment parameters is W(i)=(α1+sdp)×S(i)+(α2-sdp)×D(i)+α3×|ΔP(i) / T|; where i represents the data node number, α1 represents the sediment content weight, S(i) represents the sediment content, α2 represents the soil displacement weight, D(i) represents the soil displacement, α3 represents the power change weight, ΔP(i) represents the change in pump power between the current analysis period and the previous analysis period, T represents the duration of the preset analysis period, and sdp represents the precipitation correction parameter.

3. The water conservancy data processing method based on big data analysis according to claim 2 is characterized in that: Combine data nodes, water conservancy data and water conservancy environmental parameter analysis to construct water conservancy node characteristics; Regression analysis is performed on water conservancy data and environmental data to analyze the predicted value of water conservancy data.

4. The water conservancy data processing method based on big data analysis according to claim 3 is characterized in that: The difference between the water conservancy data collected in the current analysis period and the predicted value of the water conservancy data in the previous analysis period is taken as the data residual; Calculate the standard deviation of water level residual, sediment content residual and precipitation residual respectively. If the data residual of the current analysis period is greater than 2 times the standard deviation of the corresponding data, the current analysis data is judged to be an undetermined anomaly. Otherwise, the current analysis data is judged to be pending normal; and the data status is judged according to the number of data with pending abnormalities in water level, sediment content and precipitation. If the number of data with pending abnormalities in water level, sediment content and precipitation in the current analysis period is greater than or equal to 2, the data status of the current analysis period is judged to be abnormal, and an early warning is issued; Otherwise, the data status of the current analysis cycle is determined to be normal.

5. The water conservancy data processing method based on big data analysis according to claim 4 is characterized in that: Covariance analysis is performed on the water level residual, sediment content residual and precipitation residual with the water conservancy node characteristics to obtain the covariance matrix. The covariance matrix of the water level residual and the water conservancy node characteristics is set to Cov(RL(i), F(i, θ(i))), the covariance matrix of the sediment content residual and the water conservancy node characteristics is set to Cov(RS(i), F(i, θ(i))), and the covariance matrix of the precipitation residual and the water conservancy node characteristics is set to Cov(RM(i), F(i, θ(i))); where RL(i) represents the water level residual, RS(i) represents the sediment content residual, and M(i) represents the precipitation residual; The feature correlation is judged according to the covariance matrix. If Cov(RL(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the water level feature correlation is determined to be strong; otherwise, the water level feature correlation is determined to be weak; if Cov(RS(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the sediment content feature correlation is determined to be strong; otherwise, the sediment content feature correlation is determined to be weak; if Cov(RM(i), F(i, θ(i))) is greater than or equal to the covariance threshold, the precipitation feature correlation is determined to be strong; otherwise, the precipitation feature correlation is determined to be weak.

6. The water conservancy data processing method based on big data analysis according to claim 5 is characterized in that: According to the judgment process of updating the data status based on the feature correlation, when the data status is abnormal, if all the data with pending abnormalities have strong feature correlation, the updated data status is severe abnormality; if only one type of data with strong feature correlation exists among the data with pending abnormalities, the updated data status is slight abnormality; if the number of data with pending abnormalities is equal to 3 and the number of data with strong feature correlation is equal to 2, the updated data status is moderate abnormality; when the data status is normal, if there is one type of data with pending abnormality and all the data with pending normality have weak feature correlation, the updated data status is slight abnormality.

7. The water conservancy data processing method based on big data analysis according to claim 6 is characterized in that: The judgment process of updating the feature correlation is based on the feature determination coefficient and the water environment determination coefficient. If the water environment determination coefficient is greater than or equal to the feature determination coefficient, the covariance threshold is adjusted to update the judgment process of the feature correlation; otherwise, the judgment process of the feature correlation is not updated; the adjustment method of the covariance threshold is to increase the covariance threshold, and the increase amount is [(0.8-covariance threshold)×(water environment determination coefficient-feature determination coefficient)].

8. The water conservancy data processing method based on big data analysis according to claim 7 is characterized in that: If there are three consecutive false alarms, the feature optimization rule is triggered. The feature optimization rule is that if the water level is a pending abnormality and the precipitation is a pending normal, the pump power weight is increased by β; if the sediment content is a pending abnormality, the sediment content weight is reduced by β; if the precipitation is a pending abnormality and the water level is a pending normal, the pump power weight is reduced by β; and the weight adjustment amount is evenly divided into the weights of the other two data, where β represents a percentage parameter.

9. A water conservancy data processing device based on big data analysis, applied to the method according to any one of claims 1 to 8, characterized in that: include: The collection and processing module is used to collect water conservancy data and environmental data, and pre-process the collected data to eliminate abnormal data; Construct an analysis module to construct data nodes and assign weights to water conservancy data and environmental data to obtain water conservancy environmental parameters, and analyze water conservancy node characteristics by combining data nodes, water conservancy data, and water conservancy environmental parameters; Storage and analysis module, used to store water conservancy data and environmental data, and analyze the predicted value of water conservancy data; The state judgment module is used to analyze and calculate the predicted value of water conservancy data and the data residual of water conservancy data, and conduct coupled analysis on the data residual and water conservancy node characteristics to judge the data state and issue early warnings; The feedback optimization module is used to collect warning feedback after warning to trigger feature optimization rules.

10. A storage medium, characterized in that: The device stores instructions which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 8.

Images