A data analysis system based on intelligent water management platform

Abstract

The present invention discloses a data analysis system based on an intelligent water management platform, and the present invention relates to the field of electronic digital data processing technology. It includes a cloud system platform and several operation terminals, and the operation terminals access the cloud system platform online through the network and operate, and the cloud system platform includes: a standard database, an intelligent computing model, and a manual interaction unit; the data analysis system based on the intelligent water management platform, by establishing a standardized and unified template, is used to identify and manage the received water data, and can correct the data that does not meet the specifications, so that other personnel can also accurately identify and process it, and reduce the workload and error rate increased due to data clutter; and the unrecognizable data is transferred to manual processing and the processing rules are recorded, which, on the one hand, ensures the correct processing of the data, and on the other hand, can also be used to train the intelligent computing model, so that the intelligent computing model can gradually learn and improve its processing ability, and improve the data processing efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a data analysis system based on an intelligent water management platform. Background Art

[0002] Water affairs data refers to data related to water resources, water environment, and water conservancy projects. These data include water resources, water quality monitoring data, hydrological data, water conservancy project construction and operation data, etc. The collection, analysis, and application of water affairs data are helpful for the scientific management and protection of water resources, improving the efficiency of water resources utilization, and ensuring the sustainable development of people's lives and the ecological environment. With the development of the Internet, many water affairs data are currently managed on the Internet platform.

[0003] However, existing water management platforms usually have the following problems when processing data:

[0004] (1) Low data quality: Some water management data have quality problems, such as the format of manual input, incorrect or irregular numbers, which lead to defects in data accuracy, consistency and completeness, affecting the validity and reliability of the data.

[0005] (2) Poor data sharing and communication: In some regions, water management data lack unified standards and specifications, which leads to difficulties in data sharing and communication, affecting the dissemination and application of data.

[0006] Chinese patent CN115827621A discloses a water affairs data management system based on cloud computing and data analysis, including a water affairs terminal for collecting and uploading water affairs data; a cloud for receiving data uploaded by the water affairs terminal; randomly obtaining multiple water affairs data sets and data correctness identification results based on the water affairs data uploaded by the water affairs terminal; obtaining multiple data identification decision trees based on the water affairs data in the multiple water affairs data sets and the data correctness identification results; downloading several data identification decision trees to the water affairs terminal; the water affairs terminal is also used to identify the correctness of the received water affairs data according to the data identification decision tree to obtain cleaned water affairs data; and uploading the cleaned water affairs data to the cloud. The invention realizes efficient and accurate cleaning of water affairs data by linking data processing between the cloud and the local segment.

[0007] However, when the above patent cleans the existing water data, some data that is not problematic but only has format errors but still has use value may be cleaned out, resulting in data loss. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a data analysis system based on an intelligent water management platform to solve the above-mentioned problems.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a data analysis system based on an intelligent water management platform, including a cloud system platform and a number of operation terminals, wherein the operation terminals access the cloud system platform online through a network and perform operations, and the cloud system platform includes:

[0010] Standardized database, used to store standardized data templates, and data templates are classified into three levels: upper, middle and lower according to classification items, formats and keywords;

[0011] Intelligent computing model, used to split and process water service data, compare it with the data templates saved in the standard database, classify and save the identifiable data, transfer the unidentifiable data to manual processing, and record the manual processing rules for training the intelligent computing model, and then automatically process this type of data, and perform abnormal value detection and early warning on water service data;

[0012] The manual interaction unit is used to manually process data that cannot be recognized by the intelligent computing model.

[0013] Preferably, the classification items have several items and a subset is established as F (A1, A2, A3, ..., Ax); wherein several formats are provided below the classification items and a subset is established as Ai (B1, B2, B3, ..., By), wherein i is any natural number from 1 to x; wherein several keywords are provided below the formats and a subset is established as Bj (C1, C2, C3, ..., Cz), wherein j is any natural number from 1 to y.

[0014] Preferably, the intelligent computing model includes:

[0015] Data splitting module, used to split data according to numbers, letters, and characters;

[0016] The calculation model is used to analyze the split data and compare it with the data template to determine which project it belongs to, and convert it into data in a standardized format. The data that cannot be analyzed is transferred to the manual interaction unit;

[0017] The learning and training module is used to record the manually processed rules and input them into the computing model for training so as to update the new conversion rules.

[0018] Preferably, the steps of splitting the data to be processed by the data splitting module are:

[0019] Split the data to be processed into three categories: numbers, letters, and characters, and store them in lists a, b, and c respectively:

[0020] Number list a: a=(a1, a2, ..., ao);

[0021] Letter list b: b=(b1,b2,...,bp);

[0022] Symbol list c: c = (c1, c2, ..., cq).

[0023] Preferably, the step of processing the split data by the computing model is:

[0024] ① Initialize the template specification list T in the order of numbers, letters, and symbols, and keep the original template specification;

[0025] ② Sort the data in lists a, b, and c while maintaining the original order: sort(a, b, c), where sort() represents the sorting function;

[0026] ③ Compare the sorted data with the template specification list T character by character. If sort(a, b, c) = T, the data to be processed conforms to the specification template. Then (a, b, c) is reordered according to the corresponding template sorting method in the original specification template, and marked with a label to indicate that it has been modified. The unrecognizable output is manually processed.

[0027] Preferably, the character-by-character comparison step uses a brute force matching algorithm or a regular expression comparison calculation.

[0028] Preferably, in the case where the recognized letters belong to mathematical units, the conversion ratio of each unit is first determined, and then the conversion is performed according to its unit type:

[0029] D = f(d);

[0030] Where D represents the new unit number, d represents the old unit number, and f() represents the conversion relationship function between the new unit and the old unit. The conversion relationship is established based on the specific unit type and stored in the standard database.

[0031] Preferably, for the water affairs digital data to be processed, outlier detection is first performed using an outlier detection method, that is, outlier detection is performed based on the standard deviation and mean of the data, and the abnormal data is highlighted and warned.

[0032] Preferably, the outlier detection method is specifically as follows:

[0033] ① Calculate the mean and standard deviation of the data: Assume that the data set is E=(E1, E2, ..., E K ), where K is the number of data points, then:

[0034] The formula for calculating the mean is:

[0035] E 均 =(E1+E2+ ...+E K ) / K;

[0036] The formula for calculating the standard deviation is:

[0037] ;

[0038] ② Select the mean plus or minus 3 times the standard deviation as the outlier threshold, then the outlier threshold is 3E 均 ;

[0039] ③For each data point, calculate its deviation from the mean:

[0040] Data Point E S The deviation from the mean is: |E S -E 均 |, where S is any natural number from 1 to K;

[0041] ④ Determine whether it is an outlier: If |E S -E 均 |>3E 均 , then the data point E S Mark as an outlier.

[0042] Preferably, the water service data is uploaded by staff through an operating terminal, or some networkable devices directly upload it through the Internet of Things. Beneficial Effects

[0043] The present invention provides a data analysis system based on an intelligent water management platform. Compared with the prior art, it has the following beneficial effects:

[0044] 1. The data analysis system based on the intelligent water management platform, by establishing a standardized and unified template, is used to identify and manage the received water data, and can correct the data that does not meet the specifications, so that other personnel can also accurately identify and process it, reducing the workload and error rate caused by the data clutter; and the unrecognizable data is transferred to manual processing and the processing rules are recorded, which, on the one hand, ensures the correct processing of the data, and on the other hand, can also be used to train the intelligent computing model, so that the intelligent computing model can gradually learn and improve its processing capabilities, and improve the efficiency and accuracy of data processing. And outlier detection and early warning of water data can help to detect abnormal situations in the data early, so as to take timely measures to correct or adjust, which helps to improve data quality and accuracy. In general, this data processing method can improve data management efficiency, strengthen data quality control, and promote the continuous optimization and improvement of intelligent computing models, thereby providing more reliable support for water data analysis and decision-making.

[0045] 2. The data analysis system based on the intelligent water management platform sorts the data to be processed into three categories: numbers, letters, and characters, and sorts the standard templates in the same order to facilitate comparison between the two. By using the standard templates to compare the data to be processed, it can quickly and accurately analyze the project to which the data to be processed belongs, which is convenient for automatic classification management. Those that cannot be identified can also be processed manually. The rules of manual processing can also be analyzed to improve the standard templates and data conversion rules, making the intelligent computing model gradually powerful, so as to gradually reduce manpower in the future; and the modified data is re-sorted and labeled, which on the one hand standardizes the data format, and on the other hand helps to track the data processing history and identify the processed data, thereby improving the traceability and manageability of data processing.

[0046] 3. The data analysis system based on the intelligent water management platform converts the received water digital data into a unified format, making the data easier to process and analyze, improving the readability and operability of the data, and helping the water department to better understand and use the data. The unified data format can reduce errors and inconsistencies in the data entry and conversion process, improve the accuracy and reliability of the data, and promote data sharing and communication between different departments or organizations, improve the accessibility and availability of data, and simplify the data management and maintenance process, reducing the cost and workload of data management.

[0047] 4. The data analysis system based on the intelligent water management platform can quickly detect outliers in the data through outlier detection methods to prevent these outliers from affecting subsequent analysis and decision-making. At the same time, by highlighting and warning abnormal data, it can attract the attention of relevant personnel and prompt them to further verify and correct the data, thereby improving the accuracy and reliability of the data. After excluding abnormal data, the data can be analyzed more accurately, providing more targeted decision-making support for water management departments, helping them to better optimize water operations and management. The automated outlier detection and warning system can reduce the time and cost of manual intervention, improve work efficiency, and reduce the occurrence of human errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0049] Figure 2 It is the overall system principle block diagram of the present invention;

[0050] Figure 3 It is a principle block diagram of the intelligent computing model of the present invention;

[0051] Figure 4 It is a schematic diagram of the process of detecting abnormal values ​​of water service data according to the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The present invention provides four technical solutions:

[0054] Figure 2 The first embodiment is shown: a data analysis system based on an intelligent water management platform, including a cloud system platform and several operation terminals. The operation terminals access the cloud system platform online through the network and perform operations. The cloud system platform includes:

[0055] Standardized database, used to store standardized data templates, and data templates are classified into three levels: upper, middle and lower according to classification items, formats and keywords;

[0056] The intelligent computing model is used to split and process water service data, compare it with the data templates stored in the standard database, classify and store identifiable data, transfer unidentifiable data to manual processing, and record manual processing rules for training intelligent computing models. This will automatically process this type of data and detect and warn water service data of abnormal values. Water service data is uploaded by staff through operating terminals, or directly uploaded by some networked devices through the Internet of Things.

[0057] The manual interaction unit is used to manually process data that cannot be recognized by the intelligent computing model.

[0058] By establishing a standardized and unified template for identifying and managing the received water data, data that does not meet the specifications can be corrected, so that other personnel can also accurately identify and process it, reducing the workload and error rate caused by data clutter; and transferring unrecognizable data to manual processing and recording processing rules, on the one hand, to ensure the correct processing of data, on the other hand, it can also be used to train intelligent computing models, so that the intelligent computing model can gradually learn and improve its processing capabilities, improve data processing efficiency and accuracy. And outlier detection and early warning of water data can help to detect abnormal situations in the data early, so that timely measures can be taken to correct or adjust, which helps to improve data quality and accuracy. In general, this data processing method can improve data management efficiency, strengthen data quality control, and promote the continuous optimization and improvement of intelligent computing models, thereby providing more reliable support for water data analysis and decision-making.

[0059] Figure 1 and3 A second implementation is shown, the main difference from the first implementation is that: there are several classification items and a subset is set as F (A1, A2, A3, ..., Ax); wherein several formats are set below the classification items and a subset is set as Ai (B1, B2, B3, ..., By), wherein i is any natural number from 1 to x; wherein several keywords are set below the format and a subset is set as Bj (C1, C2, C3, ..., Cz), wherein j is any natural number from 1 to y.

[0060] Intelligent computing models include:

[0061] Data splitting module, used to split data according to numbers, letters, and characters;

[0062] The calculation model is used to analyze the split data and compare it with the data template to determine which project it belongs to, and convert it into data in a standardized format. The data that cannot be analyzed is transferred to the manual interaction unit;

[0063] The learning and training module is used to record the manually processed rules and input them into the computing model for training so as to update the new conversion rules.

[0064] The steps for the data splitting module to split the data to be processed are:

[0065] Split the data to be processed into three categories: numbers, letters, and characters, and store them in lists a, b, and c respectively:

[0066] Number list a: a=(a1, a2, ..., ao);

[0067] Letter list b: b=(b1,b2,...,bp);

[0068] Symbol list c: c = (c1, c2, ..., cq).

[0069] The steps of computing the model to process the split data are:

[0070] ① Initialize the template specification list T in the order of numbers, letters, and symbols, and keep the original template specification;

[0071] ② Sort the data in lists a, b, and c while maintaining the original order: sort(a, b, c), where sort() represents the sorting function;

[0072] ③ Compare the sorted data with the template specification list T character by character. If sort(a, b, c) = T, the data to be processed conforms to the specification template. Then (a, b, c) is reordered according to the corresponding template sorting method in the original specification template, and marked with a label to indicate that it has been modified. The unrecognizable output is manually processed.

[0073] The character-by-character comparison step uses a brute force matching algorithm or a regular expression comparison calculation.

[0074] The brute force matching algorithm is also called the naive matching algorithm. The basic idea of ​​this algorithm is to compare the first character of the text string with the first character of the pattern string. If they match, the next character will be compared. If they do not match, the pattern string will be moved back one position and the comparison will be restarted from the current position of the text string. This process continues until the pattern string is completely matched or the text string is traversed.

[0075] A regular expression is a tool used to match string patterns. It consists of a series of characters and operators that can be used to check whether a string conforms to a certain pattern or to extract specific information from a string. Regular expressions are very useful in text processing, search and replace, and can help users process text data quickly and efficiently.

[0076] By splitting the data to be processed into three categories of numbers, letters, and characters and sorting them, and sorting the standard templates in the same order, it is convenient to compare the two. By using the standard templates to compare the data to be processed, it is possible to quickly and accurately analyze the project to which the data to be processed belongs, which is convenient for automatic classification management. Those that cannot be identified can also be processed manually. The rules of manual processing can also be analyzed to improve the standard templates and data conversion rules, making the intelligent computing model gradually powerful, so as to gradually reduce manpower in the future; and the modified data is re-sorted and labeled, which on the one hand standardizes the data form, and on the other hand helps to track the data processing history and identify the processed data, improving the traceability and manageability of data processing.

[0077] Figure 3 A third embodiment is shown, which differs from the first embodiment mainly in that, for the case where the recognized letters belong to mathematical units, the conversion ratio of each unit is first determined, and then the conversion is performed according to its unit type:

[0078] D = f(d);

[0079] Where D represents the new unit number, d represents the old unit number, and f() represents the conversion relationship function between the new unit and the old unit. The conversion relationship is established based on the specific unit type and stored in the standard database.

[0080] When converting, first determine the conversion ratio of each unit, for example:

[0081] The conversion ratio between inches (in) and centimeters (cm) is 1in=2.54cm;

[0082] The conversion ratio between Celsius (℃) and Fahrenheit (℉) is ℃=(℉-32) / 1.8;

[0083] Conversion formula for length units: new unit = old unit × conversion ratio;

[0084] Conversion formula for temperature unit: new unit = (old unit - constant) / coefficient;

[0085] Through the above formula, the identified mathematical units in different formats can be converted into a unified format, and the corresponding digital adjustments can be made to facilitate subsequent mathematical calculations and management.

[0086] Converting the received water digital data into a unified format can make the data easier to process and analyze, improve the readability and operability of the data, and help water departments better understand and use the data. The unified data format can reduce errors and inconsistencies in the data entry and conversion process, improve the accuracy and reliability of the data, and promote data sharing and communication between different departments or organizations, improve the accessibility and usability of data, and simplify the data management and maintenance process, reducing the cost and workload of data management.

[0087] Figure 4 A fourth implementation is shown, which mainly differs from the first implementation in that, for the water digital data to be processed, outlier detection is first performed using an outlier detection method, that is, outlier detection is performed based on the standard deviation and mean of the data, and abnormal data is highlighted and warned.

[0088] Outlier detection is a statistical analysis method used to identify abnormal values ​​or outliers in a data set. Outliers are usually values ​​that are significantly different from the majority of values ​​in the data set. They may be caused by measurement error, data entry errors, or true anomalies.

[0089] The specific outlier detection method is as follows:

[0090] ① Calculate the mean and standard deviation of the data: Assume that the data set is E=(E1, E2, ..., E K ), where K is the number of data points, then:

[0091] The formula for calculating the mean is:

[0092] E 均 =(E1+E2+ ...+E K ) / K;

[0093] The formula for calculating the standard deviation is:

[0094] ;

[0095] ② Select the mean plus or minus 3 times the standard deviation as the outlier threshold, then the outlier threshold is 3E 均 ;

[0096] ③For each data point, calculate its deviation from the mean:

[0097] Data Point E S The deviation from the mean is: |E S -E 均 |, where S is any natural number from 1 to K;

[0098] ④ Determine whether it is an outlier: If |E S -E 均 |>3E 均 , then the data point E S Mark as an outlier.

[0099] Through the outlier detection method, outliers in the data can be quickly discovered to prevent these outliers from affecting subsequent analysis and decision-making. At the same time, by highlighting and warning abnormal data, it can attract the attention of relevant personnel and prompt them to further verify and correct the data, thereby improving the accuracy and reliability of the data. After excluding abnormal data, the data can be analyzed more accurately, providing more targeted decision-making support for water management departments, helping them to better optimize water operations and management. Through the automated outlier detection and warning system, the time and cost of manual intervention can be reduced, work efficiency can be improved, and the occurrence of human errors can also be reduced.

[0100] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0101] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0102] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data analysis system based on an intelligent water management platform, comprising a cloud system platform and a number of operation terminals, wherein the operation terminals access the cloud system platform online through a network and perform operations, characterized in that: The cloud system platform includes: Standardized database, used to store standardized data templates, and data templates are classified into three levels: upper, middle and lower according to classification items, formats and keywords; Intelligent computing model, used to split and process water service data, compare it with the data templates saved in the standard database, classify and save the identifiable data, transfer the unidentifiable data to manual processing, and record the manual processing rules for training the intelligent computing model, and then automatically process it, and detect and warn the water service data for abnormal values; A manual interaction unit is used to manually process data that cannot be recognized by the intelligent computing model; The classification items have several items and the subset is F (A1, A2, A3, ..., Ax); the classification items have several formats and the subset is Ai (B1, B2, B3, ..., By), where i is any natural number from 1 to x; the formats have several keywords and the subset is Bj (C1, C2, C3, ..., Cz), where j is any natural number from 1 to y; The intelligent computing model includes: Data splitting module, used to split data according to numbers, letters, and characters; The calculation model is used to analyze the split data and compare it with the data template to determine which project it belongs to, and convert it into data in a standardized format. The data that cannot be analyzed is transferred to the manual interaction unit; The learning and training module is used to record the manually processed rules and input them into the computing model for training so as to update the new conversion rules.

2. The data analysis system based on the intelligent water management platform according to claim 1 is characterized by: The steps of splitting the data to be processed by the data splitting module are as follows: Split the data to be processed into three categories: numbers, letters, and characters, and store them in lists a, b, and c respectively: Number list a: a=(a1, a2, ..., ao); Letter list b: b=(b1,b2,...,bp); Symbol list c: c = (c1, c2, ..., cq).

3. The data analysis system based on the intelligent water management platform according to claim 2 is characterized by: The steps of the computational model for processing the split data are as follows: ① Initialize the template specification list T in the order of numbers, letters, and symbols, and keep the original template specification; ② Sort the data in lists a, b, and c while maintaining the original order: sort(a, b, c), where sort() represents the sorting function; ③ Compare the sorted data with the template specification list T character by character. If sort(a, b, c) = T, the data to be processed conforms to the specification template. Then (a, b, c) is reordered according to the corresponding template sorting method in the original specification template, and marked with a label to indicate that it has been modified. The unrecognizable output is manually processed.

4. The data analysis system based on the intelligent water management platform according to claim 3 is characterized by: The character-by-character comparison step uses a brute force matching algorithm or a regular expression comparison calculation.

5. The data analysis system based on the intelligent water management platform according to claim 1 is characterized by: For the case where the recognized letters belong to mathematical units, first determine the conversion ratio of each unit, and then convert them according to their unit type: D = f(d); Where D represents the new unit number, d represents the old unit number, and f() represents the conversion relationship function between the new unit and the old unit. The conversion relationship is established based on the specific unit type and stored in the standard database.

6. The data analysis system based on the intelligent water management platform according to claim 1 is characterized by: For the water affairs digital data to be processed, outlier detection is first performed using the outlier detection method, that is, outlier detection is performed based on the standard deviation and mean of the data, and abnormal data is highlighted and warned.

7. The data analysis system based on the intelligent water management platform according to claim 6 is characterized by: The specific outlier detection method is as follows: ① Calculate the mean and standard deviation of the data: Assume that the data set is E=(E1, E2, ..., E K ), where K is the number of data points, then: The formula for calculating the mean is: THE 均 =(E1+E2+ …+E K ) / K; The formula for calculating the standard deviation is: ； ② Select the mean plus or minus 3 times the standard deviation as the outlier threshold, then the outlier threshold is 3E 均 ; ③For each data point, calculate its deviation from the mean: Data Point E S The deviation from the mean is: |E S -E 均 |, where S is any natural number from 1 to K; ④ Determine whether it is an outlier: If |E S -E 均 |>3E 均 , then the data point E S Mark as an outlier.

8. The data analysis system based on the intelligent water management platform according to claim 1 is characterized by: The water affairs data is uploaded by the staff through the operation terminal, or directly uploaded by some networked devices through the Internet of Things.

Images