Water quality monitoring data analysis method and system based on sponge city

Abstract

The invention relates to a water quality monitoring data analysis method and system based on a sponge city, and the method comprises the steps: responding to an analysis demand and range, and constructing and configuring a water quality monitoring data analysis processing layered architecture; collecting water quality monitoring data and carrying out pretreatment and storage treatment; performing batch processing on the static data, and / or performing streaming processing on the real-time dynamic data to generate a preliminary analysis result; importing the preliminary analysis result into a pre-constructed data model for processing, and iteratively optimizing parameters of the corresponding data model and algorithm parameters deployed by the data model; and actual monitoring data after a deep analysis result output by the application data model is observed and tracked, the architecture is actually optimized and adjusted, and a data result is converted into a visual form, encrypted and provided for an access object. The full-automatic management process provided by the invention not only improves the efficiency of data acquisition and analysis, but also greatly enhances the accuracy and intelligent level of the result, and reduces the supervision cost.

Description

Technical Field

[0001] This invention relates to the field of water quality monitoring technology, and in particular to a method and system for analyzing water quality monitoring data based on sponge cities. Background Technology

[0002] With the continued acceleration of urbanization, urban water system management, especially the processing and analysis of water quality monitoring data, is becoming increasingly important. Traditional water quality monitoring methods mainly rely on manual sampling, laboratory analysis, and basic statistical analysis. These methods are adequate for processing small-scale data from a single source, but they fall short in several aspects, including timeliness, monitoring coverage, data accuracy, intelligence level, and cost-effectiveness, when faced with large-scale, diversified, real-time updated, and highly complex water quality monitoring data.

[0003] In recent years, with the development of Internet of Things (IoT) technology, the methods of collecting water quality monitoring data have been revolutionized. Data can be collected in real time and directly stored in the database through IoT devices. However, this data collection method lacks an effective preprocessing mechanism, resulting in inconsistent quality of the stored data, which brings difficulties to subsequent data analysis and application.

[0004] In terms of data storage, traditional relational databases or Elasticsearch are widely used. However, these solutions often have high hardware resource requirements and poor scalability when dealing with massive amounts of water quality monitoring data, making it difficult to meet the needs of continuous data growth.

[0005] Data processing also faces challenges. Currently, relying on traditional relational databases results in significantly reduced computational efficiency when handling large-scale data, making real-time processing difficult. Furthermore, data reported by IoT devices in complex environments can be affected by latency, noise, and other factors, impacting the accuracy of analysis results. Adding to the complexity, water quality monitoring involves vast amounts of spatial data, which is complex and multidimensional; relational databases have limited performance in processing this type of data, further increasing the difficulty of data processing. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a water quality monitoring data analysis method and system based on sponge cities, which solves the technical problems of lack of effective preprocessing of water quality monitoring data collection, high hardware requirements and poor scalability of storage schemes, low data processing efficiency and limited accuracy of results.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0010] In a first aspect, embodiments of the present invention provide a method for analyzing water quality monitoring data in sponge cities, including:

[0011] In response to the analysis needs and scope of the acquired water quality monitoring data, a hierarchical architecture for water quality monitoring data analysis and processing is constructed, and the data flow paths, processing procedures and output requirements between each level are configured.

[0012] In the constructed architecture, water quality monitoring data is collected from at least one data source, and the water quality monitoring data is preprocessed and stored.

[0013] Perform batch processing on stored static data and / or stream processing on acquired real-time dynamic data to generate preliminary analysis results;

[0014] The preliminary analysis results are imported into at least one pre-built data model through the query engine for in-depth analysis and processing, and the parameters of the corresponding data model and the algorithm parameters deployed in the data model are iteratively optimized.

[0015] The system observes and tracks the actual monitoring data after the deep analysis results output by the applied data model. Based on the actual monitoring data, it optimizes and adjusts the architecture hierarchy, flow paths between each hierarchy, processing flow and output requirements. It also transforms the data results processed by each data model into at least one visualization form and provides them to the accessing users through preset encryption measures.

[0016] Optionally, in response to the analytical needs and scope of the acquired water quality monitoring data, a hierarchical architecture for water quality monitoring data analysis and processing is constructed, and the data flow paths, processing procedures, and output requirements between each level are configured, including:

[0017] Receive requests for water quality monitoring data analysis and determine the scope of data analysis, including specific regions, time periods, or data types.

[0018] Based on the analysis needs and scope of water quality monitoring data, a water quality monitoring data analysis and processing architecture is constructed, which includes a data acquisition layer, a data preprocessing layer, a data storage layer, a data processing layer, and a data application layer.

[0019] Configure the flow path, processing flow, and output requirements of data between different levels.

[0020] Optionally, in the constructed architecture, water quality monitoring data is collected from at least one data source, and the water quality monitoring data is preprocessed and stored, including:

[0021] Identify and connect to at least one water quality monitoring data source, and collect, buffer, and flow control the water quality monitoring data from at least one data source through a Kafka message queue, and perform preliminary preprocessing operations;

[0022] The water quality monitoring data after preliminary pretreatment is intelligently distributed to multiple processing nodes to achieve parallel preliminary cleaning treatment;

[0023] At each processing node, the water quality monitoring data after the initial cleaning process is monitored in real time according to the pre-configured anomaly detection rules. This allows for the automatic identification of abnormal data and the automatic execution of numerical interpolation or weighted replacement of neighboring data for correctable abnormal data. Unique quality marker codes are added to uncorrectable abnormal data. Additionally, the newly added and changed data portions are identified in real time, and a preset incremental data cleaning strategy is adopted to clean only the newly added and changed data portions.

[0024] After anomaly handling, the data is indexed by combining the data source and the collection timestamp to form a static dataset.

[0025] Optionally, batch processing is performed on the stored static data, and / or streaming processing is performed on the acquired real-time dynamic data to generate preliminary analysis results, including:

[0026] Configure an Apache Spark job with specified processing logic, memory, number of CPU cores, and input / output paths based on a predefined batch processing task;

[0027] The configured job is submitted to a pre-defined Apache Spark batch processing unit containing a Spark cluster for scheduling, so that the job can be executed according to the pre-defined plan or triggering conditions.

[0028] The Apache Spark batch processing unit, which has already submitted the job, performs data transformation, aggregation, and sorting on the stored static data to generate batch processing results.

[0029] The Apache Flink CDC module of the pre-defined stream processing unit extracts the required monitoring index data from the message queue of real-time dynamic data, and dynamically adjusts the data ingestion rate through a backpressure mechanism, automatically scales processing resources, and combines the SQL module of the stream processing unit to generate a detailed table containing the required monitoring index.

[0030] The rolling time window module built into the stream processing unit is used to perform statistics on the detailed table to output the stream processing results that include the range of changes in the monitored indicators within a specific time window.

[0031] Preliminary analysis results of water quality monitoring data are generated based on the batch and / or flow processing results.

[0032] Optionally, the preliminary analysis results can be imported into at least one pre-built data model for in-depth analysis processing via a query engine, and the parameters of the corresponding data model and the algorithm parameters deployed on the data model can be iteratively optimized, including:

[0033] Use the SQL query engine to query and retrieve the preliminary analysis results of the generated water quality monitoring.

[0034] Characteristic engineering analysis is performed on the preliminary analysis results to obtain characteristic engineering analysis results of characteristic indicators that include distribution patterns, volatility, correlation between indicators, and synergistic change relationships between indicators;

[0035] Based on the feature engineering analysis results, candidate models matching the data features are selected from a pre-built data model library; among them, candidate models include at least one of the following: ARIMA model, statistical anomaly detection method combined with machine learning model, and SVM model;

[0036] Based on preliminary analysis results or user input analysis requirements, determine at least one of the following target requirements: accuracy of information from data mining, accuracy of pattern recognition, and error range of trend prediction.

[0037] Based on the target requirements, models that can support the achievement of the target requirements are selected from the candidate models to serve as in-depth analysis models for water quality monitoring.

[0038] The Min-Max normalization method is used to map the data of the preliminary analysis results to the [0,1] interval. According to the set ratio, the normalized data is divided into training set, validation set and test set.

[0039] Based on the input layer structure requirements of the selected water quality monitoring depth analysis model, determine that the number of input layer nodes matches the amount of data in the preliminary analysis results, and convert the data of the preliminary analysis results to be analyzed according to the data format required by the model;

[0040] The selected water quality monitoring depth analysis model is trained using the training set, the model's performance is evaluated and the model parameters are tuned on the validation set, and the model's generalization ability is evaluated using the test set.

[0041] The trained model is used to perform in-depth analysis on the preliminary analysis results, including at least one of the following: data mining, pattern recognition, and trend prediction.

[0042] Determine the corresponding evaluation indicators based on the needs and objectives;

[0043] The process of data import, model training, performance evaluation, and parameter adjustment is repeated continuously. Each iteration further optimizes the model and algorithm parameters based on the feedback from new evaluation results and preliminary analysis results, so that the model's performance gradually approaches the target.

[0044] When the target requirement is achieved or a specific stopping condition is met, the iterative optimization stops, and the in-depth analysis results output by the data model at this point are organized and summarized.

[0045] in,

[0046] The distribution pattern is as follows:

[0047] ;

[0048] In the formula, The mean, Standard deviation;

[0049] Volatility is:

[0050] ;

[0051] In the formula, n represents the number of water quality monitoring data in the preliminary analysis results, and x i For the i-th water quality monitoring data value, This represents the data mean.

[0052] The correlation between the indicators is:

[0053] ;

[0054] In the formula, and These are the observed values ​​of two variables in the water quality monitoring data. and These are the mean values ​​of the observed values ​​of the two variables in the quality monitoring data;

[0055] The synergistic relationship between the indicators is as follows:

[0056] ;

[0057] In the formula, and These are two multivariate time series of feature indicators to be compared. The regularized path weights are determined based on expert experience. It is a variable and The distance between them is measured in Euclidean distance;

[0058] The ARIMA model is:

[0059] ;

[0060] In the formula, For autoregressive operators, B is the moving average operator, and A is the lag operator. For difference operators, For time series data, It is a white noise sequence;

[0061] The SVM model is:

[0062] ;

[0063] In the formula, For Lagrange multipliers, For the category label of the sample, is the kernel function, and b is the bias term.

[0064] Optionally, observe and track the actual monitoring data after applying the deep analysis results output by the data model, optimize and adjust the architecture hierarchy, flow paths between each level, processing flow, and output requirements based on the actual monitoring data, and transform the data results processed by each data model into at least one visualization form, providing them to the access recipients through preset encryption measures, including:

[0065] Continuously observe and record the in-depth analysis results output by the data model and the corresponding actual monitoring data;

[0066] Based on feedback from actual monitoring data, the architecture is optimized by adding, reducing, or reorganizing layers, adjusting the data flow paths between layers, improving the processing flow by improving algorithms, adjusting parameter settings, or introducing new processing modules, and adjusting output requirements.

[0067] Using visualization tools, the data results processed by various data models are transformed into at least one visualization form to obtain visualized data;

[0068] Based on multi-factor characteristics including attributes, features, importance, and sensitivity, visualized data is classified and graded.

[0069] Based on the classification and grading results of the visualized data, appropriate encryption processing is applied to the transmission and storage of visualized data of different categories and levels.

[0070] Based on the classification and grading results of the data, different data access permissions are assigned to different users and programs.

[0071] Secondly, embodiments of the present invention provide a water quality monitoring data analysis system for sponge cities, the system being used to execute the method described above, the system comprising:

[0072] The data acquisition layer is used to acquire water quality monitoring data from at least one data source;

[0073] The data preprocessing layer is used to preprocess water quality monitoring data and merge it with pre-stored historical data to form a standardized dataset;

[0074] The data storage layer is used to store the data output by the data preprocessing module in a structured database or an unstructured database.

[0075] The data processing layer is used to perform batch processing on stored static data and / or stream processing on acquired real-time dynamic data to generate preliminary analysis results.

[0076] The data application layer is used to import preliminary analysis results into at least one pre-built data model for in-depth analysis processing through a query engine, and continuously adjust and optimize the parameters of the corresponding data model and the algorithm parameters deployed in the data model based on the preliminary analysis results; observe and track the actual monitoring data after the in-depth analysis results output by the applied data model, optimize and adjust the architecture hierarchy, the flow path between each layer, the processing flow and output requirements based on the actual monitoring data, and transform the data results processed by each data model into at least one visualization form, and provide them to the access object through preset encryption measures.

[0077] Optionally,

[0078] The data acquisition layer includes: a front-end detection terminal, used to collect multi-dimensional water quality monitoring data uploaded and / or manually entered from key locations pre-set in the sponge city's full-chain facilities, and also used to acquire pre-stored historical data; and a transmission queue unit, used to collect, buffer, and flow control water quality monitoring data from at least one data source through a Kafka message queue.

[0079] The data preprocessing layer includes: a preliminary preprocessing operation unit, used to intelligently distribute the pre-processed water quality monitoring data to multiple processing nodes to achieve parallel preliminary cleaning; a multi-processing node unit, used to monitor the pre-cleaned water quality monitoring data in real time at each processing node according to pre-configured anomaly detection rules, to automatically identify abnormal data and automatically perform numerical interpolation or neighbor data weighted replacement operations on correctable abnormal data, and add unique quality marker codes to uncorrectable abnormal data; an incremental cleaning unit, used to identify newly added and changed data in real time, and use a preset incremental data cleaning strategy to clean only the newly added and changed data; and a static dataset formation unit, used to create an index by combining the data source and collection timestamp of the anomaly-handled data to form a static dataset.

[0080] The data storage layer includes: an MPP database unit for processing the water quality monitoring data after streaming and storing it as a large-scale water quality monitoring dataset using GreenPlum; and a NoSQL database unit for storing the streaming water quality monitoring data as semi-structured data via Cassandra or MongoDB, or as unstructured data via HDFS or Elasticsearch.

[0081] The data processing layer includes: a batch processing unit, which uses Apache Spark to process the stored large-scale water quality monitoring dataset, including data transformation, aggregation, and sorting, to generate batch processing results; a stream processing unit, which ingests the required monitoring indicator data from the message queue of the acquired real-time dynamic data, dynamically adjusts the data ingestion rate through a backpressure mechanism, automatically scales processing resources, and, in conjunction with the SQL module of the stream processing unit, generates a detailed table containing the required monitoring indicators; and through the built-in rolling time window module of the stream processing unit, performs statistics on the detailed table to output the stream processing results containing the range of changes in monitoring indicators within a specific time window.

[0082] In addition, the data application layer includes: an SQL query engine unit, which provides the ability to execute SQL queries on big data through Apache Hive or Presto; and a machine learning unit, which integrates Apache Spark MLlib to perform data mining tasks.

[0083] Optionally, it also includes:

[0084] The data visualization module includes a reporting tool unit and a data exploration unit. The reporting tool unit is used to generate dynamic data reports and dashboards using Tableau or Power BI; the data exploration unit provides a graphical interface.

[0085] Access the data classification and grading unit, which is used to classify and grade visualized data based on multi-factor features including attributes, characteristics, importance, and sensitivity;

[0086] The data encryption unit is used to apply corresponding encryption processing to the transmission and storage of visualization data of different categories and levels based on the classification and grading results of the visualization data.

[0087] The access control unit is used to assign different data access permissions to different users and programs based on the classification and grading results of the data.

[0088] Optionally, it also includes:

[0089] The resource management unit is used to manage computing resources through YARN to ensure that resources are allocated reasonably to each job;

[0090] The monitoring unit is used to monitor system performance and processing flows at various levels.

[0091] (III) Beneficial Effects

[0092] The beneficial effects of this invention are as follows: This invention covers multiple key aspects, providing a fully intelligent water quality management system. First, based on specific needs, a water quality monitoring data analysis and processing architecture with one or more layers is constructed, and the data flow paths, processing steps, and final output requirements between these layers are meticulously planned. Next, water quality monitoring data is efficiently collected from multiple data sources, and then this data undergoes meticulous preprocessing, streaming processing, and proper storage. Afterwards, the stored static data is batch-processed, or the dynamically acquired data is analyzed in real-time, thereby generating preliminary analysis results.

[0093] To further explore the value of the data, this invention also imports these preliminary results into a pre-built data model for in-depth analysis through a query engine. At the same time, based on the preliminary analysis results, the parameters of each data model and the parameters of the applied algorithms are continuously adjusted and optimized, thereby continuously improving the accuracy and intelligence of the analysis.

[0094] Finally, the deep analysis results output by the applied data model are continuously observed and tracked. Feedback and optimization are then performed based on actual monitoring data. This involves not only adjusting the architecture's hierarchy and processing flow but also presenting the processed data results in an intuitive visualization format and providing them to specific access points through secure encryption. This series of processes not only improves the efficiency of data collection and analysis but also significantly enhances the accuracy and intelligence of the results. The storage solution used has low hardware requirements and high scalability, reducing operating costs and providing strong technical support for sponge city construction and urban water system management. Attached Figure Description

[0095] Figure 1 A flowchart illustrating a water quality monitoring data analysis method based on sponge cities, provided as an embodiment of the present invention;

[0096] Figure 2 This is a schematic diagram illustrating the specific process of step S1 in a water quality monitoring data analysis method based on sponge cities, as provided in an embodiment of the present invention.

[0097] Figure 3 This is a schematic diagram illustrating step S2 of a water quality monitoring data analysis method based on sponge cities provided in an embodiment of the present invention.

[0098] Figure 4 This is a schematic diagram of the processing nodes of a water quality monitoring data analysis method based on sponge cities provided in an embodiment of the present invention.

[0099] Figure 5 A schematic diagram illustrating step S3 of a water quality monitoring data analysis method based on sponge cities provided in an embodiment of the present invention;

[0100] Figure 6 A schematic diagram illustrating step S4 of a water quality monitoring data analysis method based on sponge cities provided in an embodiment of the present invention;

[0101] Figure 7 A schematic diagram of step S5 of a water quality monitoring data analysis method based on sponge cities provided in an embodiment of the present invention;

[0102] Figure 8 This is a schematic diagram of the composition of a water quality monitoring data analysis system for sponge cities provided in an embodiment of the present invention;

[0103] Figure 9 This image shows the real-time monitoring effect of a water quality monitoring device in a sponge city water quality monitoring data analysis system provided in an embodiment of the present invention. Detailed Implementation

[0104] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0105] like Figure 1 As shown in the embodiment of the present invention, a water quality monitoring data analysis method based on sponge cities includes: responding to the analysis needs and scope of the acquired water quality monitoring data, constructing a hierarchical architecture for water quality monitoring data analysis and processing, and configuring the data flow path, processing flow, and output requirements between each level; in the constructed architecture, collecting water quality monitoring data from at least one data source, and performing preprocessing and storage processing on the water quality monitoring data; performing batch processing on the stored static data, and / or performing streaming processing on the acquired real-time dynamic data to generate preliminary analysis results; importing the preliminary analysis results into at least one pre-constructed data model for in-depth analysis processing through a query engine, and iteratively optimizing the parameters of the corresponding data model and the algorithm parameters deployed in the data model; observing and tracking the actual monitoring data after applying the in-depth analysis results output by the data model, optimizing and adjusting the hierarchy of the architecture, the flow path between each level, the processing flow, and the output requirements according to the actual monitoring data, and converting the data results processed by each data model into at least one visualization form, and providing them to the access object through preset encryption measures.

[0106] This invention covers several key aspects, providing a comprehensive intelligent water quality management system. First, responding to specific user needs, it constructs a one- or more-layered water quality monitoring data analysis and processing architecture, meticulously planning the data flow paths, processing steps, and final output requirements between these layers. Next, it efficiently collects water quality monitoring data from multiple data sources, then performs detailed preprocessing, streaming processing, and proper storage on this data. Finally, it performs batch processing on the stored static data or real-time analysis on the acquired dynamic data to generate preliminary analysis results.

[0107] To further explore the value of the data, this invention also imports these preliminary results into a pre-built data model for in-depth analysis through a query engine. At the same time, based on the preliminary analysis results, the parameters of each data model and the parameters of the applied algorithms are continuously adjusted and optimized, thereby continuously improving the accuracy and intelligence of the analysis.

[0108] Finally, the deep analysis results output by the applied data model are continuously observed and tracked. Feedback and optimization are then performed based on actual monitoring data. This involves not only adjusting the architecture's hierarchy and processing flow but also presenting the processed data results in an intuitive visualization format and providing them to specific access points through secure encryption. This series of processes not only improves the efficiency of data collection and analysis but also significantly enhances the accuracy and intelligence of the results. The storage solution used has low hardware requirements and high scalability, reducing operating costs and providing strong technical support for sponge city construction and urban water management.

[0109] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0110] Specifically, embodiments of the present invention provide a method for analyzing water quality monitoring data based on sponge cities, comprising:

[0111] S1. In response to the analysis needs and scope of the acquired water quality monitoring data, construct a hierarchical architecture for water quality monitoring data analysis and processing, and configure the data flow path, processing flow and output requirements between each level.

[0112] Furthermore, such as Figure 2 As shown, step S1 includes:

[0113] S11. Receive water quality monitoring data analysis requests from users, and determine the scope of data analysis, including specific regions, time periods, or data types, based on user requirements.

[0114] Specifically, the first step is to receive and analyze water quality monitoring data analysis requests from users. These requests may involve specific monitoring areas, time periods, or data types.

[0115] Specific area: Determine the geographical scope of data analysis based on user needs, such as a specific river, lake, or sea area.

[0116] Time period: Specify the time span that the user wants to analyze, such as historical data from the most recent month, a quarter, a year, or longer.

[0117] Data type: Determine the type of water quality parameters that users are concerned about, such as dissolved oxygen, turbidity, total phosphorus, ammonia nitrogen, etc., as well as the source of this data (such as IoT monitoring data, manually collected data, web scraping, or related stored files and documents, etc.).

[0118] S12. Based on the analysis needs and scope of water quality monitoring data, construct a water quality monitoring data analysis and processing architecture that includes a data acquisition layer, a data preprocessing layer, a data storage layer, a data processing layer, and a data application layer.

[0119] Having clarified user needs and the scope of data analysis, the next step is to construct a comprehensive and efficient water quality monitoring data analysis and processing architecture. This architecture should include at least the following key layers: data acquisition layer, data preprocessing layer, data storage layer, data processing layer, and data application layer. Specifically, the data acquisition layer collects and preprocesses data; the data storage layer stores large-scale datasets; the data processing layer performs batch and stream processing; the data application layer provides an SQL query engine and executes data mining and machine learning tasks; additionally, a data integration module can handle data extraction, transformation, loading, and integration; a data security module provides access control and data encryption; a data visualization module provides reporting tools and data exploration; and an operations and maintenance management module manages and monitors resources.

[0120] Therefore, by constructing the above-mentioned hierarchical and fully functional water quality monitoring data analysis and processing architecture, a solid framework can be provided for subsequent data processing work.

[0121] S13. Configure the data flow path, processing flow, and output requirements between different levels.

[0122] First, the specific flow path of data from the data acquisition layer to the data preprocessing layer, then to the data storage layer, data processing layer, and data application layer must be clearly defined. This includes data transmission methods (such as real-time transmission, periodic batch transmission, etc.) and data security and integrity assurance measures. Next, the data processing steps and methods at each layer should be planned in detail. For example, data cleaning is required in the data preprocessing layer to remove outliers or fill in missing values; specific algorithms or models may be used for data analysis in the processing layer. Finally, based on the specific needs of the user, the final data output format and content should be designed.

[0123] In addition, high-performance servers and storage devices need to be configured, and a stable and reliable operating system and necessary software framework need to be installed to provide strong hardware and software support for data processing.

[0124] S2. In the constructed architecture, water quality monitoring data is collected from at least one data source, and the water quality monitoring data is preprocessed and stored.

[0125] Furthermore, such as Figure 3 As shown, step S2 includes:

[0126] S21. Identify and connect to at least one water quality monitoring data source, and collect, buffer, and flow control the water quality monitoring data from at least one data source through a Kafka message queue, and perform preliminary preprocessing operations including removing useless data and standardizing data format.

[0127] This step first identifies at least one water quality monitoring data source. These sources include high-sensitivity sensors deployed at key locations within the urban water system, capable of real-time monitoring of key water quality indicators (such as pH, dissolved oxygen, turbidity, and heavy metal content). Next, a Kafka message queue system is configured to collect, buffer, and control water quality monitoring data from these data sources in real time. Kafka's efficiency and reliability ensure the stability and real-time nature of data transmission. After the data enters the Kafka queue, preliminary preprocessing operations are performed, such as data format conversion and deduplication, to improve data quality for subsequent processing steps.

[0128] S22. The pre-treated water quality monitoring data is intelligently distributed to multiple processing nodes to achieve parallel preliminary cleaning treatment.

[0129] After initial preprocessing, the water quality monitoring data is intelligently distributed to multiple processing nodes. This step leverages the advantages of a distributed processing architecture, distributing the data cleaning task across various nodes for parallel execution, thereby significantly improving the efficiency and speed of data cleaning. Each processing node possesses corresponding processing capabilities and resources, enabling it to independently undertake a portion of the data cleaning work.

[0130] S23. At each processing node, the water quality monitoring data after preliminary cleaning is monitored in real time according to the pre-configured anomaly detection rules. This is to automatically identify abnormal data and automatically perform numerical interpolation or neighboring data weighted replacement operations on correctable abnormal data. Unique quality mark codes are added to uncorrectable abnormal data. In addition, newly added and changed data parts in the data are identified in real time, and a preset incremental data cleaning strategy is adopted to clean only the newly added and changed data parts.

[0131] like Figure 4 As shown, at each processing node, water quality monitoring data after preliminary cleaning is monitored in real time according to pre-configured anomaly detection rules. These rules may include data range checks and consistency verification, used to automatically identify anomalies in the data. Once anomaly data is detected, it will be corrected or explicitly marked according to a preset strategy. The correction and replacement measures aim to replace the anomaly data with reasonable values. Specifically, a preset algorithm automatically identifies the type of anomaly, distinguishing between correctable anomalies (such as short-term fluctuations and sensor drift) and uncorrectable anomalies (such as equipment failure and data interruption). For the former, numerical interpolation or weighted replacement of data from adjacent time periods is automatically performed. Marking is to identify and process these data in subsequent analysis by adding a quality identification field with a unique traceability code. This step ensures the accuracy and integrity of the data.

[0132] Based on real-time water quality monitoring data, newly added and changed portions of the data are identified. For these portions, a pre-defined incremental data cleaning strategy is employed for targeted processing. This strategy avoids repeatedly cleaning the entire dataset, focusing only on newly added and changed data, thereby further improving the timeliness and efficiency of data processing. The application of the incremental data cleaning strategy enables the system to quickly adapt to data changes and maintain real-time data processing.

[0133] S24. Index the data after anomaly handling by combining the data source and the collection timestamp to form a static dataset.

[0134] S3. Perform batch processing on the stored static data and / or stream processing on the acquired real-time dynamic data to generate preliminary analysis results.

[0135] Furthermore, such as Figure 5 As shown, step S3 includes:

[0136] S31. Configure an Apache Spark job with specified processing logic, memory, number of CPU cores, and input / output paths based on a predefined batch processing task.

[0137] S32. Submit the configured job to the preset Apache Spark batch processing unit containing the Spark cluster for scheduling, ensuring that the job can be executed according to the preset plan or triggering conditions.

[0138] After configuration, the job is submitted to the batch processing unit containing the Apache Spark cluster. This step ensures that the job can be executed according to the preset plan or specific triggering conditions (such as scheduled tasks, data arrival, etc.). The batch processing unit is responsible for job scheduling and management, ensuring that resources are allocated reasonably and the job can run efficiently.

[0139] S33. Utilize the Apache Spark batch processing unit, which has submitted a good job, to perform data transformation, aggregation, and sorting on the stored static data, generating batch processing results.

[0140] S34. Using the Apache Flink CDC module of the preset stream processing unit, the required monitoring indicator data is extracted from the message queue of real-time dynamic data. The data extraction rate is dynamically adjusted through the backpressure mechanism, and the processing resources are automatically scaled. Combined with the SQL module of the stream processing unit, a detailed table containing the required monitoring indicators is generated.

[0141] It's important to understand that the back pressure mechanism is an adaptive flow control strategy in streaming data processing systems, primarily used to address system overload caused by a mismatch between data production and consumption rates. Its core principle is to dynamically adjust the upstream data injection rate based on real-time status feedback from downstream processing nodes, forming a negative feedback loop. The back pressure mechanism in streaming processing can automatically adapt to changes in data flow rate and dynamically adjust processing resources as needed. This approach ensures the continuity and stability of the data flow, providing solid support for subsequent data analysis and decision-making. The water quality monitoring data processed by streaming is stored in a corresponding type of database, forming static data for subsequent analysis and querying.

[0142] S35. The detailed table is statistically analyzed using the built-in rolling time window module of the stream processing unit to output the stream processing results that include the range of changes in the monitored indicators within a specific time window.

[0143] Apache Flink's CDC (Change Data Capture) module can ingest the required monitoring metric data from a message queue of real-time dynamic data. After ingesting the data, it is processed using Flink's SQL module to generate a detailed table containing the required monitoring metrics. Flink's built-in rolling time window module is then used to perform statistical analysis on the detailed table. The rolling time window allows for data aggregation operations within a specific time range, thereby obtaining the range of changes in the monitoring metrics within that time window, and ultimately producing the stream processing results.

[0144] In one specific embodiment, the steps for using a stream processing unit to perform real-time statistical analysis of the dissolved oxygen (CODcr) variation range within a certain time window are as follows: First, the stream processing unit uses Apache Flink CDC technology to ingest dissolved oxygen (CODcr) data from the message queue Apache Kafka in real time, and combines it with Flink SQL to generate a detailed table of the dissolved oxygen (CODcr) monitoring indicator; second, based on the above detailed table and the rolling time window in the stream processing unit Apache Flink SQL, the variation range of dissolved oxygen (CODcr) data within a certain time window is statistically analyzed.

[0145] S36. Based on the results of batch processing and / or flow processing, generate preliminary analysis results of water quality monitoring data.

[0146] S4. Import the preliminary analysis results into at least one pre-built data model through the query engine for in-depth analysis and processing, and iteratively optimize the parameters of the corresponding data model and the algorithm parameters deployed in the data model.

[0147] Step S4 involves the following steps: First, a comprehensive analysis of the preliminary analysis results is conducted, including the data distribution pattern (e.g., normal or skewed distribution), volatility (measured by standard deviation), and correlation between different indicators (calculating correlation coefficients). For example, if water quality indicators exhibit periodic fluctuations, a time series analysis model is preferred; if complex nonlinear relationships exist, machine learning models capable of handling nonlinearity, such as neural network models, are considered. Second, based on the actual application scenario, the objectives of in-depth analysis are determined, such as uncovering potential factors, identifying abnormal patterns, or predicting trends. For example, if early warning of water pollution is required, data models capable of real-time pattern recognition and short-term trend prediction are prioritized. Third, a suitable model is matched and selected: based on an understanding of the characteristics of the preliminary analysis results and the needs of in-depth analysis, at least one suitable model is selected from several pre-built data models (e.g., linear regression models, decision tree models, support vector machine models, long short-term memory network models, etc.). For example, if predicting water quality indicator trends based on historical data with linear and seasonal characteristics, the ARIMA model can be chosen; if considering the complex influence of multiple factors and the discovery of nonlinear relationships, the random forest model is more suitable. Next, the preliminary analysis results are normalized to ensure that data from different magnitudes fall within the same range, facilitating model training and computation. The training, validation, and test sets are divided according to a certain ratio (e.g., 7:2:1). Then, based on the input requirements of the data model, the processed data is accurately imported into the selected model, ensuring data compatibility with the model's input layer structure. Furthermore, evaluation metrics and stopping conditions are set. Based on the data characteristics and current model performance feedback from the preliminary analysis results, the data model parameters are dynamically adjusted. This process of data import, model training, performance evaluation, and parameter adjustment is repeated. Each iteration optimizes the model and algorithm parameters based on the new evaluation and preliminary analysis results, recording the model parameters and evaluation metric results for each iteration, gradually improving model performance to approach the preset analysis target. Iterative optimization stops when the preset analysis target is reached or the stopping condition is met. Finally, the deep analysis results output by the data model are compiled and summarized.

[0148] Furthermore, such as Figure 6 As shown, step S4 includes:

[0149] S41. Use the SQL query engine to query and retrieve the preliminary analysis results of the generated water quality monitoring.

[0150] By using the SQL query engine (Greenplum) integrated into the water quality monitoring system, SQL queries can be executed to retrieve and obtain the generated preliminary water quality monitoring results. These preliminary analysis results may include statistical information from the raw data, calculation results of basic indicators, etc.

[0151] The SQL query engine can efficiently process large-scale datasets and provide fast query responses, thus ensuring that users can obtain the analytical data they need in a timely manner.

[0152] S42. Perform feature engineering analysis on the preliminary analysis results to obtain feature engineering analysis results of feature indicators that include distribution patterns, volatility, correlation between indicators, and synergistic change relationships between indicators.

[0153] S43. Based on the feature engineering analysis results, select candidate models from the pre-built data model library that match the data features, including at least one of the following: ARIMA model, statistical anomaly detection method combined with machine learning model, and SVM model.

[0154] S44. Based on the preliminary analysis results or the analysis requirements input by the user, determine the required objective including at least one of the following: the accuracy of information from data mining, the accuracy of pattern recognition, and the error range of trend prediction.

[0155] S45. Based on the required objectives, select the models that can support the achievement of the required objectives from the candidate models, and use them as the in-depth analysis models for water quality monitoring.

[0156] In one specific embodiment, when considering the characteristics of the preliminary analysis results of water quality monitoring data, in addition to visually observing the distribution pattern, fluctuation and correlation between indicators, mathematical formulas can also be used for precise quantification.

[0157] The probability density function is used to describe the shape of the data distribution. For the normal distribution, its probability density function is:

[0158] ;

[0159] In the formula, The mean, The standard deviation is the mean and standard deviation of the sample data. By calculating the mean and standard deviation of the sample data and substituting the data into this formula, it can be determined whether the data approximately follows a normal distribution. If the data deviates significantly from a normal distribution, it may exhibit other forms such as a skewed distribution.

[0160] The magnitude of data volatility can be accurately calculated using the standard deviation formula:

[0161] ;

[0162] In the formula, n represents the number of water quality monitoring data in the preliminary analysis results, and x i For the i-th water quality monitoring data value, The standard deviation represents the mean of the data; the larger the standard deviation, the greater the volatility of the data.

[0163] Next, considering that the strength of the correlation between different indicators can be measured by calculating the Pearson correlation coefficient to determine whether there is a linear relationship, the correlation between the indicators is as follows:

[0164] ;

[0165] In the formula, and These are the observed values ​​of two variables in the water quality monitoring data. and These are the mean values ​​of the observed values ​​of the two variables in the quality monitoring data;

[0166] It is important to clarify that changes in water quality indicators are often dynamic and influenced by a combination of factors. These changes may not be simple linear relationships but involve time delays and complex dynamic patterns. Therefore, the Multivariate Dynamic Time Warping (MDTW) algorithm is introduced to analyze the dynamic trends of water quality changes and historical big data. The MDTW algorithm simultaneously considers the differences and similarities in the changes of multiple water quality indicators along the time axis, calculating the collaborative changes among the indicators as follows:

[0167] ;

[0168] In the formula, and These are two multivariate time series of feature indicators to be compared. The regularized path weights are determined based on expert experience. It is a variable and The distance between them is measured by Euclidean distance; in this way, the synergistic relationship between water quality indicators can be captured more accurately, the key driving factors affecting water quality changes can be identified, and the shortcomings of traditional univariate analysis methods that cannot fully reflect the complex dynamic relationship of water quality can be overcome.

[0169] Next, we need to clarify the requirements for in-depth analysis and determine the goals that the in-depth analysis aims to achieve. In actual water quality monitoring applications, the requirements for in-depth analysis vary. For example, it may be to uncover the potential factors hidden behind changes in water quality indicators, to accurately predict future trends in water quality, or to identify specific patterns of water quality anomalies.

[0170] If the goal is to uncover the underlying factors hidden behind changes in water quality indicators, it is necessary to analyze the complex relationships between multiple water quality indicators. In this case, it is necessary to select a model that can handle multivariate relationships and has a good ability to uncover nonlinear relationships.

[0171] For accurate prediction of future water quality trends (trend forecasting needs), such as predicting water quality changes over the next month to allow for proactive measures, preliminary analysis results may show linear trends and seasonal characteristics in water quality indicators. For example, water temperature data may be higher in summer and show a slow, linear upward trend year by year, with similar seasonal fluctuations annually. In such cases, time series analysis models can be used for prediction, with the ARIMA model being a commonly used choice. The ARIMA model is as follows:

[0172] ;

[0173] In the formula, For autoregressive operators, B is the moving average operator, and A is the lag operator. For difference operators, For time series data, The model consists of a white noise sequence. By analyzing historical data, the model's order (autoregressive order), (moving average order), and differencing order are determined, thereby enabling the prediction of future water quality indicators.

[0174] When identifying specific water quality anomaly patterns in sponge cities, such as detecting sponge facility malfunctions or sudden water quality deterioration caused by extreme rainfall events, a combination of statistical anomaly detection methods and machine learning models can be considered for real-time monitoring and rapid identification. For example, a normal fluctuation range for a water quality indicator can be defined (based on historical data statistical analysis). When new monitoring data exceeds this range to a certain extent, further pattern recognition analysis is triggered. Using a Support Vector Machine (SVM) model, the basic idea is to find an optimal hyperplane to separate data of different categories. For a binary classification problem (normal water quality vs. abnormal water quality), the decision function is:

[0175] ;

[0176] In the formula, For Lagrange multipliers, For the category label of the sample, Here, b is the kernel function, and b is the bias term. By training an SVM model, normal and abnormal patterns in the water quality data of sponge cities can be identified, allowing for timely detection of abnormal operation of sponge facilities or water pollution risks, thus ensuring the water ecological environment quality of sponge cities.

[0177] Furthermore, based on the analysis of the characteristics of the preliminary analysis results of sponge city water quality monitoring and the needs for in-depth analysis, model matching and selection are carried out.

[0178] Taking water quality monitoring in a pilot area of ​​a sponge city as an example, preliminary analysis results show that the pH value of the water has been relatively stable over the past monitoring period, with small fluctuations (standard deviation). Furthermore, the total nitrogen (TN) content exhibits a weak linear correlation with other water quality indicators (such as conductivity and heavy metal content) (the absolute values ​​of the correlation coefficients are all less than 0.3). The TN content data shows some seasonal fluctuations and has a potential correlation with the proportion of green space in sponge cities, rainfall runoff coefficient, and the intensity of surrounding agricultural activities (if any), with a slightly skewed data distribution. Simultaneously, from the perspective of in-depth analysis, it is necessary to predict the TN content trend over the next six months to assess the effectiveness of sponge cities in controlling nitrogen pollution, and to explore the potential relationship between TN content and related factors of sponge cities.

[0179] In this context, for the trend prediction of TN content, due to seasonal fluctuations and certain nonlinear characteristics, the Seasonal Autoregressive Moving Average (SARIMA) model is more suitable. Its model expression is based on the ARIMA model but considers seasonal factors:

[0180] ;

[0181] in, and These are the seasonal autoregressive and moving average operators, respectively, where S is the seasonal period and D is the seasonal difference frequency. The model parameters p, q, d, P, Q, D, and S are determined by analyzing historical TN content data to predict future TN content.

[0182] For the data mining portion that explores the relationship between TN content and sponge city-related factors, considering the existence of multivariate and nonlinear relationships, a random forest model was chosen. This model is an ensemble learning model that constructs multiple decision trees and integrates their results for prediction or classification. When constructing the decision trees, for each node, a subset of features (such as green space area, rainfall runoff coefficient, agricultural fertilizer application, etc.) is randomly selected for splitting. Through multiple random samplings and decision tree constructions, a relatively stable model with good fitting ability to complex relationships is finally obtained. This allows for the discovery of key sponge city factors affecting TN content and their interrelationships, providing data support for the planning, design, and management optimization of sponge cities.

[0183] Through the quantitative analysis of the preliminary analysis results of sponge city water quality monitoring, the clarification of the needs for in-depth analysis, and the matching and selection of appropriate models, we can more accurately construct in-depth analysis models for sponge city water quality monitoring, improve the utilization value of sponge city water quality monitoring data, and provide strong technical support for water resource protection, pollution control, and ecosystem health maintenance in sponge cities.

[0184] S46. Using the Min-Max normalization method, the data of the preliminary analysis results are mapped to the [0,1] interval to eliminate the difference in data magnitude; according to the set ratio, the normalized data is divided into training set, validation set and test set.

[0185] S47. Based on the input layer structure requirements of the selected water quality monitoring depth analysis model, determine that the number of input layer nodes matches the amount of data in the preliminary analysis results, and convert the data of the preliminary analysis results to be analyzed according to the data format required by the model.

[0186] S48. Train the selected water quality monitoring depth analysis model using the training set, evaluate the model's performance and adjust the model parameters on the validation set, and evaluate the model's generalization ability using the test set; perform at least one of the following depth analysis tasks on the preliminary analysis results using the trained model: data mining, pattern recognition, and trend prediction.

[0187] In another specific embodiment, to eliminate the impact of differences in the magnitude of different water quality indicators on model training and calculation, multiple normalization methods are employed. For example, for common numerical water quality indicators such as dissolved oxygen (DO), chemical oxygen demand (COD), and ammonia nitrogen (NH3-N), the Min-Max normalization method can be used. Let the original data be x. i The normalized data is as follows:

[0188] ;

[0189] Where, x min and x max These are the minimum and maximum values ​​in the water quality index dataset, respectively. This normalization process maps all index data to the interval [0,1], ensuring the data are on the same order of magnitude, which facilitates subsequent model training and parameter optimization.

[0190] Next, the preprocessed data is divided into training, validation, and test sets in a 7:2:1 ratio. This ensures that the model can fully learn the data features while simultaneously optimizing its parameters using the validation set and evaluating its generalization ability on an independent test set. For example, for water quality monitoring data containing a certain number of samples, the first 0.7n samples are selected as the training set for parameter learning; then 0.2n samples are selected as the validation set, and during model training, the model parameters are adjusted based on performance metrics (such as accuracy and mean squared error) on the validation set; finally, the last 0.1n samples are used as the test set to evaluate the model's final performance on unseen data.

[0191] Next, based on the input layer structure requirements of the selected data model (such as a neural network model, decision tree model, etc.), the processed data is accurately imported. For neural network models, it is necessary to ensure that the number of input layer nodes matches the number of water quality indicators. For example, if there are m water quality indicators as input features, then the neural network input layer should have m nodes. Simultaneously, the data is converted according to the data format required by the model (such as arrays, tensors, etc.) to ensure that the data can be successfully used for forward and backward propagation calculations within the model.

[0192] It is important to emphasize that techniques such as association rule mining algorithms can be used in data models to explore correlation patterns between different water quality indicators. For example, analyzing the variation range of chemical oxygen demand (COD) within a specific dissolved oxygen content range; using clustering algorithms to group monitoring samples with similar water quality characteristics to discover differences in water quality categories across different regions and time periods. Alternatively, statistical anomaly detection methods (setting thresholds to determine anomalies), deep learning techniques such as convolutional neural networks (CNNs, used for feature extraction and pattern recognition of water quality image data), or hidden Markov models (HMMs, used to identify hidden state patterns of water quality indicators over time) can be employed to identify abnormal patterns or specific regularities in water quality monitoring data. For example, determining whether specific pollutant forms exist in the water or the transition pattern from normal to polluted water quality.

[0193] Thus, based on the predictive capabilities of the data model and combined with preliminary analysis results from historical water quality monitoring, future trends are predicted. Time series analysis models (such as ARIMA) determine parameters and predict future changes in water quality indicators through differencing, autocorrelation, and partial autocorrelation analyses; machine learning regression models (such as multiple linear regression and support vector regression) use historically relevant influencing factors as input variables to predict changes in water quality indicators; deep learning models (such as Long Short-Term Memory networks LSTM) capture long-term dependencies in time series data and accurately predict trends in water quality indicators over longer periods.

[0194] S49. Determine the corresponding evaluation indicators based on the target requirements; continuously repeat the process of data import, model training, performance evaluation, and parameter adjustment. In each iteration, further optimize the model and the algorithm parameters deployed by the model based on the feedback of new evaluation results and preliminary analysis results, so that the performance of the model gradually improves and gets closer to the target requirements; when the target requirements are met or specific stopping conditions are satisfied, stop the iterative optimization and organize and summarize the deep analysis results output by the data model at this time.

[0195] In another specific embodiment, based on the objectives of deep analysis (such as the accuracy of information mined from data mining, the accuracy of pattern recognition, and the error range of trend prediction), appropriate evaluation metrics are determined. For example, for pattern recognition, metrics such as accuracy, recall, and F1 score can be used to measure model performance; for trend prediction, metrics such as mean squared error (MSE) and mean absolute error (MAE) can be used to evaluate the accuracy of prediction results. Simultaneously, specific stopping conditions are set, such as stopping the iterative optimization process when the model's evaluation metrics on the validation set no longer show significant improvement after multiple consecutive iterations (e.g., 5 times), or when the model's training time reaches a preset maximum duration.

[0196] Based on the data characteristics reported in the preliminary analysis and the performance of the current model during training and validation, the parameters of each data model (such as regression coefficients in a linear regression model, tree depth in a decision tree model, number of neurons and learning rate in a neural network model) and the parameters of the algorithms deployed in the model (such as step size and momentum parameters in optimization algorithms) are dynamically adjusted. For example, if overfitting is found during training (performing well on the training set but poorly on the test set), the complexity of the neural network model can be appropriately reduced (reducing the number of neurons, layers, etc.), or the coefficient of the regularization term can be increased to suppress overfitting; if the model converges too slowly, the learning rate and other parameters can be appropriately increased to speed up the model's training.

[0197] The process of data import, model training, performance evaluation, and parameter tuning is repeated continuously. Each iteration uses new evaluation results and preliminary analysis results to further optimize the model and algorithm parameters, gradually improving the model's performance and bringing it closer to the preset analysis goals. During the iteration process, detailed information such as model parameters and evaluation metrics are recorded for each iteration, facilitating subsequent analysis and summarization of the entire optimization process.

[0198] When the preset analysis objectives are achieved (such as the model's evaluation indicators reaching the expected threshold requirements) or specific stopping conditions are met, the iterative optimization stops, and the in-depth analysis results output by the data model at this point are organized and summarized. For example, if trend prediction is performed, the predicted water quality index values ​​for each future time period and the corresponding confidence intervals are organized into visual reports or data files; for pattern recognition results, detailed information such as the time, location, and characteristics of the identified abnormal patterns is output; for data mining results, valuable association rules, cluster group characteristics, and other content are presented in a clear and easy-to-understand form, ultimately providing complete, reliable, and practically valuable in-depth analysis results.

[0199] S5. Observe and track the actual monitoring data after the deep analysis results output by the application data model, optimize and adjust the architecture hierarchy, flow path between each level, processing flow and output requirements according to the actual monitoring data, and transform the data results processed by each data model into at least one visualization form, and provide them to the access object through preset encryption measures.

[0200] Furthermore, such as Figure 7 As shown, step S5 includes:

[0201] S51. Continuously observe and record the in-depth analysis results output by the data model and the corresponding actual monitoring data.

[0202] S52. Based on feedback from actual monitoring data, optimize and adjust the architecture hierarchy by adding, reducing, or reorganizing the hierarchy, adjust the data flow paths between each hierarchy, adjust the processing flow by improving algorithms, adjusting parameter settings, or introducing new processing modules, and adjust output requirements.

[0203] After collecting sufficient actual monitoring data, this data needs to be analyzed, and the data processing architecture needs to be optimized and adjusted accordingly based on the analysis results. This may include adding or reducing the number of layers in the architecture, reorganizing the relationships between layers, and adjusting data flow paths. For example, to meet the high-frequency early warning needs during heavy rain, a direct connection channel of "collection layer → preprocessing layer → processing layer → application layer" can be established, bypassing the conventional storage layer to achieve second-level data flow.

[0204] Simultaneously, adjustments to the processing flow are necessary, such as improving algorithms, adjusting parameter settings, or introducing new processing modules, to enhance the efficiency and accuracy of data processing. Furthermore, output requirements may need to be adjusted based on changes in actual needs to ensure the system outputs results that meet user requirements.

[0205] S53. Using visualization tools, the data results processed by each data model are transformed into at least one visualization form to obtain visualized data.

[0206] To make data results more intuitive and easier to understand, visualization tools are needed to transform the data results processed by the data model into at least one visual form, such as charts or reports. This not only helps users better understand the data but also improves decision-making efficiency and accuracy.

[0207] S54. Classify and grade the visualized data based on multi-factor characteristics including attributes, features, importance, and sensitivity.

[0208] After transforming the data into a visual format, the system also needs to classify and categorize the visualized data based on factors such as its attributes, characteristics, importance, and sensitivity. This is done to better manage and protect the data, ensuring that different categories and levels of data receive appropriate processing and protection.

[0209] S55. Based on the classification and grading results of the visualized data, apply corresponding encryption processing to the transmission and storage of visualized data of different categories and levels.

[0210] Based on the classification and grading results of the visualized data, the system needs to apply appropriate encryption processing to data of different categories and levels. The purpose of encryption processing is to protect the security and privacy of the data, preventing unauthorized access or leakage.

[0211] S56. Based on the data classification and grading results, assign different data access permissions to different users and programs.

[0212] In this step, different data access permissions need to be assigned to different users and programs based on the data classification and grading results, as well as user needs and roles. This is to ensure that users can only access the data they are authorized to access, thereby protecting data security and integrity. Access permission settings should follow the principle of least privilege, that is, granting users only the minimum permissions necessary to complete their tasks. At the same time, a sound permission management mechanism needs to be established, including processes for permission application, approval, modification, and revocation, to ensure the standardization and effectiveness of permission management.

[0213] Additionally, embodiments of the present invention provide a water quality monitoring data analysis system for sponge cities, the system being used to execute the methods described above, such as... Figure 8As shown, the system includes: a data acquisition layer for acquiring water quality monitoring data from at least one data source; a data preprocessing layer for preprocessing the water quality monitoring data and fusing it with pre-stored historical data to form a standardized dataset; a data storage layer for storing the data output from the data preprocessing module in a structured or unstructured database; a data processing layer for performing batch processing on stored static data and / or streaming processing on acquired real-time dynamic data to generate preliminary analysis results; and a data application layer for importing the preliminary analysis results into at least one pre-built data model for in-depth analysis processing via a query engine, continuously adjusting and optimizing the parameters of the corresponding data model and the algorithm parameters deployed in the data model based on the preliminary analysis results; observing and tracking the actual monitoring data after applying the in-depth analysis results from the data model, optimizing and adjusting the architecture hierarchy, flow paths between layers, processing procedures, and output requirements based on the actual monitoring data, and converting the data results processed by each data model into at least one visualization form, providing them to the accessing object through preset encryption measures.

[0214] More specifically, the data acquisition layer includes: a front-end detection terminal, used to collect multi-dimensional water quality monitoring data uploaded and / or manually entered from key locations pre-set in the sponge city's full-chain facilities, and also used to acquire pre-stored historical data; and a transmission queue unit, used to collect, buffer, and flow control water quality monitoring data from at least one data source through a Kafka message queue.

[0215] like Figure 9 As shown, the front-end detection terminal is one of the core components of the data acquisition layer. The detection terminal includes water quality and hydrological monitoring equipment such as pH meters, dissolved oxygen meters, and COD meters. In water quality monitoring scenarios, these terminals are typically deployed in key locations such as wastewater treatment plants for real-time monitoring and collection of water quality data. Specifically, the front-end detection terminal can measure and record key indicators in wastewater, such as pH, dissolved oxygen, and chemical oxygen demand. These indicators are crucial for assessing water quality, identifying potential pollution problems, and developing corresponding treatment measures. Furthermore, water quality monitoring data sources include not only reports from the aforementioned equipment but also manual data entry if equipment malfunctions.

[0216] The transmission queue unit is responsible for data transmission and management. This system uses Kafka message queue technology to implement data acquisition, buffering, and flow control. Kafka is a high-performance distributed messaging system capable of efficiently handling large data streams and providing reliable data transmission services. Through the Kafka message queue, water quality monitoring data from front-end monitoring terminals and other data sources is acquired in real time and transmitted to subsequent processing units. Simultaneously, Kafka provides data buffering and flow control mechanisms to ensure the stability and reliability of data during transmission.

[0217] Thus, through the efficient collaboration between the front-end detection terminal and the transmission queue unit, the data acquisition layer achieves comprehensive, real-time, and reliable acquisition of water quality monitoring data, providing strong support for subsequent data processing, analysis, and decision-making.

[0218] The data preprocessing layer includes: a preliminary preprocessing operation unit, used to intelligently distribute the pre-processed water quality monitoring data to multiple processing nodes to achieve parallel preliminary cleaning; a multi-processing node unit, used to monitor the pre-cleaned water quality monitoring data in real time at each processing node according to pre-configured anomaly detection rules, to automatically identify abnormal data and automatically perform numerical interpolation or neighbor data weighted replacement operations on correctable abnormal data, and add unique quality marker codes to uncorrectable abnormal data; an incremental cleaning unit, used to identify newly added and changed data in real time, and use a preset incremental data cleaning strategy to clean only the newly added and changed data; and a static dataset formation unit, used to create an index by combining the data source and collection timestamp of the anomaly-handled data to form a static dataset.

[0219] The data storage layer includes: an MPP database unit for processing the stream-processed water quality monitoring data and storing it as a large-scale water quality monitoring dataset using GreenPlum; and a NoSQL database unit for storing the stream-processed water quality monitoring data as semi-structured data via Cassandra or MongoDB, or as unstructured data via HDFS or Elasticsearch.

[0220] It's important to understand that three database units are established here to store data based on data adaptability and system performance optimization needs. The MPP database (GreenPlum), with its powerful structured data processing capabilities and high data consistency guarantee, can efficiently process and store large-scale, high-precision streaming water quality monitoring data, meeting the requirements of complex correlation analysis and long-term data backtracking. The NoSQL database unit addresses the diversity and flexibility of water quality monitoring data. Cassandra or MongoDB effectively handles semi-structured data, while HDFS or Elasticsearch properly handles unstructured data, adapting to distributed data sources and changing data formats. This improves data storage compatibility and system scalability, collectively building a comprehensive, efficient, and adaptable storage system for water quality monitoring data management, ensuring the maximum overall benefit of data storage and application.

[0221] In one embodiment, the MPP database unit is specifically designed for processing stream-processed water quality monitoring data. Utilizing GreenPlum technology, it can efficiently store large-scale water quality monitoring datasets. As a PostgreSQL-based massively parallel processing (MPP) data warehouse system, GreenPlum offers powerful data processing and storage capabilities, making it ideal for storing and analyzing massive amounts of water quality monitoring data. Through the MPP database unit, the system can quickly query and analyze these datasets, providing real-time and accurate data support for decision support.

[0222] NoSQL database units handle data that is unsuitable for storage in traditional relational databases. Specifically, it can store streaming water quality monitoring data as semi-structured or unstructured data. For semi-structured data, the system uses Cassandra or MongoDB for storage. Both are popular NoSQL solutions well-suited for storing and processing flexible and varied data structures. For unstructured data, such as text, images, or videos, the system uses HDFS (Hadoop Distributed File System) or Elasticsearch. HDFS, as part of the Hadoop ecosystem, provides highly scalable distributed file storage capabilities; while Elasticsearch is a powerful search engine capable of efficiently indexing and querying unstructured data.

[0223] The data processing layer includes: a batch processing unit, which uses Apache Spark to process the stored large-scale water quality monitoring dataset, including data transformation, aggregation, and sorting, to generate batch processing results; a stream processing unit, which ingests the required monitoring indicator data from the message queue of acquired real-time dynamic data, dynamically adjusts the data ingestion rate through a backpressure mechanism, automatically scales processing resources, and, in conjunction with the SQL module of the stream processing unit, generates a detailed table containing the required monitoring indicators; and a built-in rolling time window module of the stream processing unit performs statistics on the detailed table to output the stream processing results containing the range of changes in monitoring indicators within a specific time window.

[0224] In one specific embodiment, the batch processing unit performs deep processing on a stored large-scale water quality monitoring dataset using Apache Spark. Leveraging Spark's powerful distributed computing capabilities, the batch processing unit can efficiently execute complex batch processing tasks, including data transformation, aggregation, and sorting. These processing steps are crucial for cleaning data, eliminating outliers, extracting features, and preparing the dataset for subsequent analysis. After batch processing, the generated results not only improve data quality but also lay the foundation for further data mining and pattern recognition.

[0225] Unlike batch processing units, stream processing units focus on processing real-time data streams. Through a tight integration of Apache Kafka and Apache Flink, they enable the real-time ingestion of required monitoring indicator data from message queues. The stream processing unit's SQL module can instantly query and process this real-time data, generating detailed tables containing key monitoring indicators. These tables reflect the latest dynamics of water quality, providing decision-makers with a real-time data view.

[0226] In addition, the stream processing unit is equipped with a built-in rolling time window module. This module can perform statistical analysis on the detailed tables and output the range of changes in monitored indicators within a specific time window. This time window-based statistical method helps to capture trends and abnormal fluctuations in water quality, providing strong support for timely response and early warning.

[0227] Thus, through the collaborative operation of batch processing and stream processing units, the data processing layer achieves efficient processing of large-scale water quality monitoring datasets and instant analysis of real-time data streams. This not only improves the efficiency and accuracy of data processing but also provides solid technical support for the intelligent and real-time operation of water quality monitoring systems.

[0228] In addition, the data application layer includes: an SQL query engine unit, which provides the ability to execute SQL queries on big data through Apache Hive or Presto; and a machine learning unit, which integrates Apache Spark MLlib to perform data mining tasks.

[0229] Specifically, the SQL query engine unit integrates with advanced SQL query engines such as Apache Hive or Presto, providing users with the ability to execute SQL queries on large-scale water quality monitoring datasets. Whether for simple data retrieval or complex data analysis, the SQL query engine unit delivers fast and accurate results. This SQL-based query approach not only lowers the barrier to data analysis but also improves the flexibility and efficiency of data processing.

[0230] The machine learning unit is another crucial component of the data application layer. It integrates advanced machine learning libraries such as Apache SparkMLlib to perform complex data mining tasks. By leveraging machine learning algorithms, this unit can extract hidden patterns, trends, and association rules from massive amounts of water quality monitoring data, providing decision-makers with in-depth insights and predictive capabilities. This machine learning-based data analysis approach not only enhances the intelligence of data processing but also provides strong data support for the optimization and improvement of water quality monitoring systems.

[0231] Additionally, the system also includes:

[0232] The data visualization module includes a reporting tool unit and a data exploration unit. The reporting tool unit is used to generate dynamic data reports and dashboards using Tableau or Power BI. The data exploration unit provides a graphical interface to help users explore and analyze data and discover relationships between data.

[0233] It is important to emphasize that the data visualization module is a crucial window for system-user interaction, primarily composed of a reporting tool unit and a data exploration unit. The reporting tool unit utilizes advanced tools such as Tableau or Power BI to generate dynamic data reports and dashboards based on user needs, thus intuitively displaying various indicators and trends in water quality monitoring data. The data exploration unit provides users with a flexible data query and analysis environment, supporting in-depth data exploration to discover potential patterns and value.

[0234] Access the data classification and grading unit, which is used to classify and grade visualized data based on multi-factor features including attributes, characteristics, importance, and sensitivity.

[0235] To ensure data security and fairness, this invention introduces a data classification and grading unit. This unit comprehensively considers multiple factors such as data attributes, characteristics, importance, and sensitivity to scientifically and reasonably classify and grade the visualized data. This classification and grading mechanism not only helps improve data management efficiency but also provides strong support for subsequent data encryption and access control.

[0236] The data encryption unit is used to apply appropriate encryption processing to the transmission and storage of visualization data of different categories and levels based on the classification and grading results. The data encryption unit plays a crucial role in the data transmission and storage process. It applies appropriate encryption processing to different categories and levels of data according to the classification and grading results of the visualization data. This encryption mechanism can effectively protect data from unauthorized access and ensure the confidentiality and integrity of the data.

[0237] The access control unit is used to assign different data access permissions to different users and programs based on the classification and grading of data. It is a key component for ensuring data access security. Through this access control mechanism, the system can strictly control user access behavior to data, preventing data leakage and misuse.

[0238] Meanwhile, to improve the system's computing resource utilization efficiency, this invention introduces a resource management unit. This unit manages computing resources through YARN, ensuring that resources are allocated rationally to each job. This resource management mechanism not only helps improve the overall system performance but also reduces resource waste and operating costs. A monitoring unit is also introduced to monitor system performance and processing flows at each level, promptly identifying and addressing potential problems and faults. Through this comprehensive monitoring mechanism, the system ensures smooth data processing, providing users with stable and reliable service.

[0239] In summary, this invention provides a water quality monitoring data analysis method and system based on sponge cities. Its key features are: First, this invention possesses strong system integration capabilities, constructing a comprehensive big data analysis and processing system covering data acquisition, storage, processing, analysis, integration, security, visualization, and system management, specifically designed for water quality monitoring. Second, in terms of data acquisition, this invention employs high-sensitivity sensors and IoT technology to capture key water quality indicators in real time, while supporting multi-source data access to ensure data comprehensiveness and diversity, laying a solid foundation for subsequent analysis. Third, regarding storage technology, this invention utilizes MPP databases (GreenPlum), NoSQL databases (Cassandra, MongoDB), and advanced storage technologies such as HDFS and Elasticsearch to achieve efficient and flexible storage of large-scale datasets, supporting seamless integration of structured and unstructured data.

[0240] Furthermore, this invention also excels in data processing. By combining Apache Spark for batch processing and Apache Kafka and Apache Flink for stream processing, it enables real-time and complex processing of water quality monitoring data, improving both efficiency and accuracy.

[0241] Finally, at the data analysis level, this invention uses the SQL query engine (Greenplum) and the machine learning unit (Apache Spark MLlib) to perform deep data mining and machine learning tasks, uncovering potential correlations between data and providing intelligent analysis support for water quality monitoring.

[0242] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0243] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the technical solution should be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0244] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the invention's technical solutions and their equivalents, then the invention should also include these modifications and variations.

Claims

1. A method for analyzing water quality monitoring data in sponge cities, characterized in that, include: In response to the analysis needs and scope of the acquired water quality monitoring data, a hierarchical architecture for water quality monitoring data analysis and processing is constructed, and the data flow paths, processing procedures and output requirements between each level are configured. In the constructed architecture, water quality monitoring data is collected from at least one data source, and the water quality monitoring data is preprocessed and stored. Perform batch processing on stored static data and / or stream processing on acquired real-time dynamic data to generate preliminary analysis results; The preliminary analysis results are imported into at least one pre-built data model through the query engine for in-depth analysis and processing, and the parameters of the corresponding data model and the algorithm parameters deployed in the data model are iteratively optimized. The system observes and tracks the actual monitoring data after the deep analysis results output by the applied data model. Based on the actual monitoring data, it optimizes and adjusts the architecture hierarchy, flow paths between each hierarchy, processing flow and output requirements. It also transforms the data results processed by each data model into at least one visualization form and provides them to the accessing users through preset encryption measures.

2. The water quality monitoring data analysis method based on sponge cities as described in claim 1, characterized in that, In response to the analytical needs and scope of the acquired water quality monitoring data, a hierarchical architecture for water quality monitoring data analysis and processing is constructed, and the data flow paths, processing procedures, and output requirements between each level are configured, including: Receive requests for water quality monitoring data analysis and determine the scope of data analysis, including specific regions, time periods, or data types. Based on the analysis needs and scope of water quality monitoring data, a water quality monitoring data analysis and processing architecture is constructed, which includes a data acquisition layer, a data preprocessing layer, a data storage layer, a data processing layer, and a data application layer. Configure the flow path, processing flow, and output requirements of data between different levels.

3. The water quality monitoring data analysis method based on sponge cities as described in claim 1, characterized in that, In the constructed architecture, water quality monitoring data is collected from at least one data source, and the preprocessing and storage of the water quality monitoring data include: Identify and connect to at least one water quality monitoring data source, and collect, buffer, and flow control the water quality monitoring data from at least one data source through a Kafka message queue, and perform preliminary preprocessing operations; The water quality monitoring data after preliminary pretreatment is intelligently distributed to multiple processing nodes to achieve parallel preliminary cleaning treatment; At each processing node, the water quality monitoring data after the initial cleaning process is monitored in real time according to the pre-configured anomaly detection rules. This allows for the automatic identification of abnormal data and the automatic execution of numerical interpolation or weighted replacement of neighboring data for correctable abnormal data. Unique quality marker codes are added to uncorrectable abnormal data. Additionally, the newly added and changed data portions are identified in real time, and a preset incremental data cleaning strategy is adopted to clean only the newly added and changed data portions. After anomaly handling, the data is indexed by combining the data source and the collection timestamp to form a static dataset.

4. The water quality monitoring data analysis method based on sponge cities as described in claim 1, characterized in that, Perform batch processing on stored static data and / or streaming processing on acquired real-time dynamic data to generate preliminary analysis results, including: Configure an Apache Spark job with specified processing logic, memory, number of CPU cores, and input / output paths based on a predefined batch processing task; The configured job is submitted to a pre-defined Apache Spark batch processing unit containing a Spark cluster for scheduling, so that the job can be executed according to the pre-defined plan or triggering conditions. The Apache Spark batch processing unit, which has already submitted the job, performs data transformation, aggregation, and sorting on the stored static data to generate batch processing results. The Apache Flink CDC module of the pre-defined stream processing unit extracts the required monitoring index data from the message queue of real-time dynamic data, and dynamically adjusts the data ingestion rate through a backpressure mechanism, automatically scales processing resources, and combines the SQL module of the stream processing unit to generate a detailed table containing the required monitoring index. The rolling time window module built into the stream processing unit is used to perform statistics on the detailed table to output the stream processing results that include the range of changes in the monitored indicators within a specific time window. Preliminary analysis results of water quality monitoring data are generated based on the batch and / or flow processing results.

5. The water quality monitoring data analysis method based on sponge cities as described in claim 1, characterized in that, The preliminary analysis results are imported into at least one pre-built data model through a query engine for in-depth analysis and processing. The parameters of the corresponding data model and the algorithm parameters deployed in the data model are iteratively optimized, including: Use the SQL query engine to query and retrieve the preliminary analysis results of the generated water quality monitoring. Characteristic engineering analysis is performed on the preliminary analysis results to obtain characteristic engineering analysis results of characteristic indicators that include distribution patterns, volatility, correlation between indicators, and synergistic change relationships between indicators; Based on the feature engineering analysis results, candidate models matching the data features are selected from a pre-built data model library; among them, candidate models include at least one of the following: ARIMA model, statistical anomaly detection method combined with machine learning model, and SVM model; Based on preliminary analysis results or user input analysis requirements, determine at least one of the following target requirements: accuracy of information from data mining, accuracy of pattern recognition, and error range of trend prediction. Based on the target requirements, models that can support the achievement of the target requirements are selected from the candidate models to serve as in-depth analysis models for water quality monitoring. The Min-Max normalization method is used to map the data of the preliminary analysis results to the [0,1] interval. According to the set ratio, the normalized data is divided into training set, validation set and test set. Based on the input layer structure requirements of the selected water quality monitoring depth analysis model, determine that the number of input layer nodes matches the amount of data in the preliminary analysis results, and convert the data of the preliminary analysis results to be analyzed according to the data format required by the model; The selected water quality monitoring depth analysis model is trained using the training set, the model's performance is evaluated and the model parameters are tuned on the validation set, and the model's generalization ability is evaluated using the test set. The trained model is used to perform in-depth analysis on the preliminary analysis results, including at least one of the following: data mining, pattern recognition, and trend prediction. Determine the corresponding evaluation indicators based on the needs and objectives; The process of data import, model training, performance evaluation, and parameter adjustment is repeated continuously. Each iteration further optimizes the model and algorithm parameters based on the feedback from new evaluation results and preliminary analysis results, so that the model's performance gradually approaches the target. When the target requirement is achieved or a specific stopping condition is met, the iterative optimization stops, and the in-depth analysis results output by the data model at this point are organized and summarized. in, The distribution pattern is as follows: ； In the formula, The mean, Standard deviation; Volatility is: ； In the formula, n represents the number of water quality monitoring data in the preliminary analysis results, and x i Let i be the i-th water quality monitoring data value. This represents the data mean. The correlation between the indicators is: ； In the formula, and These are the observed values ​​of two variables in the water quality monitoring data. and These are the mean values ​​of the observed values ​​of the two variables in the quality monitoring data; The synergistic relationship between the indicators is as follows: ； In the formula, and These are two multivariate time series of feature indicators to be compared. The regularized path weights are determined based on expert experience. It is a variable and The distance between them is measured in Euclidean distance; The ARIMA model is: ； In the formula, For autoregressive operators, B is the moving average operator, and A is the lag operator. For difference operators, For time series data, It is a white noise sequence; The SVM model is: ； In the formula, For Lagrange multipliers, For the category label of the sample, is the kernel function, and b is the bias term.

6. The water quality monitoring data analysis method based on sponge cities as described in any one of claims 1-5, characterized in that, The system observes and tracks actual monitoring data after applying the deep analysis results of the data model output. Based on the actual monitoring data, it optimizes and adjusts the architecture hierarchy, flow paths between levels, processing procedures, and output requirements. It also transforms the data results processed by each data model into at least one visualization format and provides them to access recipients through pre-defined encryption measures. Continuously observe and record the in-depth analysis results output by the data model and the corresponding actual monitoring data; Based on feedback from actual monitoring data, the architecture is optimized by adding, reducing, or reorganizing layers, adjusting the data flow paths between layers, improving the processing flow by improving algorithms, adjusting parameter settings, or introducing new processing modules, and adjusting output requirements. Using visualization tools, the data results processed by various data models are transformed into at least one visualization form to obtain visualized data; Based on multi-factor characteristics including attributes, features, importance, and sensitivity, visualized data is classified and graded. Based on the classification and grading results of the visualized data, appropriate encryption processing is applied to the transmission and storage of visualized data of different categories and levels. Based on the classification and grading results of the data, different data access permissions are assigned to different users and programs.

7. A water quality monitoring data analysis system for sponge cities, characterized in that, The system is used to perform the method as described in any one of claims 1-6, the system comprising: The data acquisition layer is used to acquire water quality monitoring data from at least one data source; The data preprocessing layer is used to preprocess water quality monitoring data and merge it with pre-stored historical data to form a standardized dataset; The data storage layer is used to store the data output by the data preprocessing module in a structured database or an unstructured database. The data processing layer is used to perform batch processing on stored static data and / or stream processing on acquired real-time dynamic data to generate preliminary analysis results. The data application layer is used to import preliminary analysis results into at least one pre-built data model for in-depth analysis processing through a query engine, and continuously adjust and optimize the parameters of the corresponding data model and the algorithm parameters deployed in the data model based on the preliminary analysis results; observe and track the actual monitoring data after the in-depth analysis results output by the applied data model, optimize and adjust the architecture hierarchy, the flow path between each layer, the processing flow and output requirements based on the actual monitoring data, and transform the data results processed by each data model into at least one visualization form, and provide them to the access object through preset encryption measures.

8. The water quality monitoring data analysis system for sponge cities as described in claim 7, characterized in that, The data acquisition layer includes: a front-end detection terminal, used to collect multi-dimensional water quality monitoring data uploaded and / or manually entered from key locations pre-set in the sponge city's full-chain facilities, and also used to acquire pre-stored historical data; and a transmission queue unit, used to collect, buffer, and flow control water quality monitoring data from at least one data source through a Kafka message queue. The data preprocessing layer includes: a preliminary preprocessing operation unit, used to intelligently distribute the pre-processed water quality monitoring data to multiple processing nodes to achieve parallel preliminary cleaning; a multi-processing node unit, used to monitor the pre-cleaned water quality monitoring data in real time at each processing node according to pre-configured anomaly detection rules, to automatically identify abnormal data and automatically perform numerical interpolation or neighbor data weighted replacement operations on correctable abnormal data, and add unique quality marker codes to uncorrectable abnormal data; an incremental cleaning unit, used to identify newly added and changed data in real time, and use a preset incremental data cleaning strategy to clean only the newly added and changed data; and a static dataset formation unit, used to create an index by combining the data source and collection timestamp of the anomaly-handled data to form a static dataset. The data storage layer includes: an MPP database unit for processing the water quality monitoring data after streaming and storing it as a large-scale water quality monitoring dataset using GreenPlum; and a NoSQL database unit for storing the streaming water quality monitoring data as semi-structured data via Cassandra or MongoDB, or as unstructured data via HDFS or Elasticsearch. The data processing layer includes: a batch processing unit, which uses Apache Spark to process the stored large-scale water quality monitoring dataset, including data transformation, aggregation, and sorting, to generate batch processing results; a stream processing unit, which ingests the required monitoring indicator data from the message queue of the acquired real-time dynamic data, dynamically adjusts the data ingestion rate through a backpressure mechanism, automatically scales processing resources, and, in conjunction with the SQL module of the stream processing unit, generates a detailed table containing the required monitoring indicators; and through the built-in rolling time window module of the stream processing unit, performs statistics on the detailed table to output the stream processing results containing the range of changes in monitoring indicators within a specific time window. In addition, the data application layer includes: an SQL query engine unit, which provides the ability to execute SQL queries on big data through Apache Hive or Presto; and a machine learning unit, which integrates Apache Spark MLlib to perform data mining tasks.

9. The water quality monitoring data analysis system for sponge cities as described in claim 7, characterized in that, Also includes: The data visualization module includes a reporting tool unit and a data exploration unit. The reporting tool unit is used to generate dynamic data reports and dashboards using Tableau or Power BI; the data exploration unit provides a graphical interface. Access the data classification and grading unit, which is used to classify and grade visualized data based on multi-factor features including attributes, characteristics, importance, and sensitivity; The data encryption unit is used to apply corresponding encryption processing to the transmission and storage of visualization data of different categories and levels based on the classification and grading results of the visualization data. The access control unit is used to assign different data access permissions to different users and programs based on the classification and grading results of the data.

10. The water quality monitoring data analysis system for sponge cities as described in claim 7, characterized in that, Also includes: The resource management unit is used to manage computing resources through YARN to ensure that resources are allocated reasonably to each job; The monitoring unit is used to monitor system performance and processing flows at various levels.

Images