Hydrological sensor data analysis system based on flink and construction method thereof

Abstract

The application discloses a hydrological sensor data analysis system based on Flink and a construction method thereof, and comprises the following steps: (1) installing and configuring Apache Flink, Apache Kafka and Apache IoTDB; (2) in the data sending layer, the hydrological flow data of different sensors is subjected to format conversion and is sent to an upstream message transmission layer; (3) in the upstream message transmission layer, the data sending layer and a data processing layer are connected by using Kafka and an asynchronous message transmission mode; (4) in the data processing layer, Flink is used for statistical analysis of the hydrological sensor data; (5) in a downstream message transmission layer, the data processing layer, a persistent layer and a visualization layer are connected by using Kafka and the asynchronous message transmission mode; (6) in the persistent layer, the analysis result is persisted by using IoTDB; and (7) in the visualization layer, the display of real-time analysis results and the query of historical analysis results are realized. The application realizes real-time analysis of high-speed unbounded hydrological sensor data.

Description

Technical Field

[0001] This invention relates to a hydrological sensor data analysis system based on Flink and its construction method, belonging to the field of stream data processing and statistical analysis technology. Background Technology

[0002] Hydrological sensors, as a crucial component of water conservancy research and construction, play a vital role in collecting and processing hydrological data for flood control, drought prevention, and water hazard mitigation. Typically, hydrological sensors collect data at regular time intervals and transmit it as streaming data to an analysis platform, ultimately forming hydrological time series for storage and computation. Statistical analysis is fundamental to the study of hydrological sensor data. Many current hydrological analyses are based on assumptions about certain characteristics of the data, hoping that deviations from reality will not affect the analysis results. However, hydrological analysis based on flawed assumptions can lead to unreliable and incorrect results. Therefore, it is essential to use statistical analysis to verify the characteristics of the hydrological data before further processing.

[0003] Traditional statistical analysis methods are typically based on database technology. This involves first storing the data in a database, then using tools like R, SPSS, and Matlab on a single machine to load the dataset into memory and perform analysis using feature testing algorithms. This method is common for analyzing static datasets, but it's unsuitable for hydrological sensor data. Hydrological sensor data is characterized by its large volume and high flow rate, meaning that the memory resources of a single computer are often insufficient. It's necessary to continuously load new data into the database to free up memory space, and then retrieve the data for computation. This process incurs additional time overhead, and when the computation speed is slower than the flow rate of the streaming data, analysis results cannot be provided in a timely manner. Furthermore, current mainstream big data frameworks like Spark and Flink do not provide adequate support for statistical analysis in streaming scenarios. Summary of the Invention

[0004] Purpose of the invention: This invention provides a hydrological sensor data analysis system based on Flink and its construction method, offering a complete solution for the statistical analysis of large-scale, high-speed hydrological sensor data.

[0005] Technical solution: A hydrological sensor data analysis system based on Flink, comprising a data transmission layer, an upstream message transmission layer, a data processing layer, a downstream message transmission layer, a persistence layer, and a visualization layer arranged from top to bottom.

[0006] The data transmission layer collects flow data from different hydrological sensors, converts the data into a format, merges it into a single data stream, and then transmits it to the upstream message transmission layer.

[0007] The upstream message transmission layer is used to realize data transmission between the data sending layer and the data processing layer. It uses the message queue Kafka and adopts an asynchronous message passing method to connect the data sending layer and the data processing layer.

[0008] The data processing layer processes and analyzes the hydrological sensor data and sends the analysis results to the downstream message transmission layer.

[0009] The downstream message transmission layer is used to realize data transmission between the data processing layer, persistence layer, and visualization layer. It uses the message queue Kafka and adopts an asynchronous message passing method to connect the data processing layer, persistence layer, and visualization layer.

[0010] The persistence layer uses the time-series database IoTDB to persist the hydrological sensor data analysis results output by the data processing layer.

[0011] The visualization layer includes visualizations of real-time analysis results and visualizations of historical analysis results, providing real-time display and historical data query for users to conduct further analysis.

[0012] A method for constructing a hydrological sensor data analysis system based on Flink includes the following steps:

[0013] (1) Install and configure the stream computing framework Apache Flink, the message queue Apache Kafka, and the time series database Apache IoTDB to ensure smooth communication between the stream computing framework Apache Flink, the message queue Apache Kafka, and the time series database Apache IoTDB;

[0014] (2) In the data sending layer, the Spring Boot framework is used to receive hydrological flow data from different sensors, convert the format of each data into a unified JSON format, cache it in a queue, and send the data to the upstream message transmission layer through Kafka Producer.

[0015] (3) In the upstream message transmission layer, create a topic named Source Topic to cache the source data transmitted by the data sending layer, and set the number of partitions and replicas of the Topic according to the number of nodes in the cluster where the system is located and the user's needs.

[0016] (4) In the data processing layer, Flink's time window mechanism and window functions are used to partition, preprocess, aggregate, and analyze hydrological sensor data. The data processed by Flink is sent to the downstream message transmission layer. On the one hand, the visualization layer can pull the latest analysis results from the downstream message transmission layer in real time. On the other hand, the persistence layer can pull the analysis results and persist them. The downstream message transmission layer uses Kafka for asynchronous communication and peak shaping.

[0017] (5) In the downstream message transmission layer, create a topic named Result Topic to cache the result data transmitted by the data processing layer, and set the number of partitions and replicas of the Topic according to the number of nodes in the system cluster and user needs.

[0018] (6) In the persistence layer, the analysis results are persisted using the time series database IoTDB.

[0019] (7) In the visualization layer, implement the persistence layer and the downstream message transmission layer.

[0020] It provides a 0-connection interface; at the same time, it utilizes the WebSocket protocol and the ECharts framework to visualize real-time analysis results and query historical analysis results.

[0021] In step (2), the implementation of the data sending layer specifically includes the following steps:

[0022] Step (21) deserialize the hydrological sensor data and extract the data information;

[0023] Step (22): Create a queue and cache the data information in the queue;

[0024] Step (23) involves parsing the components of the data information and converting them into a unified JSON format;

[0025] Step (24): Using the send method in Kafka Producer, the source data is packaged into a message, the message key is set to the sensor number, and the data in the queue is sent to the upstream message transport layer asynchronously according to the Kafka Topic name and Topic partition number.

[0026] Step (25) uses the Spring Boot framework to start a Java backend service to continuously convert and output the hydrological sensor data in steps (21)-(24).

[0027] In step (4), the implementation of the data processing layer specifically includes the following steps:

[0028] Step (41): Using the FlinkKafkaConsumer method, continuously monitor the Source Topic in the Kafka Broker. If new data exists in the Topic, immediately pull the data.

[0029] Step (42): Using the KeyBy operator, the hydrological data is partitioned according to the different sensor numbers, and the data with different numbers are isolated from each other;

[0030] Step (43) establish a rolling time window as a preprocessing window to preprocess the hydrological data and solve the problems of redundancy, duplication, anomaly and missing data.

[0031] Step (44) involves aggregating hydrological data based on a rolling time window to reduce the data size.

[0032] Step (45) applies the statistical test algorithm to the window function ProcessWindowFunction to analyze the normality, stationarity, uniformity and trend of hydrological sensor data;

[0033] Step (46) uses the FlinkKafkaProducer method to send the analysis results to the Kafka ResultTopic, where the key is set to the sensor number.

[0034] In step (43), the hydrological data preprocessing specifically includes the following steps:

[0035] Step (43a): Establish a scrolling window W0 as a preprocessing window, the size of which depends on the sampling frequency of the hydrological sensor;

[0036] Step (43b) parses the JSON format data, retains the timestamp, sensor number and hydrological data information, and solves the data redundancy problem;

[0037] Step (43c) compares the timestamps of the data in the window. If the data with the same timestamp is found, the duplicate data is deleted to solve the data duplication problem.

[0038] Step (43d) uses Tukey Test to detect outliers. Based on the interquartile range (IQR), the first quartile (Q1) and the third quartile (Q3), the upper and lower boundaries (R1 and R3) are calculated. Values ​​less than R1 or greater than R3 are identified as outliers and removed to solve the data anomaly problem.

[0039] In step (43e), if the proportion of missing data in the preprocessing window does not exceed the set proportion, then the window is filled using linear interpolation; if the proportion of missing data exceeds the set proportion, then the data is considered to have lost statistical significance, and the window is discarded to resolve the missing data problem.

[0040] Step (43f) outputs the preprocessed data stream S0.

[0041] In step (44), the specific steps for realizing hydrological data aggregation based on time windows include the following:

[0042] Step (44a): Establish a rolling window W1 with a size of 1 hour, take the average value of the preprocessed data stream S0 within the window to measure the concentration position of the data within the window, and then output the average value as data stream S1.

[0043] In step (44b), a rolling window W2 with a size of 1 day is created. The average value of the data stream S1 within the window is taken to measure the concentration of the data within the window. Then the average value is output as data stream S2.

[0044] In step (45), the statistical analysis specifically includes the following steps:

[0045] Step (45a): Open three sliding windows W3, W4 and W5 for the aggregated data stream S2. The window sizes are 7 days (week), 10 days (ten-day period) and 30 days (month), respectively, with a step size of 1 day.

[0046] In step (45b), a total of 12 testing algorithms for four characteristics—normality, stationarity, uniformity, and trend—are applied to the Flink window function ProcessWindowFunction in three windows, as shown in Table 1. Three algorithms are used to comprehensively evaluate each characteristic, and the hydrological analysis results are output as three data streams: S3, S4, and S5.

[0047] Table 1 Feature Detection Algorithms Used in the Data Processing Layer

[0048]

[0049] In step (6), implementing the persistence layer specifically includes the following steps:

[0050] Step (61): Create a storage group and set the storage path for the analysis results to "root.object.sensor_type.sensor_id.analysis", where:

[0051] ①root: The default root node;

[0052] ②object: represents the object of observation, such as rivers, reservoirs, etc.;

[0053] ③sensor_type: Represents the sensor type, such as water level, flow rate, precipitation sensor, etc.;

[0054] ④sensor_id: Represents the sensor number, used to locate a specific sensor;

[0055] ⑤analysis: Represents the analysis results, that is, the final result of each feature testing algorithm.

[0056] Step (62): Use Kafka Consumer to continuously monitor the Kafka Result Topic. If Flink writes new analysis results to the Topic, immediately pull the data.

[0057] Step (63) write the retrieved hydrological analysis results into the buffer. If the number of data entries in the buffer exceeds 10 or the time since the last data writing exceeds 1 second, then write the data in the buffer into the database.

[0058] In step (7), the implementation of the visualization layer specifically includes the following steps:

[0059] Step (71): Use Kafka Consumer to continuously monitor the Kafka Result Topic. If new data flows into the Topic, immediately pull the data.

[0060] Step (72): Using the WebSocket protocol, a full-duplex TCP connection is established between the client and the server, and the server actively pushes real-time hydrological analysis results to the client.

[0061] Step (73): In the client, the hydrological analysis results pushed by the client are plotted into charts in real time using the ECharts framework;

[0062] Step (74) uses IoTDB JDBC to establish a connection between the visualization layer and the persistence layer. Historical analysis results can be queried based on time range, feature selection and sensor number, and then plotted into charts using ECharts for display.

[0063] Beneficial Effects: Compared with existing technologies, the hydrological sensor data analysis system and its construction method based on Flink provided by this invention enable real-time analysis of high-speed, unbounded hydrological sensor data, providing a complete solution for the statistical analysis of current hydrological sensor data. This system utilizes components such as Flink, IoTDB, and ECharts to acquire, preprocess, statistically analyze, store, and visualize hydrological sensor data. Simultaneously, it uses the message queue Kafka to decouple different layers, making the system more modular and flexible. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the system structure according to an embodiment of the present invention. Detailed Implementation

[0065] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0066] like Figure 1 As shown, a hydrological sensor data analysis system based on Flink includes: a data transmission layer, an upstream message transmission layer, a data processing layer, a downstream message transmission layer, a persistence layer, and a visualization layer, arranged in order from top to bottom.

[0067] The data transmission layer is used to collect flow data from different hydrological sensors, convert the data into a unified format, and aggregate it into a data stream to send to the upstream message transmission layer, providing the system with a unified data source.

[0068] The upstream message transport layer connects the data sending layer and the data processing layer, enabling data transmission between them. Utilizing the Kafka message queue and employing asynchronous message passing, the data sending and processing layers are decoupled, preventing blocking of the data sending layer due to untimely data consumption by the data processing layer. Furthermore, partitioning and replicating topics in Kafka further improves throughput and availability.

[0069] The data processing layer, implemented using the Flink streaming framework, is the core layer of this system, where a large number of complex calculations occur. The data processing layer uses a time window operator to implement 12 feature verification algorithms for hydrological sensor data in Java, and applies them to the window function `ProcessWindowFunction` to analyze the four characteristics of hydrological data: normality, stationarity, trend, and uniformity. The verification algorithms used are shown in Table 1. However, before analyzing the data, some processing is required. First, the data is partitioned according to different sensor sources to avoid data from different sensors being mixed up. Second, a rolling time window is established to address data redundancy, duplication, anomalies, and missing data within the window. Finally, to improve computational efficiency, a time window-based data aggregation method is implemented to reduce the data size.

[0070] The downstream message transport layer connects the data processing layer with the persistence and visualization layers, enabling data transmission between them. Utilizing the Kafka message queue, an asynchronous message passing method is employed to decouple the three layers. Furthermore, topics in Kafka are partitioned and replicated to further improve throughput and availability.

[0071] The persistence layer is responsible for storing the analysis results. Since the analysis results are presented in time series format, the time series database IoTDB is used as the persistence method to provide feedback on historical analysis result queries initiated by users.

[0072] The visualization layer is used to display the analysis results of hydrological sensor data, including real-time and historical analysis results. For real-time analysis results, the system listens to the downstream message transmission layer, pulls result data from Kafka, and sends it to the client via the WebSocket protocol. The ECharts framework then plots the data into real-time charts. For historical analysis results, the system connects to the persistent layer IoTDB, queries historical results based on user query criteria, and returns the results. The ECharts framework then plots the data into charts.

[0073] This invention also discloses a method for constructing a hydrological sensor data analysis system based on Flink, comprising the following steps:

[0074] Step 1: Install and configure the stream computing framework Apache Flink, the message queue Apache Kafka, and the time series database Apache IoTDB. Because this system is modular, it can be installed on a single machine or in a cluster with multiple nodes. The following steps, using a cluster with three nodes as an example, explain the configuration process for these three components:

[0075] (1) Configure the Flink cluster. The Flink cluster consists of JobManager and TaskManager. The JobManager is deployed on node 1 to allocate, monitor and manage tasks for TaskManager. Then, TaskManager is deployed on the three nodes to execute Flink jobs.

[0076] (2) Configure the Kafka cluster. To manage the Kafka cluster, a Zookeeper cluster needs to be configured on 3 nodes, with the Client port set to 2181 to provide services for external access; and a Kafka cluster needs to be configured on 3 nodes, with the Bootstrap port set to 9092 to listen for data transmission or pull requests;

[0077] (3) Configure IoTDB. Install and configure IoTDB on node 3, set the database account password, and set the listening port to 6667. Clients can access IoTDB through this port.

[0078] (4) Start each component in the cluster, test the data connectivity of each port, and ensure smooth communication between components.

[0079] Step 2: In the data transmission layer, using the Spring Boot framework, hydrological flow data from different sensors is received, each data point is converted to a uniform format, cached in a queue, and then sent to the upstream message transmission layer. This involves the following steps:

[0080] (1) Deserialize the hydrological sensor data and extract the data information;

[0081] (2) Establish a queue and cache the data information in the queue;

[0082] (3) Parse the components of the data information and convert them into a unified JSON format;

[0083] (4) Using the send method in Kafka Producer, set the key to the sensor number, and send the data in the queue to the upstream message transport layer asynchronously according to the Topic name and partition number;

[0084] (5) Use the Spring Boot framework to start a Java backend service to continuously convert and output hydrological sensor data.

[0085] Step 3: In the upstream message transmission layer, create a Source Topic to cache the source data transmitted by the data sending layer. Configure the Source Topic with 3 partitions and 3 replicas, storing them in the Broker nodes on 3 different nodes. Partitioning improves Kafka's throughput, while replication enhances its reliability and prevents data loss.

[0086] Step 4: In the data processing layer, leveraging the Flink framework's support for stream computing, the hydrological sensor data is partitioned, preprocessed, aggregated, and statistically analyzed. This includes the following steps:

[0087] (1) Use the FlinkKafkaConsumer method to continuously monitor the Source Topic in Kafka Broker. If there is new data in the Topic, immediately pull the data.

[0088] (2) Using the KeyBy operator, the hydrological data is partitioned according to the different sensor numbers, and the data with different numbers are isolated from each other;

[0089] (3) A rolling time window is established as a preprocessing window to preprocess the hydrological data and address redundancy, duplication, anomalies, and missing data. First, a rolling window W0 is established as the preprocessing window, with its size determined by the sampling frequency of the hydrological sensors. Second, the JSON-formatted data is parsed, retaining timestamps, sensor numbers, and hydrological data information to resolve data redundancy. Then, the timestamps of the data within the window are compared; if duplicate timestamps are found, the duplicate data is deleted to resolve data duplication. Finally, Tukey Test is used to detect outliers, and the upper and lower boundaries R0 and R1 are estimated according to the formula:

[0090]

[0091] In the formula, Q1 is the first quartile, Q3 is the third quartile, and IQR is the interquartile range.

[0092] IQR = Q3 - Q1

[0093] Values ​​less than R0 or greater than R1 are identified as outliers and removed to resolve data anomaly issues. For missing data, two approaches are used depending on the situation: if the missing data ratio within the preprocessing window does not exceed 25%, it is filled using linear interpolation; if the missing data ratio exceeds 25%, the data is considered statistically insignificant, and the window is discarded to resolve the missing data issue. Finally, the preprocessed data stream S0 is output.

[0094] (4) Based on the rolling time window, the hydrological data is aggregated and the data size is reduced. First, a rolling window W1 with a size of 1 hour is established. The average value of the preprocessed data S0 within the window is taken to measure the concentration of the data within the window. Then the average value is output as data stream S1. Then, a rolling window W2 with a size of 1 day is established. The average value of S1 within the window is taken to measure the concentration of the data within the window. Then the average value is output as data stream S2.

[0095] (5) Statistical testing algorithms are applied to the window function ProcessWindowFunction to analyze the normality, stationarity, uniformity, and trend of hydrological sensor data. First, three sliding windows W3, W4, and W5 are opened for the aggregated data stream S2, with window sizes of 7 days (week), 10 days (ten-day periods), and 30 days (month), respectively, and a step size of 1 day. Then, within these three windows, a total of 12 testing algorithms targeting the four characteristics of normality, stationarity, uniformity, and trend are applied to the Flink window function ProcessWindowFunction, as shown in Table 1. Three algorithms are used to comprehensively evaluate each characteristic. Taking stationarity analysis as an example, the evaluation criteria are as follows:

[0096] ① For each feature, if the number of algorithms that do not reject the null hypothesis is greater than or equal to 2 out of the 3 algorithms used, then the hydrological data is determined to be stationary.

[0097] ② If the number of algorithms that do not reject the null hypothesis is 1 or 0, then the hydrological data is determined to be non-stationary. Finally, the determination results are output as three data streams: S3, S4, and S5.

[0098] (6) Use the FlinkKafkaProducer method to send the analysis results to the Kafka Result Topic, where the key is set to the sensor number.

[0099] Step 5: In the downstream message transmission layer, create a Result Topic to cache the analysis results from the data processing layer. Configure the Result Topic with 3 partitions and 3 replicas, storing them on 3 different Broker nodes. Partitioning improves Kafka's throughput, while replication enhances its reliability and prevents data loss.

[0100] Step 6: In the persistence layer, the analysis results are persisted using the time-series database IoTDB. This includes the following steps:

[0101] (1) Establish a storage group and divide the storage path for hydrological analysis results. Set the storage path for the result data to "root.object.sensor_type.sensor_id.analysis", where "root" is the default root node, "object" represents the observed object, such as rivers, reservoirs, etc., "sensor_type" represents the sensor type, such as water level, flow, precipitation sensors, etc., "sensor_id" represents the sensor number, used to locate a specific sensor, and "analysis" represents the analysis result, that is, the final result of each feature verification algorithm. Using this directory tree structure storage path, the entity data of different observed objects can be stored in different folders on the disk, thereby achieving physical isolation.

[0102] (2) Use Kafka Consumer to continuously monitor Kafka Result Topic. If new data flows into the Topic, immediately pull the data.

[0103] (3) Store the retrieved hydrological analysis results into the write buffer. To avoid the large network overhead caused by frequent data writing, optimize the IoTDB write strategy for the following two conditions:

[0104] ① The buffer contains 10 data entries;

[0105] ②The buffer contains data and it has been 1 second since the last write.

[0106] If the IoTDB write buffer meets either of the above two conditions, the data in the buffer will be inserted into the database, thereby reducing communication time and improving write efficiency.

[0107] Step 7: In the visualization layer, establish connections with the persistence layer and message transmission layer; simultaneously, utilize the WebSocket protocol and ECharts framework to visualize real-time analysis results and query historical analysis results. This includes the following steps:

[0108] (1) For real-time visualization of analysis results, the Kafka Consumer continuously listens to the Kafka Result Topic. If new data flows into the Topic, it is immediately pulled. Then, using the WebSocket protocol, a full-duplex TCP connection is established between the client and the server, and the server actively pushes real-time hydrological analysis results to the client. Then, in the client, the ECharts framework is used to plot the hydrological analysis results pushed by the client into charts in real time. Users can select to view the real-time analysis results according to the sensor number and characteristics.

[0109] (4) In terms of historical result query, the IoTDB native Java API is used to establish a connection between the visualization layer and the persistence layer. Historical analysis results can be queried according to time range, feature selection and sensor number, and then displayed as charts using ECharts.

Claims

1. A method for constructing a Flink-based hydrological sensor data analysis system, characterized in that, It comprises the following steps: (1) Install and configure the stream computing framework Apache Flink, the message queue Apache Kafka and the time series database Apache IoTDB, so that the stream computing framework Apache Flink, the message queue Apache Kafka and the time series database Apache IoTDB can communicate with each other; (2) In the data sending layer, use the Spring Boot framework to receive hydrological stream data from different sensors, convert the format of each piece of data into a unified JSON format, and cache it in the queue, and send the data to the upstream message transmission layer through the Kafka Producer; (3) In the upstream message transmission layer, create a Source Topic for caching source data transmitted by the data sending layer, and set the number of partitions and the number of replicas of the Topic according to the number of nodes in the cluster where the system is located and user demand; (4) In the data processing layer, use the time window mechanism and window function of Flink to realize the partitioning, preprocessing, aggregation and analysis calculation of hydrological sensor data; (5) In the downstream message transmission layer, create a Result Topic for caching result data transmitted by the data processing layer, and set the number of partitions and the number of replicas of the Topic according to the number of nodes in the cluster where the system is located and user demand; (6) In the persistence layer, use the time series database IoTDB to realize the persistence of the analysis results; (7) In the visualization layer, realize the connection with the persistence layer and the downstream message transmission layer; at the same time, use the WebSocket protocol and the ECharts framework to realize the visualization of real-time analysis results and the query of historical analysis results; In the (4), the data processing layer specifically comprises the following steps: Step (41), use the FlinkKafkaConsumer method to continuously monitor the Source Topic in the Kafka Broker, and if there is new data in the Topic, immediately pull the data; Step (42), use the KeyBy operator to partition the hydrological data according to the different sensor numbers, and isolate the data with different numbers from each other; Step (43), establish a rolling time window as a preprocessing window to preprocess the hydrological data and solve the problems of redundancy, repetition, abnormality and missing in the data; Step (44), based on the rolling time window, realize the aggregation of the hydrological data to reduce the data size; Step (45), apply the statistical test algorithm to the window function ProcessWindowFunction to realize the analysis of the normality, stationarity, homogeneity and trend of the hydrological sensor data; Step (46), use the FlinkKafkaProducer method to send the analysis results to the Result Topic of Kafka, wherein the key is set as the sensor number; In the step (43), the hydrological data preprocessing specifically comprises the following steps: Step (43a), a rolling time window is established as a preprocessing window, and the window size is determined according to the sampling frequency of the hydrological sensor; Step (43b), the JSON format data is parsed, and the timestamp, sensor number and hydrological data information are retained, and the data redundancy problem is solved; Step (43c), the timestamps of the data in the window are compared, and if the same timestamp data appears, the repeated data is deleted, and the data repetition problem is solved; Step (43d), the Tukey Test is used to detect outliers, and according to the interquartile range IQR, the first quartile Q1 and the third quartile Q3, the upper and lower boundaries R1 and R3 are calculated, and the values less than R1 or greater than R3 are determined as outliers and then discarded, and the data anomaly problem is solved; Step (43e), if the missing data proportion in the preprocessing window does not exceed the preset value, the linear interpolation filling method is used to fill it; if the missing data proportion exceeds the preset value, it is considered that the data has lost statistical significance, and the window is discarded, and the data missing problem is solved; Step (43f), the preprocessed data stream S0 is output; In the step (44), the hydrological data aggregation based on the time window is implemented, and specifically includes the following steps: Step (44a), a rolling time window W1 with a size of 1 hour is established, the average value of the preprocessed data stream S_0 in the window is taken to measure the centralized position of the data in the window, and then the average value is output as a data stream S1; Step (44b), a rolling window W2 with a size of 1 day is established, the average value of the data stream S1 in the window is taken to measure the centralized position of the data in the window, and then the average value is output as a data stream S2; In the step (45), the statistical analysis is implemented, and specifically includes the following steps: Step (45a), three rolling time windows W3, W4 and W5 are started for the aggregated data stream S2, and the window sizes are 7 days, 10 days and 30 days respectively, and the step length is 1 day; Step (45b), in the three windows, a total of 12 test algorithms for four characteristics of normality, stationarity, homogeneity and trend are applied to the window function ProcessWindowFunction of Flink, 3 algorithms are used for each characteristic for comprehensive judgment, and the judgment results are output as three data streams S3, S4 and S5; The three algorithms for normality evaluation are: Kolmogorov-Smirnov, Jarque-Bera and Geary's test; The three algorithms for stationarity evaluation are: Students T, Simple T and Mann-Whitney test; The three algorithms for trend evaluation are: Kendall Rank, Mann-Kendall and SROC test; The three algorithms for homogeneity evaluation are: Bayesian, Dunnett and Von-Neumann test.

2. The method of constructing a Flink-based hydrological sensor data analysis system according to claim 1, wherein, In the (2), the data sending layer is implemented, and specifically includes the following steps: Step (21), hydrological sensor data is deserialized to extract data information; Step (22), a queue is established to cache data information in the queue; Step (23), the components of the data information are parsed and converted into a unified JSON format; Step (24), the send method in Kafka Producer is used to set the key as the sensor number, and the data in the queue is sent to the upstream message transmission layer in an asynchronous manner according to the Topic name and partition number; Step (25), Java background service is started using the Spring Boot framework to continuously convert and output hydrological sensor data. 3.The method of constructing a Flink-based hydrological sensor data analysis system according to claim 1, wherein, In step (6), the persistent layer specifically includes the following steps: Step (61), a storage group is established to divide the storage path of the hydrological analysis result; Step (62), Kafka Consumer is used to continuously monitor the Result Topic of Kafka, and if new data flows into the Topic, the data is immediately pulled; Step (63), the pulled hydrological analysis result is written into the buffer, and if the number of data in the buffer exceeds 10 or the time since the last data writing exceeds 1 second, the data in the buffer is written into the database.

4. The method of constructing a Flink-based hydrological sensor data analysis system according to claim 1, wherein, In step (7), the visualization layer specifically includes the following steps: Step (71), Kafka Consumer is used to continuously monitor the Result Topic of Kafka, and if new data flows into the Topic, the data is immediately pulled; Step (72), a full-duplex TCP connection between the client and the server is established using the WebSocket protocol, and the server actively pushes real-time hydrological analysis results to the client; Step (73), in the client, the ECharts framework is used to draw the hydrological analysis results pushed by the client into a chart in real time; Step (74), IoTDB JDBC is used to establish a connection between the visualization layer and the persistent layer, and historical analysis results can be queried according to time range, feature selection, and sensor number, and ECharts is used to draw and display the chart.

5. A Flink-based hydrological sensor data analysis system implemented by the method of any one of claims 1-4, characterized in that, The data sending layer, the upstream message transmission layer, the data processing layer, the downstream message transmission layer, the persistent layer, and the visualization layer are arranged from top to bottom; The data sending layer collects stream data from different hydrological sensors and performs format conversion on these data, and transmits the fused data stream to the upstream message transmission layer; The upstream message transmission layer is used to realize data transmission between the data sending layer and the data processing layer, and uses message queue Kafka to connect the data sending layer and the data processing layer in an asynchronous message transmission manner; The data processing layer processes and analyzes hydrological sensor data and sends the analysis results to the downstream message transmission layer; The downstream message transmission layer is used for realizing data transmission of the data processing layer and the persistent layer and the visualization layer, utilizes a message queue Kafka, adopts an asynchronous message transmission mode, and connects the data processing layer and the persistent layer and the visualization layer; The persistent layer utilizes a time series database IoTDB to persist the hydrological sensor data analysis result output by the data processing layer; The visualization layer comprises real-time analysis result visualization and historical analysis result visualization, and provides real-time display and historical data query.

Images