A multi-frequency asynchronous streaming data collection method for intelligent water affairs system

Abstract

The application discloses a kind of multi-frequency asynchronous stream data collection methods for intelligent water system, comprising the following steps: S1: according to the output frequency requirement of intelligent water system, set the window time length in data collection module, set window time margin;S2: set the data pool storage space corresponding to different frequency, current data window and second data window storage space;S3: for the current data window or second data window corresponding to one kind of frequency, record current time and start current data window;S4: whenever a data arrives, read the information of current arrival data, judge the space that each data should store, and check whether current window needs to trigger window calculation;S5: re-adjust window, and update the start and end time of current window;S6: repeat steps S3 to S5 until terminate the operation of intelligent water system.The application can improve data utilization, adapt to different frequency sensors and instruments.

Description

Technical Field

[0001] This invention relates to the field of intelligent wastewater treatment technology, and in particular to a multi-frequency asynchronous streaming data collection method for intelligent water systems. Background Technology

[0002] Wastewater treatment facilities are indispensable in modern urban infrastructure and play a vital role in environmental protection and sustainable social development. With the development of artificial intelligence technology, various intelligent water systems are emerging. These intelligent water systems all require the collection of sensor data from wastewater treatment facilities, which is then input into models such as intelligent control, intelligent prediction, or intelligent decision-making to obtain corresponding output results.

[0003] To ensure that the water quality indicators after wastewater treatment meet discharge standards, it is necessary to install various water quality indicator detection instruments or sensors not only at the influent and effluent points, but also at key process sections. These different water quality indicator detection methods include those based on biochemical reactions, as well as those based on physical or chemical reactions. The reaction time required from water sample collection to obtaining test results varies, corresponding to differences in the frequency of data generation by instruments or sensors. Furthermore, some water quality indicators have multiple detection instruments or sensors based on different technologies. Due to operational control requirements for detection frequency and cost considerations, even for the same type of water quality indicator, different detection instruments or sensors may be used at different locations on the same wastewater treatment process line. The data generated by these sensors and instruments is streaming data (streaming data is a large, ordered sequence of data that is continuously generated and arrives in chronological order) and is transmitted through a mixed network of the smart water system.

[0004] In wastewater treatment processes, different sensors generate data at varying frequencies. The system output frequency required for operational control, including equipment operation, condition prediction, and adjustment decisions, often differs from the sensor data generation frequency. Current technologies mostly collect data at fixed frequencies, either by downsampling high-frequency data to align with the lowest frequency, or by selecting an appropriate intermediate frequency to downsample high-frequency data while interpolating low-frequency data. For downsampling, when the frequencies of different sensor data are not exactly multiples of each other, the question arises of how to select which data to use or how to interpolate to generate the sampling time data. Furthermore, interpolating low-frequency data to generate higher-frequency data can lead to significant additional delays in the system if interpolation is performed between already generated sensor data, resulting in untimely system output. Conversely, using extrapolation interpolation may cause the interpolation result to deviate significantly from future actual operating conditions.

[0005] For any type of smart water system, whether it's intelligent control, intelligent prediction, or intelligent decision-making, relying solely on a single data point at a particular moment as the representative input value for each output is insufficient and easily affected by outliers. A common approach in existing technologies is to collect all data over a period of time, calculate its average, and use this average as the representative value for that period when inputting it into the smart water system. This is effective to some extent, but it also has the side effect that occasional surges in inflow load data may be averaged out, leading to insufficient subsequent operational intensity and the risk of exceeding effluent standards.

[0006] Furthermore, the wastewater treatment process is lengthy, involving not only physical and chemical processes but also complex biochemical treatments. This complexity is reflected in smart water systems, where different components experience varying data transmission delays. This can lead to data delays during data collection. Consequently, while the smart water system is performing calculations, some data may still be in the transmission link, resulting in data loss and affecting the final output.

[0007] Therefore, there is an urgent need to provide a novel multi-frequency asynchronous streaming data collection method for smart water systems to solve the above problems. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a multi-frequency asynchronous streaming data collection method for intelligent water systems, which can improve data utilization, optimize system response time, and adapt to sensors and instruments of different frequencies.

[0009] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: to provide a method for multi-frequency asynchronous streaming data collection in a smart water system, comprising the following steps:

[0010] S1: Based on the output frequency F requirement of the intelligent water system, set the window time length ΔT and the window time margin ε in the data collection module, where ΔT = 1 / F and ε refers to the maximum time difference that the system allows for out-of-order arrival of data;

[0011] S2: Set up data pool storage space, current data window and second data window storage space for different frequencies. The data pool is used to store data of sensors or instruments with output frequencies lower than F; the current data window and second data window are used to store data of sensors or instruments with output frequencies equal to F.

[0012] S3: For the current data window or the second data window corresponding to one of the frequencies, record the current time Tb and open the current data window. At the same time, calculate the end time Te of the current window, Te = Tb + ΔT. Te is also the start time of the second window.

[0013] S4: Whenever a piece of data arrives, read the information of the currently arriving data, determine the space that each piece of data should be stored in, and check whether the data in the current window needs to trigger window calculations;

[0014] S5: Resize the window, making the original second window the new current window, making the original current window the new second window, and update the start and end times of the current window: Tb = Te, Te = Tb + ΔT;

[0015] S6: Repeat steps S3 to S5 until the smart water system is terminated.

[0016] In a preferred embodiment of the present invention, in step S1, the output frequency of the intelligent water system is not higher than the highest output frequency of the sensor or instrument.

[0017] In a preferred embodiment of the present invention, in step S2, when the intelligent water system has multiple outputs and the output frequencies are inconsistent, the inconsistent frequency F i Set up a current window and a second window with corresponding time lengths. The data pool is then used to store output frequencies lower than min(F). i (Data from sensors or instruments)

[0018] In a preferred embodiment of the present invention, in step S4, the information of the currently arriving data includes a timestamp, an indicator name, and an indicator value. The timestamp is denoted as T, which identifies the output time of the sensor or instrument for the data. The indicator name identifies the water quality indicator content of a certain location in the smart water system. The indicator value identifies the specific value of the data, that is, the indicator detection value output by the sensor or instrument.

[0019] Furthermore, in step S4, the specific steps for determining the space where each piece of data should be stored include:

[0020] S401: If the arriving data belongs to a sensor or instrument with an output frequency lower than F, record the data information corresponding to the data in the data pool; if the sensor or instrument to which the data belongs has not yet recorded it, create a new record; if it has already been recorded, update the timestamp field value and index value field value of the record; return to wait for the next data to be collected.

[0021] S402: If the timestamp of the arriving data is T≤Te, then record the data information corresponding to the data in the current window and return to wait for the next data to be collected; otherwise, if T>Te, then record the data information corresponding to the data in the second window and proceed to step S403.

[0022] S403: Calculate dT = T - ε, where dT is used to confirm whether data has arrived beyond the window time margin ε;

[0023] S404: If dT≤Te, return and wait to collect the next data; otherwise, proceed to the next step to check whether the data in the current window needs to trigger window calculation.

[0024] In a preferred embodiment of the present invention, the step of triggering window calculation in step S4 is as follows:

[0025] For each sensor or instrument, check all its data in the current window, eliminate abnormal data, and then calculate the average value of the remaining data. Use this average value as the representative value of the sensor or instrument's data within the current window time period and send it to the control, prediction, or decision-making calculation module. At the same time, the data in the data pool is also sent to the control, prediction, or decision-making calculation module.

[0026] The beneficial effects of this invention are as follows: This invention proposes a multi-frequency streaming data collection method based on a window mechanism and applies it to a smart water management system. By collecting data through the window mechanism of this invention, data from sensors / instruments with different acquisition and output frequencies can be fully utilized and effectively coordinated within the smart water management system, while also eliminating abnormal data. Furthermore, considering the issue of data latency, this method reserves a window time margin ε to facilitate the collection of late data, thereby effectively preventing data loss and improving the accuracy and reliability of the smart water management system. Specifically:

[0027] (1) Improve data utilization: By designing a window mechanism and introducing tolerance time, the problem of data transmission delay is effectively solved, ensuring that the sensor / instrument data that should be included in the calculation can be collected, thereby improving the data utilization.

[0028] (2) Optimize system response time: By setting a reasonable tolerance time, both excessive system waiting time is avoided and data integrity is ensured, thus achieving a balance between system response time and output accuracy.

[0029] (3) Adapting to sensors and instruments of different frequencies: This method can flexibly respond to sensor and instrument data of different frequencies, ensuring that different frequency data between sensors / instruments can work together effectively, improving the system's compatibility, flexibility and generalization. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the multi-frequency asynchronous streaming data collection method for intelligent water systems according to the present invention.

[0031] Figure 2This is a schematic diagram of the window mechanism described above;

[0032] Figure 3 This is a schematic diagram illustrating the relationship between the settings of different window time lengths ΔT and tolerance time ε and the data collection rate for simulated data.

[0033] Figure 4 This is a schematic diagram illustrating the relationship between the settings of different window time lengths ΔT and tolerance time ε and the data collection rate for simulated data 2. Detailed Implementation

[0034] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0035] Please see Figure 1 The embodiments of the present invention include:

[0036] In intelligent water systems, the data streams generated by various sensors and instruments are continuous, from system startup to shutdown. To accommodate different data acquisition frequencies of sensors / instruments and varying system output frequencies, and to avoid data loss due to transmission delays, this invention designs a window mechanism:

[0037] The window described here has a start and end time; each control, prediction, or decision calculation relies only on the data within a window from the start time to the end time; the window size, or the length of the window, is determined by the required output frequency for control, prediction, or decision-making in the system. Specifically, if the required output frequency is once every n minutes, then the window size is n minutes; for each window, all valid sensor or instrument data within that window time period are collected and processed, and then the average value after excluding abnormal data is calculated. This average value is used as the representative value of the data within that time period and participates in the control, prediction, or decision-making calculation.

[0038] More specifically, the data output by the sensor or instrument is recorded as a structure containing a timestamp, an indicator name, and an indicator value, expressed in the following structure format containing three "domain name:domain value" pairs:

[0039]

[0040] The timestamp identifies the sensor or instrument output time of this data. The field name is timestamp, and the field value can be a string containing the year, month, and day separated by hyphens (YYYY for 4-digit year, MM for 2-digit month, DD for 2-digit day), a space, and the hours, minutes, and seconds separated by colons (hh for 2-digit hours in 24-hour format, mm for 2-digit minutes, ss for 2-digit seconds). The field value can also be expressed as a time-type numerical value (unit: seconds).

[0041] The indicator name identifies which water quality indicator is being measured at which location. The domain name is sensor_name, and the field value is a string containing the name (XX) of the indicator being detected by the sensor or instrument and a string (pp) indicating its location on the wastewater treatment process line.

[0042] The index value identifies the specific numerical value of the data, that is, the index detection value output by the sensor or instrument.

[0043] A typical data record example is as follows:

[0044]

[0045] or

[0046]

[0047]

[0048] This indicates that the data was output at 17:30:23 on September 25, 2024, and the total nitrogen value of the effluent was 8.21.

[0049] See Figure 1 The multi-frequency streaming data collection method based on the window mechanism of the present invention is used in the data collection module of a smart water system, and includes the following steps:

[0050] S1: When the system starts up, determine the system's output frequency F, and set the window time length ΔT and window time margin ε in the data collection module according to the output frequency F requirement of the smart water system. Here, ΔT = 1 / F, and ε refers to the maximum time difference that the system allows for out-of-order arrival of data. It is determined according to the transmission characteristics of the communication network used by the system in the actual application scenario to ensure that the data delay problem can be effectively dealt with while maintaining the efficient operation of the system.

[0051] In wastewater treatment engineering practice, the required output frequency of a smart water system must not exceed the maximum output frequency of the sensor or instrument; otherwise, it would be meaningless. In other words, there must be at least one data point within the window time length ΔT. Therefore, the range of values ​​for F or ΔT can be as low as the maximum output frequency of the sensor or instrument, and as high as 120 minutes, or 2 hours (this 2-hour figure comes from the "GB 18918-2002 Standard for Pollutant Discharge from Urban Wastewater Treatment Plants," which stipulates that water quality sampling should be conducted at least once every 2 hours).

[0052] Furthermore, in engineering practice, network latency is a common problem that can cause data with earlier timestamps to arrive late. This latency can lead to a situation where data with later timestamps arrives before the window expires, triggering window calculations and transmitting the results to the system's control, prediction, or decision-making units, while data with earlier timestamps, due to network latency, has not yet arrived and cannot be included in the calculations in time, resulting in a biased final result.

[0053] To avoid the above situation, the calculation of the window in the method of the present invention is not triggered immediately after the last piece of data is received, but a certain time margin ε, or tolerance time ε, is reserved to receive data that may be late. That is, the tolerance time refers to the maximum time difference that the system allows data to arrive out of order.

[0054] S2: Set up data pool storage space, current data window, and second data window storage space corresponding to different frequencies. The data pool is used to store data from sensors or instruments with output frequencies lower than F; the current data window and second data window are used to store data from sensors or instruments with output frequencies equal to F. Sensor / instrument data is classified and recorded according to different output frequencies, which facilitates the management of data required for calculations and avoids the loss of low-frequency data when switching high-frequency data windows. For low-frequency sensor / instrument data, records are stored in the data pool using a unique recording method, ensuring that old data remains valid until new data arrives, thus guaranteeing the integrity of the data required by the system for control, prediction, or decision-making calculations.

[0055] When a smart water system has multiple outputs and the output frequencies are inconsistent, then the inconsistent frequency F i Set up a current window and a second window with corresponding time lengths. The data pool is then used to store output frequencies lower than min(F). iThe data comes from sensors or instruments. The second data window stores all data beyond the current data window, not just data within time ε. When new data arrives and the time exceeds (the last time of the current window Te+ε), the calculation of the current data window is triggered.

[0056] Regardless of whether the system outputs only one frequency or multiple frequencies, the subsequent operations in the data collection module follow the same logic, regardless of whether there is one or multiple windows. Therefore, the following details the subsequent methods of this invention using a single window.

[0057] S3: For the current data window or the second data window corresponding to one of the frequencies, record the current time Tb and open the current data window. At the same time, calculate the end time Te of the current window, Te = Tb + ΔT. Te is also the start time of the second window.

[0058] S4: Whenever a piece of data arrives, read the information of the currently arriving data, including the timestamp, indicator name, and indicator value. The timestamp value is denoted as T. Determine the space that each piece of data should be stored in, and check whether the data in the current window needs to trigger window calculations. The specific steps are as follows:

[0059] S401: If the arriving data belongs to a sensor or instrument with an output frequency lower than F, then record the data information corresponding to the data (in the form of the aforementioned data structure) in the data pool: if the sensor or instrument to which the data belongs has not yet been recorded, then create a new record; if it has already been recorded, then update the timestamp field value and index value field value of the record; return to wait for the next data to be collected.

[0060] S402: If the timestamp of the arriving data is T≤Te, then the data information corresponding to the data (in the form of the aforementioned data structure) is recorded in the current window, and the process returns to wait for the collection of the next data; otherwise, if T>Te, then the data information corresponding to the data is recorded in the second window, and the process proceeds to step S403.

[0061] S403: Calculate dT = T - ε, where dT is used to confirm whether data has arrived beyond the window time margin ε;

[0062] S404: If dT≤Te, return and wait to collect the next data; otherwise, proceed to the next step to check if the data in the current window needs to trigger window calculation.

[0063] For each sensor or instrument, check all its data in the current window, eliminate abnormal data, and then calculate the average value of the remaining data. Use this average value as the representative value of the sensor or instrument's data within the current window time period and send it to the control, prediction, or decision-making calculation module. At the same time, the data in the data pool is also sent to the control, prediction, or decision-making calculation module.

[0064] S5: Rearrange the windows, turning the original second window into the new current window and the original current window into the new second window. This can be achieved by changing the value of the window pointer. Each window object has a window pointer. When you need to change the window, swap the window objects that the two pointers point to, and the current window will become the second window, and the second window will become the current window. At the same time, update the start and end times of the current window: Tb = Te, Te = Tb + ΔT.

[0065] Switching between the current window and the second window is achieved by changing the value of the window pointer, avoiding the time and space consumption caused by moving data between the two window storage spaces, thus improving the efficiency of system operation and calculation.

[0066] S6: Repeat steps S3 to S5 until the smart water system is terminated.

[0067] The following is combined Figure 2 The window mechanism of this invention will be explained in detail.

[0068] Figure 2 In this context, the start time of the current window is Tb, and the end time is Te. It acts as a container, collecting sensor / instrument data of all corresponding frequencies within the current window period. ε is the tolerance time, which is the maximum time difference between out-of-order arrivals of data that the system can tolerate for normal and effective operation. When new data arrives, and the timestamp T of the new data minus the tolerance time ε is greater than the window end time Te, it can be considered that all data within the current window period has arrived and been collected. At this point, window calculation can be triggered, inputting all the data required for control, prediction, or decision-making into the system's calculation module.

[0069] In the window mechanism of this invention, window calculation is passively triggered; that is, it is only checked whether the data in the current window should be calculated whenever a new piece of data arrives. The advantage of this approach is that it eliminates the need for real-time monitoring of whether the current window should be closed, saving computational resources and simplifying program implementation.

[0070] The method described in this invention not only effectively processes multi-frequency asynchronous data streams from different sensors / instruments, but also ensures that control, prediction, or decision-making is based on filtered and smoothed data, thereby improving the accuracy and reliability of control, prediction, or decision-making. Furthermore, the window mechanism designed in this invention implicitly expands low-frequency data, aligning data of different frequencies and enabling the system to utilize the most appropriate data for control, prediction, or decision-making calculations.

[0071] In this invention example, the relationship between window time length ΔT, tolerance time ε, and sensor / instrument data collection rate was also explored through simulation experiments.

[0072] Each experiment uses simulation tools to generate a batch of simulated multi-frequency asynchronous streaming data with different network delays. Different window time lengths ΔT and tolerance times ε are set respectively. The arrival time of the data is examined, and the data arriving within the tolerance time is statistically analyzed to calculate the corresponding data collection rate.

[0073] See Figure 3 and Figure 4 The figure illustrates the relationship between the data collection rate and different window time lengths ΔT and tolerance time ε for two batches of simulation data. The three axes in the figure represent the window time length ΔT, tolerance time ε, and data collection rate, respectively.

[0074] from Figure 3 and Figure 4 The results of the two experiments presented show that, for the same window duration, a shorter tolerance time results in a lower data collection rate; similarly, for the same tolerance time setting, a shorter window duration leads to a lower data collection rate. Uncollected values ​​are still in transit during window computation, resulting in data loss and causing the system's control, prediction, or decision-making computations to fail. Increasing the tolerance time ε (i.e., increasing the time spent waiting for data in transit) or increasing the window size ΔT (a larger window reduces the impact of data in transit on subsequent computations) can both increase the data collection rate. This experiment demonstrates the necessity of waiting a certain period after the current window closes before triggering window computation.

[0075] This invention utilizes a window mechanism for data classification, recording, and storage, enabling intelligent water systems to receive multi-frequency asynchronous sensor / meter data from various sources as input, and to meet the needs of different locations for control, prediction, or decision-making outputs at different frequencies during water management operations. This provides intelligent water systems employing the method of this invention with sufficient flexibility and versatility.

[0076] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for multi-frequency asynchronous streaming data collection in a smart water system, characterized in that, Includes the following steps: S1: Based on the output frequency F requirement of the intelligent water system, set the window time length ΔT and the window time margin ε in the data collection module, where ΔT = 1 / F and ε refers to the maximum time difference that the system allows for out-of-order arrival of data; S2: Set up data pool storage space, current data window and second data window storage space for different frequencies. The data pool is used to store data of sensors or instruments with output frequencies lower than F; the current data window and second data window are used to store data of sensors or instruments with output frequencies equal to F. S3: For the current data window or the second data window corresponding to one of the frequencies, record the current time Tb and open the current data window. At the same time, calculate the end time Te of the current window, Te = Tb + ΔT. Te is also the start time of the second window. S4: Whenever a piece of data arrives, read the information of the currently arriving data, determine the space that each piece of data should be stored in, and check whether the data in the current window needs to trigger window calculations; S5: Resize the window, making the original second window the new current window, making the original current window the new second window, and update the start and end times of the current window: Tb = Te, Te = Tb + ΔT; S6: Repeat steps S3 to S5 until the smart water system is terminated. In step S4, the information read from the currently arriving data includes the timestamp, indicator name, and indicator value. The timestamp, denoted as T, identifies the output time of the data from the sensor or instrument. The indicator name identifies the water quality indicator content of a specific location within the smart water system. The indicator value identifies the specific numerical value of the data, i.e., the indicator detection value output by the sensor or instrument. In step S4, the specific steps for determining the space where each piece of data should be stored include: S401: If the arriving data belongs to a sensor or instrument with an output frequency lower than F, record the data information corresponding to the data in the data pool; if the sensor or instrument to which the data belongs has not yet recorded it, create a new record; if it has already been recorded, update the timestamp field value and index value field value of the record; return to wait for the next data to be collected. S402: If the timestamp of the arriving data is T≤Te, then record the data information corresponding to the data in the current window and return to wait for the next data to be collected; otherwise, if T>Te, then record the data information corresponding to the data in the second window and proceed to step S403. S403: Calculate dT = T - ε, where dT is used to confirm whether data has arrived beyond the window time margin ε; S404: If dT≤Te, return and wait to collect the next data; otherwise, proceed to the next step to check whether the data in the current window needs to trigger window calculation.

2. The multi-frequency asynchronous streaming data collection method for intelligent water systems according to claim 1, characterized in that, In step S1, the output frequency of the intelligent water system is not higher than the highest output frequency of the sensor or instrument.

3. The multi-frequency asynchronous streaming data collection method for intelligent water systems according to claim 1, characterized in that, In step S2, when the intelligent water system has multiple outputs and the output frequencies are inconsistent, the inconsistent frequency F i Set up a current window and a second window with corresponding time lengths. The data pool is then used to store output frequencies lower than min(F). i (Data from sensors or instruments) 4. The multi-frequency asynchronous streaming data collection method for intelligent water systems according to claim 1, characterized in that, In step S4, the step that triggers the window calculation is as follows: For each sensor or instrument, check all its data in the current window, exclude abnormal data, and then calculate the average value of the remaining data. Use this average value as the representative value of the sensor or instrument's data within the current window time period and send it to the control, prediction, or decision-making calculation module. At the same time, the data in the data pool is also sent to the control, prediction, or decision-making calculation module.

Images