Water affair monitoring data additional recording method and device, electronic equipment and medium

Abstract

The invention provides an additional recording method and device for water affair monitoring data, electronic equipment and a medium. The method comprises the steps of obtaining time sequence data of water affair monitoring; segmenting the time sequence data into a series of data windows according to a preset time window size and a sliding step length; counting data points in each data window to judge whether missing data points exist in the window or not; and when missing data points are detected, marking the corresponding windows as abnormal windows, and taking corresponding additional recording measures based on the number of the missing data points. The missing or damaged data points in the water affair monitoring data are effectively identified and repaired, and the integrity and accuracy of the data are ensured. According to the method, a small number of missing value windows and a batch of missing value windows are distinguished through the preset threshold value, and a corresponding additional recording strategy is adopted, so that the problem of data missing under different conditions can be adaptively solved, and the robustness under the situation of water affair monitoring data loss is remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data cleaning, and in particular to a method, device, electronic equipment and medium for supplementing water monitoring data. Background Art

[0002] In the field of outdoor water environment monitoring, data collection and transmission are key steps to achieve environmental monitoring. However, due to the particularity of the outdoor water environment, monitoring equipment often needs to perform long-term data collection in open and complex waters. Such a working environment poses multiple challenges to data transmission, especially signal instability, which often leads to data loss or damage during transmission and storage. Traditional monitoring systems often rely on strengthening signal transmission capabilities and improving the reliability of storage devices to solve the above problems. However, these methods usually require high costs and cannot completely avoid the risk of data loss or damage. In addition, once data is lost or damaged during transmission, the existing system lacks an effective mechanism to identify and repair these problems, which affects the integrity and reliability of the data set.

[0003] Remote transmission devices (gateways) play a vital role in this process. They are the bridge between monitoring equipment and data processing centers. However, abnormalities in the gateway device itself can also affect the integrity and accuracy of the data. For example, hardware failures, software errors, or external interference in the gateway device can all lead to data loss or damage. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present disclosure is to provide a method, device, electronic equipment and medium for re-recording water monitoring data to solve the technical problem that data cannot be re-recorded when lost.

[0005] The first aspect of the present disclosure provides a method for supplementing water monitoring data, comprising: acquiring time series data of water monitoring; dividing the time series data into a series of data windows according to a preset time window size and a sliding step; counting the data points in each data window to determine whether there are missing data points in the window; when missing data points are detected, marking the corresponding window as an abnormal window, and taking corresponding supplementary recording measures based on the number of missing data points.

[0006] In an embodiment of the first aspect, the data points in each data window are counted to determine whether there are missing data points in the window; a first threshold and a second threshold for the number of data points in the window are set, wherein the first threshold is higher than the second threshold; when the number of data points in the window is lower than the first threshold and higher than the second threshold, the window is determined to be a small number of missing value windows; when the number of data points in the window is lower than the second threshold, the window is determined to be a batch missing value window.

[0007] In an embodiment of the first aspect, the supplementary recording measures for the small number of missing value windows include: obtaining data points of the front and rear adjacent windows of the small number of missing value windows to form a candidate data set; calculating the distance between each data point in the candidate data set and each missing data point in the small number of missing value windows; based on the calculated distance, selecting the K data points with the closest distance from the candidate data set to form a supplementary data set, where K is a preset positive integer; calculating the weighted average of the K data points in the supplementary data set, and supplementing the corresponding missing data points in the small number of missing value windows with the weighted average.

[0008] In an embodiment of the first aspect, when calculating the distance between each data point in the candidate data set and the missing data points in a small number of missing value windows, the Minkowski distance formula is used; and the weights of the K data points are allocated according to the inverse of the Minkowski distance of each data point to the missing data point.

[0009] In an embodiment of the first aspect, the filling-in processing of the batch missing value windows includes: inputting the batch missing value windows into a pre-trained missing value prediction model to obtain predicted values ​​of missing data points; assigning weights to the predicted values ​​of the batch missing value windows according to preset conditions; calculating the weighted average of the predicted values ​​according to the assigned weights; and filling the corresponding missing data points in the batch missing value windows with the weighted average.

[0010] In an embodiment of the first aspect, the preset conditions include: one or more combinations of: a proportion of valid data in the batch missing value window, and a length of an interval between the batch missing value window and the current time.

[0011] In an embodiment of the first aspect, the training step of the missing value prediction model includes: extracting time series data of windows that are not marked with abnormalities as a training data set; inputting the training data set into a pre-built missing value prediction model to obtain a prediction result; optimizing a loss function based on the difference between the prediction result and the actual data value; and repeating the optimization process until the value of the loss function converges to a predetermined threshold.

[0012] In an embodiment of the first aspect, the missing value prediction model adopts an LSTM architecture, and the missing value prediction model includes: an input layer for receiving time series data; one or more LSTM layers for learning and capturing long-term dependencies in time series data; and an output layer for generating a final prediction value.

[0013] The second aspect of the present disclosure discloses a device for supplementing water monitoring data, including: an acquisition module, used to acquire time series data of water monitoring; a window division module, used to divide the time series data into a series of data windows according to a preset time window size and a sliding step; a judgment module, used to count the data points in each data window to determine whether there are missing data points in the window; and a supplementary recording module, used to mark the corresponding window as an abnormal window when missing data points are detected, and take corresponding supplementary recording measures based on the number of missing data points.

[0014] The third aspect of the present disclosure discloses an electronic device, comprising: a processor and a memory; wherein the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method for supplementing water monitoring data described in any embodiment of the first aspect.

[0015] A fourth aspect of the present disclosure discloses a computer-readable storage medium having a computer program stored thereon. When the program is executed by an electronic device, the method for supplementing water monitoring data described in any embodiment of the first aspect is implemented.

[0016] As described above, the water affairs monitoring data supplementation method, device, electronic device and medium disclosed in the present disclosure at least include the following technical effects:

[0017] (1) Through the supplementary recording method disclosed in the present invention, missing or damaged data points can be effectively identified and repaired, ensuring the integrity and accuracy of the data set, and providing a reliable basis for subsequent analysis.

[0018] (2) By presetting thresholds to distinguish between a small number of missing value windows and a large number of missing value windows and adopting different supplementary recording strategies, the system can adaptively handle data missing problems in different situations and enhance the robustness of the system in the face of data loss.

[0019] (3) The Minkowski distance formula and weighted average method are used to supplement the missing data points in a small number of missing value windows, and the trained LSTM model is used to predict the missing data points in a batch of missing value windows, which optimizes the data processing process and improves the efficiency and accuracy of supplementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a method for supplementing water monitoring data in an embodiment of the present disclosure is shown.

[0021] Figure 2 A schematic diagram showing a flow chart of determining whether a window is missing points in an embodiment of the present disclosure.

[0022] Figure 3 A schematic diagram of the process of supplementing a small number of missing value windows in one embodiment of the present disclosure is shown.

[0023] Figure 4 A schematic diagram of the process of supplementing batch missing value windows in one embodiment of the present disclosure is shown.

[0024] Figure 5 Schematic diagram showing the process of training a missing value prediction model in an embodiment of the present disclosure

[0025] Figure 6 A schematic diagram of a module showing a device for supplementing recording water monitoring data in an embodiment of the present disclosure.

[0026] Figure 7 A schematic diagram of the circuit structure of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0029] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] like Figure 1 As shown, a flow chart of a method for supplementing water monitoring data in an embodiment of the first aspect of the present disclosure is shown, including steps S11-S14.

[0031] Step S11: Acquire time series data of water affairs monitoring.

[0032] Specifically, in the field of water monitoring, time series data include key indicators such as water quality parameters and water flow. These data are recorded in time series and continuously and orderly reflect the changing process of water resources on the time axis. Using these time series data, trends, periodic changes and abnormal events can be analyzed, thus providing a scientific basis for monitoring and evaluating water resources.

[0033] In water monitoring, time series data covers multiple indicators closely related to water resources, such as water level, flow rate, concentration of chemical substances in water quality (such as dissolved oxygen, ammonia nitrogen, phosphate, etc.), pH value, turbidity and temperature. These parameters are crucial to assessing the health of water resources. For example, by analyzing the time series data of water quality parameters, changes in the degree of water pollution can be observed and timely treatment measures can be taken; monitoring of water flow helps to understand the distribution and utilization of water resources, and plays an important role in flood warning and drought management.

[0034] In order to obtain accurate time series data, it is usually necessary to rely on a series of sensors, monitoring stations and automatic sampling equipment. These devices can collect key data points regularly or continuously at fixed time intervals and record them in digital form. By processing and analyzing these data, one or more sets of monitoring indicator combinations can be designed, such as a combination of flow rate and flow velocity, chemical oxygen demand (COD) and ammonia nitrogen (NH3-N or NH4 + -N) to meet specific business needs.

[0035] Step S12: Divide the time series data into a series of data windows according to a preset time window size and sliding step size.

[0036] Specifically, when processing time series data, splitting the data into a series of data windows involves two parameters: the time window size and the sliding step size. The time window size defines the time range covered by each data window, while the sliding step size specifies the time interval between adjacent data windows.

[0037] The time window size is usually determined based on the sampling frequency of the data and the analysis objectives. For example, in water quality monitoring, if the data is sampled once an hour, then a suitable time window size may be 24 hours, which ensures that each data window contains a sufficient number of data points to calculate the daily average. The selection of the time window size needs to balance the statistical significance of the data and the sensitivity to trend changes. Larger time windows may mask short-term changes, while smaller time windows may cause large fluctuations in the data, which is not conducive to the analysis of long-term trends.

[0038] The sliding step size determines the degree of overlap between data windows. A smaller sliding step size will increase the number of data windows and improve the resolution of the analysis, but it will also increase the amount of calculation. A larger sliding step size will reduce the number of data windows and reduce the computational burden, but may miss some key dynamic changes. In practical applications, the choice of sliding step size is often based on the expected speed of data changes and the consideration of data processing capabilities.

[0039] Specifically, in the process of creating a data window, suppose there is a list of time series data, where each data point has a timestamp. First, the start and end time points of each window are determined according to the preset time window size. Then, the sliding step is used to create the next data window, and the process is repeated until all the data are assigned to a data window.

[0040] Step S13: Count the data points in each data window to determine whether there are any missing data points in the window.

[0041] Specifically, each data window represents a specific time interval, which contains all the data points collected during that time period. For each data window, the data points are counted. This is usually achieved by traversing all data points in the window and accumulating their number. During the counting process, the validity of the data points needs to be considered, that is, any obvious erroneous values ​​or outliers are excluded, which may be marked as invalid or missing.

[0042] Once the count is complete, the total number of data points is compared to the expected number. The expected number is based on the size of the time window and the sampling frequency of the data. For example, if the data is sampled once an hour and the time window size is 24 hours, the expected number of data points should be 24. If the actual number of counts is less than expected, this indicates that there are missing data points within that data window.

[0043] In some embodiments, step S13 includes steps S131-S133, such as Figure 2 As shown, a flow chart of step S13 is shown, wherein:

[0044] Step S131: setting a first threshold and a second threshold for the number of data points in the window, wherein the first threshold is higher than the second threshold.

[0045] Specifically, these two thresholds are predetermined based on the characteristics of historical data and business requirements through scientific statistical analysis methods and are used for subsequent data window classification.

[0046] Step S132: when the number of data points in the window is lower than the first threshold and higher than the second threshold, the window is determined to be a window with a small number of missing values.

[0047] Specifically, when the number of data points in the window is lower than the first threshold and higher than the second threshold, the window is determined to be a window with a small number of missing values, which means that the number of data points in the window is not complete, but the degree of missing is relatively light.

[0048] Step S133: When the number of data points in the window is lower than the second threshold, the window is determined to be a batch missing value window.

[0049] Specifically, when the number of data points in the window is lower than the second threshold, the window is determined to be a batch missing value window, which indicates that the number of missing data points in the window is large and a more complex supplementary recording strategy is required.

[0050] By setting these two thresholds, the system can automatically identify different types of missing values ​​and adopt corresponding supplementary strategies according to the specific circumstances, thereby improving the completeness of the data set and the accuracy of subsequent analysis.

[0051] Step S14: When missing data points are detected, the corresponding window is marked as an abnormal window, and corresponding supplementary recording measures are taken based on the number of missing data points.

[0052] Specifically, the abnormal windows can be divided into a small number of missing value windows and a large number of missing value windows, and different supplementary recording measures can be taken for different types of abnormal windows.

[0053] In some embodiments, the supplementary recording measures for a small number of missing value windows include steps S21-S24, such as Figure 3 As shown, a flow chart showing the supplementary admission measures is shown, in which:

[0054] Step S21: obtaining data points of the windows adjacent to and before the small number of missing value windows to form a candidate data set.

[0055] Specifically, determine the location of the window with a small number of missing values, that is, find the time point where a small number of missing values ​​appear in the time series. Locate the window before and after the window with a small number of missing values, and the data points in these adjacent windows will become part of the candidate data set. Extract the data points of the adjacent windows before and after the window with a small number of missing values, merge the extracted data points, and form a set containing multiple data points, namely the candidate data set.

[0056] Step S22: Calculate the distance between each data point in the candidate data set and each missing data point in a small number of missing value windows.

[0057] Specifically, the system determines the number and location of missing data points in a small number of missing value windows, and then extracts each data point in the candidate dataset. These data points are a key source of information for predicting and filling in missing data in a small number of missing value windows. For each missing data point, the system calculates the distance between it and each data point in the candidate dataset. This distance can be based on numerical differences or other metrics, such as similarity of time series.

[0058] Distance can be calculated using various mathematical methods, such as Euclidean distance, Manhattan distance, Minkowski distance, etc. These methods can help the system quantify the degree of difference between two data points. By calculating the distance between each missing data point and each data point in the candidate data set, the system can obtain a distance matrix that contains the distance information between all missing data points and candidate data points.

[0059] Step S23: Based on the calculated distance, select K data points with the closest distance from the candidate data set to form a complementary data set, where K is a preset positive integer.

[0060] Specifically, the system will select the K closest data points from the candidate data set according to the distance information calculated in step S22. Here, K is a preset positive integer representing the number of selected supplementary data points. The system will sort the data points in the candidate data set according to the size of the distance, and the data points with smaller distance will have higher priority.

[0061] Step S24: Calculate the weighted average of the K data points in the supplementary data set, and use the weighted average to supplement the corresponding missing data points in the small number of missing value windows.

[0062] Specifically, the weighted average of K data points in the supplementary data set is calculated. The calculation of the weighted average takes into account the distance between data points, that is, the closer the data points are, the higher the weight is. The calculated weighted average is then used to supplement the corresponding missing data points in a small number of missing value windows.

[0063] In some embodiments, when calculating the distance between each data point in the candidate data set and the missing data points in the small number of missing value windows, the Minkowski distance formula is used. The Minkowski distance is a general form of a class of distance metrics, which includes Euclidean distance, Manhattan distance, etc. as special cases. The selection of appropriate Minkowski distance parameters can be adjusted according to actual conditions to best reflect the similarity between data points.

[0064] In a preferred embodiment of the present invention, the weights of the K data points are assigned according to the inverse of the Minkowski distance of each data point to the missing data point. This means that data points with closer distances are given higher weights because they are considered more reliable sources of supplementary recording. This method ensures the accuracy and reliability of the supplementary recording data.

[0065] The supplementary recording measures for batch missing value windows include steps S31-S34, such as Figure 4 As shown, a flow chart showing the supplementary admission measures is shown, in which:

[0066] Step S31: input the batch missing value windows into a pre-trained missing value prediction model to obtain the predicted values ​​of the missing data points.

[0067] Specifically, identify windows with batch missing values, where the number of data points in these windows is lower than the preset second threshold, indicating that there is a large amount of data missing, and then input the data of these batch missing value windows into a pre-trained missing value prediction model. This model is trained based on a large number of time series data of normal data windows, and can capture patterns and trends in time series data. The model processes the input data and generates predicted values ​​for missing data points. These predicted values ​​are based on the model's learning and understanding of normal data windows, as well as its ability to predict time series data.

[0068] In some embodiments, the training step of the missing value prediction model includes steps S41-S44, such as Figure 5 As shown, a schematic diagram of the model training process is shown, where:

[0069] Step S41: extracting time series data of windows that are not marked as abnormal as a training data set.

[0070] Specifically, we first need to prepare a training data set. This involves extracting data from windows that are not marked as abnormal from the existing time series data. These data will be used as training data sets to train the missing value prediction model. The selection of windows that are not marked as abnormal is to ensure the quality and accuracy of the training data, because abnormal data may affect the prediction performance of the model.

[0071] Step S42: input the training data set into the pre-built missing value prediction model to obtain the prediction result.

[0072] Specifically, in the process of training the missing value prediction model, the extracted training data set is input into the pre-built model. This model can be any type of machine learning model, such as decision tree, random forest, neural network, etc., depending on the characteristics of the data and business needs. The output of the model is the predicted value for the data points that may be missing in each data window.

[0073] Step S43: Optimizing the loss function based on the difference between the prediction result and the actual data value.

[0074] Specifically, during the model training phase, a loss function is used to evaluate the difference between the model's predictions and the actual data values. The choice of loss function depends on the type of model and the nature of the prediction task. For example, for regression problems, mean square error (MSE) or absolute error may be used as the loss function. By minimizing the loss function, the model's parameters can be gradually optimized.

[0075] Step S44: Repeat the optimization process until the value of the loss function converges to a predetermined threshold.

[0076] Specifically, the model training process involves repeated iterations, and each iteration adjusts the model parameters according to the value of the loss function. This process continues until the value of the loss function converges to a predetermined threshold, or a preset number of training rounds is reached. The convergence criterion can be set according to actual needs and model performance to ensure that the model has sufficient predictive power.

[0077] Exemplarily, the missing value prediction model adopts an LSTM architecture, and the missing value prediction model includes: an input layer for receiving time series data; one or more LSTM layers for learning and capturing long-term dependencies in time series data; and an output layer for generating a final prediction value.

[0078] Specifically, the input layer is responsible for receiving time series data, which usually includes timestamps and related monitoring indicators, such as water quality parameters, flow, etc. The input layer converts this data into a format that the model can process and passes it to the next layer.

[0079] The LSTM layer is the core part of the model, which includes LSTM units that are able to learn and capture long-term dependencies in time series data. LSTM units control the flow of information through gating mechanisms (forget gate, input gate, and output gate) to remember long-term information and ignore noise. One or more LSTM layers can be stacked together to enhance the learning and expression capabilities of the model.

[0080] The output layer is responsible for generating the final predicted value, which is the value added to the missing data point. The output layer can be a simple linear layer or other types of layers, depending on the specific needs of the model.

[0081] Through this LSTM architecture, the missing value prediction model can effectively learn patterns and trends in time series data, thereby accurately predicting and filling in missing data points. This model is particularly suitable for processing data with complex time dependencies, such as water monitoring data, where indicators such as water quality parameters and flow rates change over time and may be affected by multiple factors.

[0082] Step S32: assigning weights to the predicted values ​​of the batch missing value windows according to preset conditions.

[0083] Specifically, according to the preset conditions, weights are assigned to the predicted values ​​of the batch missing value windows. In some embodiments, the preset conditions may include multiple factors, such as the proportion of valid data in the window and the length of the interval between the window and the current time. Among them, the proportion of valid data refers to the ratio of the number of data points actually existing in the window to the number of expected data points, and this ratio can be used as an indicator to measure the integrity of the window data; the length of the interval between the window and the current time refers to the distance between the latest data point in the window and the current time point, and this interval length can be used as an indicator to measure the timeliness of the window data.

[0084] According to these preset conditions, different weights are assigned to the predicted values ​​of batch missing value windows. The weight reflects the integrity and timeliness of the window data. The higher the weight, the more important the data.

[0085] Step S33: Calculate the weighted average of the predicted values ​​according to the assigned weights.

[0086] Specifically, according to the weights assigned in step S32, the predicted values ​​of the batch missing value windows are weighted averaged, and the weighted average is obtained by multiplying each predicted value by its corresponding weight and then summing them. The higher the weight, the greater the contribution of the predicted value in the weighted average calculation. The weighted average reflects the overall trend and change of the data in the batch missing value window, and can better supplement the missing data points.

[0087] Step S34: Fill the weighted average into the corresponding missing data points in the batch missing value window.

[0088] Specifically, by supplementing the weighted average, a more accurate and reliable prediction value can be obtained, thereby improving the completeness of the data set and the accuracy of subsequent analysis.

[0089] In some embodiments, when it is found that consecutive windows meet the batch missing value anomaly condition, the system will identify them as a set, and predict each window in the set separately. By treating consecutive "batch missing value anomaly" windows as a set, repeated calculations can be reduced and the accuracy of predictions can be improved, and missing data points can be predicted and supplemented more efficiently.

[0090] Furthermore, the quality of the supplemented data is evaluated through multi-dimensional indicators, including but not limited to the difference between the predicted value and the actual value, data continuity, etc. This evaluation process is crucial to understanding the prediction accuracy and data continuity of the current LSTM model. Based on the evaluation results, precise adjustments can be made to improve model performance.

[0091] According to the above evaluation results, multiple parameters of the LSTM model are automatically adjusted, including the learning rate, the number of LSTM units, and the ratio of the Dropout layer. In addition, parameters such as the window size and sliding step size are also adjusted as needed. This parameter optimization is based on model performance feedback and aims to make the window division strategy more consistent with the characteristics of the data, thereby improving data processing efficiency and accuracy. The weight allocation strategy in the weighted average rule is further refined to more accurately map the relationship between data points and ensure the accuracy and consistency of the supplementary recording process.

[0092] This continuous iterative optimization process is a key feature of the present invention, which ensures the stability and reliability of the data cleaning and analysis process and provides high-quality data support for subsequent data analysis and decision-making.

[0093] It should be particularly noted that the process or method represented by the flowchart of the above embodiment of the present disclosure can be understood as representing a module, fragment or part of a code including one or more sets of executable instructions configured to implement the steps of a specific logical function or process. And the scope of the preferred embodiment of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0094] like Figure 3 As shown, the water conservancy monitoring data supplementary recording device 60 of the present disclosure is shown. It should be noted that the principle and technical implementation of the water conservancy monitoring data supplementary recording device can refer to the water conservancy monitoring data supplementary recording method embodiment in the previous embodiment (for example Figure 1 ), so it will not be repeated in this embodiment.

[0095] Specifically, the water affairs monitoring data supplementary recording identification device 60 includes: an acquisition module 61, a window division module 62, a judgment module 63, and a supplementary recording module 64, wherein:

[0096] The acquisition module 61 is used to acquire time series data of water affairs monitoring;

[0097] The window division module 62 is used to divide the time series data into a series of data windows according to a preset time window size and sliding step size;

[0098] The judging module 63 is used to count the data points in each data window to judge whether there are missing data points in the window;

[0099] The supplementary recording module 64 is used to mark the corresponding window as an abnormal window when missing data points are detected, and take corresponding supplementary recording measures based on the number of missing data points.

[0100] It should be noted that Figure 6 Each functional module in the embodiment can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a program instruction product. The program instruction product includes one or a group of program instructions. When the program instruction is loaded and executed on a computer, the process or function according to the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The program instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0101] and, Figure 6 The device disclosed in the embodiment can be implemented by other module division methods. The device embodiments shown above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as a group of modules or modules can be combined or dynamically added to another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical or other forms.

[0102] in addition, Figure 6 Each functional module and submodule in the embodiment can be dynamically in a processing component, or each module can exist physically separately, or two or more modules can be dynamically in one component. The above-mentioned dynamic components can be implemented in the form of hardware or in the form of software functional modules. If the above-mentioned dynamic components are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0103] like Figure 7 As shown, a schematic diagram of the structure of an electronic device in an embodiment of the present disclosure is shown.

[0104] The electronic device can execute the following steps by running the computer program instructions: Figure 1 The method in any one of the items. Exemplarily, the electronic device may be a server group / server, a desktop computer, a laptop computer, etc., or a server / server group in the cloud that communicates remotely with a local terminal, a distributed computing node system, etc.

[0105] The electronic device 70 includes a bus 71, a processor 72, and a memory 73. The processor 72 and the memory 73 can communicate with each other through the bus 71. The memory 73 can store program instructions. The processor 72 implements the method steps in the previous embodiment by running the program instructions in the memory 73, such as Figure 1 Any method.

[0106] The bus 71 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, although only one thick line is used in the figure, it does not mean that there is only one bus or one type of bus.

[0107] In some embodiments, the processor 72 may be implemented as a central processing unit (CPU), a microprocessing unit (MCU), a system on chip (System On Chip), or a field programmable logic array (FPGA). The memory 73 may include a volatile memory (Volatile Memory) for temporary storage of data when running a program, such as a random access memory (Random Access Memory, RAM).

[0108] The memory 73 may also include a non-volatile memory (non-volatile memory) for data storage, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state disk (SSD).

[0109] In some embodiments, the electronic device 70 may further include a communicator 74. The communicator 74 is used to communicate with the outside. In a specific example, the communicator 74 may include one or a group of wired and / or wireless communication circuit modules. For example, the communicator 74 may include one or more of a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include: for example, near field communication (NFC) technology, infrared (IR) technology, global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth (BT), global navigation satellite system (GNSS), etc. One or more.

[0110] In an embodiment of the present disclosure, a computer-readable storage medium may be provided, wherein program instructions are stored, and the program instructions are executed by running, for example Figure 1 A method for supplementing water monitoring data in an embodiment.

[0111] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or are implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA).

[0112] In summary, the water monitoring data supplementation method, device, equipment and medium disclosed in the present invention effectively identify and repair missing or damaged data points in water monitoring data, ensuring the integrity and accuracy of the data. By presetting a threshold to distinguish between a small number of missing value windows and a batch of missing value windows, and adopting corresponding supplementation strategies, it is possible to adaptively deal with data missing problems in different situations, and significantly improve the robustness in the scenario of water monitoring data loss. In addition, the Minkowski distance formula and weighted average method are used to supplement the missing data points in a small number of missing value windows, and at the same time, the well-trained LSTM model is used to predict and supplement the missing data points in the batch of missing value windows, which optimizes the entire data processing process and significantly improves the efficiency and accuracy of data supplementation.

[0113] The above embodiments are merely illustrative of the principles and effects of the present disclosure, and are not intended to limit the present disclosure. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present disclosure. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present disclosure shall still be covered by the claims of the present disclosure.

Claims

1. A method for supplementing water monitoring data, characterized in that: include: Obtain time series data for water monitoring; According to a preset time window size and sliding step size, the time series data is divided into a series of data windows; Count the data points in each data window to determine whether there are missing data points in the window; When missing data points are detected, the corresponding window is marked as an abnormal window, and corresponding supplementary recording measures are taken based on the number of missing data points.

2. The method for supplementing water affairs monitoring data according to claim 1, characterized in that: The data points in each data window are counted to determine whether there are missing data points in the window; Setting a first threshold and a second threshold for the number of data points in the window, wherein the first threshold is higher than the second threshold; When the number of data points in the window is lower than the first threshold and higher than the second threshold, the window is determined to be a window with a small number of missing values; When the number of data points in the window is lower than the second threshold, the window is determined to be a batch missing value window.

3. The method for supplementing water monitoring data according to claim 2, characterized in that: The supplementary recording measures for the small number of missing value windows include: Acquire data points of the windows adjacent to and before the small number of missing value windows to form a candidate data set; Calculating the distance between each data point in the candidate data set and each missing data point in a small number of missing value windows; Based on the calculated distance, the K data points with the closest distance are selected from the candidate data set to form a complementary data set, where K is a preset positive integer; The weighted average of the K data points in the supplementary data set is calculated, and the weighted average is used to supplement the corresponding missing data points in the small number of missing value windows.

4. The method for supplementing water monitoring data according to claim 3, characterized in that: The Minkowski distance formula is used to calculate the distance between each data point in the candidate data set and the missing data points in a small number of missing value windows; The weights of the K data points are assigned according to the inverse of the Minkowski distance from each data point to the missing data point.

5. The method for supplementing water affairs monitoring data according to claim 2, characterized in that: The processing of supplementing the batch missing value windows includes: Inputting the batch missing value windows into a pre-trained missing value prediction model to obtain predicted values ​​of missing data points; According to preset conditions, weights are assigned to the predicted values ​​of the batch missing value windows; wherein the preset conditions include: a proportion of valid data in the batch missing value windows, a length of an interval between the batch missing value windows and the current time, or one or more combinations thereof; Calculating a weighted average of the predicted values ​​according to the assigned weights; The weighted average is used to fill in the corresponding missing data points in the batch missing value window.

6. The method for supplementing water affairs monitoring data according to claim 5, characterized in that: The training steps of the missing value prediction model include: Extract the time series data of the window without abnormal labeling as the training data set; Input the training data set into the pre-built missing value prediction model to obtain the prediction result; Optimizing the loss function based on the difference between the predicted result and the actual data value; The optimization process is repeated until the value of the loss function converges to a predetermined threshold.

7. The method for supplementing water affairs monitoring data according to claim 5, characterized in that: The missing value prediction model adopts an LSTM architecture, and the missing value prediction model includes: An input layer to receive time series data; One or more LSTM layers to learn and capture long-term dependencies in time series data; An output layer that generates the final predictions.

8. A device for supplementing recording water affairs monitoring data, characterized in that: include: An acquisition module is used to obtain time series data of water affairs monitoring; A window partitioning module is used to partition the time series data into a series of data windows according to a preset time window size and sliding step size; A judgment module is used to count the data points in each data window to determine whether there are missing data points in the window; The supplementary recording module is used to mark the corresponding window as an abnormal window when missing data points are detected, and take corresponding supplementary recording measures based on the number of missing data points.

9. An electronic device, characterized in that: The electronic device comprises: Processor and memory; Wherein, the memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the electronic device executes the method for supplementing water monitoring data as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the method for supplementing the water monitoring data described in any one of claims 1 to 7 is implemented.

Images