A hydrological data anomaly identification and repair method based on a federated learning framework

Abstract

The application discloses a hydrological data anomaly detection and repair method based on a federal learning framework, which comprises a model training process and an identification and repair process, and the model training process comprises the following steps: firstly, pre-processing and abnormalizing hydrological data for training; secondly, building a federal learning architecture, initializing model parameters on a server, and sending a global model to each client; after the client receives the model, learning local data features is started, a bidirectional LSTM with an attention mechanism is used to extract context information, then, the functions of anomaly detection and data repair are simultaneously optimized through an adversarial learning mode, and finally, the functions of the model are updated through iterative interaction between the client and the server. Through the new method, the hydrological telemetry data privacy is protected, the anomaly identification and repair are simultaneously performed, the hydrological prediction performance is improved, and the loss caused by uncertain disasters is reduced.

Description

Technical Field

[0001] This invention relates to the field of hydrological data processing, and in particular to a method for identifying and repairing hydrological data anomalies based on a federated learning framework. Background Technology

[0002] With increasing uncertainty surrounding global natural disasters, the development of smart hydrology is receiving growing attention. This aims to build an integrated space-air-ground hydrological telemetry system based on technologies such as cloud computing and big data, enabling more real-time and accurate observation and recording of hydrological phenomena and providing a data foundation for hydrological research. Clearly, as the primary source of hydrological data, hydrological telemetry equipment bears the heavy responsibility of data collection and storage. The accuracy and reliability of the hydrological data provided by telemetry equipment directly impacts fundamental decisions regarding flood control and drought relief, ecological environmental protection, and comprehensive water resource development. However, in actual operation, telemetry equipment often experiences system failures, equipment aging, and weak signals due to remote locations, leading to numerical errors, partial omissions, and severe gaps in the collected hydrological data. This seriously affects the integrity, accuracy, and reliability of the hydrological data, and directly reduces the statistical analysis capabilities of various hydrological models. Therefore, identifying anomalies in hydrological data, mining potential data characteristics, and repairing abnormal data are crucial for improving hydrological forecasting performance and reducing losses caused by uncertain disasters.

[0003] However, existing methods for anomaly identification and restoration of hydrological data mainly suffer from the following problems: 1) In practice, the identification and restoration of anomaly data often need to be addressed simultaneously, but most studies focus more on anomaly detection and neglect the importance of restoring anomaly data; 2) Most models do not take into account the potential temporal information of features such as water level, rainfall, and flow in hydrological telemetry data, resulting in low accuracy of anomaly identification and poor data restoration; 3) They ignore the privacy issues contained in telemetry data. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention provides a generative adversarial network model based on a federated learning framework and a long short-term memory network, so as to achieve the identification and repair of anomalies in hydrological telemetry data while protecting data privacy.

[0005] This invention first structures and models the raw hydrological data from each client (i.e., hydrological telemetry device) in the federated learning architecture into corresponding time-series data. Then, it waits for the server to initialize parameters and sends the generative adversarial network (GAN) model to be optimized and the global model parameters to each client. Upon receiving the data, each client inputs its processed time-series data into the GAN. A discriminator network identifies anomalous sequences, while a generator network reconstructs and repairs them. These two networks are trained and optimized adversarially. Simultaneously, a long short-term memory (LSTM) network is embedded, and an attention mechanism is introduced to learn the latent features of the data and capture their temporal dependencies. Afterward, each client sends its trained local model parameters to the server, which integrates them into new global model parameters and resends them to the clients. Ultimately, while protecting the privacy of hydrological telemetry station data, this invention enables both data repair of hydrological telemetry data and identification of anomalous data.

[0006] This invention achieves the above objectives through the following technical solution: a method for anomaly identification and repair of hydrological data based on a federated learning framework, comprising a model training process and an identification and repair process, wherein the model training process includes the following steps:

[0007] S1: Preprocess and anomaly-inducing the training hydrological data;

[0008] S2: Build the federated learning architecture and initialize the model parameters;

[0009] S3: The client-side optimizes anomaly detection and data repair functions through adversarial learning;

[0010] S4: The local client interacts with the server to update global parameters;

[0011] The identification and restoration process is as follows: the original hydrological data is preprocessed and then input into the trained model, and the output is the restored data.

[0012] Preferably, step S1 specifically includes the following steps:

[0013] S1.1: Screen the hydrological data, remove noisy and duplicate data, and select the data that is of research significance.

[0014] S1.2: The filtered hydrological data are normalized and processed into a matrix sequence F with the same time slot. T ;

[0015] S1.3: Process the matrix sequence F TA predetermined percentage (approximately 10%) of the data is artificially created to generate abnormal data for testing data anomaly detection and repair functions. This artificial creation includes adding one or more of the following: offset anomalies, sequence anomalies, and extreme value anomalies.

[0016] During the formal identification and repair process, the data preprocessing includes S1.1 and S1.2, but does not include S1.3.

[0017] Preferably, step S2 specifically includes the following steps:

[0018] S2.1: Build a federated learning framework by using K hydrological telemetry stations as clients and a cloud server as the server (i.e., a trusted third party); K is the total number of hydrological telemetry stations.

[0019] S2.2: Define the dataset size for the k-th client as Data k If 1 ≤ k ≤ K, then the total dataset size used for local training is

[0020] S2.3: The server initializes the global model parameters, that is, it generates the training parameters for each layer of the adversarial network and LSTM network, and sends the global model and initial parameters to each client.

[0021] Preferably, the model includes a generator and a discriminator. The generator includes an LSTM network and a fully connected layer, and the discriminator includes a bidirectional LSTM network with an attention mechanism and a fully connected layer.

[0022] When predicting hydrological data at time T+1, it is necessary to process the hydrological data before time T+1 to obtain a matrix sequence F. T The input matrix sequence first passes through a neuron containing three control unit state gates, which are:

[0023] Forget gate, used to retrieve information to be discarded. t :

[0024] f t =σ(W f x t +W f h t-1 +b f )

[0025] Input gate, used to obtain the information i to be memorized. t And storing current cell state information (Intermediate variable):

[0026] i t =σ(W i x t +W i ht-1 +b i )

[0027]

[0028] Based on the above forgotten information f t and update information i t A new cell state C is obtained t (f t ×C t-1 This indicates a selective discarding of past information, while This indicates that selective retention is possible.

[0029]

[0030] And output gate:

[0031] o t =σ(W o x t +W o h t-1 +b o )

[0032] h t =o t ·tanh(C t )

[0033] Finally, the generator output hidden state h at time t is calculated. t C t Used to retrieve information from long-term memory, while h t This is used to obtain short-term memory information, and the initial C0 and h0 are defaulted to an all-zero matrix; where x t For matrix sequence F T The input at time point t, h t-1 Let W represent the hidden state at time t-1, where W represents the weight vector of the corresponding gated unit or cell state based on its index, and b represents the offset of the corresponding gated unit or cell state based on its index. f It is the weight vector of the forget gate, b f It is the offset of the forget gate, W i It is the weight vector of the input gate, b i It is the offset of the input gate, W c b is the weight vector of the current cell state. c It is the offset of the current cell state, i.e., W. o It is the weight vector of the output gate, b o σ is the offset of the output gate, σ is the activation function, typically the sigmoid function, and tanh is the activation function. This process involves three inputs: the input value x at time t.t , the hidden layer state h of the generator output at the previous moment t-1 and the cell state C of the neuron t-1 . The output at the current moment includes the hidden layer state h of the generator t and the cell state C t . The final LSTM output is obtained from the hidden layer state h at the last time step T that synthesizes all the useful information before, and then sent to the fully connected layer network: T to obtain the predicted value x at the next moment

[0034] x T+1 = Linear(h T )

[0035] Linear represents the fully connected layer. The above is the network structure of the generator, which uses gated settings to control the transmission state, remember the information that needs to be remembered for a long time, and forget the unimportant content, so as to explore the time series F T+1 . The time series change rules such as relatively long intervals and delays in T . Through the network structure of LSTM, assuming that a certain segment x T in the sequence F t1 ,..., x tu (tu < T) is missing or has other anomalies, we can use the information of all the nodes before this node in the sequence for cyclic regression prediction, that is, predict x t1 based on the data nodes between t1 moments, and then fill the originally abnormal part with x t1 , and combine the information of the previous nodes to predict x t2 again, and so on to complete the data repair process;

[0036] The bidirectional LSTM network of the discriminator includes a forward LSTM network layer and a backward LSTM network layer. The structures of the forward LSTM network layer and the backward LSTM network layer are the same as those of the LSTM network layer in the generator. The matrix sequence input to the forward LSTM network layer is the forward input, and the forward output hidden layer state at time t is denoted as The matrix sequence input to the backward LSTM network is the backward input, and the forward output hidden layer state at time t is denoted as The hidden layer state h of the bidirectional LSTM t is calculated by the following formula, where is the Concat() function, combining the forward and backward hidden layer state information:

[0037]

[0038] Furthermore, to improve the discriminator's learning ability, an attention mechanism was introduced. The attention layer extracts the weight matrix using the following formula:

[0039] α = softmax(w T tanh(H))

[0040] The product r of H and the weight matrix α is used as the output of the attention layer.

[0041] r = Hα T

[0042] Where H is the output of the bidirectional LSTM layer, i.e., the hidden state information {h1,...,h} at all time points. T}, of size v*T, where v is the hidden state information h. T The number of embedded features, T is the length of the sequence, w T This is the transpose of the parameter vector matrix obtained through training, which is continuously optimized through model training. α is the weight matrix, and r is the output of this layer. Then it enters the fully connected layer Linear() and the sigmoid activation function fixes the values ​​in the [0,1] interval.

[0043] PSY T =Sigmoid(Linear(r))

[0044] Obtain sequence F T The probability that each timestamp t is true (PSY) T PSY T The size is T*1, and all values ​​are between [0,1]. This is the overall structure of the discriminator network, using sequence F as an example. T As input, with PSY T As the final output, anomaly detection is performed on each timestamp of the sequence. By adding the above structural improvements to the generative adversarial network, the discriminator's ability to detect anomalies and the generator's ability to fit data can be enhanced simultaneously, thereby improving the overall performance of the model.

[0045] Preferably, step S3 specifically includes the following steps:

[0046] S3.1: Initialize and fix the generator G, and start training the discriminator D; use real data F T and the data forged by G F T As input to D, the result is passed through a bidirectional LSTM layer, an attention layer, and a fully connected layer, ultimately outputting the discrimination result; if the discriminator input is F... T If the data is real and normal, then the output result P is... TThe decision value at all time points should be as close to 1 as possible; otherwise, the output should tend to 0. Since the classifier D generally uses the sigmoid function, training the discriminator D is a process of minimizing its cross-entropy, and the loss function is as follows:

[0047]

[0048] in For real data samples, This refers to the generation of data or abnormal data samples. Indicates that x belongs to This means that y belongs to Obviously, for the discriminator D, whether it is generated data or abnormal data, the output result should be as close to 0 as possible;

[0049] S3.2: Generator Optimization: with the sequence F to be repaired T As input, the signal passes through an LSTM layer and a fully connected layer, and outputs G(F). T ) = F T For the generator, the goal is to generate data that can deceive the discriminator as much as possible. Therefore, its training is a process of maximizing cross-entropy, with the following loss function:

[0050]

[0051] Finally, if and only if At this time, the globally optimal solution is obtained;

[0052] S3.3: K clients calculate their respective loss gradients L k (w) Used to update the local model:

[0053]

[0054] Where s(*) is the regularization function, l j (*) represents the loss of the j-th sample, w is the local weight parameter, and λ∈[0,1] is used to balance the loss.

[0055] Preferably, step S4 specifically includes the following steps:

[0056] All K clients will assign their local model training parameters w k Send to the server to update the global parameter W z+1 Unlike traditional centralized training methods, federated learning updates the globally trained model through a secure parameter aggregation mechanism. Furthermore, to reduce communication overhead during model parameter transmission, a federated averaging algorithm is used to accelerate model convergence; that is, the server updates the model based on:

[0057]

[0058] To update the global model parameters; where n k and These represent the number of samples and local weights on client k, respectively, where n is the total number of samples from all selected clients. The latest global weight W is then obtained. z+1 Then, the updated global model is sent to client k for the next round of optimization and updates.

[0059] As a preferred embodiment, the identification and repair process includes the following steps:

[0060] The k-th client processes the hydrological data into a matrix sequence F. kT Subsequently, the last updated global weight parameters W are downloaded from the server; at this point, both the local generator G and discriminator D possess optimal data repair and anomaly detection capabilities; therefore, the k-th client first sends the matrix sequence F... kT The input is fed into the discriminator of the local model to obtain D(F). kT ) = PSY T That is, the sequence F kT The probability that each timestamp t is normal is defined as [0,1], and a value greater than 0.5 is considered normal.

[0061] If it contains outliers, then the matrix sequence F to be repaired will be... kT The data is input into the generator to reconstruct it, i.e., data repair G(F) kT ) = F kT Finally, the parts identified as anomalous will be replaced by the repaired data, and the repaired sequence F will be analyzed using inverse normalization. kT Restore the data to its original state. This concludes the data anomaly detection and repair process.

[0062] The beneficial effect of this invention is that it can simultaneously identify and repair anomalies in hydrological telemetry data while protecting data privacy, with high accuracy and reliability. Attached Figure Description

[0063] Figure 1 This is a flowchart of a model training process according to the present invention;

[0064] Figure 2 This is a training structure diagram of a federated learning framework according to the present invention;

[0065] Figure 3 This is a diagram illustrating the internal structure of a long short-term memory network according to the present invention.

[0066] Figure 4 This is a structural diagram of a bidirectional long short-term memory network model based on an attention mechanism according to the present invention.

[0067] Figure 5 This is a structural diagram of a generative adversarial network model according to the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below.

[0069] This invention provides a method for hydrological data anomaly detection and data restoration based on a federated learning framework, including a model training process and an identification and restoration process, such as... Figure 1 and Figure 2 As shown, the training process includes:

[0070] S1: Preprocess and anomaly-detection the training hydrological data, that is, extract useful data from the original dataset. The steps are as follows:

[0071] S1.1: The raw data cleaning process involved cleaning hydrological telemetry data collected from four hydrological telemetry stations in Hangzhou, Jinhua, Shaoxing, and Lishui cities in Zhejiang Province over a period of 90 days, from January 1, 2022 to March 31, 2022, ensuring the authenticity and reliability of the data source for model training and testing. The data attributes include:

[0072] serial number name Notes 1 sid Telemetry Terminal SID Encoding 2 Longitude_hy Longitude of hydrological telemetry station 3 Latitude_hy Latitude of hydrological remote sensing station 4 Terminal_na Terminal Name 5 Current_sta Current status 6 Telemetry_pro Telemetry Project 7 Monitoring_ele Monitoring elements 8 Watershed_na River Basin Name 9 Rain_area catchment area 10 Responsible_un Responsible unit 11 Sensor_model Sensor model 12 Device_In Equipment Information 13 …… ……

[0073] Table 1

[0074] As can be seen from the table, the original dataset contains rich and complex data, including a significant amount of privacy information. Clearly, our focus for anomaly detection and repair is on sensor monitoring data. Furthermore, due to differences in equipment models and geographical locations at different telemetry stations, data recording intervals and collected data attributes may vary. Therefore, we selected to filter out redundant data attributes, extract the common attributes of the hydrological equipment at these four telemetry stations, and statistically analyze the collected data at five-minute intervals, significantly reducing the data volume and facilitating subsequent analysis and calculations. The extracted data attributes are as follows:

[0075]

[0076]

[0077] After data extraction, it was found that some attributes remained unchanged for extended periods under normal conditions. These attributes were deemed unsuitable for experimental research. For example, regarding rainfall, if there were no rainfall for several consecutive days, the attribute value would remain at 0 for an extended period. Furthermore, data analysis revealed that due to the small fluctuations in water level, the current water level and the 5-minute water level attribute value were often identical; therefore, these were considered for removal. Thus, the hydrological telemetry data targeted in this invention primarily includes the following attributes:

[0078] serial number name Notes 1 Sid Telemetry Terminal SID Encoding 2 Flow flow 3 Tem Ambient temperature 4 Cwl Current water level 5 Vol power supply voltage 6 Ifr Indicator flow rate 7 Iwt Instantaneous water temperature 8 Cifr Current instantaneous flow rate

[0079] Table 3S1.2: Data normalization is performed using the following formula

[0080]

[0081] Where x represents the data before normalization, max and min represent the maximum and minimum values ​​of the attribute data, respectively, and X represents the data after normalization. After normalizing the data for all attributes, the common time point for each attribute is used as the feature of that time point. That is, each time point includes the flow rate, ambient temperature, current water level, power supply voltage, index flow rate, instantaneous water temperature, and current instantaneous flow rate at that moment. To better extract the sequence features, a window size of two hours, or 24 time points, is used as the sequence length, thereby constructing each matrix sequence F. T .

[0082] S1.3: To test the performance of the method of the invention, the matrix sequence F constructed in S1.2 is subjected to... T The process of creating a dirty structure test set involves several steps, including: 1. Selecting approximately 10% of the data F. T Using human-made dirt 2, masking matrix F anomaly For F T 3. Masking to create missing or abnormal data; T Add offset anomalies, sequence anomalies, and extreme value anomalies.

[0083] S2: Set up the federated learning architecture and initialize the model parameters. The specific steps are as follows:

[0084] S2.1: Four hydrological telemetry stations in Hangzhou, Jinhua, Shaoxing, and Lishui in Zhejiang Province serve as local clients, while a cloud server acts as a trusted third party to build the federated learning framework. The server configuration includes an Ubuntu 18.04 operating system, 128GB of RAM, an Intel(R) Xeon(R) Gold CPU with 16 cores and dual threads, and an NVIDIA Quadro P6000 graphics card.

[0085] S2.2: The dataset for each client consists of hydrological telemetry data collected over 90 days, from January 1, 2022 to March 31, 2022. The data from the first 30 days is used as the training set, and the entire 90 days' data is used as the test set. Approximately 10% (9 days) of the test data is processed as outlier data (F). anomaly Define the dataset size for K clients as Data. K Then the total size of the local training dataset is Data = Data1 + Data2 + Data3 + Data4.

[0086] S2.3: The server initializes the model parameters W and sends the global model to each client so that they can train their own datasets.

[0087] The model includes a generator and a discriminator. The generator consists of an LSTM network and fully connected layers, while the discriminator consists of a bidirectional LSTM network with attention mechanism and fully connected layers. The specific process of the model extracting temporal feature information is as follows:

[0088] First, each client receives the global model sent from the server, and then begins training its local dataset. This training process is completed synchronously by all four local clients. Simultaneously, the loss gradient of the local data is calculated (where s(*) is the regularization function, w is the local weight parameter, and λ∈[0,1]):

[0089]

[0090] The processed matrix sequence F T As input to the model, hydrological data, being a collection of hydrological elements such as water level, flow rate, and voltage observed over time, often exhibits significant time-series characteristics. Clearly, neither the generator nor the discriminator has a specific structure for processing time-series data. Therefore, failing to consider the unique time-series characteristics during hydrological data training is a major reason for the generator's weak ability to fit real data and the discriminator's low accuracy in identifying outliers. To learn the latent spatial distribution of the data, a bidirectional LSTM with attention mechanism is used to extract time-series feature information. For the generator, such as... Figure 3 As shown, with F T As the input sequence, it first passes through three gates controlling the control unit state: the input gate, the forget gate, and the output gate. Then, according to the aforementioned gate functions and formulas:

[0091] h t =o t ·tanh(C t )

[0092] Finally, the hidden state h at time t is calculated.t .

[0093] For a discriminator, in order to better learn the features of the data, such as Figure 4 As shown, a reverse LSTM layer is added to the existing forward LSTM network layer to consider the context from both directions, increasing the network's available information. This network structure consists of two forward-propagating LSTM layers. and backward transmission Then the hidden state h at this time t Calculated using the following formula, where The `Concat()` function combines forward and backward hidden state information:

[0094]

[0095] Furthermore, to improve the discriminator's learning ability, an attention mechanism is introduced. This layer extracts the weight matrix using the following formula:

[0096] α = softmax(w T tanh(H))

[0097] The product r of H and the weight matrix α is used as the output of the attention layer.

[0098] r = Hα T

[0099] Where H is the output of the bidirectional LSTM layer, i.e., the hidden state information {h1,...,h} at all time points. T}, of size v*T, where v is the hidden layer information h T The number of embedded features, T is the length of the sequence, w T This is the transpose of a parameter vector matrix, continuously optimized through model training. α is the weight matrix, and r is the output of this layer. Then, it enters the fully connected layer Linear() and the sigmoid activation function fixes the values ​​in the [0,1] interval.

[0100] PSY T =Sigmoid(Linear(r))

[0101] Obtain sequence F T The probability that each timestamp t is true (PSY) T PSY T The size is T*1, and all values ​​are between [0,1]. This is the overall structure of the discriminator network, using sequence F as an example. T As input, with PSY TAs the final output, anomaly detection is performed on each timestamp of the sequence. By adding the above structural improvements to the generative adversarial network, the discriminator's ability to detect anomalies and the generator's ability to fit data can be enhanced simultaneously, thereby improving the overall performance of the model.

[0102] S3: The client optimizes anomaly detection and data repair functions through adversarial learning. The specific steps are as follows:

[0103] S3.1: After extracting time series features, it is necessary to balance the adversarial learning processes of the generator and discriminator in the generative adversarial network (GAN) to optimize anomaly detection and data repair functions. GANs optimize through a "binary game" adversarial approach, requiring the discriminator to outperform the generator; otherwise, gradient vanishing is likely. Therefore, the discriminator D is often trained multiple times before the generator is trained once. First, the generator G is initialized and fixed, and then the discriminator D is trained using real data F. T and G's forged data F T The input to D is passed through a bidirectional LSTM layer, an attention layer, and a fully connected layer, and the final output is the discrimination result PSY. T If the discriminator input is F T If the data is real and normal, then the output result P is... T The decision value at all time points should be as close to 1 as possible; otherwise, the output should tend to 0. Clearly, for the discriminator D, whether dealing with generated or outlier data, the output should be as close to 0 as possible.

[0104] S3.2: The optimization of the generator is similar to the training process of traditional generative adversarial network models, using F... T As input, the signal passes through a fully connected layer and an LSTM layer, and outputs G(F). T ) = F T For the generator, the goal is to generate data that can deceive the discriminator as much as possible, using this as an optimization objective to gradually approximate the original data, thereby achieving the effect of data repair. Finally, if and only if P FT =P FT′ When the time is right, the global optimal solution is obtained.

[0105] S4: The local client interacts with the server to update global parameters. The specific steps are as follows:

[0106] After each client completes its first round of local training, the weights w obtained from training are... k Send to the server to update the global model parameters W for the next round. z+1Unlike traditional centralized training methods, federated learning updates the globally trained model through a secure parameter aggregation mechanism. Furthermore, to reduce communication overhead during model parameter transmission, a federated averaging algorithm is used to accelerate model convergence. That is, the server updates the model based on:

[0107]

[0108] To update the global model parameters. After obtaining the latest W... z+1 Then, the updated global model is sent to client k for the next round of optimization and updates.

[0109] The overall generative adversarial network model is as follows: Figure 5 As shown.

[0110] The identification and repair process is as follows: The client processes its own hydrological data into a matrix sequence F. kT The test set includes outliers. Then, the last updated global weight parameters W are downloaded from the server. At this point, both the local generator G and discriminator D possess optimal data repair and anomaly detection capabilities. Therefore, the client first processes the matrix sequence F... kT The input is fed into the discriminator of the local model to obtain D(F). kT This represents the probability (0-1, with values ​​exceeding 0.5 considered normal) of each timestamp t in the sequence being normal. If outliers are included, then F... kt The data is input into the generator to regenerate it, i.e., data repair G(F) kT ) = F kT Finally, the parts identified as anomalous will be replaced by the repaired data. Simultaneously, inverse normalization will be used to optimize the repaired sequence F. kT Restore the data to its original state. This concludes the data anomaly detection and repair process.

[0111] Application cases demonstrate that the hydrological telemetry data anomaly detection and repair method based on federated learning proposed in this invention is effective. Compared to other design methods, this invention uses a federated learning architecture to protect data privacy, while the discriminator and generator in the generative adversarial network are used for data anomaly detection and data repair, respectively. To improve the model's ability to extract temporal features, a bidirectional long short-term memory network based on an attention mechanism and a regular long short-term memory network are embedded into the model's discriminator and generator, respectively. The model processes the hydrological data from the hydrological telemetry equipment into a temporal matrix sequence as input. The bidirectional long short-term memory network layer in the discriminator extracts relevant temporal information, and the result, i.e., the hidden layer state, is used as input to the attention layer to obtain the weight matrix. Finally, the fully connected layer outputs the identification result, completing the data anomaly identification. In addition, the matrix sequence of data determined to be anomalous by the discriminator is also input to the generator, which uses its ability to fit the data distribution to complete the data repair. Experiments using real hydrological datasets from four telemetry stations in Hangzhou, Jinhua, Shaoxing, and Lishui provided by the Zhejiang Provincial Hydrological Communication Platform fully demonstrate the feasibility and superiority of the model.

[0112] The above description describes specific embodiments of the present invention and the technical principles employed. If changes are made in accordance with the concept of the present invention, and the resulting functions do not exceed the spirit covered by the specification and drawings, they should still fall within the protection scope of the present invention.

Claims

1. A method for anomaly identification and repair of hydrological data based on a federated learning framework, comprising a model training process and an identification and repair process. The model training process includes the following steps: S1: Preprocess and anomaly-detection of the hydrological data used in training; S2: Build the federated learning architecture and initialize the model parameters; S3: The client optimizes anomaly detection and data repair functions through adversarial learning; S4: The local client interacts with the server to update global parameters; The identification and restoration process is as follows: the original hydrological data is preprocessed and then input into the trained model, and the output is the restored data; Step S2 includes the following steps: S2.1: Will Each hydrological telemetry station acts as a client, and a cloud server acts as the server to build a federated learning framework; K represents the total number of hydrological telemetry stations. S2.2: Define the dataset size for the k-th client as... If 1 ≤ k ≤ K, then the total dataset size used for local training is ; S2.3: The server initializes the global model parameters, that is, it generates the training parameters of each layer of the adversarial network and LSTM network, and sends the global model and initial parameters to each client; The model includes a generator and a discriminator. The generator includes an LSTM network and a fully connected layer, and the discriminator includes a bidirectional LSTM network with an attention mechanism and a fully connected layer. When predicting hydrological data at time T+1, it is necessary to process the hydrological data before time T+1 to obtain a matrix sequence F. T The input matrix sequence first passes through a neuron containing three control unit state gates, which are: Forget gate, used to retrieve information that needs to be discarded. : Input gate, used to obtain the information to be memorized. And storing current cell state information : Based on the information to be discarded above and the information to be memorized A new cell state is obtained: And output gate: Finally, the generator output hidden state at time t is calculated. ; Used to retrieve information from long-term memory, and It is then used to acquire short-term memory information, and initially as well as The default value is an all-zero matrix; where, For matrix sequences The input at time point t, Refers to the hidden layer state at time t-1, W represents the weight vector of the corresponding gated unit or cell state according to the subscript, and b represents the offset of the corresponding gated unit or cell state according to the subscript. It is the activation function; The final output of the LSTM network is the hidden state at the last time step T, which integrates all the useful information from before. Once obtained, it is then fed into the fully connected layer network: Obtain the predicted value at the next time step. ; The discriminator's bidirectional LSTM network consists of a forward LSTM layer and a backward LSTM layer. The structures of both the forward and backward LSTM layers are identical to those in the generator's LSTM layer. The matrix sequence input to the forward LSTM layer is the forward input, and the forward output hidden state at time t is denoted as... The matrix sequence input to the inverse LSTM network is the inverse input, and the forward output hidden state at time t is denoted as... Hidden states of a bidirectional LSTM Calculated using the following formula, where The `Concat()` function combines forward and backward hidden state information: The attention layer extracts its weight matrix using the following formula: and through With weight matrix product As the output of the attention layer: in, It is the output of the bidirectional LSTM layer, that is, the hidden state information at all time points. Size is ,in Hidden state information Number of embedded features It is the length of the sequence. It is the transpose of the parameter vector obtained through training. This is the weight matrix. This is the output of this layer; then it enters the fully connected layer network Linear() and the activation function sigmoid fixes the value in the range [0,1]. Obtain the sequence The probability that each timestamp t is true ,in The size is T*1, and all are between [0,1].

2. The hydrological data anomaly identification and repair method based on a federated learning framework as described in claim 1, characterized in that: Step S1 includes the following steps: S1.1: Filter the hydrological data to remove noisy and duplicate data; S1.2: The filtered hydrological data are normalized and processed into a matrix sequence with the same time slot. ; S1.3: Process the matrix sequence The set proportions were partially manipulated to create abnormal data.

3. The hydrological data anomaly identification and repair method based on a federated learning framework as described in claim 1, characterized in that: Step S3 includes the following steps: S3.1: Initialize the generator And fix it, and begin training the discriminator. ; with matrix sequence and Fake data As The input is processed through a bidirectional LSTM layer, an attention layer, and a fully connected layer, and finally the discrimination result is output. If the discriminator input is If the data is real and normal, the output is 1; otherwise, the output is 0. (Discriminator) The training loss function is as follows: in For the true data distribution, Here, represents the distribution of generated or anomalous data, x is a sample that follows the distribution of real data, and y is a sample that follows the distribution of generated or anomalous data. D(x) represents the probability that the discriminator classifies x as real data, and D(y) represents the probability that the discriminator classifies y as real data. Indicates that x belongs to , This means that y belongs to ; S3.2: Generator Optimization: Using Matrix Sequences As input, it passes through an LSTM layer and a fully connected layer before outputting. The generator's training loss function is as follows: Finally, if and only if At this time, the globally optimal solution is obtained; S3.3: K clients calculate their respective loss gradients. Used to update the local model: in It is a regularization function. Let the loss of the j-th sample be denoted as . For local weight parameters, Used to balance losses.

4. The hydrological data anomaly identification and repair method based on a federated learning framework as described in claim 1, characterized in that: Step S4 includes the following steps: All K clients will assign their local model training parameters Send to server to update global parameters and Clients respectively The number of samples and local weights, Let z represent the total number of samples from all selected clients, and z represent the training epoch; after obtaining the latest... Then, the updated global model is sent to the client. Then, we will carry out the next round of optimization and updates.

5. The hydrological data anomaly identification and repair method based on a federated learning framework as described in claim 1, characterized in that: The identification and repair process includes the following steps: No. Each client processes hydrological data into a matrix sequence. Then, download the latest updated global weight parameters from the server. , No. The client first sends the matrix sequence The input is fed into the discriminator of the local model to obtain... That is, the sequence Each timestamp The probability is considered normal, with a value in the range [0,1]. A value greater than 0.5 is considered normal. If the matrix sequence contains outliers, then the matrix sequence to be repaired will be... The data is input into the generator to reconstruct it, i.e., data repair. The parts identified as anomalous will be replaced with the repaired data, and finally, inverse normalization will be used to transform the repaired sequence. Restore to the original data.

Images