River monitoring data processing method and device and electronic equipment

Abstract

The invention discloses a river monitoring data processing method and device and electronic equipment, and relates to the field of water conservancy and the field of data processing. The method comprises the following steps: fusing river channel monitoring data from different monitoring devices in a tensor form to obtain fused target data; performing local manifold structure analysis on the target data to obtain local reconstruction weights among different local data in the target data; performing dimension reduction processing on the target data according to the local reconstruction weight to obtain target feature data of the target data; determining missing data in the target data according to the target feature data; missing data is repaired by analyzing the spatial-temporal information and parameter incidence relation before different data in the target data. The technical problem that in the prior art, due to the fact that river monitoring data belongs to a large amount of multi-source heterogeneous data, the processing precision of abnormal data such as missing data is low is solved.

Description

Technical Field

[0001] The present application relates to the fields of water conservancy and data processing, and specifically to a method, device and electronic equipment for processing river monitoring data. Background Art

[0002] River monitoring data governance is a key technology for water resource management and disaster prevention. However, traditional methods have many flaws. First, data silos are serious, and multi-source sensor data with different protocols and sampling frequencies are difficult to achieve spatiotemporal alignment, resulting in the failure of cross-dimensional correlation analysis. Second, because river monitoring data is a large amount of multi-source heterogeneous data, existing technologies have low accuracy in processing abnormal data such as missing data, cannot adapt to dynamic environments, and have a high false alarm rate. In addition, traditional interpolation methods suffer from severe distortion when filling missing values, and prediction errors are large in continuous missing scenarios. Finally, database architectures that rely on offline batch processing cannot meet real-time requirements and have long data processing delays.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method, device and electronic device for processing river monitoring data to at least solve the technical problem of low accuracy in processing abnormal data such as missing data caused by the fact that river monitoring data belongs to a large amount of multi-source heterogeneous data in the prior art.

[0005] According to one aspect of an embodiment of the present application, a method for processing river monitoring data is provided, including: fusing river monitoring data from different monitoring devices in the form of tensors to obtain fused target data; performing local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; the local reconstruction weights are used to constrain the relative positional relationship of different data points of the target data in different dimensional spaces to remain unchanged; performing dimensionality reduction processing on the target data according to the local reconstruction weights to obtain target feature data of the target data; determining missing data in the target data according to the target feature data; and repairing the missing data by analyzing the spatiotemporal information and parameter association relationships between different data in the target data.

[0006] Optionally, the river monitoring data from different monitoring devices are fused in the form of tensors to obtain fused target data, including: converting the river monitoring data into the original tensor; decomposing the original tensor into a core tensor and N factor matrices, wherein the core tensor includes the relationship between the feature information of the original tensor and different dimensional spaces; the N factor matrices are used to characterize the feature transformation of the original tensor in the N dimensional space; performing modular product operations on the core tensor and the N factor matrices to obtain the target data.

[0007] Optionally, in the process of performing local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, the method also includes: taking each local data in the target data as a data point, wherein the data point is represented in the form of a vector; taking k data points in the target data whose distance from the i-th data point is less than a preset threshold as the k nearest neighbor points of the i-th data point, wherein i is an integer greater than or equal to 1, and k is an integer greater than 1; performing norm calculation based on the i-th data point, the k nearest neighbor points and the weight value of each nearest neighbor point to obtain the local reconstruction weight between the i-th data point and the k nearest neighbor points.

[0008] Optionally, the target data is subjected to dimensionality reduction processing according to the local reconstruction weights to obtain target feature data of the target data, including: inputting the local reconstruction weights between different local data in the target data and the target data into a deep autoencoder, wherein the number of neurons in the input layer and the number of neurons in the output layer of the deep autoencoder are the same as the number of data dimensions of the target data, removing redundant information in the target data by the deep autoencoder, and extracting feature data from the remaining information; performing dimensionality reduction processing on the extracted feature data by the deep autoencoder based on the local reconstruction weights between different local data in the target data to obtain target feature data.

[0009] Optionally, missing data in the target data is determined based on the target feature data, including: inputting the target feature data into a target model, converting the target feature data into a feature vector through the target model, detecting the vector distance between the feature vector and the normal vector of the hyperplane, and determining the missing data in the target data based on the vector distance; wherein the target model is composed of a vector machine model and a Gaussian mixture model, and the normal vector of the hyperplane is used to define the boundary of normal data.

[0010] Optionally, before inputting the target feature data into the target model, slack variables, target parameters and objective functions are obtained, wherein the slack variables are used to constrain the amount of data allowed to appear on the error side of the hyperplane, the target parameters are used to control the number of support vectors of the vector machine model and the error tolerance of the vector machine model, and the objective function is used to map the target feature data to the target dimensional feature space; based on the slack variables, target parameters, objective function, normal vector and intercept of the hyperplane, the constraints of the vector machine model are determined.

[0011] Optionally, missing data is repaired by analyzing the spatiotemporal information and parameter associations between different data in the target data, including: obtaining the i-th data point of a known data value in the target data, wherein the position corresponding to the i-th data point is the first position, and the collection moment corresponding to the i-th data point is the first moment; determining the interpolation weight corresponding to the missing data at the second position and collected at the second moment based on the spatial distance between the first position and the second position and the time interval between the first moment and the second moment; determining the data value of the missing data based on the interpolation weight corresponding to the missing data and the known data value of the i-th data point; and filling the data value of the missing data into the second position of the target data to repair the missing data.

[0012] According to another aspect of an embodiment of the present application, a device for processing river monitoring data is also provided, including: a fusion unit, used to fuse river monitoring data from different monitoring equipment in tensor form to obtain fused target data; an analysis unit, used to perform local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; the local reconstruction weights are used to constrain the relative positional relationship of different data points of the target data in different dimensional spaces to remain unchanged; a processing unit, used to perform dimensionality reduction processing on the target data according to the local reconstruction weights to obtain target feature data of the target data; a determination unit, used to determine missing data in the target data based on the target feature data; and a repair unit, used to repair missing data by analyzing the spatiotemporal information and parameter association relationships between different data in the target data.

[0013] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is run, the device where the computer-readable storage medium is located executes the above-mentioned method for processing river monitoring data.

[0014] According to another aspect of an embodiment of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned method for processing river monitoring data.

[0015] From the above content, it can be seen that the present application fuses the river monitoring data from different monitoring equipment in the form of tensors to obtain the fused target data; performs local manifold structure analysis on the target data to obtain the local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; the local reconstruction weights are used to constrain the relative position relationship of different data points of the target data in different dimensional spaces to remain unchanged; performs dimensionality reduction processing on the target data according to the local reconstruction weights to obtain the target feature data of the target data; determines the missing data in the target data based on the target feature data; and repairs the missing data by analyzing the spatiotemporal information and parameter correlation relationship between different data in the target data.

[0016] In an embodiment of the present application, tensor decomposition and local manifold structure analysis are adopted to fuse river monitoring data from different monitoring equipment in the form of tensors, and local manifold structure analysis is performed to determine the intrinsic manifold structure information and local reconstruction weights between data points, thereby achieving the purpose of efficient fusion and feature extraction of multi-source heterogeneous data, thereby realizing dimensionality reduction processing of target data and accurate extraction of target feature data, and further solving the technical problem of low accuracy in processing abnormal data such as missing data caused by the fact that river monitoring data belongs to a large amount of multi-source heterogeneous data in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a flowchart of an optional method for processing river monitoring data according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of another optional method for processing river monitoring data according to an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of an optional device for processing river monitoring data according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and the relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain the relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0024] According to an embodiment of the present application, an embodiment of a method for processing river monitoring data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] Optionally, according to an embodiment of the present application, a river monitoring data processing system (hereinafter referred to as the system) is provided as the execution subject of the river monitoring data processing method of the embodiment of the present application, wherein the system can be a software system or an embedded system combining software and hardware. Of course, the method execution subject in the embodiment of the present application can also be other forms of execution subjects, such as devices, equipment, etc. Those skilled in the art should know that this application does not specifically limit the specific form of expression of the method execution subject.

[0026] Figure 1 is a flow chart of an optional method for processing river monitoring data according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps:

[0027] Step S101 , the river monitoring data from different monitoring devices are fused in the form of tensors to obtain fused target data.

[0028] Optionally, the system can obtain fused target data by applying multi-source heterogeneous data fusion technology.

[0029] Optionally, the system can input river monitoring data transmitted from various monitoring devices into a data fusion module. This module first performs a preliminary data format conversion on the river monitoring data, converting various non-standard data formats into a unified format (such as JSON) for subsequent processing. Furthermore, the module performs basic cleaning operations on the river monitoring data, removing data records with obvious errors or illogicality, such as abnormal timestamps and monitoring values ​​outside a reasonable range.

[0030] Subsequently, the data fusion module constructs the original tensor based on the source and dimension information of the river monitoring data. For example, for river monitoring data containing three dimensions: water quality parameters, monitoring time, and monitoring location, the water quality parameters can be used as the first dimension, the monitoring time as the second dimension, and the monitoring location as the third dimension to construct a three-dimensional tensor. Assuming there are 10 water quality parameters, 100 monitoring time points, and 20 monitoring locations, the original tensor .

[0031] Next, the data fusion module initializes the core tensor G and factor matrix . Set the dimension of the core tensor G to ,Right now , , The data fusion module then performs iterative calculations using the Alternating Least Squares (ALS) method. In each iteration, one of the factor matrices or core tensors is updated while the other factor matrices and core tensors are fixed.

[0032] For example, in the first iteration, fix , by minimizing the reconstruction error To update The iterative process is repeated until the reconstruction error converges to a smaller threshold (such as ).

[0033] Subsequently, the data fusion module analyzes and processes the decomposed tensors to remove noise and redundant information. In addition, the data fusion module combines the core tensor G and the factor matrix through n-modular product operation. Recombine to obtain the fused target data.

[0034] Optionally, this process includes format conversion and preliminary cleaning of the river monitoring data, as well as in-depth analysis and processing of the data. For example, tensor decomposition can effectively handle missing values ​​and outliers in the river monitoring data. When constructing the tensor, the system considers the time series characteristics and spatial location information of the river monitoring data, so that the fused target data can better reflect the spatiotemporal dynamic changes of the river monitoring data.

[0035] Alternatively, through Tucker decomposition, the system can align river monitoring data from different sources into the same temporal and spatial framework, thereby resolving data silos caused by differing protocols and sampling frequencies. This approach not only improves data integration efficiency but also facilitates subsequent data processing. Furthermore, the fused target data can be used in a variety of applications, such as river ecological and environmental assessments, pollution warnings, and water resource management, providing more accurate and reliable data support for river management.

[0036] Step S102 : performing local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data.

[0037] In step S102, local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; local reconstruction weights are used to constrain the relative position relationship of different data points of the target data in different dimensional spaces to remain unchanged.

[0038] Optionally, the system can input the fused target data into the LLE (Locally Linear Embedding) algorithm module for feature extraction. The LLE algorithm module can determine the local neighborhood for each data point. For example, the neighborhood size k can be set to 10, then for each data point The LLE algorithm module calculates the Euclidean distance between the data point and its 10 nearest neighbors in the local neighborhood, thereby determining the nearest neighbor point set N(i). The LLE algorithm module then constructs a linear equation system based on the nearest neighbor point set to solve the local reconstruction weight. By solving this system of equations, the local reconstruction weight of each data point is obtained.

[0039] Alternatively, local manifold structure analysis can effectively handle the complex structures in high-dimensional river monitoring data, especially when the data has nonlinear structures, and can better capture the intrinsic characteristics of river monitoring data. By using local reconstruction weights, high-dimensional river monitoring data can be mapped to a low-dimensional space while preserving the local structure of the river monitoring data.

[0040] Step S103 , performing dimensionality reduction processing on the target data according to the local reconstruction weight to obtain target feature data of the target data.

[0041] Optionally, the system can input the target data processed by LLE into a DAE (Deep Autoencoder) for further processing. The DAE's network structure is designed as follows: the number of neurons in the input layer matches the dimensionality of the target data after LLE processing, ensuring seamless integration of the target data into the DAE for processing. The DAE has three hidden layers, with 50, 30, and 10 neurons in each layer, respectively. This structure facilitates layer-by-layer feature extraction and dimensionality reduction. The output layer has the same number of neurons as the input layer to ensure that the integrity of the original river monitoring data is preserved after encoding and decoding. When training the DAE, the system can use a stochastic gradient descent algorithm, setting the learning rate to p (e.g., 0.001) and performing q (e.g., 100) iterations of training.

[0042] Optionally, through the above training process, DAE can learn high-level abstract features of the target data while removing redundant information in the target data, and ultimately obtain a low-dimensional feature representation (i.e., target feature data), providing more refined and effective data features for subsequent data analysis and processing.

[0043] Step S104: determining missing data in the target data according to the target feature data.

[0044] Optionally, the system can input the reduced feature data (i.e., target feature data) into the One-Class SVM model, select the radial basis kernel function (RBF) as the kernel function, and set the parameters , The system can train the One-Class SVM using training data and solve the objective function , get the optimal hyperplane normal vector and intercept , thereby determining the boundary model of normal data.

[0045] Then, the system substitutes all data points into the boundary model, and data points are marked as preliminary outliers.

[0046] Finally, for the initially identified abnormal points, the system can use GMM (Gaussian Mixture Model) for further analysis. Set the number of Gaussian components K=3 and initialize the parameters of the Gaussian mixture model. 、 and , and iteratively calculates using the expectation maximization (EM) algorithm. In the E step, GMM can calculate the posterior probability of each preliminary outlier belonging to each Gaussian component based on the current parameters; in the M step, GMM can update the parameters based on the posterior probability. 、 and GMM repeats the E and M steps until the parameters converge. Finally, the system calculates the probability value P(x) of the preliminary outlier under the Gaussian mixture model. If P(x) is lower than the set threshold (such as 0.1), the data point is determined to be a true outlier.

[0047] Step S105 , repairing missing data by analyzing the spatiotemporal information and parameter association relationships between different data in the target data.

[0048] Optionally, the system can repair missing data using a data repair module based on spatiotemporal interpolation and Bayesian network correction technology.

[0049] Optionally, the system can use the data repair module to make a preliminary estimate of the missing data detected based on the time series and spatial location information (i.e., spatiotemporal information) of the missing data using the spatiotemporal kriging interpolation method. First, the data repair module can calculate the spatiotemporal distance between known data points and construct a semivariogram. The data repair module can select a spherical model as the semivariogram model and fit the semivariogram using the least squares method to obtain the model parameters. Subsequently, the data repair module can calculate the interpolation weights based on the fitted semivariogram model. , and then substitute into the formula Calculate the estimated values ​​of the missing data points to obtain preliminary repair data.

[0050] Optionally, the data repair module can also construct a Bayesian network model, which uses various parameters of river monitoring data, such as water quality parameters, flow, water level, etc., as nodes in the network. Then, with the help of the knowledge of domain experts and analysis of data correlation, the causal relationship and correlation between these nodes are clarified, and the structure of the entire network is built. After the Bayesian network structure is built, the data repair module uses known data to learn and determine the conditional probability distribution of the Bayesian network. When faced with a data point after preliminary repair, the data repair module will use the joint tree algorithm to perform Bayesian reasoning based on the structure of the Bayesian network and the values ​​of the known nodes to calculate the probability distribution of the data point. Finally, the data repair module will select the largest probability value from the probability distribution and use it as the final repair value of the data point to obtain complete and intact repaired data.

[0051] Optionally, Figure 2 is a flow chart of another optional method for processing river monitoring data according to an embodiment of the present application, such as Figure 2 As shown in the figure, the system first performs a multi-source heterogeneous data fusion step, integrating river monitoring data from different monitoring devices. This river monitoring data includes various data types, such as water quality parameters, flow, and water level. Next, during the data access and preprocessing phase, the received river monitoring data undergoes format conversion and cleansing to ensure its quality and consistency. After preprocessing, the river monitoring data enters the tensor construction phase. The data is constructed into a tensor form. Tucker decomposition decomposes the tensor into a core tensor and a factor matrix, thereby reconstructing and fusing the river monitoring data and obtaining the fused target data.

[0052] Subsequently, the system performs feature extraction and dimensionality reduction on the fused target data, using local linear embedding and deep autoencoder processing to extract the key features of the target data and reduce its dimension, thereby improving analysis efficiency.

[0053] In the anomaly detection stage, the system constructs the boundary of normal data by training the One-Class SVM model, and uses Gaussian mixture model (GMM) analysis to further identify and determine anomalies.

[0054] Once the outliers are identified, the data repair process begins. This involves using spatiotemporal interpolation methods to make initial estimates of the missing data, followed by corrections through a Bayesian network. The Bayesian network uses a joint tree algorithm to perform Bayesian inference to calculate the probability distribution of the data points and select the value with the highest probability as the final repair value.

[0055] Finally, the restored data is organized and formatted during the data output and application phase, and output through push notifications, APIs, reports, charts, and other formats. This entire process ensures the high-precision management and effective utilization of river monitoring data, providing a reliable and accurate data foundation for river management.

[0056] From the above content, it can be seen that the present application adopts the method of tensor decomposition and local manifold structure analysis. By fusing the river monitoring data from different monitoring equipment in the form of tensors, and performing local manifold structure analysis to determine the intrinsic manifold structure information and local reconstruction weights between data points, the purpose of efficient fusion and feature extraction of multi-source heterogeneous data is achieved, thereby realizing the dimensionality reduction processing of target data and accurate extraction of target feature data, and thus solving the technical problem of low accuracy in processing abnormal data such as missing data caused by the fact that river monitoring data belongs to a large amount of multi-source heterogeneous data in the existing technology.

[0057] In an optional embodiment, river monitoring data from different monitoring devices are fused in tensor form to obtain fused target data, including: converting the river monitoring data into an original tensor; decomposing the original tensor into a core tensor and N factor matrices, wherein the core tensor includes the relationship between the feature information of the original tensor and different dimensional spaces; the N factor matrices are used to characterize the feature transformation of the original tensor in N dimensional spaces; and performing modular product operations on the core tensor and the N factor matrices to obtain the target data.

[0058] Optionally, the system can achieve effective fusion of multi-source data through a multi-source heterogeneous data fusion module. This module employs a multi-source data fusion method based on tensor decomposition, constructing river monitoring data from different sources and dimensions (such as the multiple parameter dimensions of water quality data and the spatiotemporal dimensions of flow data) into a tensor format. Furthermore, using the Tucker decomposition algorithm, the high-dimensional tensor is decomposed into a core tensor and multiple factor matrices. By analyzing and reconstructing the core tensor and factor matrices, potential relationships between the data are explored, thereby achieving effective fusion of multi-source data.

[0059] Specifically, the Tucker decomposition algorithm is as follows: First, the original tensor is a multidimensional array, and each dimension corresponds to different data features and attributes. Let the original tensor be ,in Represents the size of the tensor in each dimension. For example, for river monitoring data containing three dimensions of water quality parameters (such as pH value, dissolved oxygen, conductivity, etc.), monitoring time, and monitoring location, a three-dimensional tensor can be constructed, where It can be the number of water quality parameters, It can be the number of monitoring time points, This could be the number of monitoring locations.

[0060] Next, formula (1) is obtained by Tucker decomposition.

[0061] As shown in formula (1), is the core tensor, is the size of the core tensor in each dimension, and is usually ,The core tensor contains the main feature information of the original tensor and the relationship between different dimensions. is a factor matrix, which represents the feature transformation of the original tensor in the nth dimension. For example, The information of the original tensor in the first dimension (such as the water quality parameter dimension) can be converted into a new feature space, thereby achieving effective extraction and conversion of data features.

[0062] (1)

[0063] Finally, the n-module product operation is a specific operation method for tensors and matrices. Through the n-module product operation, the core tensor and factor matrix can be recombined to obtain the fused target data. This process can not only retain the main features and dimensional relationships of the original data, but also explore the potential relationships between the data and achieve effective fusion of multi-source heterogeneous data. Among them, formula (2) is the n-module product operation formula of tensor A and matrix B. As shown in formula (2), Represents the n-module product operation of tensor and matrix, tensor ,matrix .

[0064] (2)

[0065] Alternatively, multi-source heterogeneous data fusion technology, based on the Tucker algorithm for tensor decomposition, can effectively fuse multi-source, high-dimensional, and heterogeneous river monitoring data, mining the underlying complex relationships between the data to improve data consistency and integrity. Furthermore, reconstruction of the fused data can fill in missing data points, providing a more reliable data foundation for subsequent data analysis and applications.

[0066] In an optional embodiment, in the process of performing local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, the method also includes: taking each local data in the target data as a data point, wherein the data point is represented in the form of a vector; taking k data points in the target data whose distance to the i-th data point is less than a preset threshold as the k nearest neighbor points of the i-th data point, wherein i is an integer greater than or equal to 1, and k is an integer greater than 1; performing norm calculation based on the i-th data point, the k nearest neighbor points and the weight value of each nearest neighbor point to obtain the local reconstruction weight between the i-th data point and the k nearest neighbor points.

[0067] Optionally, the system can complete the above process through a feature extraction and dimensionality reduction module. Specifically, the feature extraction and dimensionality reduction module can use an improved LLE algorithm combined with DAE to perform feature extraction and dimensionality reduction.

[0068] Optionally, the specific steps of the LLE algorithm are as follows: first, the LLE algorithm is used to perform local manifold structure analysis on the fused data to calculate the local reconstruction weights between data points.

[0069] For example, let the data point There are k nearest neighbors in the local neighborhood of ,but The reconstruction weight is .

[0070] in, is a vector representing the i-th data point in the dataset, whose dimension depends on the number of features of the original data. Represents data points In its local neighborhood, use the jth nearest neighbor point The weight when reconstructing, and satisfying . N(i) represents the data point The set of neighboring points in the local neighborhood of . Represents the norm of a vector, usually the Euclidean norm.

[0071] Optionally, the system can quantify the local similarity between data points by calculating a local reconstruction weight between each data point and its nearest neighbors. This local reconstruction weight not only reflects the distance between data points but also takes into account the relative position of the data points in the local structure, thereby better preserving the local geometric structure of the target data.

[0072] In an optional embodiment, the target data is subjected to dimensionality reduction processing according to the local reconstruction weights to obtain target feature data of the target data, including: inputting the local reconstruction weights between different local data in the target data and the target data into a deep autoencoder, wherein the number of neurons in the input layer and the number of neurons in the output layer of the deep autoencoder are the same as the number of data dimensions of the target data, removing redundant information in the target data by the deep autoencoder, and extracting feature data from the remaining information; and performing dimensionality reduction processing on the extracted feature data by the deep autoencoder based on the local reconstruction weights between different local data in the target data to obtain target feature data.

[0073] For example, the network structure of a DAE (Deep Autoencoder) is as follows: the number of neurons in the input layer matches the dimension of the data after LLE processing; the hidden layers are set to three, with 50, 30, and 10 neurons, respectively; and the output layer has the same number of neurons as the input layer. The system can train the DAE using a stochastic gradient descent algorithm, with a learning rate of 0.001 and 100 iterations. Through training, the DAE learns high-level abstract features of the data, removes redundant information, and obtains a low-dimensional feature representation, thereby obtaining the target feature data.

[0074] Optionally, the local linear embedding algorithm combined with the deep autoencoder can not only retain the local manifold structure information of the target data, but also learn the high-level abstract features of the target data, achieve efficient dimensionality reduction of the target data, and reduce the complexity of data processing.

[0075] In an optional embodiment, missing data in target data is determined based on target feature data, including: inputting the target feature data into a target model, converting the target feature data into a feature vector through the target model, detecting the vector distance between the feature vector and the normal vector of the hyperplane, and determining the missing data in the target data based on the vector distance; wherein the target model is composed of a vector machine model and a Gaussian mixture model, and the normal vector of the hyperplane is used to define the boundary of normal data.

[0076] Optionally, the system can complete the above steps through an anomaly detection module (i.e., target model). Specifically, the anomaly detection module performs anomaly detection based on a method combining One-Class SVM (one-class support vector machine) and Gaussian mixture model (GMM).

[0077] Alternatively, the specific process of using a One-Class SVM and a Gaussian mixture model to identify abnormal data such as missing data is as follows: First, the anomaly detection module uses a One-Class SVM to train the target feature data after dimensionality reduction, converting it into a feature vector to construct a boundary model for normal data. Formula (3) is the objective function of the One-Class SVM.

[0078] (3)

[0079] In formula (3), the constraint condition of the objective function is , w is the normal vector of the hyperplane, which is used to define the boundary of the normal data. is a slack variable used to allow some data points to be on the wrong side of the hyperplane to handle noise and anomalies in the data. is the intercept of the hyperplane. v is a parameter that controls the number of support vectors and the error tolerance. . n is the number of training data. The data points Functions that map to high-dimensional feature spaces (usually using kernel functions such as radial basis kernel functions) ,in, are the parameters of the kernel function).

[0080] Alternatively, a method combining a vector machine model and a Gaussian mixture model can leverage the advantages of the vector machine model in boundary delineation and the Gaussian mixture model in data distribution modeling. This allows for more accurate identification of missing data within the target data, improving the precision and reliability of anomaly detection and effectively addressing noise and anomalies in river monitoring data. Furthermore, by measuring the vector distance between the eigenvector and the hyperplane normal vector, the system can quantify the degree of deviation of each data point from the normal data boundary, effectively distinguishing between normal and missing data.

[0081] In an optional embodiment, before inputting the target feature data into the target model, slack variables, target parameters and objective functions are obtained, wherein the slack variables are used to constrain the amount of data allowed to appear on the error side of the hyperplane, the target parameters are used to control the number of support vectors of the vector machine model and the error tolerance of the vector machine model, and the objective function is used to map the target feature data to the target dimensional feature space; based on the slack variables, target parameters, objective function, normal vector and intercept of the hyperplane, the constraints of the vector machine model are determined.

[0082] Alternatively, in practical applications, since the data may contain noise or outliers, it is unrealistic to classify all data points correctly. Therefore, introducing slack variables to allow some data points to be on the wrong side of the hyperplane can improve the robustness of the model.

[0083] Optionally, the target parameter v directly affects the complexity and generalization ability of the target model. A smaller v value means that the target model is less tolerant to outliers, while a larger v value allows more data points to be considered as outliers.

[0084] Optionally, the target data can be mapped from the original space to a high-dimensional feature space (i.e., target dimension feature space) through the objective function, so as to better separate normal data and abnormal data.

[0085] In an optional embodiment, missing data is repaired by analyzing the spatiotemporal information and parameter associations between different data in the target data, including: obtaining the i-th data point of a known data value in the target data, wherein the position corresponding to the i-th data point is the first position, and the collection moment corresponding to the i-th data point is the first moment; determining the interpolation weight corresponding to the missing data at the second position and collected at the second moment based on the spatial distance between the first position and the second position and the time interval between the first moment and the second moment; determining the data value of the missing data based on the interpolation weight corresponding to the missing data and the known data value of the i-th data point; and filling the data value of the missing data into the second position of the target data to repair the missing data.

[0086] Optionally, the system can complete the above process through a data repair module. Specifically, the data repair module can adopt a joint data repair method based on spatiotemporal interpolation and Bayesian network.

[0087] Optionally, for missing data, the data repair module can make a preliminary estimate based on the time series and spatial location information of the target data using a spatiotemporal interpolation method (such as spatiotemporal Kriging interpolation). The missing points are estimated by space-time kriging interpolation. The value of .in, represents the value of a known data point at spatial location i and time t. represents a missing data point at spatial location j and time s that needs to be estimated. is the interpolation weight, which can be calculated based on the temporal and spatial correlation between data points and is usually determined by the semivariogram. Describes the degree of difference between data points in space and time, and h represents the temporal and spatial distance between two data points. Among them, the interpolation weight can be calculated by fitting the semivariogram model (such as the spherical model, exponential model, etc.) .

[0088] Alternatively, a joint data repair method based on spatiotemporal interpolation and Bayesian networks can fully utilize the spatiotemporal information of the target data and the causal relationship between parameters to accurately repair missing data and abnormal data, thereby improving the accuracy and availability of the target data.

[0089] According to another aspect of the embodiment of the present application, a device for processing river monitoring data is also provided, wherein: Figure 3 is a schematic diagram of an optional device for processing river monitoring data according to an embodiment of the present application, such as Figure 3 As shown, the apparatus for processing river monitoring data includes: a fusion unit 301 , an analysis unit 302 , a processing unit 303 , a determination unit 304 and a repair unit 305 .

[0090] Optionally, the fusion unit 301 is used to fuse the river monitoring data from different monitoring equipment in the form of tensors to obtain fused target data; the analysis unit 302 is used to perform local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; the local reconstruction weights are used to constrain the relative position relationship of different data points of the target data in different dimensional spaces to remain unchanged; the processing unit 303 is used to perform dimensionality reduction processing on the target data according to the local reconstruction weights to obtain target feature data of the target data; the determination unit 304 is used to determine the missing data in the target data based on the target feature data; the repair unit 305 is used to repair the missing data by analyzing the spatiotemporal information and parameter association relationship between different data in the target data.

[0091] Optionally, the fusion unit 301 includes: a conversion subunit, a decomposition subunit, and a first calculation subunit. The conversion subunit is used to convert the river monitoring data into an original tensor; the decomposition subunit is used to decompose the original tensor into a core tensor and N factor matrices, wherein the core tensor includes the relationship between the feature information of the original tensor and different dimensional spaces; the N factor matrices are used to represent the feature transformation of the original tensor in the N dimensional space; and the first calculation subunit is used to perform a modular product operation on the core tensor and the N factor matrices to obtain target data.

[0092] Optionally, the analysis unit 302 includes: a first processing subunit, a second processing subunit, and a second calculation subunit. The first processing subunit is configured to treat each local data in the target data as a data point, wherein the data point is represented in the form of a vector; the second processing subunit is configured to treat k data points in the target data whose distance from the i-th data point is less than a preset threshold as k neighboring points of the i-th data point, wherein i is an integer greater than or equal to 1, and k is an integer greater than 1; and the second calculation subunit is configured to perform norm calculation based on the i-th data point, the k neighboring points, and the weight value of each neighboring point to obtain a local reconstruction weight between the i-th data point and the k neighboring points.

[0093] Optionally, the processing unit 303 includes: an input subunit, an extraction subunit, and a third processing subunit. The input subunit is configured to input the local reconstruction weights between different local data in the target data and the target data into a deep autoencoder, wherein the number of neurons in the input layer and the number of neurons in the output layer of the deep autoencoder are the same as the number of data dimensions of the target data; the extraction subunit is configured to remove redundant information in the target data through the deep autoencoder and extract feature data from the remaining information; and the third processing subunit is configured to perform dimensionality reduction processing on the extracted feature data based on the local reconstruction weights between different local data in the target data through the deep autoencoder to obtain target feature data.

[0094] Optionally, the determination unit 304 includes a fourth processing subunit. The fourth processing subunit is configured to input the target feature data into a target model, convert the target feature data into a feature vector using the target model, detect a vector distance between the feature vector and a normal vector of a hyperplane, and determine missing data in the target data based on the vector distance; wherein the target model comprises a vector machine model and a Gaussian mixture model, and the normal vector of the hyperplane is used to define a boundary of normal data.

[0095] Optionally, the determination unit 304 further includes: a first acquisition subunit and a first determination subunit. The first acquisition subunit is configured to acquire slack variables, target parameters, and an objective function, wherein the slack variables are used to constrain the amount of data allowed to appear on the error side of the hyperplane, the target parameters are used to control the number of support vectors of the vector machine model and the error tolerance of the vector machine model, and the objective function is used to map the target feature data to a target dimensional feature space; and the determination subunit is configured to determine the constraint conditions of the vector machine model based on the slack variables, target parameters, objective function, normal vector, and intercept of the hyperplane.

[0096] Optionally, the repair unit 305 includes: a second acquisition subunit, a second determination subunit, a third determination subunit, and a repair subunit. The second acquisition subunit is used to acquire the i-th data point of a known data value in the target data, wherein the position corresponding to the i-th data point is the first position and the collection time corresponding to the i-th data point is the first moment; the second determination subunit is used to determine the interpolation weight corresponding to the missing data at the second position and collected at the second moment based on the spatial distance between the first position and the second position and the time interval between the first moment and the second moment; the third determination subunit is used to determine the data value of the missing data based on the interpolation weight corresponding to the missing data and the known data value of the i-th data point; and the repair subunit is used to fill the data value of the missing data into the second position of the target data to repair the missing data.

[0097] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is run, the device where the computer-readable storage medium is located executes the above-mentioned method for processing river monitoring data.

[0098] According to another aspect of an embodiment of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned method for processing river monitoring data.

[0099] The above-mentioned embodiments or examples disclosed in this application are not exhaustive, but are only illustrations of some embodiments or examples, and are not intended to be specific limitations on the scope of protection disclosed in this application. In the absence of contradiction, each step in a certain embodiment or example in this application can be implemented as an independent example, and the steps can be arbitrarily combined. For example, the solution after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be arbitrarily combined; in addition, the various embodiments or examples can be arbitrarily combined. For example, some or all of the steps in different embodiments or examples can be arbitrarily combined, and a certain embodiment or example can be arbitrarily combined with the optional methods or optional examples of other embodiments or examples.

[0100] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0104] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, and other media that can store program code.

[0106] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing river monitoring data, characterized in that: include: The river monitoring data from different monitoring equipment are fused in the form of tensors to obtain the fused target data; Performing a local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine intrinsic manifold structure information between different data points in the target data; and the local reconstruction weights are used to constrain the relative positional relationships between different data points of the target data in different dimensional spaces to remain unchanged; Performing dimensionality reduction processing on the target data according to the local reconstruction weight to obtain target feature data of the target data; Determining missing data in the target data according to the target feature data; The missing data is repaired by analyzing the spatiotemporal information and parameter association relationships between different data in the target data.

2. The method according to claim 1, characterized in that The river monitoring data from different monitoring devices are fused in tensor form to obtain the fused target data, including: Converting the river monitoring data into a raw tensor; Decomposing the original tensor into a core tensor and N factor matrices, wherein the core tensor includes the relationship between the feature information of the original tensor and different dimensional spaces; the N factor matrices are used to represent the feature transformation of the original tensor in the N dimensional spaces; Perform a modular product operation on the core tensor and the N factor matrices to obtain the target data.

3. The method according to claim 1, characterized in that In the process of performing local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, the method further includes: Taking each local data in the target data as a data point, wherein the data point is represented in a vector form; k data points in the target data whose distance to the i-th data point is less than a preset threshold are used as k neighboring points of the i-th data point, where i is an integer greater than or equal to 1 and k is an integer greater than 1; A norm calculation is performed based on the i-th data point, the k neighboring points, and the weight value of each neighboring point to obtain a local reconstruction weight between the i-th data point and the k neighboring points.

4. The method according to claim 1, wherein Performing dimensionality reduction processing on the target data according to the local reconstruction weight to obtain target feature data of the target data includes: Inputting the local reconstruction weights between different local data in the target data and the target data into a deep autoencoder, wherein the number of neurons in the input layer and the number of neurons in the output layer of the deep autoencoder are the same as the number of data dimensions of the target data, Removing redundant information from the target data by the deep autoencoder and extracting feature data from the remaining information; The deep autoencoder performs dimensionality reduction processing on the extracted feature data based on the local reconstruction weights between different local data in the target data to obtain the target feature data.

5. The method according to claim 1, characterized in that Determining missing data in the target data according to the target feature data includes: The target feature data is input into a target model, the target feature data is converted into a feature vector through the target model, the vector distance between the feature vector and the normal vector of the hyperplane is detected, and the missing data in the target data is determined based on the vector distance; wherein, the target model is composed of a vector machine model and a Gaussian mixture model, and the normal vector of the hyperplane is used to define the boundary of normal data.

6. The method according to claim 5, characterized in that Before inputting the target feature data into the target model, the method further includes: Obtaining slack variables, target parameters, and an objective function, wherein the slack variables are used to constrain the amount of data allowed to appear on the error side of the hyperplane, the target parameters are used to control the number of support vectors of the vector machine model and the error tolerance of the vector machine model, and the objective function is used to map the target feature data to a target dimensional feature space; The constraint conditions of the vector machine model are determined according to the slack variables, the target parameters, the objective function, the normal vector, and the intercept of the hyperplane.

7. The method according to claim 1, characterized in that Repairing the missing data by analyzing the temporal and spatial information and parameter association relationships between different data in the target data includes: Acquire an i-th data point of a known data value in the target data, wherein a position corresponding to the i-th data point is a first position, and a collection time corresponding to the i-th data point is a first time; Determining, based on the spatial distance between the first position and the second position and the time interval between the first moment and the second moment, an interpolation weight corresponding to the missing data at the second position and collected at the second moment; Determining the data value of the missing data according to the interpolation weight corresponding to the missing data and the known data value of the i-th data point; The data value of the missing data is filled into the second position of the target data to repair the missing data.

8. A device for processing river monitoring data, characterized in that: include: A fusion unit is used to fuse the river monitoring data from different monitoring devices in the form of tensors to obtain fused target data; an analysis unit, configured to perform a local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine intrinsic manifold structure information between different data points in the target data; and the local reconstruction weights are used to constrain the relative positional relationships between different data points of the target data in different dimensional spaces to remain unchanged; a processing unit, configured to perform dimensionality reduction processing on the target data according to the local reconstruction weight to obtain target feature data of the target data; a determining unit, configured to determine missing data in the target data based on the target feature data; The repair unit is used to repair the missing data by analyzing the spatiotemporal information and parameter association relationship between different data in the target data.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the method for processing river monitoring data according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the river monitoring data processing method described in any one of claims 1 to 7.

Images