River monitoring data processing method, device and electronic equipment

Abstract

The application discloses a river monitoring data processing method and device and electronic equipment, and relates to the water conservancy field and the data processing field. The method comprises the following steps: fusing river monitoring data from different monitoring devices in the form of a tensor to obtain target data after fusion; performing local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data; performing dimension reduction processing on the target data according to the local reconstruction weights to obtain target feature data of the target data; determining missing data in the target data according to the target feature data; and repairing the missing data by analyzing the spatiotemporal information and parameter correlation relationship between different data in the target data. The application solves the technical problem of low processing accuracy of abnormal data such as missing data caused by the fact that river monitoring data belongs to a large amount of multi-source heterogeneous data.

Description

Technical Field

[0001] This application relates to the fields of water conservancy and data processing, and more specifically, to a method, apparatus, and electronic equipment for processing river monitoring data. Background Technology

[0002] River monitoring data governance is a key technology for water resource management and disaster prevention. However, traditional methods have many shortcomings. First, data silos are severe; multi-source sensor data with different protocols and sampling frequencies are difficult to align spatiotemporally, leading to the failure of cross-dimensional correlation analysis. Second, existing technologies suffer from low accuracy in handling missing data and other anomalies due to the large volume of heterogeneous, multi-source river monitoring data, making them unsuitable for dynamic environments and resulting in high false alarm rates. Furthermore, traditional interpolation methods suffer from severe distortion when filling in missing values, leading to large prediction errors in scenarios with continuous missing values. Finally, database architectures relying on offline batch processing cannot meet real-time requirements, resulting in long data processing delays.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for processing river monitoring data, which at least solves the technical problem of low accuracy in processing abnormal data such as missing data caused by the large amount of multi-source heterogeneous data in the prior art.

[0005] According to one aspect of the embodiments of this application, a method for processing river monitoring data is provided, comprising: fusing river monitoring data from different monitoring devices in tensor form to obtain fused target data; performing local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; the local reconstruction weights are used to constrain the relative positional relationship between different data points in different dimensional spaces of the target data to remain unchanged; performing dimensionality reduction processing on the target data according to the local reconstruction weights to obtain target feature data of the target data; determining missing data in the target data according to the target feature data; and repairing the missing data by analyzing the spatiotemporal information and parameter correlation between different data in the target data.

[0006] Optionally, river monitoring data from different monitoring devices are fused in tensor form to obtain fused target data, including: converting the river monitoring data into original tensors; decomposing the original tensors into a core tensor and N factor matrices, wherein the core tensor includes the feature information of the original tensor and the interrelationships between different dimensional spaces; the N factor matrices are used to characterize the feature transformations of the original tensor in the N dimensional spaces; and performing a modular product operation on the core tensor and the N factor matrices to obtain the target data.

[0007] Optionally, in the process of performing local manifold structure analysis on the target data to obtain the local reconstruction weights between different local data in the target data, the method further includes: treating each local data in the target data as a data point, wherein the data point is represented in vector form; treating the k data points in the target data whose distance to the i-th data point is less than a preset threshold as the k nearest neighbors of the i-th data point, wherein i is an integer greater than or equal to 1, and k is an integer greater than 1; calculating the norm based on the i-th data point, the k nearest neighbors, and the weight value of each nearest neighbor to obtain the local reconstruction weights between the i-th data point and the k nearest neighbors.

[0008] Optionally, the target data is dimensionality reduced based on the local reconstruction weights to obtain the target feature data. This includes: inputting the local reconstruction weights between different local data in the target data and the target data into a deep autoencoder, wherein the number of neurons in the input layer and the number of neurons in the output layer of the deep autoencoder are the same as the number of data dimensions of the target data; removing redundant information from the target data through the deep autoencoder and extracting feature data from the remaining information; and performing dimensionality reduction on the extracted feature data based on the local reconstruction weights between different local data in the target data through the deep autoencoder to obtain the target feature data.

[0009] Optionally, determining missing data in the target data based on the target feature data includes: inputting the target feature data into the target model, converting the target feature data into feature vectors through the target model, detecting the vector distance between the feature vectors and the normal vector of the hyperplane, and determining the missing data in the target data based on the vector distance; wherein the target model consists of a vector machine model and a Gaussian mixture model, and the normal vector of the hyperplane is used to define the boundary of the normal data.

[0010] Optionally, before inputting the target feature data into the target model, slack variables, target parameters, and objective function are obtained. The slack variables are used to constrain the number of data points allowed on the error side of the hyperplane, the target parameters are used to control the number of support vectors and the error tolerance of the vector machine model, and the objective function is used to map the target feature data to the target dimension feature space. The constraints of the vector machine model are determined based on the slack variables, target parameters, objective function, normal vector, and the intercept of the hyperplane.

[0011] Optionally, missing data can be repaired by analyzing the spatiotemporal information and parameter correlations between different data in the target data. This includes: obtaining the i-th data point with known data values ​​in the target data, where the location of the i-th data point is the first position and the acquisition time of the i-th data point is the first time; determining the interpolation weight corresponding to the missing data located at the second position and acquired at the second time based on the spatial distance between the first and second positions and the time interval between the first and second times; determining the data value of the missing data based on the interpolation weight corresponding to the missing data and the known data value of the i-th data point; and filling the missing data value into the second position of the target data to repair the missing data.

[0012] According to another aspect of the embodiments of this application, a processing apparatus for river monitoring data is also provided, comprising: a fusion unit for fusing river monitoring data from different monitoring devices in tensor form to obtain fused target data; an analysis unit for performing local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; and the local reconstruction weights are used to constrain the relative positional relationship between different data points in different dimensional spaces of the target data to remain unchanged; a processing unit for performing dimensionality reduction processing on the target data according to the local reconstruction weights to obtain target feature data of the target data; a determination unit for determining missing data in the target data according to the target feature data; and a repair unit for repairing the missing data by analyzing the spatiotemporal information and parameter correlation between different data in the target data.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described method for processing river monitoring data.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described method for processing river monitoring data.

[0015] As described above, this application fuses river monitoring data from different monitoring devices in tensor form to obtain fused target data; performs local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; the local reconstruction weights are used to constrain the relative positional relationship between different data points in different dimensional spaces to remain unchanged; performs dimensionality reduction processing on the target data based on the local reconstruction weights to obtain target feature data of the target data; determines the missing data in the target data based on the target feature data; and repairs the missing data by analyzing the spatiotemporal information and parameter correlation between different data in the target data.

[0016] In this embodiment, tensor decomposition and local manifold structure analysis are used to fuse river monitoring data from different monitoring devices in tensor form and perform local manifold structure analysis to determine the intrinsic manifold structure information and local reconstruction weights between data points. This achieves the goal of efficient fusion and feature extraction of multi-source heterogeneous data, thereby realizing dimensionality reduction of target data and accurate extraction of target feature data. This solves the technical problem of low accuracy in processing abnormal data such as missing data caused by the large amount of multi-source heterogeneous river monitoring data in the prior art. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart of an optional method for processing river monitoring data according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of another optional method for processing river monitoring data according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of an optional river monitoring data processing device according to an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0024] According to an embodiment of this application, an embodiment of a method for processing river monitoring data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] Optionally, according to the embodiments of this application, a river monitoring data processing system (hereinafter referred to as the system) is provided as the execution subject of the river monitoring data processing method of the embodiments of this application. The system can be a software system or an embedded system combining software and hardware. Of course, the method execution subject in the embodiments of this application can also be other forms of execution subject, such as devices, equipment, etc. It should be known by those skilled in the art that this application does not particularly limit the specific form of the method execution subject.

[0026] Figure 1 This is a flowchart of an optional method for processing river monitoring data according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0027] Step S101: The river monitoring data from different monitoring devices are fused in tensor form to obtain the fused target data.

[0028] Optionally, the system can obtain the fused target data by using multi-source heterogeneous data fusion technology.

[0029] Optionally, the system can input river monitoring data transmitted from different monitoring devices into the data fusion module. The data fusion module first performs a preliminary data format conversion on the river monitoring data, transforming various non-standard formats into a unified format (such as JSON) for subsequent processing. Simultaneously, the data fusion module performs basic data cleaning operations, removing obviously erroneous or illogical data records, such as abnormal timestamps and monitoring values ​​exceeding reasonable ranges.

[0030] Subsequently, the data fusion module constructs an original tensor based on the source and dimensional information of the river monitoring data. For example, for river monitoring data containing three dimensions—water quality parameters, monitoring time, and monitoring location—water quality parameters can be used as the first dimension, monitoring time as the second dimension, and monitoring location as the third dimension, thus constructing a three-dimensional tensor. Assuming there are 10 water quality parameters, 100 monitoring time points, and 20 monitoring locations, the original tensor... .

[0031] Next, the data fusion module initializes the core tensor G and the factor matrix. Let the dimension of the core tensor G be . ,Right now , , The data fusion module then uses ALS (Alternating Least Squares) for iterative calculations. In each iteration, the other factor matrices and the core tensor are fixed, and one of the factor matrices or the core tensor is updated.

[0032] For example, in the first iteration, fix By minimizing the reconstruction error Update Repeat the iterative process until the reconstruction error converges to a small threshold (e.g., ...). ).

[0033] Subsequently, the data fusion module analyzes and processes the decomposed tensors, removing noise and redundant information. Furthermore, the data fusion module uses n-modulus operations to combine the core tensor G and the factor matrix. The data is then recombined to obtain the fused target data.

[0034] Optionally, the above process includes format conversion and preliminary cleaning of the river monitoring data, as well as in-depth analysis and processing of the river monitoring data. For example, tensor decomposition can effectively handle missing and outlier values ​​in the river monitoring data. When constructing tensors, the system considers the time-series characteristics and spatial location information of the river monitoring data, enabling the fused target data to better reflect the spatiotemporal dynamic changes of the river monitoring data.

[0035] Optionally, through Tucker decomposition, the system can align river monitoring data from different sources into the same temporal and spatial framework, thereby resolving the data silo problem caused by protocol differences and varying sampling frequencies. This method not only improves data integration efficiency but also facilitates subsequent data processing. Furthermore, the fused target data can be used for various applications, such as river ecological environment assessment, pollution early warning, and water resource management, providing more accurate and reliable data support for river management.

[0036] Step S102: Perform local manifold structure analysis on the target data to obtain the local reconstruction weights between different local data in the target data.

[0037] In step S102, local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; local reconstruction weights are used to constrain the relative positional relationships of different data points in different dimensional spaces of the target data to remain unchanged.

[0038] Optionally, the system can input the fused target data into the LLE (Locally Linear Embedding) algorithm module for feature extraction. The LLE algorithm module can determine a local neighborhood for each data point; for example, the neighborhood size k can be set to 10, then for each data point... The LLE algorithm module calculates the Euclidean distance between the data point and its 10 nearest neighbors in the local neighborhood, thereby determining the set of nearest neighbors N(i). Subsequently, the LLE algorithm module constructs a system of linear equations to solve for the local reconstruction weights based on the set of nearest neighbors, and obtains the local reconstruction weights for each data point by solving this system of equations.

[0039] Optionally, local manifold structure analysis can effectively handle the complex structures in high-dimensional river monitoring data, especially when the data has a nonlinear structure, and can better capture the intrinsic characteristics of the river monitoring data. By reconstructing weights locally, high-dimensional river monitoring data can be mapped to a low-dimensional space while preserving the local structure of the river monitoring data.

[0040] Step S103: Dimensionality reduction is performed on the target data based on the local reconstruction weights to obtain the target feature data of the target data.

[0041] Optionally, the system can input the LLE-processed target data into a DAE (Deep Autoencoder) for further processing. The DAE's network structure is designed as follows: the number of neurons in the input layer matches the dimensionality of the LLE-processed target data to ensure seamless integration into the DAE. The DAE has three hidden layers with 50, 30, and 10 neurons respectively. This design facilitates feature extraction and gradual dimensionality reduction. The output layer has the same number of neurons as the input layer to maintain the integrity of the original river monitoring data after encoding and decoding. During DAE training, the system can employ a stochastic gradient descent algorithm with a learning rate of p (e.g., 0.001) and perform q iterations (e.g., 100 iterations).

[0042] Optionally, through the above training process, DAE can learn high-level abstract features of the target data, while removing redundant information in the target data, and finally obtain a low-dimensional feature representation (i.e. target feature data), providing more refined and effective data features for subsequent data analysis and processing.

[0043] Step S104: Determine the missing data in the target data based on the target feature data.

[0044] Optionally, the system can input the dimensionality-reduced feature data (i.e., the target feature data) into the One-Class SVM model, select the radial basis function (RBF) as the kernel function, and set the parameters. , The system can train a One-Class SVM using training data and solve the objective function. The optimal hyperplane normal vector is obtained. and intercept This allows us to determine the boundary model for normal data.

[0045] Subsequently, the system substitutes all data points into the boundary model, and for those that satisfy... The data points were marked as preliminary outliers.

[0046] Finally, for the outliers initially identified, the system can use a Gaussian Mixture Model (GMM) for further analysis. The number of Gaussian components, K=3, is set, and the parameters of the Gaussian Mixture Model are initialized. , and The Gaussian Model (GMM) is iteratively computed using the Expectation-Maximization (EM) algorithm. In the E-step, the GMM calculates the posterior probability of each initial outlier belonging to each Gaussian component based on the current parameters; in the M-step, the GMM updates the parameters based on the posterior probabilities. , and The GMM repeats the E-step and M-step until the parameters converge. Finally, the system calculates the probability value P(x) of the initial outlier under the Gaussian mixture model. If P(x) is lower than a set threshold (e.g., 0.1), the data point is determined to be a true outlier.

[0047] Step S105: By analyzing the spatiotemporal information and parameter correlations between different data in the target data, the missing data is repaired.

[0048] Optionally, the system can use a data repair module based on spatiotemporal interpolation and Bayesian network correction techniques to repair missing data.

[0049] Optionally, the system can use the data repair module to perform preliminary estimation of the detected missing data based on the time series and spatial location information (i.e., spatiotemporal information) of the missing data using the spatiotemporal Kriging interpolation method. First, the data repair module can calculate the spatiotemporal distance between known data points and construct a semivariogram. The data repair module can choose a spherical model as the semivariogram model and fit the semivariogram using the least squares method to obtain the model parameters. Subsequently, the data repair module can calculate the interpolation weights based on the fitted semivariogram model. Then substitute into the formula Calculate the estimated values ​​of the missing data points to obtain preliminary repaired data.

[0050] Optionally, the data repair module can also construct a Bayesian network model, which uses various parameters of river monitoring data, such as water quality parameters, flow rate, and water level, as nodes in the network. Then, leveraging domain expert knowledge and data correlation analysis, the causal relationships and correlations between these nodes are clarified, thus establishing the structure of the entire network. After the Bayesian network structure is built, the data repair module uses known data to learn and determine the conditional probability distribution of the Bayesian network. When faced with initially repaired data points, the data repair module uses a joint tree algorithm to perform Bayesian inference based on the structure of the Bayesian network and the known node values, thereby calculating the probability distribution of the data point. Finally, the data repair module selects the highest probability value from the probability distribution as the final repaired value for the data point, thus obtaining complete and intact repaired data.

[0051] Optionally, Figure 2 This is a flowchart of another optional method for processing river monitoring data according to an embodiment of this application, such as... Figure 2 As shown, the system first performs a multi-source heterogeneous data fusion step, integrating river monitoring data from different monitoring devices. This river monitoring data includes various data types, such as water quality parameters, flow rate, and water level. Next, in the data access and preprocessing stage, the accessed river monitoring data undergoes format conversion and cleaning to ensure its quality and consistency. After preprocessing, the river monitoring data enters the tensor construction stage, where it is constructed into tensor form. Tucker decomposition decomposes the tensor into a core tensor and a factor matrix, thereby achieving the reconstruction and fusion of the river monitoring data to obtain the fused target data.

[0052] Subsequently, the system performs feature extraction and dimensionality reduction on the fused target data. By using local linear embedding and deep autoencoder processing, the system extracts key features of the target data and reduces its dimensionality, thereby improving analysis efficiency.

[0053] During the anomaly detection phase, the system constructs the boundaries of normal data by training a One-Class SVM model and uses Gaussian Mixture Model (GMM) analysis to further identify and determine outliers.

[0054] Once outliers are identified, the data repair process begins. This involves making an initial estimate of the missing data using spatiotemporal interpolation methods, followed by refinement using a Bayesian network. The Bayesian network uses a joint tree algorithm for Bayesian inference to calculate the probability distribution of the data points and selects the value with the highest probability as the final repaired value.

[0055] Finally, the repaired data is organized and formatted during the data output and application phase, and output through message push, API interface, reports, charts, and other forms. This entire process ensures high-precision management and effective utilization of river monitoring data, providing a reliable and accurate data foundation for river management.

[0056] As can be seen from the above, this application adopts tensor decomposition and local manifold structure analysis to fuse river monitoring data from different monitoring devices in tensor form and perform local manifold structure analysis to determine the intrinsic manifold structure information and local reconstruction weights between data points. This achieves the goal of efficient fusion and feature extraction of multi-source heterogeneous data, thereby realizing dimensionality reduction of target data and accurate extraction of target feature data. It also solves the technical problem of low accuracy in processing abnormal data such as missing data caused by the large amount of multi-source heterogeneous river monitoring data in existing technologies.

[0057] In one optional embodiment, river monitoring data from different monitoring devices are fused in tensor form to obtain fused target data, including: converting the river monitoring data into an original tensor; decomposing the original tensor into a core tensor and N factor matrices, wherein the core tensor includes the feature information of the original tensor and the relationship between different dimensional spaces; the N factor matrices are used to characterize the feature transformation of the original tensor in the N dimensional spaces; and performing a modular product operation on the core tensor and the N factor matrices to obtain the target data.

[0058] Optionally, the system can achieve effective fusion of multi-source data through a multi-source heterogeneous data fusion module. This module employs a tensor decomposition-based multi-source data fusion method, constructing river monitoring data from different sources and dimensions (such as multiple parameter dimensions of water quality data, spatiotemporal dimensions of flow data, etc.) into tensor form. Furthermore, using the Tucker decomposition algorithm, the high-dimensional tensor is decomposed into a core tensor and multiple factor matrices. Through analysis and reconstruction of the core tensor and factor matrices, potential relationships between the data are uncovered, thereby achieving effective fusion of multi-source data.

[0059] Specifically, the Tucker decomposition algorithm works as follows: First, the original tensor is a multidimensional array, where each dimension corresponds to a different data feature and attribute. Let the original tensor be... ,in These represent the magnitude of the tensor in each dimension. For example, for river monitoring data containing three dimensions—water quality parameters (such as pH, dissolved oxygen, conductivity, etc.), monitoring time, and monitoring location—a three-dimensional tensor can be constructed, where... It can be the number of water quality parameters. It could be the number of monitoring time points. This could be the number of monitoring locations.

[0060] Next, formula (1) is the expression obtained by Tucker decomposition.

[0061] As shown in formula (1), For the core tensor, It is the size of the core tensor in each dimension, and usually... The core tensor contains the main feature information of the original tensor and the interrelationships between different dimensions. Let be the factor matrix, representing the feature transformation of the original tensor in the nth dimension. For example, The information of the original tensor in the first dimension (such as the water quality parameter dimension) can be transformed into a new feature space, thereby realizing the effective extraction and transformation of data features.

[0062] (1)

[0063] Finally, the n-modulus product operation is a specific operation method between tensors and matrices. Through the n-modulus product operation, the core tensor and factor matrix can be recombined to obtain the fused target data. This process can not only preserve the main features and dimensional relationships of the original data, but also mine the potential relationships between data, and realize the effective fusion of multi-source heterogeneous data. Among them, formula (2) is the formula for the n-modulus product operation of tensor A and matrix B, as shown in formula (2). This represents the n-modulus product operation between a tensor and a matrix. ,matrix .

[0064] (2)

[0065] Optionally, multi-source heterogeneous data fusion technology, based on the Tucker algorithm of tensor decomposition, can effectively fuse multi-source, high-dimensional, and heterogeneous river monitoring data, uncovering potential complex relationships between data points and thus improving data consistency and integrity. Furthermore, through the reconstruction of the fused data, missing parts can be filled, providing a more reliable data foundation for subsequent data analysis and applications.

[0066] In an optional embodiment, during the process of performing local manifold structure analysis on the target data to obtain the local reconstruction weights between different local data in the target data, the method further includes: treating each local data in the target data as a data point, wherein the data point is represented in vector form; treating the k data points in the target data whose distance to the i-th data point is less than a preset threshold as the k nearest neighbors of the i-th data point, wherein i is an integer greater than or equal to 1, and k is an integer greater than 1; and performing norm calculation based on the i-th data point, the k nearest neighbors, and the weight value of each nearest neighbor to obtain the local reconstruction weights between the i-th data point and the k nearest neighbors.

[0067] Optionally, the system can complete the above process through a feature extraction and dimensionality reduction module. Specifically, the feature extraction and dimensionality reduction module can use an improved LLE algorithm combined with DAE for feature extraction and dimensionality reduction.

[0068] Optionally, the specific steps of the LLE algorithm are as follows: First, the LLE algorithm is used to perform local manifold structure analysis on the fused data and calculate the local reconstruction weights between data points.

[0069] For example, let's say data points There are k nearest neighbors in the local neighborhood. ,but The reconstruction weights are .

[0070] in, It is a vector representing the i-th data point in the dataset, and its dimension depends on the number of features in the original data. Representing data points Within its local neighborhood, use the j-th nearest neighbor. The weights during reconstruction, and satisfying N(i) represents the data points. The set of nearest neighbors within a local neighborhood. The norm of a vector is usually represented by the Euclidean norm.

[0071] Optionally, the system can quantify the local similarity between data points by calculating the local reconstruction weights between each data point and its nearest neighbors. These local reconstruction weights not only reflect the distance between data points but also consider their relative positions within the local structure, thus better preserving the local geometric structure information of the target data.

[0072] In one optional embodiment, the target data is dimensionality reduced based on local reconstruction weights to obtain target feature data. This includes: inputting the local reconstruction weights between different local data in the target data and the target data into a deep autoencoder, wherein the number of neurons in the input layer and the number of neurons in the output layer of the deep autoencoder are the same as the number of data dimensions of the target data; removing redundant information from the target data through the deep autoencoder and extracting feature data from the remaining information; and performing dimensionality reduction on the extracted feature data based on the local reconstruction weights between different local data in the target data through the deep autoencoder to obtain the target feature data.

[0073] For example, the network structure of a DAE (Deep Autoencoder) is as follows: the number of neurons in the input layer is the same as the dimensionality of the data processed by LLE; the hidden layers are set to 3 layers with 50, 30, and 10 neurons respectively; and the number of neurons in the output layer is the same as the input layer. The system can use the stochastic gradient descent algorithm to train the DAE, setting the learning rate to 0.001 and the number of iterations to 100. Through training, the DAE learns the high-level abstract features of the data, removes redundant information, and obtains a low-dimensional feature representation, thereby obtaining the target feature data.

[0074] Optionally, the local linear embedding algorithm combined with a deep autoencoder can not only preserve the local manifold structure information of the target data, but also learn the high-level abstract features of the target data, thereby achieving efficient dimensionality reduction of the target data and reducing the complexity of data processing.

[0075] In one optional embodiment, determining missing data in target data based on target feature data includes: inputting target feature data into a target model, converting the target feature data into feature vectors through the target model, detecting the vector distance between the feature vectors and the normal vector of the hyperplane, and determining missing data in target data based on the vector distance; wherein the target model consists of a vector machine model and a Gaussian mixture model, and the normal vector of the hyperplane is used to define the boundary of normal data.

[0076] Optionally, the system can complete the above steps through an anomaly detection module (i.e., the target model). Specifically, the anomaly detection module performs anomaly detection based on a combination of One-Class SVM and Gaussian Mixture Model (GMM).

[0077] Optionally, the specific process of using One-Class SVM and Gaussian mixture model to identify anomalous data such as missing data is as follows: First, the anomaly detection module uses One-Class SVM to train the dimensionality-reduced target feature data, transforming it into feature vectors, thereby constructing a boundary model for normal data. Among them, formula (3) is the objective function of One-Class SVM.

[0078] (3)

[0079] In formula (3), the constraint condition of the objective function is: w is the normal vector of the hyperplane, used to define the boundary of the normal data. It is a slack variable used to allow some data points to be on the wrong side of the hyperplane in order to handle noise and anomalies in the data. It is the intercept of the hyperplane. v is a parameter that controls the number of support vectors and the error tolerance. n is the number of training data. It is to put data points Functions mapped to high-dimensional feature spaces (usually kernel functions, such as radial basis function kernels) ,in, (These are the parameters of the kernel function).

[0080] Optionally, combining vector machine (VMM) and Gaussian mixture model (Gaussian mixture model) approaches can fully leverage the advantages of VMM in boundary delimitation and the capabilities of Gaussian mixture model in data distribution modeling. This allows for more accurate identification of missing data in the target data, improving the precision and reliability of anomaly detection and effectively addressing noise and anomalies in river monitoring data. Furthermore, by detecting the vector distance between the feature vector and the hyperplane normal vector, the system can quantify the deviation of each data point from the normal data boundary, thereby effectively distinguishing between normal and missing data.

[0081] In an optional embodiment, before inputting the target feature data into the target model, slack variables, target parameters, and an objective function are obtained. The slack variables are used to constrain the number of data points allowed on the error side of the hyperplane, the target parameters are used to control the number of support vectors and the error tolerance of the vector machine model, and the objective function is used to map the target feature data to the target dimension feature space. The constraints of the vector machine model are determined based on the slack variables, target parameters, objective function, normal vector, and the intercept of the hyperplane.

[0082] Alternatively, in practical applications, it is unrealistic to correctly classify all data points completely due to the potential presence of noise or outliers. Therefore, introducing slack variables to allow some data points to lie on the wrong side of the hyperplane can improve the robustness of the model.

[0083] Optionally, the objective parameter v directly affects the complexity and generalization ability of the objective model. A smaller v value means that the objective model has a lower tolerance for outliers, while a larger v value allows more data points to be considered outliers.

[0084] Optionally, the objective function can be used to map the target data from the original space to a high-dimensional feature space (i.e., the target dimension feature space), thereby better separating normal data from abnormal data.

[0085] In one optional embodiment, missing data is repaired by analyzing the spatiotemporal information and parameter correlations between different data in the target data, including: obtaining the i-th data point with known data values ​​in the target data, wherein the location corresponding to the i-th data point is the first position, and the acquisition time corresponding to the i-th data point is the first time; determining the interpolation weight corresponding to the missing data located at the second position and acquired at the second time based on the spatial distance between the first position and the second position and the time interval between the first time and the second time; determining the data value of the missing data based on the interpolation weight corresponding to the missing data and the known data value of the i-th data point; and filling the missing data value into the second position of the target data to achieve the repair of missing data.

[0086] Optionally, the system can complete the above process through a data repair module. Specifically, the data repair module can employ a joint data repair method based on spatiotemporal interpolation and Bayesian networks.

[0087] Optionally, for missing data, the data repair module can perform preliminary estimation based on the time series and spatial location information of the target data using spatiotemporal interpolation methods (such as spatiotemporal kriging interpolation). Let the known data points... The value is estimated by spatiotemporal kriging interpolation to determine the missing points. The value is .in, This represents the value of a known data point at spatial location i and time t. This indicates missing data points at spatial location j and time s that need to be estimated. The interpolation weights can be calculated based on the spatiotemporal correlation between data points, and are typically determined using a semi-variogram. This describes the degree of spatial and temporal difference between data points, where h represents the spatiotemporal distance between two data points. The interpolation weights can be calculated by fitting a semi-variogram model (such as a spherical model or an exponential model). .

[0088] Optionally, the joint data repair method based on spatiotemporal interpolation and Bayesian networks can make full use of the spatiotemporal information of the target data and the causal relationship between parameters to accurately repair missing and abnormal data, thereby improving the accuracy and usability of the target data.

[0089] According to another aspect of the embodiments of this application, a processing apparatus for river monitoring data is also provided, wherein... Figure 3 This is a schematic diagram of an optional river monitoring data processing device according to an embodiment of this application, as shown below. Figure 3 As shown, the river monitoring data processing device includes: a fusion unit 301, an analysis unit 302, a processing unit 303, a determination unit 304, and a repair unit 305.

[0090] Optionally, the fusion unit 301 is used to fuse river monitoring data from different monitoring devices in tensor form to obtain fused target data; the analysis unit 302 is used to perform local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data, wherein the local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data; the local reconstruction weights are used to constrain the relative positional relationship between different data points in different dimensional spaces of the target data to remain unchanged; the processing unit 303 is used to perform dimensionality reduction processing on the target data according to the local reconstruction weights to obtain target feature data of the target data; the determination unit 304 is used to determine the missing data in the target data according to the target feature data; and the repair unit 305 is used to repair the missing data by analyzing the spatiotemporal information and parameter correlation between different data in the target data.

[0091] Optionally, the fusion unit 301 includes: a transformation subunit, a decomposition subunit, and a first computation subunit. The transformation subunit converts the river monitoring data into an original tensor; the decomposition subunit decomposes the original tensor into a core tensor and N factor matrices, wherein the core tensor includes the feature information of the original tensor and the relationships between different dimensional spaces; the N factor matrices characterize the feature transformations of the original tensor in the N dimensional spaces; and the first computation subunit performs a modular product operation on the core tensor and the N factor matrices to obtain the target data.

[0092] Optionally, the analysis unit 302 includes: a first processing subunit, a second processing subunit, and a second calculation subunit. The first processing subunit is used to treat each local data point in the target data as a data point, where the data points are represented in vector form. The second processing subunit is used to treat k data points in the target data whose distance to the i-th data point is less than a preset threshold as the k nearest neighbors of the i-th data point, where i is an integer greater than or equal to 1, and k is an integer greater than 1. The second calculation subunit is used to perform norm calculation based on the i-th data point, the k nearest neighbors, and the weight value of each nearest neighbor to obtain the local reconstruction weight between the i-th data point and the k nearest neighbors.

[0093] Optionally, the processing unit 303 includes: an input subunit, an extraction subunit, and a third processing subunit. The input subunit is used to input the local reconstruction weights between different local data in the target data and the target data into a deep autoencoder, wherein the number of neurons in the input layer and the number of neurons in the output layer of the deep autoencoder are the same as the number of data dimensions of the target data. The extraction subunit is used to remove redundant information from the target data through the deep autoencoder and extract feature data from the remaining information. The third processing subunit is used to perform dimensionality reduction processing on the extracted feature data based on the local reconstruction weights between different local data in the target data through the deep autoencoder to obtain the target feature data.

[0094] Optionally, the determining unit 304 includes a fourth processing subunit. The fourth processing subunit is used to input target feature data into the target model, convert the target feature data into feature vectors through the target model, detect the vector distance between the feature vectors and the normal vector of the hyperplane, and determine the missing data in the target data based on the vector distance. The target model consists of a vector machine model and a Gaussian mixture model, and the normal vector of the hyperplane is used to define the boundary of the normal data.

[0095] Optionally, the determining unit 304 further includes: a first acquisition subunit and a first determining subunit. The first acquisition subunit is used to acquire slack variables, target parameters, and an objective function. The slack variables are used to constrain the number of data points allowed on the error side of the hyperplane. The target parameters are used to control the number of support vectors and the error tolerance of the vector machine model. The objective function is used to map the target feature data to the target dimension feature space. The determining subunit is used to determine the constraints of the vector machine model based on the slack variables, target parameters, objective function, normal vector, and the intercept of the hyperplane.

[0096] Optionally, the repair unit 305 includes: a second acquisition subunit, a second determination subunit, a third determination subunit, and a repair subunit. The second acquisition subunit is used to acquire the i-th data point of known data value in the target data, where the location corresponding to the i-th data point is the first location, and the acquisition time corresponding to the i-th data point is the first time. The second determination subunit is used to determine the interpolation weight corresponding to the missing data located at the second location and acquired at the second time, based on the spatial distance between the first location and the second location and the time interval between the first time and the second time. The third determination subunit is used to determine the data value of the missing data based on the interpolation weight corresponding to the missing data and the known data value of the i-th data point. The repair subunit is used to fill the missing data value into the second location of the target data, thereby repairing the missing data.

[0097] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described method for processing river monitoring data.

[0098] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described method for processing river monitoring data.

[0099] The above-described embodiments or examples disclosed in this application are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this application. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, various embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0100] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0101] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0106] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing river monitoring data, characterized in that, include: River monitoring data from different monitoring devices are fused in tensor form to obtain the fused target data; Local manifold structure analysis is performed on the target data to obtain local reconstruction weights between different local data points in the target data. The local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data. The local reconstruction weights are used to constrain the relative positional relationship between different data points in the target data in different dimensional spaces to remain unchanged. The target data is reduced in dimensionality based on the local reconstruction weights to obtain the target feature data of the target data. Based on the target feature data, determine the missing data in the target data; The missing data is repaired by analyzing the spatiotemporal information and parameter correlations between different data in the target data. In the process of performing local manifold structure analysis on the target data to obtain the local reconstruction weights between different local data in the target data, each local data in the target data is treated as a data point, wherein the data point is represented in vector form; the k data points in the target data whose distance to the i-th data point is less than a preset threshold are regarded as the k nearest neighbors of the i-th data point, wherein i is an integer greater than or equal to 1, and k is an integer greater than 1; the norm is calculated based on the i-th data point, the k nearest neighbors, and the weight value of each nearest neighbor to obtain the local reconstruction weights between the i-th data point and the k nearest neighbors.

2. The method according to claim 1, characterized in that, River monitoring data from different monitoring devices are fused in tensor form to obtain fused target data, including: Convert the river monitoring data into raw tensors; The original tensor is decomposed into a core tensor and N factor matrices, wherein the core tensor includes the feature information of the original tensor and the relationship between different dimensional spaces; the N factor matrices are used to characterize the feature transformation of the original tensor in N dimensional spaces. The target data is obtained by performing a modular product operation on the core tensor and the N factor matrices.

3. The method according to claim 1, characterized in that, The target data is dimensionality reduced based on the local reconstruction weights to obtain the target feature data, including: The local reconstruction weights between different local data in the target data, along with the target data itself, are input into a deep autoencoder. The number of neurons in the input layer and the number of neurons in the output layer of the deep autoencoder are both the same as the number of data dimensions of the target data. The deep autoencoder removes redundant information from the target data and extracts feature data from the remaining information. The deep autoencoder performs dimensionality reduction on the extracted feature data based on the local reconstruction weights between different local data in the target data to obtain the target feature data.

4. The method according to claim 1, characterized in that, Determining missing data in the target data based on the target feature data includes: The target feature data is input into the target model, and the target feature data is converted into feature vectors through the target model. The vector distance between the feature vectors and the normal vector of the hyperplane is detected, and the missing data in the target data is determined based on the vector distance. The target model consists of a vector machine model and a Gaussian mixture model, and the normal vector of the hyperplane is used to define the boundary of the normal data.

5. The method according to claim 4, characterized in that, Before inputting the target feature data into the target model, the method further includes: Obtain slack variables, target parameters, and an objective function, wherein the slack variables are used to constrain the number of data points allowed on the error side of the hyperplane, the target parameters are used to control the number of support vectors of the vector machine model and the error tolerance of the vector machine model, and the objective function is used to map the target feature data to the target dimension feature space; The constraints of the vector machine model are determined based on the slack variables, the target parameters, the objective function, the normal vector, and the intercept of the hyperplane.

6. The method according to claim 1, characterized in that, By analyzing the spatiotemporal information and parameter correlations between different data in the target data, the missing data is repaired, including: Obtain the i-th data point of known data value in the target data, wherein the position corresponding to the i-th data point is the first position, and the acquisition time corresponding to the i-th data point is the first time. Based on the spatial distance between the first position and the second position and the time interval between the first time and the second time, determine the interpolation weight corresponding to the missing data located at the second position and collected at the second time. The data value of the missing data is determined based on the interpolation weights corresponding to the missing data and the known data value of the i-th data point; The missing data value is filled into the second position of the target data to repair the missing data.

7. A device for processing river monitoring data, characterized in that, include: The fusion unit is used to fuse river monitoring data from different monitoring devices in tensor form to obtain the fused target data; The analysis unit is used to perform local manifold structure analysis on the target data to obtain local reconstruction weights between different local data in the target data. The local manifold structure analysis is used to determine the intrinsic manifold structure information between different data points in the target data. The local reconstruction weights are used to constrain the relative positional relationship between different data points in different dimensional spaces of the target data to remain unchanged. The processing unit is used to perform dimensionality reduction processing on the target data according to the local reconstruction weights to obtain the target feature data of the target data; A determining unit is configured to determine missing data in the target data based on the target feature data; The repair unit is used to repair the missing data by analyzing the spatiotemporal information and parameter correlation between different data in the target data; The analysis unit includes: a first processing subunit, used to treat each local data in the target data as a data point, wherein the data point is represented in vector form; a second processing subunit, used to treat k data points in the target data whose distance to the i-th data point is less than a preset threshold as the k nearest neighbors of the i-th data point, wherein i is an integer greater than or equal to 1, and k is an integer greater than 1; and a second calculation subunit, used to perform norm calculation based on the i-th data point, the k nearest neighbors, and the weight value of each nearest neighbor to obtain the local reconstruction weight between the i-th data point and the k nearest neighbors.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device containing the computer-readable storage medium performs the method for processing river monitoring data as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method for processing river monitoring data according to any one of claims 1 to 6.

Images