Power grid water regulation data management and control method based on artificial intelligence

Abstract

This invention discloses an artificial intelligence-based method for managing and controlling power grid water dispatch data, specifically relating to the field of power grid water dispatch data management technology. The method involves collecting multi-source heterogeneous water dispatch data and its metadata to construct a unified format dataset; generating a credibility score based on metadata and data features to form a credibility matrix; constructing a multi-dimensional quality assessment tensor and extracting the dominant factor subspace; inputting this tensor into a clustering perceptual neural network to identify abnormal data, outputting an abnormal dataset and abnormal labels; combining a large model enhanced by RAG retrieval with historical similar data to complete abnormal data repair; constructing a data quality improvement report using the difference vectors before and after repair and label information; and finally pushing the report to a data management platform and generating a traceable rectification work order. This method achieves automatic identification, intelligent repair, and feedback tracking of power grid water dispatch data, improving data credibility, governance efficiency, and security, and is suitable for data management needs in complex scenarios under new power systems.

Description

Technical Field

[0001] This invention relates to the field of power grid water dispatch data management technology, specifically to a power grid water dispatch data control method based on artificial intelligence. Background Technology

[0002] Against the backdrop of the rapid construction of new power systems and the advancement of "dual-carbon" goals, hydropower, as a key regulating energy source, plays a crucial supporting role in ensuring the safe and stable operation of the power grid through its dispatch data. However, with the continuous increase in the proportion of new energy sources in the power grid, wind and solar power output in regions such as Guangxi exhibits high randomness and volatility. The role of hydropower dispatch in frequent peak shaving and rapid response is becoming increasingly important, and the accuracy, real-time nature, and consistency of its basic data directly affect the scientific validity and enforceability of hydropower dispatch strategies.

[0003] In actual operation, the data acquisition chain for power grid hydropower dispatch is long and the sources are heterogeneous and complex, including various types such as hydrological conditions, engineering conditions, power generation, and power station operating status. Furthermore, the acquisition frequency and quality are inconsistent, making it highly susceptible to problems such as missing data, redundancy, conflicts, and distortion during data fusion. For example, due to the complex terrain environment of hydrological telemetry stations, extreme weather or equipment failures can easily lead to data interruptions or abnormal drift. Some power stations are not connected to the unified dispatch system, and their operating status data must be manually processed and uploaded, posing risks of delays and deviations. In addition, traditional data quality inspection processes often rely on manual spot checks or rule matching, which are inefficient and have poor coverage, failing to meet the needs of automated quality control for high-frequency, massive, and multi-source heterogeneous data. Summary of the Invention

[0004] The purpose of this invention is to provide an artificial intelligence-based method for managing and controlling power grid water dispatch data, in order to address the shortcomings of the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a power grid water dispatch data management and control method based on artificial intelligence, comprising: Collect multi-source heterogeneous raw dataset D from the water survey data source to be measured, including hydrological data, power data and operating parameter data, and attach the raw acquisition metadata M of each data source, including acquisition equipment type, timestamp accuracy, acquisition frequency and acquisition environment information. The multi-source heterogeneous original dataset D and the metadata M are subjected to format unification, unit normalization and structure mapping transformation, and a confidence prediction matrix C is constructed in combination with the metadata M to obtain the unified format dataset D0. Based on the unified format dataset D0, a multi-dimensional data quality assessment tensor Q is constructed, and its dominant quality factor subspace Q′ is extracted using tensor compression technology. The Q′ is used as a feature input to the anomaly recognition model, which identifies the abnormal data points in D0 based on the clustering perceptual neural network, and outputs the abnormal dataset Derr and its corresponding abnormal label set Terr. The Derr, Terr and the credibility matrix C are jointly input into the defect data repair model. The defect data repair model is driven by a large industry vertical model based on RAG retrieval enhancement, and achieves adaptive data repair by dynamic knowledge fusion with the historical similar water-conditioning scene database, and outputs the repaired dataset Dfix. Semantic alignment comparison is performed based on the difference vector between Dfix and D0, and a data quality improvement report is constructed by combining the anomaly label source information in Terr. The data quality improvement report is pushed to the data management platform, and the work order generator automatically associates the data source identifier to be rectified, generating a rectification task with a tracking ID.

[0006] Preferably, the step of constructing a confidence prediction matrix C by combining meta-information M to obtain a unified format dataset D0 includes the following steps: Based on the acquisition device type, sampling frequency and timestamp precision contained in the original acquisition metadata M, the multi-source heterogeneous original dataset D is subjected to structural mapping and semantic normalization processing to generate a data copy D′ with unified data field format and time alignment; Extract the corresponding metadata M for each data record in the data copy D′, and assign confidence weight values ​​to each metadata parameter by constructing a weighted factor model; Based on the volatility of the data values ​​themselves, historical stability, and meta-information weighting factors, a credibility score is generated by using a method that combines Bayesian inference and sliding window statistical analysis. All credibility scores are matrixed to construct a data credibility prediction matrix C, which is then combined with the structure-normalized data copy D′ to output a unified format dataset D0.

[0007] Preferably, the process of generating a credibility score includes the following steps: Based on the time series characteristics of various data in the unified format dataset D0, the mean, standard deviation and coefficient of variation of the data within the sliding time window are extracted to construct a set of volatility indicators; By calling up stable data from the historical database under similar scenarios and similar collection devices over multiple periods, the current data is matched for similarity and deviation is calculated to form a set of historical stability indicators. By combining the meta-information weighted factor model, the data acquisition device type, sampling frequency, and timestamp accuracy parameters associated with each data point are weighted and fused to generate a context confidence vector; The volatility index, stability index, and contextual confidence vector are input into the Bayesian inference model to estimate the confidence probability, outputting a continuous confidence score and mapping it to the confidence score matrix.

[0008] Preferably, the process of extracting its dominant quality factor subspace Q′ using tensor compression technology includes the following steps: Based on the time dimension, data field dimension, and five quality attribute dimensions in the unified format dataset D0, a third-order tensor structure Q is constructed, where the five quality attributes include completeness, consistency, accuracy, timeliness, and uniqueness. Normalize the data in each dimension of tensor Q; The tensor Q is compressed using a higher-order singular value decomposition method to separate the dominant factor space Q′ with the largest principal component variance contribution.

[0009] Preferably, the identification of anomalous data points in D0 based on a clustering perceptual neural network includes the following steps: The dominant quality factor subspace Q′ is expanded into a two-dimensional feature matrix and reconstructed into a multi-dimensional input vector set in time series order. A clustering-aware neural network model is constructed, which integrates an autoencoder and a density-aware clustering module. The autoencoder is used for feature compression and reconstruction error analysis, and the clustering module is used to measure the distance distribution between data points and local cluster centers. Each input vector in Q′ is input into a clustering perceptual neural network, and its clustering consistency and reconstruction error level in high-dimensional space are identified through the training phase. Anomaly scores are calculated by combining reconstruction error and cluster isolation degree. If the anomaly score exceeds a set threshold, it is marked as an anomalous data point. The output is the anomalous dataset Derr and its corresponding anomalous label set Terr. The label content includes the anomaly type, severity level and data field location information.

[0010] Preferably, the step of jointly inputting Derr, Terr, and the credibility matrix C into the defect data repair model includes the following steps: Based on the anomaly type and data field location information in the anomaly label set Terr, a set of data points to be repaired is extracted from Terr, and the repair priority is determined by combining the scores of the corresponding positions in the confidence matrix C. By constructing a retrieval query vector, the timestamp, spatial location, field type, and similar historical context of the data point to be repaired are encoded and input into the RAG retrieval module to dynamically retrieve similar data fragments in the industry water control scenario knowledge base. The retrieved context information is concatenated with the abnormal data points and their label information to form a structured repair suggestion, which is then input into a large vertical model pre-trained based on industry knowledge to generate repair suggestion values. The generated repair suggestion value is compared with the original data context for consistency. If the difference is less than the set threshold, it is directly replaced to form the repaired dataset Dfix.

[0011] Preferably, the semantic alignment comparison based on the difference vector between Dfix and D0 includes the following steps: The numerical differences between the repaired dataset Dfix and the unified format dataset D0 at the same field and time index position are calculated to form a set of difference vectors ΔV, which are then categorized and stored according to field type. The set of difference vectors ΔV is semantically mapped and matched with the anomaly type, severity level, and source information in the anomaly label set Terr to establish a data repair logical link. Based on the mapping results, a data repair record table is generated, recording the original value, repair value, repair reason, and credibility improvement of each anomaly data point. Combining the data repair record table and difference statistics, a data quality improvement report is automatically generated.

[0012] The technical effects and advantages provided by the present invention in the above technical solution are as follows: 1. This invention constructs a comprehensive methodology encompassing metadata modeling, data reliability quantification, tensor quality assessment, intelligent anomaly identification, enhanced retrieval and repair, and closed-loop task management. This achieves an automated processing chain for power grid water dispatch data, from "problem discovery" to "intelligent repair" and then to "management feedback." Compared to traditional quality management methods relying on manual sampling and rule-based judgment, this invention possesses multi-dimensionality, strong robustness, and adaptability, significantly improving the accuracy of data quality identification and the practicality of repair suggestions.

[0013] 2. This invention integrates RAG-based enhanced vertical domain large-scale model technology with tensor analysis, and for the first time introduces semantic awareness and contextual linkage mechanisms into hydropower dispatch data quality management. This not only improves the adaptability of anomaly identification to scene complexity but also realizes an interpretable, quantifiable, and traceable data quality improvement path. Through automated work order generation and closed-loop tracking of rectification, this invention effectively promotes the transformation of hydropower dispatch data governance from manual experience-driven to intelligent decision-driven, providing solid data support for the safe and efficient operation of new power systems. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] For examples, please refer to Figure 1 As shown in this embodiment, the power grid water dispatch data management and control method based on artificial intelligence includes: Collect multi-source heterogeneous raw dataset D from the water survey data source to be measured, including hydrological data, power data and operating parameter data, and attach raw acquisition metadata M for each data source, including acquisition equipment type, timestamp accuracy, acquisition frequency and acquisition environment information.

[0018] First, the original data from multiple heterogeneous sources is collected and its metadata is synchronously labeled. This step aims to provide a unified data foundation and semantic context for subsequent data quality assessment and repair. By constructing a structured data collection process, various types of water dispatch-related data involved in power grid dispatching operations are collected, and their collected metadata is recorded simultaneously, realizing a two-layer data acquisition mode of "data ontology + collection context".

[0019] Specifically, the collected dataset D includes, but is not limited to, the following three main data types: Hydrological data, such as water level, water flow, rainfall, runoff, sediment content, and water temperature, are collected by equipment such as hydrological telemetry stations, water level gauges, and rainfall sensors. Electricity data: including the generating power of each unit of the hydropower station, daily power generation, cumulative electricity, and abandoned water power, etc., which are obtained from the electricity metering equipment and the dispatch system interface; Operating parameter data: covering reservoir capacity, start-up and shutdown status, regulation mode, pump / unit switching status, maintenance records, etc., are collected by the power plant operation and management system.

[0020] The aforementioned raw data is highly heterogeneous, involving structured tables, semi-structured logs, and real-time streaming data from SCADA systems. The formats are inconsistent, the sampling granularity varies greatly, and the timestamp precision is inconsistent, making it difficult to directly perform fusion calculations.

[0021] To achieve more efficient data fusion and intelligent processing, this invention establishes a metadata set M while collecting data D, and synchronously marks the background semantics of each data point and its source device. The metadata M includes at least the following: Data Acquisition Device Type: Indicates the type of hardware device that acquired the data, such as radar rain gauge, hydrostatic water level gauge, distributed smart energy meter, etc. Timestamp precision: Determine the precision of the timestamp for each data item, such as whether it is in minutes, hours or seconds, to facilitate time alignment and trend calculation; Collection frequency: Specifies the sampling period of the data stream to which the data point belongs, such as 5 minutes / time, 15 minutes / time, hourly timed upload, etc.; Collect environmental information: Record the external environmental conditions at the time of data collection, including weather conditions, equipment operating status, communication link quality, and abnormal alarm status.

[0022] By jointly modeling the data ontology D and the metadata M, this invention can comprehensively consider both the "data value itself" and the "data generation context" in subsequent steps, achieving more accurate data quality identification and credibility analysis. For example, even with a sudden change in water level, if the accompanying metadata indicates that there was thunderstorm weather and unstable communication links during that period, it can be preliminarily determined that the data anomaly may be caused by acquisition errors. In addition, this metadata set M also provides semantic support for the subsequent explanationability enhancement of data anomaly labels, making the entire quality control process more transparent and traceable, and suitable for complex business applications in power grid control scenarios with high credibility and high explainability requirements.

[0023] The multi-source heterogeneous original dataset D and the metadata M are subjected to format unification, unit normalization and structure mapping transformation, and a confidence prediction matrix C is constructed in combination with the metadata M to obtain the unified format dataset D0.

[0024] First, based on parameters such as the type of acquisition device, sampling frequency, and timestamp precision contained in the original acquisition metadata, the multi-source heterogeneous original datasets are subjected to structural mapping and semantic normalization. For data fields with different formats (such as water level possibly represented by "m" or "cm"), they are uniformly converted into standard physical quantities and units through a field mapping table; for data with large differences in sampling frequency (such as 5-minute sampling and 1-hour sampling), time scale alignment is performed through interpolation algorithms to ensure that the data has uniform granularity on the time axis.

[0025] Based on the data timestamp, a time window index table is constructed to categorize all data records into windows. The output is a data copy with a uniform data field format and time alignment, denoted as data copy D′.

[0026] In the generated data copy D′, the original acquisition metadata corresponding to each data record is extracted, mainly including: acquisition device type, sampling frequency, and timestamp precision. Based on this, a metadata weighting factor model is constructed, assigning a specific weight value to each parameter.

[0027] Weight values ​​are set based on domain experience and historical reliability statistical analysis, for example: Precision-grade equipment, such as hydrostatic level gauges, is assigned a weight of 0.9, while simpler equipment, such as float level gauges, is assigned a weight of 0.6. A high sampling frequency (less than or equal to 5 minutes) is assigned a weight of 0.85, and a low sampling frequency (greater than or equal to 1 hour) is assigned a weight of 0.6. The timestamp precision is 0.95 for seconds and 0.8 for minutes.

[0028] By calculating a weighted average of the above weights, a credibility weighting factor for each data point is formed. This weighting method can be expressed as: Credibility weight W = (w1×a1 + w2×a2 + w3×a3) / (w1 + w2 + w3), where w1, w2, and w3 are the subjective weight coefficients of each meta-information parameter, and a1, a2, and a3 are their corresponding score values.

[0029] Based on the time axis, a sliding time window of length T is constructed (a value of T=6 is recommended, i.e., analyzing 6 adjacent time periods). Within each window, for the same physical quantity field, its mean, standard deviation, and coefficient of variation are calculated to describe short-term fluctuation characteristics. For example, for the water level data field H, the following indicators are calculated within the window [t, t+5]: mean μ=(H1+H2+...+H6) / 6; standard deviation σ is the standard deviation of the above sequence; coefficient of variation CV=σ / μ. These indicators constitute a set of volatility indicators for subsequent reliability assessment.

[0030] By accessing historical databases, stable water survey data collected under similar scenarios (such as the same season and the same watershed) and with similar equipment are extracted to construct historical reference benchmarks. Sequence matching is performed between current data and historical stable samples, and the deviation value Δ is calculated as a stability index.

[0031] The specific matching method is dynamic time warping (DTW) distance calculation, and the average deviation of the matching sequences is taken: if the current water level sequence H_current matches the historical sequence H_history, the deviation Δ = average |H_current - H_history|; if Δ is less than a set threshold (e.g., 0.15 meters), it is marked as high stability. The output stability index set serves as an auxiliary input for quality estimation.

[0032] The aforementioned meta-information weighting factors are fused with volatility and stability indicators to construct a contextual confidence vector. This vector is a set of features with consistent dimensions, representing the confidence level of the data point across multiple quality dimensions. This vector can be denoted as: V=[CV,Δ,W], where CV is the coefficient of variation, Δ is the historical bias, and W is the weighted confidence factor.

[0033] Using the contextual confidence vector V as input, a Bayesian inference model is constructed to calculate the confidence probability. In the model, each dimension of the confidence vector is defined as a condition variable, and the output variable is the probability P(T=1|V) that the data is "highly confident". The Bayesian scoring model is defined as follows: The prior probability P(T) is set based on historical data, such as a high-confidence sample ratio of 0.85; The likelihood function P(V|T) is fitted with a Gaussian distribution using data in the training samples that have been labeled with confidence levels. The posterior probability P(T|V) is calculated using Bayes' theorem, and the output is the credibility score, ranging from 0 to 1.

[0034] The credibility scores of each data point calculated above are matrixed and arranged in a two-dimensional manner according to the data field and time index to form a data credibility prediction matrix C.

[0035] For example, the behavior is a time point, listed as various physical fields (water level, output, flow rate, etc.), and each element in the matrix represents the confidence value of that field at the corresponding time point.

[0036] Finally, the confidence prediction matrix C is combined with the structure-normalized data copy D′ to generate a unified format dataset D0 for subsequent quality analysis and anomaly identification.

[0037] A multi-dimensional data quality assessment tensor Q is constructed based on the unified format dataset D0, and its dominant quality factor subspace Q′ is extracted using tensor compression technology. The specific processing steps are as follows: This invention defines a three-dimensional tensor structure Q based on a unified format dataset D0. This tensor Q contains the following three orthogonal dimensions: Time dimension T: Represents the time index of each data record, arranged according to a uniform sampling period, such as once per hour; Data field dimension F: corresponds to various physical quantities in the water dispatch data, such as water level, water flow, output, electricity, reservoir capacity, etc. Data quality attribute dimension A: covers five quality assessment attributes, namely completeness, consistency, accuracy, timeliness and uniqueness.

[0038] In this structure, the tensor element Q(t,f,a) represents the score value of the quality attribute a at time point t and data field f, with the score ranging from 0 to 1.

[0039] The calculation method for each quality attribute is defined as follows: Completeness: Defined as the non-null rate of a field within a continuous time window; Consistency: Defined as the proportion of time that this field satisfies rule constraints with related fields, such as the consistency between water level changes and flow rate changes. Accuracy: Defined as the inverse ratio of the error between this field and the predicted value of the reference model; Timeliness: defined as the frequency with which the absolute value of the error between the acquisition time and the theoretical time is within a tolerance threshold; Uniqueness: Defined as the proportion of records that are not duplicated within a given time period.

[0040] Based on the above definition, the evaluation results of each data point under different quality dimensions are filled into the corresponding tensor positions to complete the construction of the third-order tensor Q.

[0041] Because the five data quality attributes differ in numerical distribution, calculation method, and physical meaning, direct tensor operations would lead to uneven contributions from each dimension. Therefore, this invention normalizes the attribute values ​​of each dimension in tensor Q.

[0042] The normalization method employs a minimum-maximum scaling approach, described by the following formula: For any element in tensor Q, its normalized value Qnorm(t,f,a) equals the attribute value of that dimension minus the minimum value of that dimension, divided by the range of that dimension (maximum value minus minimum value). After normalization, all dimensions are uniformly set to [0,1], facilitating subsequent algorithm processing. This process eliminates interference caused by differences in the units and fluctuation ranges of different indicators, improving the consistency of model recognition.

[0043] To further reduce the redundancy of high-dimensional tensor data and extract its key quality features, this invention employs Higher Order Singular Value Decomposition (HOSVD) to decompose tensor Q. The specific steps are as follows: Expand the tensor Q into a two-dimensional matrix along each dimension, for example, expand the tensor into matrix QT along the time dimension; perform singular value decomposition on each expanded matrix to obtain its set of left singular vectors (i.e., principal components in each dimension); sort the singular value sets obtained by decomposition according to their contribution, and select the top few singular vectors to construct a low-rank approximate subspace; reconstruct the selected principal components into a compressed low-rank tensor, denoted as the dominant factor subspace Q′.

[0044] In this invention, the principal component selection principle is to retain the portion that explains more than 95% of the cumulative variance, ensuring that the main quality features are preserved while significantly reducing the data dimensionality. This principal component subspace Q′ retains the quality features that change most significantly across multiple time points, multiple physical fields, and various quality attributes, and uses them as input features in subsequent anomaly identification to improve the model's sensitivity and generalization ability.

[0045] Using Q′ as a feature input to the anomaly identification model, the model identifies anomalous data points in D0 based on a clustering perceptual neural network, and outputs the anomalous dataset Derr and its corresponding set of anomalous labels Terr.

[0046] First, the dominant quality factor subspace Q′ is structurally expanded. Q′ is a low-dimensional tensor extracted from tensor Q through compression, with dimensions of time point × data field × number of dominant quality factors. This tensor is expanded along the time dimension into a two-dimensional feature matrix M, where: rows represent each time point; columns represent the feature combinations of all data fields at the corresponding time point along each dominant quality factor dimension. Then, matrix M is read sequentially in chronological order, with each row being a multi-dimensional input vector, forming an input vector set V for neural network training and prediction.

[0047] A clustering-aware neural network model is constructed, which combines an autoencoder structure with a density-aware clustering analysis mechanism, and has the capabilities of feature compression, adaptive reconstruction, and spatial anomaly perception.

[0048] The autoencoder structure consists of an encoder E and a decoder D: the encoder E compresses the input vector into a low-dimensional latent representation Z; the decoder D receives the latent representation Z and reconstructs the original input, outputting the reconstructed vector V′; the reconstruction error is defined as the Euclidean distance between the input vector V and the reconstructed vector V′.

[0049] The density-aware clustering module is constructed using local neighborhood distance analysis. The specific steps are as follows: K-means clustering is performed in the encoding space Z to calculate the distance between each sample and the center of its cluster; the cluster isolation degree is defined as the ratio of the distance from the sample point to its cluster center to the average distance of all points. This network, by jointly optimizing the reconstruction error and cluster distance, enables the model to simultaneously possess local pattern learning and spatial anomaly recognition capabilities.

[0050] During the training phase, the set of dominant factor input vectors V is input one by one into the aforementioned clustering perceptual neural network, and the following operations are performed: The latent representation Z and the reconstructed vector V′ are calculated through forward propagation; the reconstruction error Erec is calculated, defined as the sum of the squared errors of the input vector and the reconstructed vector; the cluster isolation degree Diso of each sample in the latent space Z is calculated, i.e., the normalized distance from the current sample to its cluster center; the model is trained using a joint loss function, L = α × Erec + β × Diso, where α and β are adjustable weight coefficients, with recommended initial values ​​of 0.7 and 0.3, respectively. Through this training process, the model can automatically learn the mainstream patterns and anomalous distribution structures of the data.

[0051] After model training is complete, anomaly scoring is performed on each input vector, following these steps: Calculate the final anomaly score S for the current input vector, a weighted combination of reconstruction error and cluster isolation: S = γ × Erec + δ × Diso, where γ and δ are weight parameters set during the model inference phase, recommended to be 0.6 and 0.4 respectively; Set anomaly detection threshold T1, based on experience with the anomaly distribution of training data, preferably at the 95th percentile of the historical score distribution; If the score S exceeds the threshold T1, mark the data point as an anomaly; Output an anomaly dataset Derr for all anomaly data points, and generate a corresponding anomaly label set Terr. Labels include: anomaly type (e.g., mutation, missing, drift); anomaly severity level (divided into high, medium, and low based on anomaly score); the data field, time point, and data source metadata identifier. This label structure supports tracing and prioritizing in subsequent data repair and visualization reports.

[0052] The Derr, Terr, and credibility matrix C are jointly input into the defect data repair model, which is driven by a large industry vertical model based on RAG retrieval enhancement. The model achieves adaptive data repair by dynamically fusing with the knowledge in the historical similar water-conditioning scene library, and outputs the repaired dataset Dfix.

[0053] First, based on the anomaly type, severity level, and data field location information in the anomaly label set Terr, a set of data points to be repaired, Sfix, is extracted from the anomaly dataset Terr. Each data point to be repaired corresponds to a specific field location and time index in the original dataset D0.

[0054] Subsequently, the score value corresponding to each data point in Sfix is ​​extracted from the credibility matrix C. Repair priorities are assigned based on the score value; lower scores indicate poorer data quality and lower credibility, with a greater impact on overall data integrity, and should be repaired first. Repair priorities P are arranged in reverse order: if a data point's credibility score is less than 0.3, it is classified as high priority; if the score is between 0.3 and 0.6, it is classified as medium priority; and those greater than 0.6 are marked as low priority and can be repaired later.

[0055] For each data point to be repaired, a structured retrieval query vector Vquery is constructed. This vector includes the following encoded information: Timestamp information: indicates the specific time of the data point; Spatial location: The corresponding watershed, watershed location, hydropower station or telemetry point number of the data source; Data field types: such as water level, water flow, electricity consumption, reservoir capacity, etc.; Anomaly type labels: such as "mutation", "deletion", "offset", "redundancy", etc.

[0056] The above information is concatenated into a standardized retrieval template and input into the RAG (Retrieval-Augmented Generation) retrieval module. This module performs vector semantic retrieval in a pre-built industry water control scenario knowledge base to extract spatiotemporal scene data fragments that are similar to the historical data points to be repaired. These are typically complete data sequences from the same season, location, or similar operating conditions, and are used for subsequent model inference.

[0057] Similar historical fragments retrieved are used as auxiliary context and concatenated with the field values, anomaly labels, and metadata of the current data point to be repaired to generate a structured repair suggestion, Tfusion. This suggestion is then passed as input to a large-scale vertical domain model (such as a language-numerical multimodal model based on the Transformer architecture) pre-trained on water conservancy and hydropower industry data. This large-scale vertical domain model has the following capabilities: understanding the contextual evolution patterns of time-series data; identifying typical behavioral patterns and anomaly correction rules in water resources management scenarios; and generating the optimal predicted value for the current data point under contextual rationality conditions. The model output is a repair suggestion value, Rsug, which is the predicted value of the field to be repaired that best conforms to historical patterns and logical constraints under the current semantic and physical conditions.

[0058] To ensure that the repaired data does not introduce new errors, this invention introduces a consistency comparison mechanism. The generated repair suggestion value Rsug is compared with the set of historical context values ​​of that field within its time neighborhood, and the relative difference Δ is calculated as follows: Δ = |Rsug − average value (field value at nearby times)| ÷ average value. If Δ is less than the set threshold TΔ (recommended to be 0.2, i.e., relative change not exceeding 20%), the repair suggestion is considered highly consistent with the original data context and can directly replace the original outlier, forming the repaired dataset Dfix. If Δ exceeds the threshold, the data point is marked as "requiring manual review," and the original outlier is retained for subsequent verification analysis. The final repaired dataset Dfix will be used in the generation of data quality improvement reports and the closed-loop rectification process.

[0059] Semantic alignment comparison is performed based on the difference vector between Dfix and D0, and a data quality improvement report is constructed by combining the anomaly label tracing information in Terr.

[0060] First, based on the structural consistency between the repaired dataset Dfix and the unified format dataset D0, the differences between the corresponding data values ​​with the same field names and time index positions are calculated item by item, forming a difference vector set ΔV. The difference calculation method is as follows: let the value of a certain field F at time index t before repair be V0(t,F), and the value after repair be Vfix(t,F), then the difference Δ(t,F) = Vfix(t,F) − V0(t,F). The difference vector ΔV is grouped according to field F, forming a difference vector index structure with the field as the primary key, which is used for subsequent mapping and statistics. This difference set ΔV reflects the change magnitude of each field at each time point during the data repair process, quantifying the degree of impact of the repair operation on the original data.

[0061] The set of difference vectors ΔV is mapped line by line to the set of anomaly labels Terr. Each label in Terr contains the following information: anomaly type (e.g., missing, mutation, drift), severity level (high, medium, low), and source metadata (e.g., acquisition device type, data sampling precision, etc.). The mapping method is as follows: If the position corresponding to Δ(t,F) has the same time index t and field F label in Terr, then a one-to-one semantic mapping is established; Semantically link anomaly types with their corresponding difference patterns. For example, the "mutation" type should correspond to high-amplitude short-time difference values, and "missing" should correspond to the filled values ​​after null replacement. By combining the score values ​​of the original data positions in the credibility matrix C with the credibility scores after repair, the degree of improvement in credibility ΔC due to repair is recorded.

[0062] Through this semantic mapping process, a complete data repair logic chain is constructed, namely a traceable path from "identification of the source of anomalies" to "numerical changes" and then to "quality improvement".

[0063] Based on the semantic mapping results, a structured data repair record table, Rlog, is generated. This record table is in two-dimensional table format, and each record contains the following fields: data field name; time index position; original value before repair, V0; suggested value after repair, Vfix; difference Δ(t,F); corresponding anomaly type and severity level; source of repair suggestion (generated by a large model or interpolated from historical data); and confidence score improvement ΔC (i.e., the score after repair minus the original score). This record table is used for subsequent quality improvement assessment, repair plan auditing, and manual review, and possesses high interpretability and auditability.

[0064] After constructing the data repair record table, a data quality improvement report (Rreport) is automatically generated by combining the statistical characteristics of the difference vectors and the label distribution information. The report content includes, but is not limited to, the following: The report includes the following sections: Field Dimension Quality Improvement Chart: Displays the average credibility score improvement curves for each key field before and after repair; Anomaly Type Distribution Chart: Statistics on the proportion of various anomaly types (such as missing values ​​and mutations) in the overall anomaly data; Repair Magnitude Distribution Chart: Displays the distribution density of the repair difference Δ(t,F) for each field, used to identify fields with significant repairs; Source Tag Validity Analysis Table: Evaluates the differences in data anomaly frequency and repair effectiveness from different data acquisition devices; Feedback Suggestions: Generates improvement suggestions for the data acquisition system based on the report content, such as increasing sampling frequency and enhancing the inspection of specific devices. The above report can be generated as a structured document and visual charts, and supports exporting to PDF or embedding into an interactive data quality monitoring platform.

[0065] The data quality improvement report is pushed to the data management platform, and the work order generator automatically associates the data source identifier to be rectified, generating a rectification task with a tracking ID.

[0066] First, the generated data quality improvement report (Rreport) is pushed to the data management platform. This platform has interfaces to connect to various data source metadata and can identify the source identifiers of abnormal data in the report. The specific identification method is as follows: Extract the field names, time index, acquisition device number, and data source identifier of each record in the repair record table Rlog in the report; Based on the field relationships, match the data source module, site, and interface number to which the data point belongs in the metadata management table.

[0067] The identification result is Spending, which represents the data collection objects that currently have data quality defects and have been repaired but require source rectification.

[0068] This invention designs a work order generator. This component takes "Spending" as input and automatically generates a corresponding rectification task work order for each data source to be rectified. The work order task content includes: a unique identifier for the data source; the data fields and time period involved; the corresponding anomaly type and severity level; suggested rectification operations (such as equipment calibration, sampling frequency adjustment, and communication link check); and a corresponding repair record summary. The system automatically generates structured work order documents based on a unified template, ensuring that the task content is clear and the operations are explicit.

[0069] To achieve full-process tracking and closed-loop management of tasks, the system assigns a globally unique tracking ID to each rectification task when a work order is generated. This tracking ID has the following structure: The prefix indicates the data source classification code (e.g., H for hydrological data, P for electricity data); the middle section is the timestamp information, accurate to the second; the suffix is ​​a six-digit random code to avoid conflicts. The system uses this ID to achieve the following functions: linking data repair logs with the source data traceability path; binding responsible units or equipment maintenance personnel; marking rectification status, including "pending confirmation," "processing," "completed," and "abnormally suspended"; and supporting subsequent automatic sending of reminders and feedback to the data management platform.

[0070] After the rectification task is issued, the data management platform can receive feedback information from the site, including the rectification completion time, operation instructions, verification results, etc. The system updates the status of the original work order based on the feedback and writes the relevant information into the rectification log database.

[0071] If the rectification operation improves the quality score of the corresponding data field, it will be marked as "closed-loop completed" in the next data quality report. Otherwise, the system will mark it as "rectification invalid" and trigger rectification suggestions again or upgrade it to a manual review task.

[0072] This mechanism enables a complete closed-loop path from "data anomaly identification" → "data repair" → "task generation" → "task execution" → "quality improvement verification", effectively enhancing the intelligence and systematic nature of power grid water dispatch data management.

[0073] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data management and control method for power grid water dispatching based on artificial intelligence, characterized in that: include: Collect multi-source heterogeneous raw dataset D from the water survey data source to be measured, including hydrological data, power data and operating parameter data, and attach raw acquisition metadata M for each data source, including acquisition equipment type, timestamp accuracy, acquisition frequency and acquisition environment information. The multi-source heterogeneous original dataset D and the metadata M are subjected to format unification, unit normalization and structure mapping transformation, and a confidence prediction matrix C is constructed in combination with the metadata M to obtain the unified format dataset D0. Based on the unified format dataset D0, a multi-dimensional data quality assessment tensor Q is constructed, and its dominant quality factor subspace Q′ is extracted using tensor compression technology. The Q′ is used as a feature input to the anomaly recognition model, which identifies the abnormal data points in D0 based on the clustering perceptual neural network, and outputs the abnormal dataset Derr and its corresponding abnormal label set Terr. The Derr, Terr and the credibility matrix C are jointly input into the defect data repair model. The defect data repair model is driven by a large industry vertical model based on RAG retrieval enhancement, and achieves adaptive data repair by dynamic knowledge fusion with the historical similar water-conditioning scene database, and outputs the repaired dataset Dfix. Semantic alignment comparison is performed based on the difference vector between Dfix and D0, and a data quality improvement report is constructed by combining the anomaly label source information in Terr. The data quality improvement report is pushed to the data management platform, and the work order generator automatically associates the data source identifier to be rectified, generating a rectification task with a tracking ID.

2. The power grid water dispatch data management and control method based on artificial intelligence according to claim 1, characterized in that: The process of constructing a confidence prediction matrix C by combining meta-information M to obtain a unified format dataset D0 includes the following steps: Based on the acquisition device type, sampling frequency and timestamp precision contained in the original acquisition metadata M, the multi-source heterogeneous original dataset D is subjected to structural mapping and semantic normalization processing to generate a data copy D′ with unified data field format and time alignment; Extract the corresponding metadata M for each data record in the data copy D′, and assign confidence weight values ​​to each metadata parameter by constructing a weighted factor model; Based on the volatility of the data values ​​themselves, historical stability, and meta-information weighting factors, a credibility score is generated by using a method that combines Bayesian inference and sliding window statistical analysis. All credibility scores are matrixed to construct a data credibility prediction matrix C, which is then combined with the structure-normalized data copy D′ to output a unified format dataset D0.

3. The power grid water dispatch data management and control method based on artificial intelligence according to claim 2, characterized in that: The process of generating a credibility score includes the following steps: Based on the time series characteristics of various data in the unified format dataset D0, the mean, standard deviation and coefficient of variation of the data within the sliding time window are extracted to construct a set of volatility indicators; By calling up stable data from the historical database under similar scenarios and similar collection devices over multiple periods, the current data is matched for similarity and deviation is calculated to form a set of historical stability indicators. By combining the meta-information weighted factor model, the data acquisition device type, sampling frequency, and timestamp accuracy parameters associated with each data point are weighted and fused to generate a context confidence vector; The volatility index, stability index, and contextual confidence vector are input into the Bayesian inference model to estimate the confidence probability, outputting a continuous confidence score and mapping it to the confidence score matrix.

4. The power grid water dispatch data management and control method based on artificial intelligence according to claim 1, characterized in that: The process of extracting the dominant quality factor subspace Q′ using tensor compression technology includes the following steps: Based on the time dimension, data field dimension, and five quality attribute dimensions in the unified format dataset D0, a third-order tensor structure Q is constructed, where the five quality attributes include completeness, consistency, accuracy, timeliness, and uniqueness. Normalize the data in each dimension of tensor Q; The tensor Q is compressed using a higher-order singular value decomposition method to separate the dominant factor space Q′ with the largest principal component variance contribution.

5. The power grid water dispatch data management and control method based on artificial intelligence according to claim 1, characterized in that: The method for identifying anomalous data points in D0 based on a clustering perceptual neural network includes the following steps: The dominant quality factor subspace Q′ is expanded into a two-dimensional feature matrix and reconstructed into a multi-dimensional input vector set in time series order. A clustering-aware neural network model is constructed, which integrates an autoencoder and a density-aware clustering module. The autoencoder is used for feature compression and reconstruction error analysis, and the clustering module is used to measure the distance distribution between data points and local cluster centers. Each input vector in Q′ is input into a clustering perceptual neural network, and its clustering consistency and reconstruction error level in high-dimensional space are identified through the training phase. Anomaly scores are calculated by combining reconstruction error and cluster isolation degree. If the anomaly score exceeds a set threshold, it is marked as an anomalous data point. The output is the anomalous dataset Derr and its corresponding anomalous label set Terr. The label content includes the anomaly type, severity level and data field location information.

6. The power grid water dispatch data management and control method based on artificial intelligence according to claim 1, characterized in that: The step of jointly inputting Derr, Terr, and the credibility matrix C into the defect data repair model includes the following steps: Based on the anomaly type and data field location information in the anomaly label set Terr, a set of data points to be repaired is extracted from Terr, and the repair priority is determined by combining the scores of the corresponding positions in the confidence matrix C. By constructing a retrieval query vector, the timestamp, spatial location, field type, and similar historical context of the data point to be repaired are encoded and input into the RAG retrieval module to dynamically retrieve similar data fragments in the industry water control scenario knowledge base; The retrieved context information is concatenated with the abnormal data points and their label information to form a structured repair suggestion, which is then input into a large vertical model pre-trained based on industry knowledge to generate repair suggestion values. The generated repair suggestion value is compared with the original data context for consistency. If the difference is less than the set threshold, it is directly replaced to form the repaired dataset Dfix.

7. The power grid water dispatch data management and control method based on artificial intelligence according to claim 1, characterized in that: The semantic alignment comparison based on the difference vector between Dfix and D0 includes the following steps: The numerical differences between the repaired dataset Dfix and the unified format dataset D0 at the same field and time index position are calculated to form a set of difference vectors ΔV, which are then categorized and stored according to field type. The set of difference vectors ΔV is semantically mapped and matched with the anomaly type, severity level, and source information in the anomaly label set Terr to establish a data repair logical link. Based on the mapping results, a data repair record table is generated, recording the original value, repair value, repair reason, and credibility improvement of each anomaly data point. Combining the data repair record table and difference statistics, a data quality improvement report is automatically generated.

Images