A power system data storage method and system based on big data analysis

Abstract

This invention discloses a power system data storage method and system based on big data analysis. The method includes the following steps: S1, acquiring raw operating data of the power system, which includes real-time power data, near-real-time power data, and historical network-wide operating data; S2, constructing a power data semantic uncertainty assessment model based on the historical network-wide operating data, and calculating the semantic uncertainty index of each piece of real-time / near-real-time power data through the assessment model. This invention relates to the field of power system data storage technology. This power system data storage method and system based on big data analysis, through a dynamic hierarchical storage method based on semantic uncertainty assessment, allows different types of data to be stored in different storage levels, greatly improving data access efficiency. Real-time and near-real-time data can be dynamically adjusted in storage location according to their importance and uncertainty, reducing unnecessary data redundancy.

Description

Technical Field

[0001] This invention relates to the field of power system data storage, and more specifically, to a power system data storage method and system based on big data analysis. Background Technology

[0002] With the continuous development of power systems, their scale is becoming increasingly large and their structure increasingly complex. A large number of data acquisition devices, such as smart meters and sensors, are deployed in power systems. These devices collect massive amounts of power operation data in real-time and near real-time, including real-time power data, near real-time power data, and historical network-wide operation data.

[0003] Traditional power system data storage methods often employ a uniform storage strategy, failing to adequately consider the characteristics and importance differences of various data. On one hand, using the same storage architecture and redundancy levels for all data leads to unreasonable allocation of storage resources. Some critical data may suffer from inefficient access due to inappropriate storage levels, while non-critical data consumes significant amounts of valuable storage resources. On the other hand, the lack of effective assessment and handling of semantic uncertainty during data storage results in a lack of targeted and flexible data storage.

[0004] Furthermore, data on the same physical quantity in a power system exhibits different characteristics and uses at different time scales, but traditional storage methods fail to consider the consistency and correlation between these data. Data storage often involves storing data from different time scales independently, leading to high data redundancy. This not only increases storage costs but also complicates data management and analysis. When complete original data is needed for analysis or playback, the lack of an effective data reconstruction mechanism makes it difficult to accurately reconstruct the original data from the stored data, impacting the accuracy and timeliness of power system monitoring, analysis, and decision-making. Summary of the Invention

[0005] The purpose of this invention is to provide a power system data storage method and system based on big data analysis. It solves the problem that the existing power system operation data, including real-time power data, near real-time power data, and historical network operation data, are large in volume and diverse in type. Traditional storage methods are difficult to meet the needs of real-time performance, redundancy control, and data query and analysis.

[0006] This invention achieves the above objective through the following technical solution: a power system data storage method based on big data analysis, the method comprising the following steps:

[0007] S1. Obtain the raw operating data of the power system, which includes real-time power data, near real-time power data, and historical network-wide operating data;

[0008] S2. Construct a semantic uncertainty assessment model for power data based on the historical network operation data, and calculate the semantic uncertainty index of each real-time / near real-time power data using the assessment model;

[0009] S3. Based on the semantic uncertainty index, dynamically allocate the distribution ratio and redundancy level of real-time / near real-time power data in different storage levels to complete adaptive hierarchical storage.

[0010] S4. Construct a consistency model across multiple time scales to perform consistency constraint analysis on the data evolution trend of the same physical quantity at different time scales;

[0011] S5. When it is determined that the short-cycle data change is a predictable disturbance within the long-term trend, only the trend residual parameter is stored; when data playback or analysis is required, the complete original data is reconstructed through the multi-timescale consistency model to achieve redundant storage.

[0012] Furthermore, obtaining the raw operating data of the power system in step S1 includes:

[0013] By deploying various data acquisition-related devices and system interfaces in the power system, we comprehensively collect multiple types of raw operating data from various power-related devices and terminal power consumption nodes;

[0014] Perform full-process preprocessing operations on the collected raw operational data, including data cleaning, outlier removal, and format standardization, to ensure the consistency and usability of the data in subsequent processing.

[0015] Furthermore, the data cleaning includes:

[0016] Remove missing and duplicate values ​​caused by equipment failure or transmission interference, and use corresponding missing value imputation methods for different types of data;

[0017] The outlier removal is based on statistical principles. Statistical parameters of the data sequence are calculated, and data exceeding a preset range are identified as outliers and removed. At the same time, outlier events are recorded.

[0018] The format standardization process uses relevant data processing technologies to convert unstructured data into structured data, thereby unifying the data format.

[0019] Furthermore, step S2, which involves constructing a power data semantic uncertainty assessment model based on the historical network-wide operational data, includes:

[0020] Extract multi-dimensional data features from standardized historical network operation data;

[0021] The data features include at least one of the following: data correlation features, time decay features, scheduling requirement correlation features, and fault backtracking correlation features;

[0022] The weights of each data feature are determined using the analytic hierarchy process (AHP).

[0023] A semantic uncertainty assessment model based on a deep learning network is constructed, and the model is trained and optimized to obtain the final assessment model.

[0024] Furthermore, the analytic hierarchy process (AHP) determines the weights by including:

[0025] Construct a multi-layered structural model consisting of a target layer, a criterion layer, and an indicator layer;

[0026] Experts in relevant fields were invited to compare the relative importance of each feature in the criterion layer and construct a judgment matrix;

[0027] The rationality of the judgment matrix is ​​verified by consistency test. The judgment matrix that passes the test is decomposed into eigenvalues, the eigenvectors are calculated and normalized to obtain the weights of the features of each criterion layer.

[0028] Furthermore, the training steps of the semantic uncertainty assessment model include:

[0029] Divide the dataset and normalize the data;

[0030] Initialize model parameters and configure training-related parameters;

[0031] The model is trained iteratively using appropriate training methods, and an early stopping mechanism is set to prevent overfitting.

[0032] The trained model is evaluated until it meets the preset evaluation criteria.

[0033] Furthermore, the adaptive hierarchical storage in step S3 includes:

[0034] A preset semantic uncertainty index threshold range is used to allocate real-time / near real-time power data of different threshold ranges to the corresponding storage level, which includes a hot storage layer, a warm storage layer, and a cold storage layer.

[0035] Each storage tier adopts a corresponding storage architecture, storage latency control standard, and redundancy level configuration;

[0036] Establish a dynamic threshold adjustment mechanism to optimize threshold range division and redundancy level configuration based on relevant power system operation monitoring data.

[0037] Furthermore, step S4, which involves constructing a consistency model across multiple time scales, includes:

[0038] Select the core physical quantities of power system operation, extract multi-timescale data sequences of each core physical quantity from historical standardized data, and construct a multi-timescale data sample set by aligning the timestamps.

[0039] A multi-timescale consistency model is constructed based on a long short-term memory network combined with an attention mechanism. The model includes a feature extraction module and a trend prediction module.

[0040] The model is trained and optimized to obtain the final multi-timescale consistency model;

[0041] By inputting real-time data of the same physical quantity at different time scales into the model, and calculating the deviation between the predicted data and the real-time data, it is determined whether the data evolution conforms to the consistency law of multiple time scales.

[0042] Furthermore, step S5 implements redundancy removal storage, including:

[0043] A consistency judgment threshold is set, which is determined based on the fluctuation characteristics of historical data and the accuracy requirements of data reconstruction.

[0044] Calculate the deviation between short-cycle data changes and long-term trend data predicted by a multi-timescale consistency model. When the deviation meets the preset conditions, it is determined to be a predictable disturbance. The trend residual parameters are then extracted and stored.

[0045] When raw short-period data is needed, long-term trend data is generated through a multi-timescale consistency model, and the complete raw data is reconstructed by using a residual compensation algorithm in combination with the trend residual parameters.

[0046] A power system data storage system based on big data analytics, applied to the aforementioned power system data storage method based on big data analytics, the system comprising:

[0047] The data acquisition module is used to acquire the raw operating data of the power system and to preprocess the raw operating data;

[0048] The semantic uncertainty assessment module is used to build a power data semantic uncertainty assessment model based on historical network operation data and calculate the semantic uncertainty index of real-time / near real-time power data.

[0049] The adaptive tiered storage module is used to dynamically allocate the distribution ratio and redundancy level of data in different storage tiers based on the semantic uncertainty index, thereby completing adaptive tiered storage.

[0050] The multi-timescale consistency analysis module is used to build a consistency model across multiple timescales and perform consistency constraint analysis on the data evolution trend of the same physical quantity at different timescales.

[0051] The redundancy removal storage module is used to determine whether short-period data changes are predictable disturbances within the long-term trend. It only stores trend residual parameters and reconstructs the complete original data through a consistency model when needed, thus achieving redundancy removal storage.

[0052] The beneficial effects of this invention are as follows:

[0053] 1. By using a dynamic hierarchical storage method based on semantic uncertainty assessment, different types of data are stored in different storage levels, which greatly improves data access efficiency. Real-time and near-real-time data can be dynamically adjusted in storage location according to their importance and uncertainty, reducing unnecessary data redundancy.

[0054] 2. When short-period data experiences predictable disturbances in its long-term trend, only the trend residual is preserved, reducing the storage space required and achieving redundant storage. This is of great significance for the massive data in power systems and can effectively save storage costs.

[0055] 3. By constructing a consistency model across multiple time scales, the consistency of data at different time scales can be ensured, thereby improving the accuracy of data analysis. In particular, for long-running data, effective trend prediction and error correction can be performed, further enhancing the utilization value of the data.

[0056] 4. Adopt adaptive hierarchical storage technology to dynamically adjust the distribution ratio and redundancy level of data in different storage tiers based on the semantic uncertainty index and the characteristics of real-time power data, optimize storage resource configuration, and meet the needs of different power system operations.

[0057] 5. Through precise management and redundancy removal of power system data, when playback or analysis is required, the complete original data can be quickly reconstructed through a consistency model, ensuring the real-time performance and efficiency of the data. Attached Figure Description

[0058] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0059] Figure 1 This is a flowchart illustrating the overall method of the present invention;

[0060] Figure 2 This is a detailed flowchart of step S2 of the present invention;

[0061] Figure 3 This is a detailed flowchart of step S4 of the present invention;

[0062] Figure 4 This is a system block diagram of the present invention. Detailed Implementation

[0063] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0064] Example 1:

[0065] Please see Figures 1-3 This invention provides a technical solution: a power system data storage method based on big data analysis, the method comprising:

[0066] S1. Obtain raw operating data of the power system, including real-time power data, near real-time power data, and historical network-wide operating data;

[0067] The power system is a power production and consumption system composed of generation, transmission, transformation, distribution, and consumption, used to realize the supply, transmission, distribution, and consumption of electricity. Raw operating data is unprocessed data directly generated by the power system during operation. Real-time power data reflects the current operating status of the power system and has extremely high timeliness, such as real-time power, voltage, and current data at a certain moment. Near-real-time power data is similar to real-time data, but with slightly lower timeliness requirements, and is usually updated at shorter time intervals, such as every few minutes, to reflect the recent operating status of the power system. Historical network-wide operating data is the operating data of the entire power system over a period of time, covering operating information of different time points, different equipment, and different regions, and can be used to analyze the historical operating patterns and trends of the power system.

[0068] S2. Construct a semantic uncertainty assessment model for power data based on historical network operation data, and calculate the semantic uncertainty index of each real-time / near real-time power data using the assessment model;

[0069] The power data semantic uncertainty assessment model is a mathematical model used to assess the semantic uncertainty of power data. By learning and analyzing historical network operation data, it mines the semantic information behind the data and the correlation between data, thereby calculating the semantic uncertainty index of each real-time / near real-time power data. The semantic uncertainty index is used to quantify the degree of semantic uncertainty of real-time / near real-time power data. The higher the index, the more ambiguous the semantics of the data, and the more possible there are multiple interpretations or meanings. The lower the index, the more definite the semantics of the data.

[0070] S3. Based on the semantic uncertainty index, dynamically allocate the distribution ratio and redundancy level of real-time / near real-time power data in hot storage, warm storage, and cold storage to complete adaptive hierarchical storage.

[0071] Among them, warm storage is a storage method with performance and cost between hot storage and cold storage, used to store near real-time power data with moderate access frequency and less stringent real-time requirements; cold storage is a lower-cost storage method with relatively poor read / write performance, suitable for storing historical data that is not frequently accessed; distribution ratio refers to the proportion of real-time / near real-time power data in hot storage, warm storage, and cold storage, which is dynamically allocated according to the semantic uncertainty index to achieve rational and efficient data storage; redundancy level is the degree of redundant backup of stored data to ensure data reliability and availability, with different redundancy levels corresponding to different backup strategies and storage overheads, and the redundancy level of real-time / near real-time power data is dynamically determined according to the semantic uncertainty index; adaptive tiered storage automatically adjusts the distribution ratio and redundancy level of data in hot storage, warm storage, and cold storage according to the semantic uncertainty index of real-time / near real-time power data to adapt to the characteristics and storage needs of different data and achieve data storage optimization;

[0072] S4. Construct a multi-timescale consistency model spanning milliseconds, minutes, and hours to perform consistency constraint analysis on the data evolution trend of the same physical quantity at different timescales;

[0073] Among them, the multi-timescale consistency model spanning milliseconds, minutes, and hours is a model capable of handling data at different timescales, including milliseconds, minutes, and hours. This model analyzes data on the same physical quantity at different timescales, establishes the correlation between them and consistency constraints, and ensures that the data at different timescales maintains consistency in evolution trends. Physical quantities are various quantities used in power systems to describe the operating state of equipment or system characteristics, such as voltage, current, power, and frequency. Consistency constraint analysis uses the multi-timescale consistency model to analyze the data evolution trends of the same physical quantity at different timescales, and determines whether they meet the pre-set consistency constraints to ensure the accuracy and reliability of the data.

[0074] S5. When it is determined that the short-period data change is a predictable disturbance within the long-term trend, only the trend residual parameter is stored; when data playback or analysis is required, the complete original data is reconstructed through a multi-timescale consistency model to achieve redundant storage.

[0075] Among these, short-cycle data changes refer to changes in power system data over a short period of time, such as minutes or hours; long-term trends refer to the overall trend of power system data over a longer period of time, such as days, months, or years; predictable disturbances refer to factors or events that can be predicted in advance based on historical operating data and patterns of the power system, which will affect system operation; trend residual parameters are used when short-cycle data changes are determined to be predictable disturbances within the long-term trend, and only the difference between the actual data and the long-term trend is stored, i.e., the trend residual parameter, which reduces the amount of data stored and achieves redundant storage; data playback is the process of re-presenting the stored data in a certain time sequence to view and analyze the historical operation of the power system; data reconstruction is the process of restoring the data to its original state using a multi-timescale consistency model and stored trend residual parameters when complete original data is needed, in order to meet the needs of data playback and analysis; and redundant storage is the process of removing redundant information from the data by storing trend residual parameters and other methods, reducing the amount of data stored and improving storage efficiency.

[0076] It should be noted that, during use, by constructing a semantic uncertainty assessment model to calculate the index, the characteristics of the data can be accurately grasped. Based on this, storage can be dynamically allocated, and adaptive hierarchical storage allows data to be distributed on demand in hot, warm, and cold storage. This ensures fast reading of high-frequency access data while reducing storage costs. The construction of a multi-timescale consistency model can analyze and constrain the evolution trend of the same physical quantity data at different time scales, ensuring data accuracy and reliability. When short-cycle data changes are determined to be predictable disturbances within the long-term trend, only trend residual parameters remain, greatly reducing storage volume and achieving redundant storage. When needed, the complete original data can be reconstructed through the model, balancing storage efficiency and data integrity, and providing strong support for the efficient operation, data analysis, and decision-making of power systems.

[0077] In one embodiment, acquiring raw operating data of the power system includes:

[0078] By deploying intelligent sensing terminals, data acquisition units, remote terminal units, and energy management system interfaces in the power system, a wide range of original operating data from various power generation equipment, transmission lines, substation equipment, distribution equipment, and terminal power consumption nodes are comprehensively collected.

[0079] The raw operating data includes instantaneous values ​​of three-phase voltage, instantaneous values ​​of three-phase current, active power / reactive power load curves, equipment switch status, fault alarm information, environmental monitoring data, etc. The data formats include structured numerical data, log text-type unstructured data, and equipment status image-type unstructured data.

[0080] Perform full-process preprocessing on the collected raw operational data:

[0081] Data cleaning removes missing and duplicate values ​​caused by sensor malfunctions and transmission interference. For continuous data, linear interpolation is used to fill in missing values, and for discrete data, mode imputation is used.

[0082] based on Statistical principles are used to remove outliers and calculate the mean of the data series. and standard deviation It will exceed Data within a given range is identified as outliers and removed. At the same time, the outlier events are recorded for subsequent equipment status analysis.

[0083] The system performs format standardization processing. It uses natural language processing technology to extract key information from unstructured text data and convert it into key-value pair format. It uses image recognition technology to extract feature parameters from device status images and quantify them into numerical data. All data is then uniformly converted into structured data in JSON format to ensure data consistency and usability in subsequent processing.

[0084] This design comprehensively collects various types of raw operational data through multiple terminals, covering multiple data formats, and performs full-process preprocessing. Comprehensive collection enables the acquisition of all-round information about the power system, providing rich materials for subsequent analysis. The coverage of multiple data formats meets the needs of different scenarios. The full-process preprocessing operation is of great significance: data cleaning removes bad data to ensure data quality; outlier removal and recording facilitate equipment status analysis; format standardization unifies the data format, ensuring consistency and usability in subsequent processing, enabling data to participate efficiently in subsequent semantic evaluation, storage allocation, and other stages, improving the accuracy and efficiency of data storage and processing in the entire power system.

[0085] In one embodiment, a semantic uncertainty assessment model for power data is constructed based on historical network-wide operational data. The semantic uncertainty index for each piece of real-time / near-real-time power data is calculated using this assessment model, including:

[0086] From the standardized historical network operation data, a sliding window sampling method is used to extract multi-dimensional data features: data correlation features include the Pearson correlation coefficient and mutual information value of the data with other equipment operation data and scheduling instruction data, reflecting the correlation importance of the data in the system;

[0087] The time decay feature is based on the interval between the data generation time and the current time, and is calculated using an exponential decay function to reflect the impact of data timeliness on value.

[0088] The correlation characteristics of scheduling requirements are obtained by statistically analyzing the frequency and proportion of historical data in which this type of data is called in the scheduling decision-making process, directly relating to the actual application value of the data;

[0089] The correlation characteristics of fault retrospection are calculated by analyzing the contribution of this type of data in historical fault events to fault location and cause analysis, and then quantifying it by assigning weights using the analytic hierarchy process.

[0090] The rules for determining weights in the analytic hierarchy process (AHP) are as follows:

[0091] A three-layer structure model is constructed, consisting of a target layer (assessment of data semantic uncertainty), a criterion layer (data correlation, time decay, scheduling demand correlation, and fault backtracking correlation), and an indicator layer (specific quantitative indicators corresponding to each feature). Five to eight experts in the fields of power system scheduling, data storage, and fault analysis are invited to conduct pairwise comparisons of the relative importance of each feature in the criterion layer to construct a judgment matrix.

[0092] Passed the consistency test (consistency rate) Verify the validity of the judgment matrix; if it is not satisfied, readjust the judgment matrix.

[0093] The eigenvalue decomposition of the judgment matrix that passes the consistency test is performed, the eigenvector is calculated and normalized, and the weights of the features of each criterion layer are obtained: data correlation weight 0.3, time decay weight 0.2, scheduling requirement correlation weight 0.35, and fault backtracking correlation weight 0.15.

[0094] A semantic uncertainty assessment model based on a deep neural network is constructed. The model architecture includes an input layer, a hidden layer, and an output layer: the dimension of the input layer is consistent with the dimension of the extracted features, denoted as . dimension, To accommodate the number of features, four hidden layers are used, with 512, 256, 128, and 64 neurons in each layer, respectively. The ReLU activation function is chosen to mitigate the vanishing gradient problem, and the Sigmoid function is used in the output layer to map the output values ​​to... interval;

[0095] The specific training steps for the model are as follows:

[0096] The dataset is divided into training, validation and test sets in a 7:2:1 ratio after standardization of historical feature data to ensure that the value fluctuation coefficients of each dataset are distributed in a consistent manner.

[0097] Data normalization involves applying Min-Max normalization to the feature data of the training, validation, and test sets, mapping the data to... The interval, the formula is:

[0098]

[0099] in, These are the original eigenvalues. , These are the minimum and maximum values ​​of this feature in the training set, respectively.

[0100] Model initialization: Set the random seed to 42 to ensure reproducible training; use the He initialization method to initialize the hidden layer weights; initialize the bias term to 0.

[0101] Training parameters were configured as follows: the optimizer was Adam, the initial learning rate was set to 0.001, and cosine annealing was used as the learning rate decay strategy. , The batch size was set to 64, the number of training rounds was set to 200, the loss function was set to mean squared error, the regularization method was set to L2 regularization, and the weight decay coefficient was set to 0.0005.

[0102] Model training employs iterative training using mini-batch gradient descent, with computation performed on the validation set after each training round. The loss mechanism triggers an early stopping mechanism when the validation set loss fails to decrease for 10 consecutive rounds to prevent overfitting.

[0103] Model evaluation involves calculating the model's mean absolute error and root mean square error on the test set. The mean absolute error must be ≤0.02 and the root mean square error must be ≤0.03. If these conditions are met, the model is considered to have completed training. Otherwise, the number of hidden layer neurons or the learning rate are adjusted for retraining.

[0104] Using the extracted multi-dimensional data features as input vectors, and the value fluctuation coefficient of the data in scenarios such as scheduling analysis, fault backtracking, and system optimization within the next year as labels, the model training is completed according to the above steps until the loss value on the validation set converges and tends to stabilize, thus obtaining the optimized evaluation model.

[0105] Preprocessed real-time / near-real-time power data are used to generate feature vectors according to the same feature extraction rules. These vectors are then input into the optimized evaluation model, which outputs a semantic uncertainty index for each data point. ;index The range of values ​​is The larger the value, the greater the potential value fluctuation of the data in the future scheduling, analysis, and backtracking process. The higher the importance of the data to system operation decisions and fault handling, the higher the priority of storage resources required.

[0106] This design constructs an evaluation model to calculate the semantic uncertainty index of real-time / near real-time power data, extracts features from multiple dimensions to comprehensively consider data characteristics, and evaluates data from multiple aspects such as relevance, timeliness, and practical application value. The analytic hierarchy process (AHP) determines the weights to make the evaluation more scientific and reasonable. The deep neural network model can accurately learn the relationship between data features and value fluctuation coefficients and output the semantic uncertainty index. This index can quantify the potential value fluctuation range of the data, clarify the importance of the data to system operation decisions and fault handling, provide key basis for subsequent adaptive hierarchical storage, and realize the rational allocation of storage resources.

[0107] In one embodiment, based on the semantic uncertainty index, the distribution ratio and redundancy level of real-time / near real-time power data in hot storage, warm storage, and cold storage are dynamically allocated to complete adaptive hierarchical storage, including:

[0108] Preset initial semantic uncertainty index threshold range:

[0109] The hot storage threshold range is This corresponds to high-value, volatile data; the temperature storage threshold range is... This corresponds to data with moderate value fluctuations; the cold storage threshold range is... This corresponds to low-value, volatile data.

[0110] The initial threshold is determined as follows: Based on one year of historical power system operation data, the value fluctuation coefficient distribution of all data in subsequent applications is statistically analyzed; the K-means clustering algorithm is used to cluster the value fluctuation coefficients. The value fluctuation coefficient cluster centers of high, medium and low data are obtained; the semantic uncertainty index mapping value corresponding to the cluster center is used as the initial threshold benchmark. Combined with the storage resource configuration, hot storage capacity accounts for 20% of the total storage capacity, warm storage accounts for 50% and cold storage accounts for 30%. The threshold range is adjusted to match the proportion of each type of data with the proportion of storage resources, and the initial threshold is determined.

[0111] Storage tier configuration rules: When the semantic uncertainty index of data When data is distributed to a hot storage layer built with an all-flash array, the storage layer is deployed in a distributed architecture, with data read and write latency controlled within 10 milliseconds. At the same time, the redundancy level is set to the highest level, and no less than 3 data replicas are stored on different physical nodes through a multi-replica redundancy strategy. The replicas are synchronized using a synchronous write mechanism to ensure high availability and millisecond-level access response speed, meeting the high-frequency access requirements of scenarios such as real-time scheduling and emergency fault handling.

[0112] When the semantic uncertainty index of data At that time, the data is allocated to a warm storage layer using a hybrid storage architecture of flash memory + mechanical hard disk, with storage latency controlled within 100 milliseconds. The redundancy level is set to medium, and the number of redundant replicas is dynamically adjusted according to the data access frequency: 2 replicas are retained when the access frequency is higher than 10 times per day, and 1 replica is retained when the access frequency is lower than 10 times per day. The replicas are updated using an asynchronous synchronization mechanism to balance storage access speed and hardware cost investment while ensuring data reliability.

[0113] When the semantic uncertainty index of data At that time, the data is allocated to a cold storage layer built with a high-density mechanical hard disk array, mainly to meet the needs of data archiving and long-term backup. The storage latency is controlled within 1 second, the redundancy level is set to the lowest level, only one data copy is kept, and compression storage technology is adopted. The compression algorithm is LZ4 to reduce storage resource consumption. The copy is stored on offline or near offline nodes, and data integrity is verified regularly.

[0114] Establish a dynamic threshold adjustment mechanism: Monitor real-time changes in power system dispatch demand, data access frequency statistics, and storage resource occupancy. Each quarter, optimize the threshold range division criteria using a genetic algorithm based on monitoring data, with the optimization objective being to maintain storage resource utilization at a certain level. The average data access latency is below a preset threshold; simultaneously, the redundancy level configuration is dynamically adjusted based on the resource utilization of different storage tiers. For example, when the resource utilization of the hot storage tier exceeds... In some cases, the upper limit of the hot storage threshold can be temporarily adjusted to 0.85 to avoid overloading storage resources and ensure the flexibility and adaptability of tiered storage.

[0115] This design dynamically allocates data storage tiers and redundancy levels based on the semantic uncertainty index. It presets initial threshold ranges and combines historical data and storage resource conditions to make the allocation more reasonable. Different storage tiers adopt different storage architectures and latency controls for data with different value fluctuations to meet various data access needs. Setting different redundancy levels ensures data reliability while balancing costs. The dynamic threshold adjustment mechanism optimizes the thresholds based on system changes and storage resource utilization, ensuring that tiered storage can flexibly adapt to actual needs, improve storage resource utilization, and ensure effective storage and fast access to data in different scenarios.

[0116] In one embodiment, a multi-timescale consistency model spanning milliseconds, minutes, and hours is constructed to perform consistency constraint analysis on the data evolution trend of the same physical quantity at different timescales, including:

[0117] The core physical quantities of power system operation are selected, including key physical quantities that directly affect the safe and stable operation of the system, such as bus voltage, line current, system active load, system reactive load, and transformer load rate.

[0118] For each core physical quantity, multi-timescale data sequences are extracted from historical standardized data: millisecond-level data sequences are sampled at 10 millisecond intervals to capture instantaneous fluctuation characteristics; minute-level data sequences are sampled at 1 minute intervals to reflect short-term trends; and hour-level data sequences are sampled at 1 hour intervals to reflect long-term evolution patterns. Multi-timescale data sample sets are constructed by aligning with timestamps, with each sample containing three timescale data sequences of the same physical quantity within the same time interval.

[0119] A multi-timescale consistency model is constructed based on a Long Short-Term Memory (LSTM) network combined with an attention mechanism. The model consists of a feature extraction module and a trend prediction module. The feature extraction module uses a three-layer LSM network with 128 neurons in each layer to extract features from data sequences at three time scales, capturing the temporal dependencies at different scales. The attention mechanism layer assigns dynamic weights to the features extracted at each scale, highlighting the contribution of key scale features to trend prediction. The trend prediction module uses a two-layer fully connected network with 64 and 32 neurons respectively, using ReLU as the activation function. The number of neurons in the output layer is consistent with the dimension of the prediction target.

[0120] Rules for determining the dynamic weights of the attention mechanism:

[0121] Weights are adaptively assigned based on the predicted contribution of each time-scale feature. The contribution is calculated by summing the absolute values ​​of the gradients of that feature during model training; a larger sum of absolute gradient values ​​indicates a higher contribution and a larger weight. The specific calculation formula is as follows:

[0122]

[0123] in, For the first Weights of features at each time scale Corresponding to millisecond, minute, and hourly timeframes, respectively. For the model loss function on the th The gradient of the feature extraction results at each time scale is calculated through backpropagation.

[0124] The specific training steps for a multi-timescale consistency model are as follows:

[0125] Dataset preprocessing involves Z-score standardization of the multi-timescale data sample set, using the following formula:

[0126]

[0127] in, The mean of the sample set. The standard deviation of the sample set is given; the sample set is divided into training, validation, and test sets in an 8:1:1 ratio.

[0128] Model initialization: Set the random seed to 42, use orthogonal initialization for the weights of the long short-term memory network layers, use Xavier initialization for the weights of the fully connected layers, and initialize the bias term to 0.

[0129] Training parameters were configured as follows: the optimizer was AdamW, the learning rate was set to 0.0005, the weight decay coefficient was set to 0.001, the batch size was set to 32, the number of training epochs was set to 150, and gradient clipping was used. To prevent gradient explosion; mean squared error is chosen as the loss function.

[0130] During training, a time-series sequential input method is used for training. After each training round, the mean squared error loss is calculated on the validation set. Early stopping is triggered when the validation set loss does not decrease for 15 consecutive rounds. At the same time, a model checkpointing mechanism is used to save the model weights with the lowest validation set loss.

[0131] Model evaluation involves calculating prediction accuracy and mean absolute percentage error on the test set, with a target prediction accuracy of [missing value]. Mean absolute percentage error If the condition is met, the model training is considered complete; otherwise, the number of layers or neurons in the long short-term memory network is adjusted and the model is retrained.

[0132] Using a multi-timescale data sample set as training data, data sequences from two time scales are taken as input, and a third time scale data sequence is used as the prediction target. The model is trained using cross-validation following the steps described above. The optimization objective is to minimize the mean squared error between the predicted results and the actual data. Hyperparameters such as the number of network layers, neurons, and learning rate are adjusted to improve the model's prediction accuracy until the model's prediction accuracy on the test set exceeds [a certain threshold]. This yields a multi-timescale consistency model that has been trained.

[0133] Real-time data of the same physical quantity at different time scales are aligned by timestamps and input into the model. The model outputs evolution trend prediction data at each time scale, and the deviation between the prediction data and the real-time data is calculated using the following formula:

[0134]

[0135] Preset acceptable deviation threshold: For millisecond-level data, the acceptable deviation threshold is... For minute-level data, a reasonable deviation threshold is: For hourly data, a reasonable deviation threshold is: The rule for determining the reasonable range threshold for deviation is as follows: Based on multi-timescale data of the same physical quantity over the past three years, calculate the deviation distribution between the predicted and actual data under normal operating conditions, and take the 95th quantile of the deviation distribution as the reasonable range threshold to ensure... The above normal data evolution meets the consistency requirements; by comparing the deviation value with the reasonable range threshold, it is determined whether the data evolution conforms to the consistency law of multiple time scales: when the deviation value is within the preset reasonable range, it is determined that the data evolution trend is consistent; when the deviation value exceeds the preset range, it is determined that there is abnormal fluctuation, and the cause of the fluctuation needs to be further analyzed.

[0136] This design constructs a multi-timescale consistency model to analyze the evolution trend of core physical quantity data. It selects key physical quantities to focus on the core factors affecting the safe and stable operation of the system, extracts multi-timescale data sequences to comprehensively capture data features, and constructs a model by combining a long short-term memory network with an attention mechanism. This model can accurately extract features at different scales and highlight the contribution of key features. Through rigorous training and evaluation, the model's accuracy is ensured. This model can determine whether the data evolution conforms to the rules, detect abnormal fluctuations in a timely manner, and provide a basis for judgment for subsequent redundancy removal and storage, thus ensuring the accuracy and consistency of power system data at different time scales.

[0137] In one embodiment, when short-period data changes are determined to be predictable disturbances within a long-term trend, only the trend residual parameters are stored; when data playback or analysis is required, the complete original data is reconstructed through a multi-timescale consistency model to achieve redundant storage, including:

[0138] Set consistency judgment threshold threshold Based on the fluctuation characteristics determined by historical data, separate settings were set for different physical quantities: voltage and current. Values Load-related physical quantities Values Transformer load rate Values Threshold The specific determination rules are as follows: Select multi-timescale data of the same physical quantity during the normal operation period of the power system in the past year, calculate the deviation between short-period data and long-term trend data, fit the probability distribution curve of the deviation value using the kernel density estimation method, take the deviation value corresponding to the point of maximum probability density as the basic threshold, and then combine it with the data reconstruction accuracy requirements to determine the reconstruction error. The basic thresholds are then adjusted to ultimately determine the consistency judgment thresholds for various physical quantities. ;

[0139] Calculate the deviation between short-period data (milliseconds, minutes) and the long-term trend (hourly trend extension) predicted by the multi-timescale consistency model. When the deviation is ≤ When the short-cycle data change is determined to be a predictable disturbance within the long-term trend, that is, the fluctuation conforms to the inherent law of system operation and can be restored by combining the long-term trend with the disturbance characteristics;

[0140] The least squares method is used to fit the residual sequence between the predictable disturbance data and the model-predicted long-term trend data. Trend residual parameters are extracted, including: residual mean (reflecting the average strength of the disturbance), residual variance (reflecting the dispersion of the disturbance), residual change period (reflecting the periodicity of the disturbance), and residual peak value (reflecting the maximum amplitude of the disturbance). These residual parameters are then linked by timestamps and compressed for storage, discarding the complete original short-period data, which significantly reduces the amount of data stored.

[0141] When a power system needs to perform data playback, operational analysis, or fault tracing, and requires access to original short-cycle data, the system first generates long-term trend data for the corresponding time interval using a multi-timescale consistency model. Then, based on the stored trend residual parameters, a residual compensation algorithm is used to reconstruct the complete original data. The formula for the residual compensation algorithm is as follows:

[0142]

[0143] In the formula, For the reconstructed original data, For long-term trend data predicted by the model, For the stored set of trend residual parameters, This is a time-weighted function used to simulate the variation of residuals over time. Weighting rules: Based on the time distribution characteristics of the residual sequence, the weight of each time point is calculated using exponential smoothing. The weight of recent residuals is higher than that of distant residuals, and the sum of the weights is 1. This ensures the consistency between the reconstructed data and the original data, and the error of the reconstructed data is controlled within a certain range. Within a certain range, it meets the accuracy requirements of practical application scenarios.

[0144] This design allows for the storage of only trend residual parameters when short-cycle data changes are determined to be predictable disturbances. Data reconstruction is performed when necessary, and a reasonable consistency threshold is set to accurately determine the nature of data changes. Storing only trend residual parameters significantly reduces the amount of data stored, saving storage resources. The residual compensation algorithm can accurately reconstruct the complete original data during data playback or analysis, meeting the accuracy requirements of practical applications. This redundancy-free storage method effectively reduces the storage burden and improves data storage efficiency while ensuring data availability, providing strong support for power system data analysis and fault tracing.

[0145] Example 2:

[0146] Please see Figure 4 A power system data storage system based on big data analytics, applied to the aforementioned power system data storage method based on big data analytics, the system comprising:

[0147] The data acquisition module is used to acquire the raw operating data of the power system and to preprocess the raw operating data;

[0148] The semantic uncertainty assessment module is used to build a power data semantic uncertainty assessment model based on historical network operation data and calculate the semantic uncertainty index of real-time / near real-time power data.

[0149] The adaptive tiered storage module is used to dynamically allocate the distribution ratio and redundancy level of data in different storage tiers based on the semantic uncertainty index, thereby completing adaptive tiered storage.

[0150] The multi-timescale consistency analysis module is used to build a consistency model across multiple timescales and perform consistency constraint analysis on the data evolution trend of the same physical quantity at different timescales.

[0151] The redundancy removal storage module is used to determine whether short-period data changes are predictable disturbances within the long-term trend. It only stores trend residual parameters and reconstructs the complete original data through a consistency model when needed, thus achieving redundancy removal storage.

[0152] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A power system data storage method based on big data analysis, characterized in that, The method includes the following steps: S1. Obtain the raw operating data of the power system, which includes real-time power data, near real-time power data, and historical network-wide operating data; S2. Construct a semantic uncertainty assessment model for power data based on the historical network operation data, and calculate the semantic uncertainty index of each real-time / near real-time power data using the assessment model; S3. Based on the semantic uncertainty index, dynamically allocate the distribution ratio and redundancy level of real-time / near real-time power data in different storage levels to complete adaptive hierarchical storage. S4. Construct a consistency model across multiple time scales to perform consistency constraint analysis on the data evolution trend of the same physical quantity at different time scales; S5. When it is determined that the short-cycle data change is a predictable disturbance within the long-term trend, only the trend residual parameter is stored; when data playback or analysis is required, the complete original data is reconstructed through the multi-timescale consistency model to achieve redundant storage.

2. The power system data storage method based on big data analysis according to claim 1, characterized in that, The step S1 of obtaining the raw operating data of the power system includes: By deploying various data acquisition-related devices and system interfaces in the power system, we comprehensively collect multiple types of raw operating data from various power-related devices and terminal power consumption nodes; Perform full-process preprocessing operations on the collected raw operational data, including data cleaning, outlier removal, and format standardization, to ensure the consistency and usability of the data in subsequent processing.

3. The power system data storage method based on big data analysis according to claim 2, characterized in that, The data cleaning includes: Remove missing and duplicate values ​​caused by equipment failure or transmission interference, and use corresponding missing value imputation methods for different types of data; The outlier removal is based on statistical principles. Statistical parameters of the data sequence are calculated, and data exceeding a preset range are identified as outliers and removed. At the same time, outlier events are recorded. The format standardization process uses relevant data processing technologies to convert unstructured data into structured data, thereby unifying the data format.

4. The power system data storage method based on big data analysis according to claim 1, characterized in that, Step S2, which involves constructing a power data semantic uncertainty assessment model based on the historical network operation data, includes: Extract multi-dimensional data features from standardized historical network operation data; The data features include at least one of the following: data correlation features, time decay features, scheduling requirement correlation features, and fault backtracking correlation features; The weights of each data feature are determined using the analytic hierarchy process (AHP). A semantic uncertainty assessment model based on a deep learning network is constructed, and the model is trained and optimized to obtain the final assessment model.

5. The power system data storage method based on big data analysis according to claim 4, characterized in that, The analytic hierarchy process (AHP) determines the weights by including: Construct a multi-layered structural model consisting of a target layer, a criterion layer, and an indicator layer; Experts in relevant fields were invited to compare the relative importance of each feature in the criterion layer and construct a judgment matrix; The rationality of the judgment matrix is ​​verified by consistency test. The judgment matrix that passes the test is decomposed into eigenvalues, the eigenvectors are calculated and normalized to obtain the weights of the features of each criterion layer.

6. The power system data storage method based on big data analysis according to claim 4, characterized in that, The training steps for the semantic uncertainty assessment model include: Divide the dataset and normalize the data; Initialize model parameters and configure training-related parameters; The model is trained iteratively using appropriate training methods, and an early stopping mechanism is set to prevent overfitting. The trained model is evaluated until it meets the preset evaluation criteria.

7. The power system data storage method based on big data analysis according to claim 1, characterized in that, The adaptive hierarchical storage in step S3 includes: A preset semantic uncertainty index threshold range is used to allocate real-time / near real-time power data of different threshold ranges to the corresponding storage level, which includes a hot storage layer, a warm storage layer, and a cold storage layer. Each storage tier adopts a corresponding storage architecture, storage latency control standard, and redundancy level configuration; Establish a dynamic threshold adjustment mechanism to optimize threshold range division and redundancy level configuration based on relevant power system operation monitoring data.

8. The power system data storage method based on big data analysis according to claim 1, characterized in that, Step S4, which involves constructing a consistency model across multiple time scales, includes: Select the core physical quantities of power system operation, extract multi-timescale data sequences of each core physical quantity from historical standardized data, and construct a multi-timescale data sample set by aligning the timestamps. A multi-timescale consistency model is constructed based on a long short-term memory network combined with an attention mechanism. The model includes a feature extraction module and a trend prediction module. The model is trained and optimized to obtain the final multi-timescale consistency model; By inputting real-time data of the same physical quantity at different time scales into the model, and calculating the deviation between the predicted data and the real-time data, it is determined whether the data evolution conforms to the consistency law of multiple time scales.

9. The power system data storage method based on big data analysis according to claim 1, characterized in that, The redundancy removal storage implemented in step S5 includes: A consistency judgment threshold is set, which is determined based on the fluctuation characteristics of historical data and the accuracy requirements of data reconstruction. Calculate the deviation between short-cycle data changes and long-term trend data predicted by a multi-timescale consistency model. When the deviation meets the preset conditions, it is determined to be a predictable disturbance. The trend residual parameters are then extracted and stored. When raw short-period data is needed, long-term trend data is generated through a multi-timescale consistency model, and the complete raw data is reconstructed by using a residual compensation algorithm in combination with the trend residual parameters.

10. A power system data storage system based on big data analytics, characterized in that, The system is applied to the power system data storage method based on big data analysis as described in any one of claims 1-9, the system comprising: The data acquisition module is used to acquire the raw operating data of the power system and to preprocess the raw operating data; The semantic uncertainty assessment module is used to build a power data semantic uncertainty assessment model based on historical network operation data and calculate the semantic uncertainty index of real-time / near real-time power data. The adaptive tiered storage module is used to dynamically allocate the distribution ratio and redundancy level of data in different storage tiers based on the semantic uncertainty index, thereby completing adaptive tiered storage. The multi-timescale consistency analysis module is used to build a consistency model across multiple timescales and perform consistency constraint analysis on the data evolution trend of the same physical quantity at different timescales. The redundancy removal storage module is used to determine whether short-period data changes are predictable disturbances within the long-term trend. It only stores trend residual parameters and reconstructs the complete original data through a consistency model when needed, thus achieving redundancy removal storage.

Images