A smart grid monitoring data storage method and system

Abstract

The present application relates to a kind of smart grid monitoring data storage method and system, specifically relates to the field of smart grid, by lightweight feature extraction in edge side, generate the metadata rich in semantic information, significantly reduce data transmission overhead and central storage pressure;Center system is based on reinforcement learning dynamic selection encoding strategy, and according to the intelligent scheduling of algorithm characteristics to heterogeneous computing node execution, realizes the optimal balance of storage efficiency and processing speed;At the same time, through online learning mechanism continuous optimization decision model, form a self-adapting closed loop, comprehensively improve the real-time performance, reliability and resource utilization efficiency of data storage under the high concurrency monitoring scenario of smart grid.

Description

Technical Field

[0001] This invention relates to the field of smart grids, and more specifically, to a smart grid monitoring and data storage method and system. Background Technology

[0002] With the deepening of smart grid construction, the power system monitoring system has incorporated a massive number of sensing devices, including smart meters, voltage and current sensors, and fault recording devices. These devices continuously generate massive amounts of time-series monitoring data during grid operation. Especially during peak electricity consumption periods, such as the concentrated start-up of air conditioning loads during extreme high temperatures in summer or the surge in heating electricity consumption during cold waves in winter, tens of millions of terminal devices simultaneously upload high-frequency data. The data reporting frequency can reach tens of thousands of records per second, and each record contains multi-dimensional parameters such as timestamp, device identifier, voltage, current, phase, and power factor. In addition, sudden fluctuations in grid load, intermittent changes in renewable energy power generation, and extreme natural disasters further exacerbate the suddenness and uncertainty of data flow. In such scenarios, the monitoring system needs to have high-throughput, low-latency data access and storage capabilities; otherwise, it will directly affect the real-time nature of operational status perception, the accuracy of control commands, and the timeliness of fault handling.

[0003] Currently, smart grid monitoring systems generally employ time-series databases or big data storage architectures to manage this type of data, such as InfluxDB or the Hadoop-based Parquet columnar storage format. These solutions typically rely on traditional data encoding methods, such as the LZ77 series general-purpose compression algorithm or Delta encoding technology, in order to reduce storage overhead. However, these methods have significant limitations: First, general-purpose encoding algorithms do not fully consider the inherent characteristics of power time-series data, such as strong periodicity, interval sparsity, waveform similarity, and frequent abrupt changes, resulting in suboptimal compression efficiency and encoding speed; Second, Most encoding processes require multiple scans and complex calculations of data. In high-concurrency write scenarios, CPU computing resources quickly become a bottleneck, significantly increasing encoding latency and thus dragging down overall storage throughput. Thirdly, existing systems mostly adopt static encoding strategies and fixed resource allocation mechanisms, lacking the ability to perceive and adapt to dynamic changes in data flow characteristics. They cannot flexibly adjust encoding computing resources or switch encoding modes according to real-time load conditions. As a result, data backlog or even write failures occur, real-time monitoring screens are delayed or lost, and in severe cases, the dispatch center cannot detect power grid faults in a timely manner, thereby delaying fault isolation and recovery operations and increasing the risk of large-scale power outages. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a smart grid monitoring data storage method and system to solve the problems mentioned in the background.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a smart grid monitoring data storage method, specifically including the following steps: Step S1: At the edge data acquisition end, real-time feature extraction is performed on the raw monitoring data stream acquired according to a fixed sampling period. The approximate sparse entropy is obtained by calculating the prediction error of adjacent sampling points and counting the number of outliers based on a dynamic threshold. At the same time, the weighted mutation index is obtained by calculating the forward difference sequence and applying exponential weighted summation. A metadata vector containing the approximate sparse entropy and the weighted mutation index is generated. The metadata vector, together with the start timestamp of the time window, the device identifier, and the corresponding raw monitoring data stream, is encapsulated into a data packet and uploaded to the central storage system. Step S2: In the central storage system, data packets from the edge data acquisition terminal are received, metadata vectors are parsed out, and the metadata vectors, together with the current CPU utilization, memory usage, and I / O throughput of the central storage system, are input into the reinforcement learning-based dynamic coding strategy selector, which outputs a specified coding instruction. Step S3: Based on the type of encoding algorithm indicated by the encoding instruction and the ratio of computationally intensive operations to logic control operations it contains, schedule the corresponding encoding task to a distributed encoding cluster composed of heterogeneous computing nodes, have the designated computing nodes perform the encoding operation on the original monitoring data stream, and store the encoded data in the central storage system. Step S4: Collect the actual performance indicators of the data block that has been encoded and stored, including compression ratio, encoding time and decoding time. Combine the actual performance indicators with the strategy decision output by the dynamic encoding strategy selector when the data block was generated and the system state data in step S2 to form a training sample. Input the training sample into the strategy value function model in the dynamic encoding strategy selector for online fine-tuning and continuous optimization of the model. In a preferred embodiment, the specific operation of calculating the approximate sparsity entropy during real-time feature extraction at the edge data acquisition end is as follows: The raw monitoring data stream is acquired at a fixed sampling period. For adjacent sampling points in the data stream, a first-order linear prediction model is used to obtain predicted values. The absolute error between the predicted values ​​and the actual sampled values ​​is calculated to form a prediction error sequence. The median absolute deviation of the prediction error sequence is calculated, and a dynamic threshold is calculated based on the median absolute deviation and a preset sensitivity adjustment factor. The number of abnormal error points in the prediction error sequence that exceed the dynamic threshold is counted. Finally, based on the ratio of the number of abnormal error points to the total length of the sequence, an approximate sparsity entropy is obtained by combining logarithmic operations. If the number of abnormal error points is zero, the approximate sparsity entropy is defined as zero.

[0006] In a preferred embodiment, the specific operation of calculating the weighted mutation index is as follows: First, calculate the first-order forward difference sequence of the original monitoring data stream. The first-order forward difference is the difference obtained by subtracting the value of the previous adjacent sampling point from the value of the next sampling point in the sequence. All differences are arranged in order to form a difference sequence. Then, an exponentially decaying weight coefficient sequence is constructed to assign weights to each difference in the difference sequence. The weight coefficient is a function of the decay rate parameter and the difference sequence index. Specifically, the weight coefficient value decreases according to the natural exponential function as its corresponding difference sequence index value moves towards the starting point of the difference sequence, so that the difference value that is closer to the end of the difference sequence, i.e., closer to the current time, is given a higher weight. Finally, the weighted mutation index is calculated, which is equal to the sum of the products of the absolute value of each difference in the difference sequence and its corresponding weight coefficient, divided by the sum of all weight coefficients.

[0007] In a preferred embodiment, the specific process of inputting the metadata vector along with the current CPU utilization, memory usage, and I / O throughput of the central system in step S2 is as follows: First, the two feature values, approximate sparsity entropy and weighted mutation index, contained in the metadata vector are normalized to compress their numerical range to between zero and one. At the same time, the three system indicators, CPU utilization, memory usage and I / O throughput, obtained from the monitoring of the central storage system, are also normalized to compress their numerical range to between zero and one. Then, the two normalized feature values ​​and the three system indicators are concatenated in sequence to form a joint state awareness vector.

[0008] In a preferred embodiment, the decision-making process of the reinforcement learning-based dynamic coding policy selector is specifically as follows: First, a policy set containing multiple coding algorithms is pre-defined. Then, a dual-network evaluation architecture is used to process the joint state-aware vector. This architecture includes a state value function network and an advantage function network. The state value function network is used to evaluate the baseline value of the current system and data joint state, and the advantage function network is used to evaluate the unique advantage of each coding algorithm relative to the average level. The outputs of the state value function network and the advantage function network are added together to obtain the final value score of each coding strategy. Finally, the coding strategy with the highest value score is selected as the output coding instruction, which will be sent to the distributed coding cluster to execute the corresponding coding operation.

[0009] In a preferred embodiment, the specific process of scheduling the corresponding encoding task to a distributed encoding cluster composed of heterogeneous computing nodes in step S3 is as follows: First, the encoding algorithm type indicated by the encoding instruction is parsed, and the proportion of computationally intensive operations and the proportion of logic-controlled operations for that algorithm type are obtained by querying a pre-set algorithm characteristic mapping table. Then, the real-time performance scores of each heterogeneous computing node in the distributed encoding cluster are obtained, including the performance scores of computationally intensive operations and logic-controlled operations. Next, the fitness score of each computing node is calculated based on the operation proportions and performance scores. The sum of the product of the computationally intensive operation proportion and the performance score of the computationally intensive operation, and the sum of the product of the proportion of logic-controlled operations and the performance score of the logic-controlled operation constitute the numerator of the fitness score, and the sum of the modulus of the operation proportion vector and the modulus of the performance score vector constitutes the denominator of the fitness score. Finally, the computing node with the highest fitness score is selected as the task execution node, and the encoding task is scheduled to this node.

[0010] In a preferred embodiment, the specific process of the designated computing node performing the encoding operation on the original monitoring data stream is as follows: After receiving the encoding instructions and the raw monitoring data stream, the target computing node first loads the corresponding encoding algorithm library and initializes the encoding context environment. Then, following the execution flow of the encoding algorithm, it first processes the logic control operations, including parsing the data stream structure, generating control instruction sequences, and allocating computing resources. Next, it processes the computationally intensive operations, including performing data transformations, running compression algorithms, and generating encoded data blocks. During the encoding process, a dual-buffer mechanism is used to alternately read data and encode data to ensure processing continuity. After encoding, a checksum is added to the data block and data index information is generated. Finally, the encoded data block is transmitted to the central storage system through a high-speed data channel and stored in partitions according to time series and data source identifiers.

[0011] In a preferred embodiment, the specific process of constructing training samples by combining actual performance indicators with strategy decision and system state data in step S4 is as follows: First, the actual performance metrics of the encoded and stored data blocks are collected, specifically including compression ratio, encoding time, and decoding time. Then, the policy decision output by the dynamic encoding policy selector corresponding to the generation of the data block and the system status data in step S2 are obtained. Subsequently, the three performance metrics of compression ratio, encoding time, and decoding time are normalized respectively, compressing their numerical range to between zero and one. Finally, the normalized performance metrics, policy decisions, and system status data are combined into a training sample according to a specific format.

[0012] In a preferred embodiment, the specific process of online fine-tuning and continuous optimization of the strategy value function model is as follows: First, the multi-objective reward value is calculated using the constructed training samples. This reward value is calculated by weighting the normalized compression ratio with positive weights and the normalized encoding and decoding times with negative weights, and by adding an additional reward term proportional to the policy entropy to encourage policy exploration. Then, an online policy gradient algorithm is adopted, with the multi-objective reward value as the optimization objective. The product of the log probability of the selected policy decision and the advantage function is calculated as the gradient direction, where the advantage function is the difference between the multi-objective reward value and the state value estimate output by the baseline function. This baseline function is used to reduce the variance of the learning process. Next, the parameters of the policy value function model are incrementally updated along this gradient direction; Meanwhile, the parameters of the baseline function itself are updated by minimizing the error between the predicted value of the baseline function and the actual multi-objective reward value; Finally, the updated strategy value function model parameters are hot-deployed into the dynamic coding strategy selector, enabling it to make better coding strategy decisions based on historical experience.

[0013] This application also provides a smart grid monitoring and data storage system, specifically including: an edge-side feature extraction module, a central-side strategy decision-making module, a heterogeneous task scheduling and execution module, and an online optimization feedback module, wherein... Edge-side feature extraction module: Configured at the data acquisition end, it is used to extract features from the raw monitoring data stream acquired at a fixed sampling period in real time. It generates a metadata vector by calculating the approximate sparsity entropy and the weighted mutation index, and then encapsulates the metadata vector and the corresponding raw data fragments into a data packet and uploads it. Central-side policy decision module: configured in the central storage system, used to receive and parse data packets from the edge side to obtain metadata vectors, and input the metadata vectors and the real-time performance indicators of the central system into a dynamic encoding policy selector, which outputs the specified encoding instructions; Heterogeneous task scheduling and execution module: configured in the central storage system, used to schedule the encoding task to a specific node in the distributed encoding cluster composed of heterogeneous computing nodes according to the characteristics of the encoding algorithm indicated by the encoding instruction, so that the node can perform the encoding operation and store the result in the storage system; Online optimization feedback module: Configured in the central storage system, it is used to collect the actual performance indicators of the stored data blocks, and to combine the performance indicators with the corresponding historical strategy decisions and system status data to form training samples, which are then input into the strategy value function model in the dynamic encoding strategy selector for online fine-tuning and continuous optimization of the model.

[0014] The beneficial effects of this invention are as follows: by performing lightweight feature extraction at the edge, metadata rich in semantic information is generated, which significantly reduces data transmission overhead and central storage pressure; the central system dynamically selects encoding strategies based on reinforcement learning and intelligently schedules execution to heterogeneous computing nodes according to algorithm characteristics, achieving an optimal balance between storage efficiency and processing speed; at the same time, the decision model is continuously optimized through an online learning mechanism to form an adaptive closed loop, comprehensively improving the real-time performance, reliability, and resource utilization efficiency of data storage in high-concurrency monitoring scenarios of smart grids. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0018] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0019] Example 1 This embodiment provides, for example Figure 1 The method for storing monitoring data in a smart grid, as shown, specifically includes the following steps: Step S1: At the edge data acquisition end, real-time feature extraction is performed on the raw monitoring data stream acquired at a fixed sampling period. By calculating the prediction error of adjacent sampling points and counting the number of outliers based on a dynamic threshold, an approximate sparse entropy is obtained. Simultaneously, by calculating the forward difference sequence and applying exponential weighted summation, a weighted mutation index is obtained, generating a metadata vector containing the approximate sparse entropy and the weighted mutation index. And the metadata vector is matched with the start timestamp of the time window. Device identifier and the corresponding raw monitoring data stream They are encapsulated together into a data packet. Afterwards, the data is uploaded to the central storage system. In the high-concurrency monitoring scenario of smart grids, the raw monitoring data streams generated by edge devices are characterized by high dimensionality, heterogeneity, and burstiness. Traditional methods directly upload the raw data to the central storage system, leading to transmission and storage bottlenecks. Therefore, this step aims to perform lightweight, high-value feature extraction on the raw data stream at the edge, generating metadata rich in semantic information, providing accurate and low-overhead input for the subsequent intelligent coding decisions of the central storage system. Step S2: In the central storage system, data packets from edge data acquisition terminals are received, the metadata vectors are parsed out, and the metadata vectors, along with the current CPU utilization, memory usage, and I / O throughput of the central storage system, are input into a reinforcement learning-based dynamic coding strategy selector. The selector outputs a specified coding instruction. In the smart grid monitoring data stream processing, the data packets uploaded by edge devices already contain metadata representing the intrinsic characteristics of the data. To achieve efficient storage, the central storage system needs to dynamically select the optimal coding strategy based on the real-time changes in system resource status and the characteristics of the data itself. However, traditional static or rule-based strategy selection methods are difficult to cope with the dynamic coupling relationship between system load and data characteristics. Therefore, this step aims to design an intelligent decision-making mechanism that integrates internal and external states to generate accurate coding instructions. Step S3: Based on the type of encoding algorithm indicated by the encoding instruction and the ratio of computationally intensive operations to logic control operations it contains, the corresponding encoding task is scheduled to a distributed encoding cluster composed of heterogeneous computing nodes. The designated computing nodes perform the encoding operation on the original monitoring data stream and store the encoded data in the central storage system. In order to maximize the performance of the heterogeneous computing cluster, this step designs a fine-grained task scheduling mechanism that can accurately deliver the algorithm to the most suitable computing node for execution based on the algorithm's inherent computational characteristics, thereby optimizing the processing efficiency. Step S4: Collect the actual performance indicators of the encoded and stored data blocks, including compression ratio, encoding time, and decoding time. Combine the actual performance indicators with the policy decision output by the dynamic encoding policy selector corresponding to the generation of the data block and the system state data in step S2 to form a training sample. Input the training sample into the policy value function model in the dynamic encoding policy selector for online fine-tuning and continuous optimization of the model. Since the traditional offline model update method cannot adapt to the dynamic changes in the smart grid monitoring environment, this step aims to design a closed-loop feedback mechanism. By fusing multi-dimensional performance indicators to calculate reward signals, and using online learning to continuously optimize the decision model, it can learn from the effects of historical decisions, continuously improve the quality of future decisions, and form a self-evolving intelligent system.

[0020] In this embodiment, it is specifically necessary to explain the specific operation of calculating the approximate sparsity entropy during real-time feature extraction at the edge data acquisition end: Obtain raw monitoring data streams at fixed sampling intervals. (in Indicates the first The values ​​of each sampling point (such as voltage and current values). Indicates the current time window identifier. (representing the total number of sampling points within the window), for adjacent sampling points in this data stream, a first-order linear prediction model is used to obtain the predicted value, and the absolute error between the predicted value and the actual sampling value is calculated using the following formula: ; in, Indicates the first The prediction error for each sampling point is the absolute value of the difference between the current sample value and the previous sample value. Indicates the first [item] in the original monitoring data stream The actual value of each sampling point Indicates the first The predicted value of each sampling point is defined as the actual value of the previous sampling point. ), forming a prediction error sequence The median absolute deviation of the prediction error sequence is calculated using the following formula: ; in, It represents the absolute deviation of the median, and is a robust statistic used to measure the volatility of data. This represents the operation of taking the median, for example, Represents the computational error sequence The median is used as the basis for calculating the dynamic threshold based on the absolute deviation of the median and a preset sensitivity adjustment factor, using the following formula: ; in, This represents a dynamic threshold, a critical value used to determine whether an error point is "abnormal." This value is dynamically calculated based on the data characteristics. This represents the sensitivity adjustment factor, a constant parameter (usually 3), used to control the leniency of the threshold relative to the data distribution. A larger value results in fewer points being judged as outliers. It is also used to count the number of outlier error points in the prediction error sequence that exceed the dynamic threshold, denoted as [missing value]. Finally, based on the ratio of the number of outlier points to the total sequence length, and combined with logarithmic operations, the approximate sparsity entropy is obtained. If the number of outlier points is zero, then the approximate sparsity entropy is defined as zero, as shown in the formula: ; in, Indicates within the time window The approximate sparsity entropy calculated internally is used to measure the anomalies and information content of the data stream, with a value range of [0, +∞). This represents the logarithmic operation with base 2. This formula represents the number of abnormal error points and its function is: The higher the value, the more abnormal or mutation information is contained in the data stream, the greater the information entropy, the lower the compression potential but the higher the data value. This step provides an important feature indicator for the subsequent generation of metadata vectors, and this approximate sparsity entropy will be used for subsequent data storage strategy selection and other operations to ensure the accuracy and effectiveness of data feature extraction. The specific steps for calculating the weighted mutation index are as follows: First, calculate the first-order forward difference sequence of the original monitoring data stream. The first-order forward difference is the difference between the value of the next sampling point and the value of the previous adjacent sampling point in the sequence. The formula for calculating the first-order forward difference is: ; in, This represents the first-order forward difference (i.e., the difference), which is the first-order forward difference in the original monitoring data stream. The data point and the first The difference between the values ​​of individual data points reflects the speed and direction of data change over a short period of time. Indicates the first [item] in the original monitoring data stream The actual value at each sampling point is the most basic input data, such as instantaneous measurements of voltage or current. This represents the index number, used to traverse the data sequence. Its value range is typically from 1 to N (for data points). ) or from 1 to N−1 (for the difference) All differences are arranged in order to form a difference sequence. ; Then, construct an exponentially decaying sequence of weights. This is used to assign weights to each difference in the difference sequence. The weight coefficient is a function of the decay rate parameter and the difference sequence index. Specifically, the weight coefficient value decreases according to the natural exponential function as its corresponding difference sequence index value moves towards the starting point of the difference sequence. This results in the difference value that is closer to the end of the difference sequence, i.e., closer to the current time, being given a higher weight. Finally, the weighted mutation index is calculated, which is equal to the sum of the products of the absolute value of each difference in the difference sequence and its corresponding weight coefficient, divided by the sum of all weight coefficients. The formula for calculating the weighted mutation index is: ; in, Indicates the weighted mutation index (value range ≥ 0), superscript This indicates that the index belongs to the first... The data is divided into time windows to distinguish calculation results from different time periods in continuous data processing. The subscript 'm' indicates that it is used to identify the index type. This represents the weight coefficient, which is the weighting coefficient for the th element in the difference sequence. Individual differences The assigned weights are determined by the formula. Calculations show that differences closer to the current time step (the end of the sequence) are assigned higher weights. This represents the attenuation rate parameter, which is a constant greater than zero. The rate at which the weights decay is preset by the system. The larger the value, the faster the weighting coefficient decays with changes in the index. e represents the natural constant, a fundamental constant in mathematics, approximately equal to 2.71828. It forms the basis of the natural exponential function. Represents the first-order forward difference The absolute value of represents the magnitude of the change, regardless of the direction of the change (positive or negative). The summation symbol is used to represent the summation of terms from... The formula sums all the terms and its function is: The larger the value, the greater the degree of drastic change in the data recently, indicating that the data segment may contain important event information (such as fault disturbances), and its encoding quality should be prioritized. This weighted mutation index reflects the mutation situation of the data and, together with the approximate sparse entropy, constitutes a key component of the metadata vector, playing an important role in the subsequent data upload and processing.

[0021] In this embodiment, it is particularly important to explain step S2, which involves transferring the metadata vector. The current CPU utilization, memory usage, and I / O throughput of the central system. (in Indicates CPU utilization. Indicates memory usage. The specific process of the common input (representing I / O throughput) is as follows: First, the two feature values ​​in the metadata vector—approximate sparse entropy and weighted mutation index—are normalized to compress their numerical range to between zero and one. Simultaneously, the three system metrics obtained from monitoring the central storage system—CPU utilization, memory usage, and I / O throughput—are also normalized to compress their numerical range to between zero and one. Then, the two normalized feature values ​​and the three system metrics are sequentially concatenated into a joint state-aware vector. The expression for the joint state-aware vector is: ; in, This represents min-max normalization. The function of this formula is to integrate features with different dimensions and physical meanings into a unified, dimensionless state space, providing standardized input for subsequent decision-making models. The decision-making process of a reinforcement learning-based dynamic coding policy selector is as follows: First, a strategy set containing multiple encoding algorithms is pre-defined. (Including K available encoding algorithms, such as run-length encoding (RLE), dictionary encoding, and differential encoding), then a dual-network evaluation architecture is used to process the joint state-aware vector. This architecture includes a state value function network and an advantage function network. The state value function network is used to evaluate the baseline value of the current system and data joint state, and the advantage function network is used to evaluate the unique advantage of each encoding algorithm relative to the average level. The outputs of the state value function network and the advantage function network are added together to obtain the final value score of each encoding strategy, expressed as: ; in, This represents a strategy set containing multiple encoding algorithms. Indicates the first Encoding strategies, representing a set of strategies A specific encoding algorithm (such as run-length encoding (RLE), dictionary encoding, differential encoding). The joint state-aware vector is a comprehensive vector composed of metadata from step S1 (such as approximate sparse entropy and weighted mutation index) and the real-time state of the central system (such as CPU utilization, memory usage, and I / O throughput) after normalization and other processing. It comprehensively represents the current system and data state. This represents the policy value function, which is a function used to evaluate the policy value in the current state. Next, select a specific strategy. The expected long-term returns (value score) that can be generated. Represents the parameters of the neural network, and represents the value function. The set of weight parameters contained therein that needs to be learned through training. This represents the state value function, which is the output of one branch network in the "dual-network evaluation architecture" and evaluates the current state. The inherent benchmark value, independent of the chosen strategy, is an estimate of the average return that all strategies could obtain in the current state. This represents the weight parameters of the state-value function network, i.e., the set of parameters that the neural network needs to be trained on. The advantage function, representing the output of another branch network in the "dual-network evaluation architecture," evaluates the performance of a given state. Next, select a specific strategy. Compared to the additional advantages or disadvantages of choosing an averaging strategy. The weight parameters represent the set of parameters that the advantage function network needs to train. Finally, the encoding strategy with the highest value score is selected as the output encoding instruction, which is then sent to a distributed encoding cluster to perform the corresponding encoding operation. The expression is: ; in, The encoded instructions representing the final output, the output result of the decision-making process, i.e., based on value scores. From the strategy set The optimal coding strategy is selected from the options, where t represents the time or sequence in which the decision occurs.

[0022] In this embodiment, the specific process of scheduling the corresponding encoding task to the distributed encoding cluster composed of heterogeneous computing nodes in step S3 is as follows: First, parse the encoded instructions. Given the specified encoding algorithm type, query the preset algorithm characteristic mapping table to obtain the proportion of computationally intensive operations for that algorithm type. and logic control type operation ratio ,satisfy The expression is: ; in, Representation Algorithm The computational characteristic vector is a two-dimensional vector composed of the proportion of computationally intensive operations and the proportion of logic-controlled operations of the algorithm. Its function is to transform the abstract algorithm identifier into a quantifiable computational resource requirement indicator. Then, it obtains the real-time performance scores of each heterogeneous computing node in the distributed coding cluster, including the performance scores of computationally intensive operations and logic-controlled operations. Next, it calculates the fitness score for each computing node based on the operation proportion and performance score. The sum of the product of the computationally intensive operation proportion and the computationally intensive operation performance score and the product of the logic-controlled operation proportion and the logic-controlled operation performance score constitutes the numerator of the fitness score, and the sum of the modulus of the operation proportion vector and the modulus of the performance score vector constitutes the denominator of the fitness score. The formula for calculating the fitness score is: ; in, This represents the j-th computing node in the heterogeneous computing node set, where j represents the node's index number. Represents a node The performance score for computationally intensive operations is the performance of a node (such as a GPU / FPGA) in performing computationally intensive tasks such as arithmetic computation and data parallelism. A higher score indicates stronger performance. Represents a node The performance score for logic control operations is the performance of a node (such as a high-performance CPU) in executing logic-intensive tasks such as instruction control, branch prediction, and task scheduling. A higher score indicates stronger capabilities. Representation Algorithm The calculation of characteristic vectors, Representation Algorithm The proportion of computationally intensive operations, i.e., the percentage of computationally intensive operations during the execution of the algorithm. Representation Algorithm The proportion of logic control operations, that is, the percentage of logic control operations during the execution of the algorithm. This represents the magnification factor, which consists of two constants greater than 1 (usually taken as...). This is used to amplify the performance advantages of a node in a certain aspect (computation or logic) when calculating fitness, making scheduling decisions more inclined to "play to the strengths and avoid the weaknesses". This represents a very small positive number and is added to the denominator to prevent the denominator from being zero, thus ensuring the numerical stability of the formula. The norm of a vector (usually referring to the Euclidean norm, i.e., the magnitude of the vector) is used to measure the size of the vector. This represents the dynamic multidimensional adaptation function, which outputs an fitness score used in quantization encoding algorithms. With computing nodes The degree of matching between the nodes is considered; a higher score indicates a better match. Finally, the computation node with the highest fit score is selected as the task execution node, and the coding task is scheduled to that node. The expression is: ; in, The target computation node is defined as the node whose fitness function is maximized. The most suitable one to perform the current coding task was selected. The node, Represents a heterogeneous set of computing nodes, containing all schedulable computing nodes. ; The specific process by which a designated computing node performs the encoding operation on the raw monitoring data stream is as follows: Target computing node Upon receiving the encoding instructions and the raw monitoring data stream, the corresponding encoding algorithm library is first loaded, and the encoding context environment is initialized. Then, following the execution flow of the encoding algorithm, the logic control operations are processed first, including parsing the data stream structure, generating control instruction sequences, and allocating computing resources. Next, the computationally intensive operations are processed, including performing data transformations, running compression algorithms, and generating encoded data blocks. During the encoding process, a double-buffer mechanism is used to alternately read and encode data to ensure processing continuity. After encoding, a checksum is added to the data block and data index information is generated. Finally, the encoded data block is transmitted through a high-speed data channel. The data is transmitted to a central storage system and stored in partitions according to time series and data source identifiers.

[0023] In this embodiment, it is specifically necessary to explain the process in step S4 of constructing training samples by combining actual performance indicators with strategy decision and system state data: First, the actual performance metrics of the encoded and stored data blocks are collected, specifically including compression ratio, encoding time, and decoding time. Then, the strategy decision output by the dynamic encoding strategy selector corresponding to the generation of the data block and the system state data in step S2 are obtained. Subsequently, the three performance metrics of compression ratio, encoding time, and decoding time are normalized respectively, compressing their numerical range to between zero and one. Finally, the normalized performance metrics, strategy decisions, and system state data are combined into a training sample in a specific format. This training sample completely records the actual performance effect obtained after adopting a specific encoding strategy under a certain system state, providing a data foundation for subsequent model optimization. The specific process of online fine-tuning and continuous optimization of the strategy value function model is as follows: First, a multi-objective reward value is calculated using the constructed training samples. This reward value is obtained by weighting and summing the normalized compression ratio with positive weights and the normalized encoding and decoding times with negative weights, and by adding an additional reward term proportional to the policy entropy to encourage policy exploration. This comprehensively reflects the balance between compression efficiency and processing speed. The formula for calculating the multi-objective reward value is as follows: ; in, This represents a multi-objective reward value, a reward signal that comprehensively evaluates compression efficiency and processing speed. This represents the weighting coefficient, used to adjust the importance of compression ratio, encoding time, and decoding time in the reward value (positive weight represents reward, negative weight represents penalty). This represents the entropy reward coefficient, used to control the influence of policy entropy on rewards, balancing exploration and exploitation. The policy entropy function measures the randomness of the policy probability distribution and encourages exploration. The policy function represents the state. The probability distribution of the next action selection. This represents the compression ratio, a key performance indicator that reflects the efficiency of data compression. Indicates encoding time, one of the actual performance metrics, representing the time spent on encoding operations. This indicates decoding time, a key performance indicator, representing the time spent on the decoding operation. These represent the minimum and maximum compression ratios, used for normalization (usually based on historical data or theoretical values). This represents the minimum and maximum encoding time, used for normalization. These represent the minimum and maximum decoding times, used for normalization. Then, an online policy gradient algorithm is adopted, with the multi-objective reward value as the optimization objective. The product of the log probability of the selected policy decision and the advantage function is calculated as the gradient direction, where the advantage function is the difference between the multi-objective reward value and the state value estimate output by the baseline function. This baseline function is used to reduce the variance of the learning process. Next, the parameters of the policy value function model are incrementally updated along this gradient direction; Meanwhile, the parameters of the baseline function itself are updated by minimizing the error between the predicted value of the baseline function and the actual multi-objective reward value; Finally, the updated strategy value function model parameters are hot-deployed into the dynamic coding strategy selector, enabling it to make better coding strategy decisions based on historical experience.

[0024] Example 2 This embodiment provides, for example Figure 2 The smart grid monitoring and data storage system shown includes: an edge-side feature extraction module, a central-side strategy decision-making module, a heterogeneous task scheduling and execution module, and an online optimization feedback module. Edge-side feature extraction module: Configured at the data acquisition end, it is used to extract features from the raw monitoring data stream acquired at a fixed sampling period in real time. It generates metadata vector by calculating approximate sparsity entropy and weighted mutation index, and then encapsulates the metadata vector and the corresponding raw data fragments into a data packet before uploading. Central-side policy decision module: Configured in the central storage system, it is used to receive and parse data packets from the edge side to obtain metadata vectors, and input the metadata vectors and the real-time performance indicators of the central system into a dynamic encoding policy selector, which outputs the specified encoding instructions. Heterogeneous task scheduling and execution module: Configured in the central storage system, it is used to schedule encoding tasks to specific nodes in a distributed encoding cluster composed of heterogeneous computing nodes according to the characteristics of the encoding algorithm indicated by the encoding instructions. The node then performs the encoding operation and stores the results in the storage system. Online optimization feedback module: Configured in the central storage system, it is used to collect the actual performance indicators of the stored data blocks, and to combine the performance indicators with the corresponding historical strategy decisions and system status data to form training samples, which are then input into the strategy value function model in the dynamic encoding strategy selector for online fine-tuning and continuous optimization of the model.

[0025] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0026] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0027] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0028] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0029] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0030] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0031] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for storing monitoring data in a smart grid, characterized in that, Specifically, the following steps are included: Step S1: At the edge data acquisition end, real-time feature extraction is performed on the raw monitoring data stream acquired according to a fixed sampling period. The approximate sparse entropy is obtained by calculating the prediction error of adjacent sampling points and counting the number of outliers based on a dynamic threshold. At the same time, the weighted mutation index is obtained by calculating the forward difference sequence and applying exponential weighted summation. A metadata vector containing the approximate sparse entropy and the weighted mutation index is generated. The metadata vector, together with the start timestamp of the time window, the device identifier, and the corresponding raw monitoring data stream, is encapsulated into a data packet and uploaded to the central storage system. Step S2: In the central storage system, data packets from the edge data acquisition terminal are received, metadata vectors are parsed out, and the metadata vectors, together with the current CPU utilization, memory usage, and I / O throughput of the central storage system, are input into the reinforcement learning-based dynamic coding strategy selector, which outputs a specified coding instruction. Step S3: Based on the type of encoding algorithm indicated by the encoding instruction and the ratio of computationally intensive operations to logic control operations it contains, schedule the corresponding encoding task to a distributed encoding cluster composed of heterogeneous computing nodes, have the designated computing nodes perform the encoding operation on the original monitoring data stream, and store the encoded data in the central storage system. Step S4: Collect the actual performance indicators of the data blocks that have been encoded and stored, including compression ratio, encoding time and decoding time. Combine the actual performance indicators with the policy decision output by the dynamic encoding policy selector when the data block was generated and the system state data in step S2 to form a training sample. Input the training sample into the policy value function model in the dynamic encoding policy selector for online fine-tuning and continuous optimization of the model.

2. The smart grid monitoring data storage method according to claim 1, characterized in that: During real-time feature extraction at the edge data acquisition terminal, the specific operation for calculating the approximate sparsity entropy is as follows: The original monitoring data stream is acquired at a fixed sampling period. For adjacent sampling points in the data stream, a first-order linear prediction model is used to obtain the predicted value. The absolute error between the predicted value and the actual sampled value is calculated to form a prediction error sequence. The median absolute deviation of the prediction error sequence is calculated, and the dynamic threshold is calculated based on the median absolute deviation and the preset sensitivity adjustment factor. The number of abnormal error points exceeding the dynamic threshold in the statistical prediction error sequence; Finally, based on the ratio of the number of outlier error points to the total length of the sequence, the approximate sparsity entropy is obtained by combining logarithmic operations. If the number of outlier error points is zero, the approximate sparsity entropy is defined as zero.

3. The smart grid monitoring data storage method according to claim 2, characterized in that: The specific steps for calculating the weighted mutation index are as follows: First, calculate the first-order forward difference sequence of the original monitoring data stream. The first-order forward difference is the difference obtained by subtracting the value of the previous adjacent sampling point from the value of the next sampling point in the sequence. All differences are arranged in order to form a difference sequence. Then, an exponentially decaying weight coefficient sequence is constructed to assign weights to each difference in the difference sequence. The weight coefficient is a function of the decay rate parameter and the difference sequence index. Specifically, the weight coefficient value decreases according to the natural exponential function as its corresponding difference sequence index value moves towards the starting point of the difference sequence, so that the difference value that is closer to the end of the difference sequence, i.e., closer to the current time, is given a higher weight. Finally, the weighted mutation index is calculated, which is equal to the sum of the products of the absolute value of each difference in the difference sequence and its corresponding weight coefficient, divided by the sum of all weight coefficients.

4. The smart grid monitoring data storage method according to claim 3, characterized in that: In step S2, the specific process of inputting the metadata vector along with the current CPU utilization, memory usage, and I / O throughput of the central system is as follows: First, the two feature values, approximate sparsity entropy and weighted mutation index, contained in the metadata vector are normalized to compress their numerical range to between zero and one. At the same time, the three system indicators, CPU utilization, memory usage and I / O throughput, obtained from the monitoring of the central storage system, are also normalized to compress their numerical range to between zero and one. Then, the two normalized feature values ​​and the three system indicators are concatenated in sequence to form a joint state awareness vector.

5. A smart grid monitoring data storage method according to claim 4, characterized in that: The decision-making process of the reinforcement learning-based dynamic coding policy selector is as follows: First, a policy set containing multiple coding algorithms is pre-defined. Then, a dual-network evaluation architecture is used to process the joint state-aware vector. This architecture includes a state value function network and an advantage function network. The state value function network is used to evaluate the baseline value of the current system and data joint state, and the advantage function network is used to evaluate the unique advantage of each coding algorithm relative to the average level. The outputs of the state value function network and the advantage function network are added together to obtain the final value score of each coding strategy. Finally, the coding strategy with the highest value score is selected as the output coding instruction, which will be sent to the distributed coding cluster to execute the corresponding coding operation.

6. The smart grid monitoring data storage method according to claim 5, characterized in that: In step S3, the specific process of scheduling the corresponding encoding task to a distributed encoding cluster composed of heterogeneous computing nodes is as follows: First, the encoding algorithm type indicated by the encoding instruction is parsed, and the proportion of computationally intensive operations and the proportion of logic-controlled operations for that algorithm type are obtained by querying a pre-set algorithm characteristic mapping table. Then, the real-time performance scores of each heterogeneous computing node in the distributed encoding cluster are obtained, including the performance scores of computationally intensive operations and logic-controlled operations. Next, the fitness score of each computing node is calculated based on the operation proportions and performance scores. The sum of the product of the computationally intensive operation proportion and the performance score of the computationally intensive operation, and the sum of the product of the proportion of logic-controlled operations and the performance score of the logic-controlled operation constitute the numerator of the fitness score, and the sum of the modulus of the operation proportion vector and the modulus of the performance score vector constitutes the denominator of the fitness score. Finally, the computing node with the highest fitness score is selected as the task execution node, and the encoding task is scheduled to this node.

7. A smart grid monitoring data storage method according to claim 6, characterized in that: The specific process by which the designated computing node performs the encoding operation on the original monitoring data stream is as follows: After receiving the encoding instructions and the raw monitoring data stream, the target computing node first loads the corresponding encoding algorithm library and initializes the encoding context environment. Then, following the execution flow of the encoding algorithm, it first processes the logic control operation part, including parsing the data stream structure, generating control instruction sequences, and allocating computing resources. Then, it processes the computationally intensive operation part, including performing data transformation, running compression algorithms, and generating encoded data blocks. A double-buffer mechanism is used during the encoding process to alternately read data and encode it in order to achieve continuous processing. After encoding, a checksum is added to the data block and data index information is generated; finally, the encoded data block is transmitted to the central storage system through a high-speed data channel and stored in partitions according to time series and data source identifier.

8. A smart grid monitoring data storage method according to claim 7, characterized in that: In step S4, the specific process of constructing training samples by combining actual performance indicators with strategy decision and system state data is as follows: First, the actual performance metrics of the encoded and stored data blocks are collected, specifically including compression ratio, encoding time, and decoding time. Then, the policy decision output by the dynamic encoding policy selector corresponding to the generation of the data block and the system status data in step S2 are obtained. Subsequently, the three performance metrics of compression ratio, encoding time, and decoding time are normalized respectively, compressing their numerical range to between zero and one. Finally, the normalized performance metrics, policy decisions, and system status data are combined into a training sample according to a specific format.

9. A smart grid monitoring data storage method according to claim 8, characterized in that: The specific process of online fine-tuning and continuous optimization of the strategy value function model is as follows: First, the multi-objective reward value is calculated using the constructed training samples. This reward value is calculated by weighting the normalized compression ratio with positive weights and the normalized encoding and decoding times with negative weights, and by adding an additional reward term proportional to the policy entropy to encourage policy exploration. Then, an online policy gradient algorithm is adopted, with the multi-objective reward value as the optimization objective. The product of the log probability of the selected policy decision and the advantage function is calculated as the gradient direction, where the advantage function is the difference between the multi-objective reward value and the state value estimate output by the baseline function. This baseline function is used to reduce the variance of the learning process. Next, the parameters of the policy value function model are incrementally updated along this gradient direction; Meanwhile, the parameters of the baseline function itself are updated by minimizing the error between the predicted value of the baseline function and the actual multi-objective reward value; Finally, the updated strategy value function model parameters are hot-deployed into the dynamic coding strategy selector, enabling it to make better coding strategy decisions based on historical experience.

10. A smart grid monitoring and data storage system applied to the smart grid monitoring and data storage method as described in any one of claims 1-9, characterized in that: Specifically, it includes: The module includes an edge-side feature extraction module, a center-side strategy decision-making module, a heterogeneous task scheduling and execution module, and an online optimization feedback module. in, Edge-side feature extraction module: Configured at the data acquisition end, it is used to extract features from the raw monitoring data stream acquired at a fixed sampling period in real time. It generates a metadata vector by calculating the approximate sparsity entropy and the weighted mutation index, and then encapsulates the metadata vector and the corresponding raw data fragments into a data packet and uploads it. Central-side policy decision module: configured in the central storage system, used to receive and parse data packets from the edge side to obtain metadata vectors, and input the metadata vectors and the real-time performance indicators of the central system into a dynamic encoding policy selector, which outputs the specified encoding instructions; Heterogeneous task scheduling and execution module: configured in the central storage system, used to schedule the encoding task to a specific node in the distributed encoding cluster composed of heterogeneous computing nodes according to the characteristics of the encoding algorithm indicated by the encoding instruction, so that the node can perform the encoding operation and store the result in the storage system; Online optimization feedback module: Configured in the central storage system, it is used to collect the actual performance indicators of the stored data blocks, and to combine the performance indicators with the corresponding historical strategy decisions and system status data to form training samples, which are then input into the strategy value function model in the dynamic encoding strategy selector for online fine-tuning and continuous optimization of the model.

Images