Method, system and equipment for processing mass time scale measurement data of elastically telescopic power grid regulation and control system and storage medium

Abstract

The invention discloses a method, a system and equipment for processing massive time scale measurement data of an elastically telescopic power grid regulation and control system, and a storage medium, and the method comprises the steps: dynamically distributing the time scale measurement data to a plurality of fragments in a Hash ring space, and maintaining the load balance; for the time scale measurement data processing process of each fragment sent by the front acquisition subsystem to the SCADA subsystem, monitoring the data accumulation value of each level through a three-level cache feedback mechanism, and then triggering the elastic expansion and contraction of the fragment number according to a threshold value; performing time scale measurement processing on each fragment, recalculating a complete abnormal measurement correction through the SCADA subsystem, and synchronously writing into a real-time library section and a time sequence library cluster through double paths; and aggregating time sequence library cluster data through a time scale self-alignment engine, and generating and externally providing a millisecond-level whole-network same-time measurement section. According to the method, the problem of hotspot overload caused by traditional architecture fragmentation static distribution is solved, the peak throughput is 1 million / second, and high-precision time mark alignment data support is provided for power grid advanced application.

Description

TECHNICAL FIELD

[0001] The application relates to a mass time-tagged measurement data elastic processing method and system, in particular to a mass time-tagged measurement data processing method, system, equipment and storage medium of an elastic power grid regulation system. BACKGROUND

[0002] At present, the substation end has a steady-state time-tagged measurement data acquisition function, the time-tagged measurement data acquired by the substation end is strong in real-time and large in data quantity, with the promotion of the substation end reconstruction work, new time-tagged measurement data sites will continue to be connected to the main station of the regulation system, so that the existing power grid time-tagged measurement processing system is prone to data flood peaks.

[0003] From the architecture, the prior art mainly has the following problems: 1, static allocation of fragments causes hot spot overload; 2, time sequence library writing bottleneck restricts scalability; 3, time tag deviation of section data affects state estimation accuracy. SUMMARY

[0004] The purpose of the application is to provide a mass time-tagged measurement data processing method, system, equipment and storage medium of an elastic power grid regulation system, to realize high-performance processing of mass time-tagged measurement data continuously connected to the main station of the power grid regulation system, and to provide the main station of the power grid regulation system with higher quality and more accurate steady-state time-tagged measurement data.

[0005] Technical scheme: the mass time-tagged measurement data processing method of the elastic power grid regulation system comprises the following steps:

[0006] A hash ring space is constructed, time-tagged measurement data is dynamically allocated to a plurality of fragments in the hash ring space, and load balancing is maintained;

[0007] Through a three-level cache feedback mechanism, the time-tagged measurement data processing process of each fragment sent by the front-end acquisition subsystem to the SCADA subsystem is monitored, and the elastic expansion of the number of fragments is triggered according to the cache threshold value at each level; the three-level cache includes: a cache space set for the message bus channel, a circular queue configured for the time-tagged measurement processing process, and a thread dynamic queue used for data processing;

[0008] Each fragment performs time-tagged measurement processing, and after recalculating by the SCADA subsystem, synchronously stores the time-tagged measurement data to the real-time library section and the time sequence library cluster, aggregates the time sequence library cluster data using a time tag self-alignment engine, performs load balancing, and generates and provides millisecond-level whole-network simultaneous time measurement sections.

[0009] Optionally, the hash ring space is constructed as:

[0010] The hash ring space size is HashSpace=256, and the hash ring is arranged from 0 to 255 in a clockwise direction;

[0011] The system is initially divided into two slices, and when slice≥2, the slice and the time tag measurement ID are initially distributed as follows:

[0012] The hash value of slice 1 is recorded as SliceHash(1)=0;

[0013] The hash value of slice 2 is recorded as SliceHash(1)+HashSpace / 2=128;

[0014] All measurement IDs in the system are mapped to the hash ring by taking modulo 256, and the first slice found clockwise by the measurement ID is the slice to which the measurement ID belongs;

[0015] The number of measurement IDs for each hash value from 0 to 255 is recorded as idNumOfHash(hashValue);

[0016] The hash space of each slice, i.e., the start hash value and the end hash value, and the number of measurement IDs in the space are recorded as a four-tuple: SliceSpace(slice, startHash, endHash, idNumOfSlice);

[0017] wherein, slice is slice identification information, startHash and endHash are the start hash value and the end hash value of the slice on the hash ring respectively, and idNumOfSlice is the total number of measurement IDs included in the slice.

[0018] Optionally, the dynamic allocation of time tag measurement data to the plurality of slices in the hash ring space is essentially based on a slice fission algorithm for balancing the hash space measurement ID to achieve load balancing. The slice fission algorithm adopts a consistent hash slicing algorithm, including a scalable slice mapping algorithm on a slice hash ring and a power grid device mapping algorithm on a slice hash ring, to ensure the uniformity of the power grid model distributed on each slice and the smoothness during elastic scaling.

[0019] The scalable slice mapping algorithm on the slice hash ring includes the following steps:

[0020] Iterate through the slice four-tuple, and select the slice with the largest idNumOfSlice as the fission target;

[0021] Calculate the newEndHash of the new slice;

[0022] Insert the new slice in the hash ring and broadcast the elastic expansion message.

[0023] The mapping algorithm of the power grid device on the sharded hash ring comprises:

[0024] The real-time library loading of the newly added shard station model is started, and a backup shard processing process is started.

[0025] The original fission channel and the newly added shard channel are synchronized with the timestamp measurement data, and the consistency rate is checked, and when the consistency rate is greater than or equal to a given threshold, the master and backup shards are switched.

[0026] Preferably, the given threshold is set to 99.95%.

[0027] Optionally, when 1 new shard is inserted, the algorithm is as follows:

[0028] Traverse each shard quadruple to find the shard with the largest number of timestamp measurement IDs, which is the shard to be fissioned;

[0029] The minimum value of newEndHash satisfying the following formula is taken as the hash value of the newly added shard:

[0030]

[0031] The hash space information of the fissioned shard and the newly added shard is modified.

[0032] Optionally, the cache threshold at each level is initially set to:

[0033] The message bus channel accumulation threshold δ1 is 5,000 messages;

[0034] The process loop queue accumulation threshold δ2 is 10,000 messages;

[0035] The total accumulation threshold δ3 of the thread dynamic queue is 15,000 messages.

[0036] Optionally, the elastic scaling of the number of shards according to the cache threshold at each level comprises: calculating a load coefficient according to the flow imbalance degree, the cache pressure, and the processing thread blocking condition, and automatically triggering shard fission according to the load coefficient and the continuous overrun time.

[0037] Optionally, the shard fission uses a fission decision engine, which includes a load evaluation module, an automatic fission trigger, and a resource recycling module.

[0038] The load evaluation module is configured to calculate the load coefficient according to the following formula:

[0039]

[0040] α+β+γ+λ=1

[0041] The flow parameter in the above formula includes: Rin Q is the number of messages received per second by a shard, i.e., the shard's received data throughput, used to reflect the input pressure; out R is the number of processed messages, i.e., the shard's processed data throughput, representing the actual processing capacity; Q is the flow imbalance degree, if Q is greater than 1, it means that the data is accumulated.

[0042] The accumulation parameters in the above formula include: Q bus Q is the number of messages retained in the message bus channel cache, i.e., the cache provided by the message bus for each subscribing process, and the corresponding threshold is δ1. Similarly, Q loop Q is the process loop queue accumulation, i.e., the accumulation of messages taken out of the message bus channel cache by the process and put into the process loop queue, and the corresponding threshold is δ2. Q bus Q and Q loop are greater, respectively indicating that the receiving message capacity and the message distribution thread distribution capacity are insufficient. Therefore, the cache pressure of the process can be represented as Q bus / δ1, Q loop / δ2.∑Q thread is the total accumulation of all thread queue messages, i.e., the accumulation of messages obtained by the process from the process loop queue, and the corresponding threshold is δ3.∑Q thread is greater, indicating that the thread processing is not enough. The processing thread blocking condition can be represented as∑Q thread / δ3.

[0043] The weight parameters in the above formula include: α, β, γ, λ, which are weighting coefficients, and the sum of the four is 1, which is obtained according to historical data training. In the experimental environment, when the thread accumulation has the greatest impact, the optimal weight in the system is [0.2, 0.3, 0.3, 0.2].

[0044] The automatic fission trigger is configured to automatically start shard fission when the load coefficient is greater than 1.5 for 60 seconds.

[0045] The resource recycling module is configured to automatically merge shards when the load coefficient is less than 0.3.

[0046] Optionally, the each shard performs timestamp measurement processing, executes abnormal measurement correction and a double-path writing mechanism, including:

[0047] Distributed deployment is adopted for timestamp measurement data processing, and each shard starts a timestamp measurement processing process, which is responsible for timestamp measurement processing of the power grid model of the shard;

[0048] The time scale measurement processing procedure of each slice executes power grid service logic, performs abnormal measurement correction on the value and quality code of the time scale measurement through SCADA calculation, and synchronously stores to the slice real-time database and time sequence database cluster, at this time, the time scale measurement in the slice real-time database is the latest time scale measurement section of the power grid, and the time sequence database cluster contains the original time scale and the value and quality code of the calculated time scale measurement after point-to-point forwarding through the consistent hashing algorithm.

[0049] Optionally, the abnormal measurement correction comprises:

[0050] Zero drift value correction: set to 0 when |measurement value| < ε;

[0051] Bypass replacement correction: replace the line measurement with the bypass switch measurement;

[0052] Opposite side replacement correction: replace the measurement on the same side with the measurement on the opposite side.

[0053] Optionally, the time sequence database cluster realizes elastic expansion through the consistent hashing algorithm; the time scale measurement writing procedure of each time sequence database server is only responsible for writing to the local time sequence database, and multiple writing instances can be expanded as needed; a unified time scale measurement section service is used to converge and obtain the measurement section of the whole power grid at the same time, and the unified time scale measurement section service realizes load balancing through Nginx reverse proxy.

[0054] The massively time scale measurement data processing system of the elastically scalable power grid control system comprises:

[0055] A dynamic slice management subsystem is configured to construct a hash ring space, dynamically allocate time scale measurement data to multiple slices in the hash ring space, and maintain load balancing;

[0056] A three-level cache feedback control subsystem is configured to construct a three-level cache feedback mechanism, wherein the first level is a cache space set for a message bus channel, the second level is a circular queue configured for a time scale measurement processing procedure, and the third level is a thread dynamic queue used for data processing; the three-level cache feedback mechanism is used to monitor the data accumulation value of the time scale measurement data processing procedure of each slice sent by the front-end acquisition subsystem to the SCADA subsystem, and trigger the elastic scaling of the number of slices according to the cache threshold value of each level;

[0057] A time scale self-alignment service subsystem is configured to use a time scale self-alignment engine to aggregate and balance the time scale measurement data of the time sequence database cluster, generate and provide millisecond-level whole-network simultaneous-time measurement sections, and the time scale measurement data of the time sequence database cluster is recalculated by the SCADA subsystem.

[0058] The electronic device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements some or all of the steps in the above-mentioned method for processing massive time-scale measurement data of an elastically scalable power grid control system.

[0059] The computer-readable storage medium stores a computer program thereon, and is characterized in that when the computer program is executed by a processor, it implements some or all of the steps in the above-mentioned method for processing massive time-scale measurement data of a scalable power grid control system.

[0060] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0061] 1. By processing time-scaled measurement data, it reflects the real changes in power grid data, provides accurate simultaneous data sections of the entire network and continuously changing power grid data, and provides higher quality and more accurate steady-state data for advanced power grid application businesses such as analytical calculations and spot markets, thereby improving the accuracy of calculation results in advanced power grid applications.

[0062] 2. Adopting an elastically scalable processing architecture, it can cope with the time-scale measurement processing pressure in the continuous access system and realize automatic horizontal expansion of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The flexible processing architecture of the power grid time-scale measurement data of the present invention;

[0064] Figure 2 It is a schematic diagram of the structure of the pre-processor and SCADA message of the present invention;

[0065] Figure 3 This is a schematic diagram of the hash ring of the consistent hashing sharding algorithm of the present invention;

[0066] Figure 4 This is a schematic diagram of the newly added extended sharding consistent hashing algorithm of the present invention. DETAILED DESCRIPTION

[0067] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0068] like Figure 1 As shown, a method for processing massive time-scale measurement data of an elastically scalable power grid control system includes the following steps:

[0069] S0: First, build a flexible processing architecture for power grid time-scale measurement data.

[0070] The front-end acquisition subsystem FES collects the time-scale measurement of the substation equipment and serializes it through Protobuf. The structure definition is as follows Figure 2 shown.

[0071] The pre-acquisition subsystem forwards the time-tagged measurements to the data processing subsystem SCADA. Each slice corresponds to a message channel, and the starting channel number is 2100. Assuming that the station A belongs to slice 2, then the station A will send the message to the message bus of channel 2102 (i.e. 2100+2).

[0072] The SCADA subsystem divides the entire power grid model into multiple slices, and each real-time database of each slice only stores the power grid model belonging to the slice. Each slice corresponds to a time-tagged measurement processing process, such as the time-tagged measurement processing process of slice 2, which subscribes to the message bus 2102 channel and receives the time-tagged measurement message of the 2102 channel. After receiving the time-tagged measurement message of the 2102 channel, the calculated result is stored in the real-time database of the slice.

[0073] The time-tagged measurement processing process sends the calculated time-tagged measurement value and quality code to the local message channel and forwards it to the time series database cluster through the point-to-point message forwarding process. The determination of the target server for forwarding also uses the consistent hashing algorithm.

[0074] S1. The power grid model is divided into slices using the consistent hashing algorithm, and the implementation is as follows:

[0075] S1.1. A hash ring space is constructed, and the size is HashSpace = 256. The hash ring is arranged in a clockwise direction from 0 to 255. The system is initially divided into two slices, as shown in FIG. 1, and the initial hash distribution of the slice and the time-tagged measurement ID is as follows: Figure 3

[0076] The hash value of slice 1 is recorded as SliceHash(1) = 0;

[0077] The hash value of slice 2 is recorded as SliceHash(1) + HashSpace / 2 = 128;

[0078] All measurement IDs in the system are taken modulo 256 and mapped to the hash ring. The first slice found clockwise by the measurement ID is the slice to which the measurement ID belongs. For example: the id of substation A is 1123100000028, and the hash value after taking modulo 256 is 28. The first slice found clockwise along the hash ring has a hash value of 128, which is the hash value of slice 2. Therefore, the slice number of substation A is 2, and the model of substation A is stored in real-time database slice 2.

[0079] The number of measurement IDs for each hash value from 0 to 255 is recorded as idNumOfHash(hashValue);

[0080] ​After all the stations are divided into slices, the number of measurements corresponding to each hash value is recorded for subsequent slice fission calculation of the number of measurement IDs. For example, hash value 1 has 10 measurements, recorded as idNumOfHash(1) = 10, hash value 2 has 20 measurements, recorded as idNumOfHash(2) = 20, if a slice X hash space contains 1, 2 two hash values, then the total number of measurement IDs of slice X is idNumOfHash(1) + idNumOfHash(2) = 30.

[0081] Record the hash space of each slice, that is, the start hash value and the end hash value, and the number of measurement IDs in the space, recorded as a four-tuple: SliceSpace(slice, startHash, endHash, idNumOfSlice), Wherein, slice is the slice identification information, startHash and endHash are the start hash value and the end hash value of the slice on the hash ring respectively, and idNumOfSlice is the total number of measurement IDs contained in the slice. For example: slice 2 has 10000 measurement IDs, and the hash space is recorded as: SliceSpace(2, 1, 128, 10000).

[0082] S1.2, the power grid model slicing uses consistent hash slicing algorithm, including the mapping algorithm of the scalable slice on the slice hash ring and the mapping algorithm of the power grid device on the slice hash ring, to ensure the uniformity of the power grid model distributed on each slice and the smoothness when the power grid model is scaled.

[0083] S1.2.1, the mapping algorithm of the scalable slice on the slice hash ring includes the following steps:

[0084] Traverse the slice four-tuple SliceSpace(slice, startHash, endHash, idNumOfSlice), select the slice with the largest idNumOfSlice as the fission target;

[0085] Calculate the new slice hash value newEndHash;

[0086] Insert the new slice in the hash ring, broadcast the elastic expansion message;

[0087] S1.2.2, the mapping algorithm of the power grid device on the slice hash ring includes:

[0088] Load the station model in the new slice real-time library, start the standby slice processing process;

[0089] The original fission channel and the new added slice channel are synchronized with the timestamp measurement data, and the consistency rate is checked. When the consistency rate is greater than or equal to 99.95%, the master and standby slices are switched.

[0090] Further, as Figure 4 indicated, it also includes a new slice algorithm. Taking the expansion of one slice as an example, the following steps are implemented:

[0091] S1.2.3, traverse each slice quadruple, find the slice with the largest number of timestamp measurement IDs, which is the slice to be fissioned;

[0092] S1.2.3.1, assuming that the current hash ring distribution is as Figure 2 indicated, slice 1 contains 2 IDs, measurement ID #4 and measurement ID #5, slice 2 contains 3 IDs, measurement ID #1, measurement ID #2, and measurement ID #3, the number of IDs of slice 2 is greater than that of slice 1, so slice 2 needs to be fissioned into 2 slices, and the hash space of slice 2 is SliceSpace(2, 1, 128, 3).

[0093] S1.2.3.2, the number of timestamp measurement IDs in the new added slice should be greater than or equal to half of the number of timestamp measurements of the fissioned slice, so the calculation formula is as follows:

[0094]

[0095] The minimum value of newEndHash that satisfies the above inequality is the hash value of the new added slice.

[0096] Figure 2 The hash values of the three IDs are 16, 32, and 112 respectively, all of which have only 1 ID, and when the hash value is equal to 32, the new slice has 2 slices, which is greater than 3 / 2 slices, therefore, the hash value of the new added slice is 32, as Figure 3 indicated.

[0097] S1.2.3.3, modify the hash space information of the fissioned slice and the new added slice.

[0098] 2 slice hash space becomes: SliceSpace(2, 33, 128, 1), the hash space is from 33 to 128, and there is 1 ID;

[0099] 3 slice hash space is: SliceSpace(3, 1, 32, 2), the hash space is from 1 to 32, and there are 2 IDs.

[0100] S2, peak elimination through a three-level cache feedback control mechanism.

[0101] S2.1, construct a three-level cache, including:

[0102] The first level: message bus channel, setting a cache space with a maximum capacity of 16MB for each time scale measurement data processing process;

[0103] The second level: the circular queue of each time scale measurement data processing process, with a size of 32MB;

[0104] The third level: the data processing queue of each thread in each time scale measurement data processing process, using C++ standard template library queue, without setting the maximum capacity.

[0105] S2.2, each time scale measurement data processing process of each slice reports the accumulated values of each level to the elastic scaling control process every 5 seconds.

[0106] The accumulated values of each level include: received data throughput rate, processed data throughput rate, message bus data accumulated value, process circular queue accumulated value, and accumulated value of each thread data processing queue, and the data is reported to the elastic scaling control process of the slice management center through the message bus.

[0107] S2.3, execute the elastic scaling algorithm, the steps are as follows:

[0108] S2.3.1, the elastic scaling control process determines whether the slice needs to be expanded into two slices through the fission decision engine according to the data accumulation of each slice data processing process, and the judgment logic is as follows:

[0109] S2.3.1.1, the load evaluation matrix is designed as follows:

[0110]

[0111] The meanings of each parameter are shown in Table 1.

[0112] Table 1

[0113]

[0114]

[0115] The above formula contains three types of parameters:

[0116] The first type of flow parameter: R in is the number of messages received by the slice per second, reflecting the input pressure; R out is the number of processed messages, representing the actual processing capacity, and the ratio of the two is greater than 1, indicating backlog.

[0117] The second type of accumulation parameter: Q bus refers to the number of messages retained in the message bus cache, and δ1 is its threshold (5000). Similarly, Q loop is the process circular queue accumulation (δ2=10000), and ∑Qthread is the sum of all thread queues (δ3=15000).

[0118] The third type of weight parameter: α, β, γ, λ is the sum of the weighting coefficients is 1, which needs to be trained according to historical data. For example, in the actual measurement, when the thread accumulation has the greatest impact, λ can be taken as 0.4.

[0119] It needs to be specially pointed out that the setting of each threshold δ is based on: the message bus channel accumulation threshold δ1=5000 corresponds to the 16MB cache depletion value; the process loop queue accumulation threshold δ2=10000 is the upper limit of 32MB capacity; and the total thread dynamic queue accumulation threshold δ3=15000 is a safety line set to prevent memory overflow.

[0120] S2.3.1.2, fission determination logic, when the load coefficient >1.5 and the continuous over-limit time ≥60 seconds are met at the same time, the fragmentation fission is triggered. Among them, 1.5 is an empirical threshold, and the continuous over-limit time can prevent false triggering caused by transient fluctuations.

[0121] S2.3.2, if a certain fragment needs to be fissioned into 2 fragments, the following steps are executed:

[0122] S2.3.2.1, according to the above-mentioned new fragment algorithm, insert a new fragment on the hash ring, and broadcast the elastic expansion message;

[0123] S2.3.2.2, the local fragment management process of the data processing subsystem SCADA listens to the expansion message, loads the models divided into the new fragment in the real-time database of the new fragment, including the power stations and all power grid models belonging to these power stations, and starts a time tag measurement processing process on the new fragment, and sets the new fragment as the standby fragment, and the original fission fragment still operates as the master fragment and provides data to the outside;

[0124] S2.3.2.3, notify the front-end subsystem, and send a copy of the data in the original fission channel to the new fragment channel at the same time;

[0125] S2.3.2.4, the local fragment management process compares the time tag measurement values of the models in the new fragment, judges whether they are consistent with the measurement values in the original fission fragment, if the measurement consistency rate exceeds the set threshold, the new fragment is switched to the master fragment, and starts to provide data to the outside, at the same time, sends an expansion end message to the front-end subsystem;

[0126] S2.3.2.5, the front-end subsystem stops sending time tag measurement data of the power grid models divided into the new fragment to the original fission fragment, and the local fragment management process deletes the power grid models divided into the new fragment in the original fission fragment.

[0127] S3, real-time database and time sequence database double-path data processing and forwarding, the specific implementation is as follows.

[0128] S3.1, each slice performs timestamp measurement processing, and after recalculating through the SCADA subsystem, synchronously stores to the real-time database section and the time series database cluster, including:

[0129] S3.1.1, the timestamp measurement data processing adopts distributed deployment, each slice starts a timestamp measurement processing process, responsible for the timestamp measurement processing of the power grid model of the slice;

[0130] S3.1.2, the timestamp measurement processing process of each slice executes power grid business logic, recalculates the value and quality code of the timestamp measurement, and writes to the local real-time database. The timestamp measurement in the real-time database is the latest timestamp measurement section of the power grid, serving the power grid control real-time monitoring business;

[0131] S3.1.3, the original timestamp and the calculated value and quality code of the timestamp measurement are forwarded to the time series database cluster through a consistent hashing algorithm, point to point.

[0132] The timestamp measurement forwarded to the time series database cluster cannot directly use the data sent by the front-end subsystem, and must be calculated by the SCADA. Because the timestamp measurement may have some abnormal states or superimpose some states calculated by the SCADA, such as zero drift value, more reasonable upper and lower limit, working condition exit, line measurement replaced by opposite side line measurement, line measurement replaced by bypass switch measurement, etc., therefore, the calculated measurement value and quality code are corrected. The correction includes:

[0133] Zero drift value correction: when |measurement value|<ε, set to 0;

[0134] Bypass replacement correction: replace the line measurement with the bypass switch measurement;

[0135] Opposite side replacement correction: replace the opposite side line measurement with the opposite side line measurement.

[0136] S3.2, use the timestamp self-alignment engine to aggregate the time series database cluster data and perform load balancing to generate and provide millisecond-level whole network simultaneous time measurement section to the outside, and the specific implementation is as follows:

[0137] S3.2.1, the time series database cluster realizes elastic expansion through a consistent hashing algorithm;

[0138] S3.2.2, the timestamp measurement writing process of each time series database server is only responsible for writing to the local time series database, and multiple writing instances can be expanded as needed to improve writing performance;

[0139] S3.2.3, advanced applications such as power grid state estimation can obtain the measurement section of the whole power grid at the same time through the unified timestamp measurement section service;

[0140] S3.2.4, the unified time scale measurement section service realizes load balancing through an Nginx reverse proxy.

[0141] The application further provides a flexible and scalable massive time scale measurement data processing system of a power grid regulation system, comprising:

[0142] A dynamic fragmentation management subsystem is configured to construct a hash ring space, dynamically allocate time scale measurement data to multiple fragments in the hash ring space, and maintain load balancing.

[0143] A three-level cache feedback control subsystem is configured to construct a three-level cache feedback mechanism, wherein the first level is a cache space set for a message bus channel, the second level is a circular queue configured for a time scale measurement processing process, and the third level is a thread dynamic queue used for data processing; the three-level cache feedback mechanism is used to monitor data accumulation values of time scale measurement data processing processes of each fragment sent by the front-end collection subsystem to the SCADA subsystem, and trigger flexible expansion of the number of fragments according to cache thresholds at different levels.

[0144] A time scale self-alignment service subsystem is configured to use a time scale self-alignment engine to aggregate and balance time scale measurement data of the time series library cluster, generate and provide millisecond-level whole-network simultaneous time measurement sections, and complete abnormal measurement correction through recalculation of the time scale measurement data of the time series library cluster by the SCADA subsystem.

[0145] The dynamic fragmentation management subsystem can adopt a fragmentation hash ring generation module, a fission decision engine and a double-channel data verifier.

[0146] The three-level cache feedback control subsystem can adopt a message bus monitoring agent, a circular queue statistician and a thread queue accumulation analyzer.

[0147] The time scale self-alignment service subsystem can adopt an Nginx reverse proxy cluster and a section data aggregation engine.

[0148] To implement the above-mentioned flexible and scalable massive time scale measurement data processing method of a power grid regulation system, the application provides an electronic device and a computer readable storage medium.

[0149] The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements part or all steps of the above-mentioned flexible and scalable massive time scale measurement data processing method of a power grid regulation system when executing the program.

[0150] The computer readable storage medium has a computer program stored thereon, and the computer program is executed by the processor to implement part or all of the steps of the massive time-tag measurement data processing method of the elastic power grid regulation system.

Claims

1. A method for processing massive time-scale measurement data of an elastically scalable power grid control system, characterized in that: The method comprises the following steps: Constructing a hash ring space, dynamically distributing time-scale measurement data to multiple shards in the hash ring space, and maintaining load balancing; Through a three-level cache feedback mechanism, the data accumulation value of the time-scale measurement data processing process sent by the front-end acquisition subsystem to each shard of the SCADA subsystem is monitored, and the elastic expansion and contraction of the number of shards is triggered according to the cache thresholds at each level. The three-level cache includes: the first level is the cache space set for the message bus channel, the second level is the circular queue configured for the time-scale measurement processing process, and the third level is the thread dynamic queue for data processing; Each shard performs time-scale measurement processing, and after being recalculated by the SCADA subsystem, it is synchronously stored in the real-time library section and the time series library cluster. The time-scale self-alignment engine is used to aggregate the time series library cluster data and perform load balancing to generate and provide millisecond-level full-network simultaneous measurement sections to the outside world.

2. The method for processing massive time-scale measurement data of a scalable power grid control system according to claim 1, characterized in that: The hash ring space is constructed as follows: The size of the hash ring space is HashSpace = 256, and the numbers are arranged from 0 to 255 in a clockwise direction on the hash ring; The system is initially divided into two shards. When slice ≥ 2, the initial hash distribution of the shard and time-scale measurement ID is as follows: The hash value of slice 1 is recorded as SliceHash(1)=0; The hash value of slice 2 is recorded as SliceHash(1)+HashSpace / 2=128; Take the modulo 256 of all measurement IDs in the system and map them to the hash ring. The first shard found by the measurement ID in a clockwise direction is the shard to which the measurement ID belongs. Record the number of measurement IDs for each hash value from 0 to 255, recorded as idNumOfHash(hashValue); Record the hash space of each slice, i.e. the starting hash value and the ending hash value, as well as the number of measurement IDs in the space, as a four-tuple: SliceSpace(slice, startHash, endHash, idNumOfSlice); in, is the total number of measurement IDs of the slice, slice is the slice identification information, startHash and endHash are the starting hash value and ending hash value of the slice on the hash ring respectively, and idNumOfSlice is the total number of measurement IDs contained in the slice.

3. The method for processing massive time-scale measurement data of a scalable power grid control system according to claim 1 or 2, characterized in that: The time-scale measurement data is dynamically allocated to the multiple shards in the hash ring space using a consistent hash sharding algorithm, including a mapping algorithm of elastically scalable shards on the shard hash ring and a mapping algorithm of power grid equipment on the shard hash ring; The mapping algorithm of the elastically scalable shards on the shard hash ring includes the following steps: Traverse the slice quadruple SliceSpace (slice, startHash, endHash, idNumOfSlice) and select the slice with the largest idNumOf Slice as the fission target; among them, slice is the slice identification information, startHash and endHash are the starting hash value and ending hash value of the slice on the hash ring respectively, and idNumOfSlice is the total number of measurement IDs contained in the slice; Calculate the hash value newEndHash of the newly added shard; Insert new shards into the hash ring and broadcast elastic expansion messages; The mapping algorithm of the power grid equipment on the sharded hash ring includes: Load the plant model to which the new shard belongs in the real-time database and start the backup shard processing process; The original fission channel and the newly added shard channel synchronize the time-scale measurement data and perform consistency verification. When the consistency rate is ≥ a given threshold, the primary and backup shards are switched.

4. The method for processing massive time-scale measurement data of a scalable power grid control system according to claim 3, characterized in that: When inserting a new shard, the algorithm is as follows: Traverse the shard quadruple and find the shard with the largest number of time-scale measurement IDs. This shard is the shard to be split. The minimum value of newEndHash that satisfies the following formula is used as the hash value of the new shard: Modify the hash space information of the fission shards and the newly added shards.

5. The method for processing massive time-scale measurement data of a scalable power grid control system according to claim 1, characterized in that: The elastic scaling of the number of shards triggered by the cache thresholds at each level includes: calculating the load factor according to the traffic imbalance, cache pressure, and processing thread blocking situation, and automatically triggering shard fission according to the load factor and the continuous exceeding time.

6. The method for processing massive time-scale measurement data of a scalable power grid control system according to claim 1, characterized in that: Each slice performs time-scale measurement processing to execute abnormal measurement correction and dual-path writing mechanism, including: Distributed deployment is used for time-scale measurement data processing. Each shard starts a time-scale measurement processing process, which is responsible for the time-scale measurement processing of the power grid model of this shard. The time-scale measurement processing process of each shard executes the business logic of the power grid, and performs abnormal measurement correction on the time-scale measurement value and quality code through SCADA calculation, and synchronously stores them in the shard real-time library and timing library cluster. At this time, the time-scale measurement in the shard real-time library is the latest time-scale measurement section of the power grid, and the timing library cluster contains the original time-scale after the consistent hash algorithm and the calculated time-scale measurement value and quality code forwarded point-to-point.

7. The method for processing massive time-scale measurement data of a scalable power grid control system according to claim 1, characterized in that: The timing library cluster achieves elastic expansion through a consistent hashing algorithm; the time-scale measurement writing process of each timing library server is only responsible for writing to the local timing library, and multiple writing instances can be expanded as needed; through a unified time-scale measurement section service, the measurement sections of the entire power grid at the same time are aggregated and obtained, and the unified time-scale measurement section service achieves load balancing through the Nginx reverse proxy.

8. A scalable and elastic power grid control system massive time-scale measurement data processing system, characterized by: The system includes: A dynamic shard management subsystem, which is used to construct a hash ring space, dynamically distribute time-scale measurement data to multiple shards in the hash ring space, and maintain load balancing; A three-level cache feedback control subsystem is used to build a three-layer cache feedback mechanism, in which the first level is the cache space set for the message bus channel, the second level is the circular queue configured for the time-scale measurement processing process, and the third level is the thread dynamic queue for data processing. The three-layer cache feedback mechanism monitors the data accumulation value of the time-scale measurement data processing process sent by the front-end acquisition subsystem to each shard of the SCADA subsystem, and triggers the elastic expansion of the number of shards according to the cache thresholds at each level; The time-scale self-alignment service subsystem is used to use the time-scale self-alignment engine to aggregate and load balance the time-scale measurement data of the timing library cluster, generate and provide millisecond-level full-network simultaneous measurement sections to the outside world, and the time-scale measurement data of the timing library cluster are recalculated by the SCADA subsystem.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for processing massive time-scale measurement data of a scalable power grid control system as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for processing massive time-scale measurement data of a scalable power grid control system as described in any one of claims 1 to 7 are implemented.

Images