Real-time analysis method and device for electric power heterogeneous data

Abstract

The invention discloses an electric power heterogeneous data real-time analysis method and device, and relates to the field of electric power information processing, and the method comprises the steps: S1, obtaining real-time electric power heterogeneous data from a plurality of power grid measurement data source ends; using a Kafka distributed message queue to access the real-time electric power heterogeneous data to a stream computing engine; s2, performing routing, event stateful processing and cleaning on the electric power heterogeneous data by using a flow calculation engine, namely routing the electric power heterogeneous data to a corresponding analyzer according to a service domain; identifying an event type in the data, and triggering a corresponding state processing flow by a state machine to obtain analyzed standard format data; carrying out interval segmentation by using a window manager, and carrying out filtering and duplicate removal operations to obtain analyzed data; and S3, pushing the analyzed data to a model for conversion or distributed storage. According to the method, data repeated scanning is avoided by adopting event-driven stateful processing, the assembly line is dynamically optimized by using window calculation, and the real-time performance of heterogeneous data access can be greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric power information processing, and in particular to an electric power heterogeneous data real-time analysis method and device. BACKGROUND

[0002] The evolution of power grid data analysis technology is closely related to digital transformation policy. In 2022, State Grid Corporation of China released the Framework Design of Digital Technology Support System for New Power System, which requires "real-time collection of electric and non-electric data from various links" and promotes the transformation of data sharing from professional line isolation to global sharing. The background can be divided into three levels:

[0003] Policy-driven level: under the goal of carbon neutrality, the new power system needs to integrate data from the power supply side (such as wind power and photovoltaic power) and the load side (such as user energy storage) to achieve "coordinated operation". The existing collection system is mainly constructed in professional lines (such as control cloud and distribution automation), resulting in different model standards, such as the large difference between the data models of power collection and distribution.

[0004] Technical evolution level: data warehouse evolves from offline batch processing (such as Hive) to real-time data warehouse. Real-time data warehouse 1.0 / 2.0 version: 1.0 version relies on Spark Streaming for ETL, but has "high Kafka outflow" and query timeout problems; 2.0 version introduces Flink to improve low latency and Exactly-once semantics, but still does not solve the business adaptability of electric power heterogeneous data. Although the stream processing framework (such as Flink) supports state management, electric power grid data has strong business correlation (such as topology model fusion), and general technology cannot be directly applied.

[0005] Business demand level: power grid businesses such as distribution active operation need "millisecond-level quasi-real-time measurement data", but the massive data (tens of millions of terminals) exacerbate the parsing load. A certain provincial company's Internet of Things collection measurement center has tried to develop data storage and model conversion, but still has defects such as "real-time collection data not standardized and collected" and "difficult to share and use".

[0006] The existing technology has defects in real-time analysis of electric power heterogeneous data, which limits the value mining of power grid data, as follows:

[0007] Low efficiency of heterogeneous data analysis: traditional methods use rule engines or static mapping, which cannot dynamically adapt to multi-source data. "Electric power resource asset data is inconsistent" between different systems, and manual intervention is required during analysis, resulting in excessive delay. The real-time data warehouse 1.0 version causes multiple consumption problems due to "all flow data stored in the same Kafka Topic", and has insufficient throughput.

[0008] Real-time performance and reliability are difficult to balance: stream processing tasks run continuously, but fault recovery mechanisms are weak. "Real-time computing has very strict performance requirements for data processing," and existing fault-tolerance technologies (such as periodic checkpointing) are prone to failure in power grid scenarios due to data timing discrepancies. Public information shows that Flink's CheckPoint mechanism may increase latency during network jitter, while power grid data needs to guarantee causality within a second-level latency.

[0009] The complexity of business integration is high: the parsing technology is not deeply integrated with the power grid model (such as SG-CIM, the State Grid Public Information Model). The existing system is "not associated with the ledger topology," requiring business applications to repeatedly process data and increase link length. For example, the "raw data" collected by the IoT management platform cannot be directly shared and requires additional conversion, violating the principle of real-time performance.

[0010] Insufficient scalability: Existing technologies "cannot meet the increasingly urgent application needs of power grid digital transformation." With the integration of new energy sources, the amount of data is surging, and the parsing system needs to support horizontal scaling, but traditional architectures (such as database-based parsing) cannot dynamically schedule resources. Summary of the Invention

[0011] To address the above issues, this invention proposes a real-time parsing method and apparatus for heterogeneous power data. By employing event-driven stateful processing to avoid redundant data scanning and dynamically optimizing the pipeline through window calculation, it significantly improves the real-time performance of heterogeneous data access, supporting millisecond-level latency. Given the strong business dependency characteristics of power grid data, business context information is maintained in the state machine, ensuring that data processing remains relevant to the business scenario. Distributed snapshot fault tolerance provides accurate recovery; snapshots capture the complete state, and restarting from a consistent point after a fault guarantees exactly-once semantics. The state machine incorporates SG-CIM standard model conversion logic, enabling automated unified parsing of multi-source data, solving the problem of manual mapping rule configuration required in traditional methods, and reducing data integration complexity.

[0012] On the one hand, the real-time analysis method for heterogeneous power data involves the following steps:

[0013] S1: Obtain real-time heterogeneous power data from several power grid measurement data sources; use Kafka distributed message queue to connect the real-time heterogeneous power data to the stream computing engine;

[0014] S2 uses a stream computing engine to route, process stateful events, and clean heterogeneous power data, as follows:

[0015] Routing routes heterogeneous power data to the corresponding resolvers according to their business domains;

[0016] The event handling process involves a parser that identifies event types in heterogeneous power data. A state machine triggers the corresponding state handling flow for each event type, obtaining parsed standard format data and updating the corresponding state in the state machine. The state machine's state includes data mapping, parsing progress, and business context. The data mapping stores data format mapping rules. The parsing progress records the data parsing progress. The business context records the topological relationship between equipment and ledgers, and / or the event dependencies of continuous measurement data.

[0017] The data is cleaned by using a window manager to segment the parsed standard format data at intervals, and then filtering and deduplicating the segmented data to obtain the parsed data.

[0018] S3 outputs the parsed data, and performs model transformation or distributed storage on the parsed data.

[0019] Preferably, step S2 further includes: using distributed snapshot fault tolerance for fault recovery; specifically: using a fault tolerance controller to periodically generate consistent snapshots among distributed nodes, storing the consistent snapshots in a distributed file system, and using the most recent consistent snapshot stored in the file system to recover data when a node failure occurs.

[0020] Preferably, the frequency of periodically generating consistent snapshots is 10-15 seconds.

[0021] Preferably, step S1 further includes: using a priority resource allocation mechanism to allocate computing resources, allocating more computing resources and bandwidth to high-priority data.

[0022] Preferably, the step of using a window manager to segment the parsed standard format data at intervals specifically involves: segmenting the parsed standard format data at intervals using time windows; the time windows include a first-level window for processing high-frequency data and a second-level window for aggregating event data; the first time window is segmented at second-level intervals; the second time window is segmented at minute-level intervals; when segmenting at intervals, the first-level window or the second-level window is dynamically selected based on the business scenario of the heterogeneous power data.

[0023] Preferably, the data format mapping rule is a conversion logic from JSON to SG-CIM.

[0024] Secondly, a real-time analysis device for heterogeneous power data includes the following:

[0025] The access module is used to obtain real-time heterogeneous power data from several power grid measurement data sources; and uses the Kafka distributed message queue to connect the real-time heterogeneous power data to the stream computing engine.

[0026] The parsing module is used to route, process stateful events, and clean up heterogeneous power data using the stream computing engine, as follows:

[0027] Routing routes heterogeneous power data to the corresponding resolvers according to their business domains;

[0028] The event handling process involves a parser that identifies event types in heterogeneous power data. A state machine triggers the corresponding state handling flow for each event type, obtaining parsed standard format data and updating the corresponding state in the state machine. The state machine's state includes data mapping, parsing progress, and business context. The data mapping stores data format mapping rules. The parsing progress records the data parsing progress. The business context records the topological relationship between equipment and ledgers, and / or the event dependencies of continuous measurement data.

[0029] The data is cleaned by using a window manager to segment the parsed standard format data at intervals, filtering and deduplicating the segmented data to obtain the parsed data.

[0030] The output module is used to output the parsed data, and to perform model transformation or distributed storage on the parsed data.

[0031] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0032] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0033] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0034] On the other hand, the real-time analysis device for heterogeneous power data includes the following:

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) This invention avoids repeated data scanning by using event-driven stateful processing and dynamically optimizes the pipeline for window calculation, which can greatly improve the real-time performance of heterogeneous data access and support millisecond-level latency.

[0037] (2) Based on the strong business dependency characteristics of power grid data, the present invention maintains business context information in the state machine, which can ensure that data processing does not deviate from the business scenario;

[0038] (3) The distributed snapshot fault tolerance of the present invention provides accurate recovery. It captures the complete state through snapshots and restarts from the consistency point after a failure, ensuring exactly-once semantics. The distributed snapshot fault tolerance reduces the recovery time to the second level and avoids business decision errors caused by data backlog.

[0039] (4) The present invention incorporates the SG-CIM standard model conversion logic into the state machine, which can realize the automated unified parsing of multi-source data, solve the problem of manual configuration of mapping rules in traditional methods, and reduce the complexity of data integration. Attached Figure Description

[0040] The present invention will now be described in further detail with reference to the accompanying drawings;

[0041] Figure 1 This is a flowchart of the real-time parsing method for heterogeneous power data according to an embodiment of the present invention;

[0042] Figure 2 This is a structural block diagram of the real-time analysis device for heterogeneous power data according to an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described below through specific embodiments.

[0045] like Figure 1 As shown, a method for real-time parsing of heterogeneous power data includes the following steps:

[0046] S1: Obtain real-time heterogeneous power data from several power grid measurement data sources; use Kafka distributed message queue to connect the real-time heterogeneous power data to the stream computing engine;

[0047] It receives massive amounts of real-time measurement data from the equipment, marketing, and scheduling sides. The data stream originates from the source (such as the electricity information collection front-end) and enters the stream computing engine via a distributed message queue (such as Kafka).

[0048] This embodiment also includes: using a priority resource allocation mechanism to allocate computing resources, allocating more computing resources and bandwidth to high-priority data. For example, identifying the "hotness" of message topics in Kafka, determining their priority, classifying frequently generated and business-related data as high-priority hot topics, and classifying infrequently generated and non-core business data as low-priority cold topics, and allocating computing resources and bandwidth according to the hotness level.

[0049] S2 uses a stream computing engine to route, process stateful events, and clean heterogeneous power data, as follows:

[0050] Routing routes heterogeneous power data to the corresponding resolvers according to their business domains;

[0051] The event handling process involves a parser that identifies event types in heterogeneous power data. A state machine triggers the corresponding state handling flow for each event type, obtaining parsed standard format data and updating the corresponding state in the state machine. The state machine's state includes data mapping, parsing progress, and business context. The data mapping stores data format mapping rules. The parsing progress records the data parsing progress. The business context records the topological relationship between equipment and ledgers, and / or the event dependencies of continuous measurement data.

[0052] The data is cleaned by using a window manager to segment the parsed standard format data at intervals, and then filtering and deduplicating the segmented data to obtain the parsed data.

[0053] Power grid data has strong business dependencies (such as measurement data needing to be associated with equipment ledgers and topology relationships). The context information maintained by the state machine in this embodiment can ensure that data processing does not deviate from the business scenario. For example, fault data can be quickly associated with the corresponding equipment file through the state.

[0054] This embodiment of Event Stateful Processing works by maintaining runtime state for event sequences of streaming data, enabling context-dependent resolution. It employs multi-level event stateful processing; for example, for continuous measurement data (such as voltage sequences), the state machine records the dependencies between preceding and subsequent events, avoiding redundant calculations. Similar to Flink's StateBackend mechanism, but optimized specifically for power data, the state includes data format mapping rules (such as JSON to SG-CIM conversion logic) and supports dynamic updates.

[0055] The specific implementation process of stateful event handling is as follows: When heterogeneous data (such as device status variables and electrical energy) flows in, the parser triggers different state handling processes based on the event type (such as alarm events and periodic collection events). This achieves "unified parsing and cleaning of heterogeneous data." For example, marketing-side data needs to identify key fields (such as user IDs), while device-side data needs to be associated with the topology. The state machine ensures that these operations are completed in memory, reducing I / O latency.

[0056] This embodiment uses layered window calculation to segment real-time heterogeneous power data at intervals. The principle is to improve real-time performance by processing data in blocks using time windows (such as scrolling windows or sliding windows). It employs "highly flexible layered window calculation," for example, the first layer (1-second level) processes high-frequency data (such as second-level measurements), and the second layer (minute-level) aggregates event data (such as fault alarms).

[0057] The implementation of layered window computing involves binding window design to business scenarios. With the goal of achieving a latency of less than 20 seconds for event-based stream computing, for example, in anomaly detection scenarios, the window size is dynamically adjusted: normal data is quickly parsed using a small window, while abnormal data is analyzed in depth using a large window. This is achieved through the stream computing engine's API ("script and JAR package upload").

[0058] The stream processing engine in this embodiment also includes Distributed Checkpoint fault tolerance for fault recovery. The principle of distributed checkpoint fault tolerance is to periodically generate consistent snapshots (Checkpoints) among distributed nodes for fault recovery. This is similar to the Chandy-Lamport algorithm: barriers are set in the stream processing topology to synchronize the state of each node, and snapshots are stored in a persistent system (such as HDFS). When a node fails, recovery is performed from the most recent snapshot, ensuring exactly-once semantics.

[0059] The specific implementation of distributed snapshot fault tolerance is as follows: Based on document business requirements, the snapshot content includes not only data offsets but also parsing status (such as partially parsed model mappings). It is necessary to "ensure orderly data processing." For example, when parsing power distribution data, the snapshot capture window calculates intermediate results to avoid timing errors caused by data rollback after a failure. Flink's CheckPoint is used in e-commerce scenarios, but this embodiment optimizes the snapshot frequency (e.g., once every 10 seconds) to meet the low latency requirements of the power grid, balancing reliability and performance.

[0060] S3 outputs the parsed data, and performs model transformation or distributed storage on the parsed data.

[0061] The following example demonstrates the workflow using the parsing of power distribution data:

[0062] Step 1: Data is received from the message queue and routed by the event dispatcher according to the business domain (such as marketing, device).

[0063] Step 2: A state resolver loads the SG-CIM model template and maps the incoming data in real time (e.g., converting non-standard data into standard measurement formats). The state machine records the mapping progress to ensure complete parsing of complex objects (e.g., transformer ledgers).

[0064] Step 3: The window manager segments the data at 500ms intervals and performs filtering and deduplication operations. Simultaneously, the fault tolerance controller triggers a snapshot every 15 seconds and stores it to the distributed file system.

[0065] Step 4: The parsed data is output to the shared topic, and the latency monitoring shows "less than 300ms".

[0066] Furthermore, this embodiment supports high performance and ensures performance through resource quota allocation ("real-time message bus") and parallel computing (such as multi-core optimization).

[0067] Compared to existing technologies (such as real-time data warehouse versions 1.0 / 2.0 or traditional parsing methods), this embodiment, through stateful event processing and distributed fault tolerance mechanisms, offers significant advantages in real-time performance, reliability, and business integration. Specifically, these advantages include the following:

[0068] Real-time performance is significantly improved, supporting millisecond-level latency. Existing technologies such as Real-Time Data Warehouse 1.0 rely on Spark Streaming, and the batch processing micro-batch architecture introduces inherent latency (usually over a second), leading to "query timeouts." This embodiment, however, uses event-driven state processing to achieve "data access latency below 300ms." The advantages are: stateful processing avoids redundant data scanning, window computation dynamically optimizes the pipeline, and it is hundreds of times faster than traditional ETL (such as those based on Hive).

[0069] Enhanced reliability ensures zero data loss. Existing fault-tolerance technologies (such as periodic logging) are prone to failure under high concurrency in power grids, and suffer from the problem that "streaming tasks run continuously once started, resulting in high computational costs." This embodiment's distributed snapshot fault tolerance provides precise recovery: snapshots capture the complete state, and restarts from a consistent point after a failure, guaranteeing exactly-once semantics. Public comparisons show that Flink's Checkpoint is superior to Spark Streaming's WAL (Write-Ahead Log). This embodiment further optimizes this by reducing recovery time to the second level through "distributed snapshot fault tolerance," avoiding business decision errors caused by data backlog.

[0070] High business adaptability reduces integration complexity. Existing parsing technologies require manual configuration of model mappings, resulting in issues such as inconsistent standards for measurement data from different disciplines and complex integration methods. This embodiment achieves "preliminary conversion of multi-source data models" through stateful processing of built-in business logic (such as SG-CIM model conversion). Advantages include: automated parsing reduces manual intervention, configurable models, and improves the matching degree between shared data and application requirements to over 90%. Compared to IoT data collection and measurement centers, this invention directly solves the problem of "raw data not associated with ledgers," shortening the data application chain.

[0071] It boasts strong scalability, supporting massive data scenarios. While Real-Time Data Warehouse 2.0 incorporates Flink, it suffers from the issue of "traffic splitting requiring manual operation." This embodiment's layered windowing and resource quota mechanisms (such as message topic hot / cold strategies) support dynamic expansion. Its advantages lie in achieving "tens of thousands of conversions per second" through parallel computing (such as multi-core technology), adapting to the data growth brought about by new energy access. This embodiment is specifically designed for tens of millions of terminals in the power grid, improving resource utilization by more than 50%.

[0072] This embodiment reduces the manpower costs of repetitive governance through standardized parsing; its fault-tolerance mechanism reduces operation and maintenance expenses, aligning with the goal of "lean management." Compared to existing systems, it is expected to save more than 20% in the construction costs of various professional information systems.

[0073] like Figure 2 As shown, the present invention also discloses a real-time analysis device for heterogeneous power data, comprising:

[0074] The access module 201 is used to obtain real-time heterogeneous power data from several power grid measurement data sources; and to use the Kafka distributed message queue to connect the real-time heterogeneous power data to the stream computing engine.

[0075] Parsing module 202 is used to route, process stateful events, and clean heterogeneous power data using the stream computing engine, as follows:

[0076] Routing routes heterogeneous power data to the corresponding resolvers according to their business domains;

[0077] The event handling process involves a parser that identifies event types in heterogeneous power data. A state machine triggers the corresponding state handling flow for each event type, obtaining parsed standard format data and updating the corresponding state in the state machine. The state machine's state includes data mapping, parsing progress, and business context. The data mapping stores data format mapping rules. The parsing progress records the data parsing progress. The business context records the topological relationship between equipment and ledgers, and / or the event dependencies of continuous measurement data.

[0078] The data is cleaned by using a window manager to segment the parsed standard format data at intervals, filtering and deduplicating the segmented data to obtain the parsed data.

[0079] Output module 203 is used to output the parsed data, and to perform model transformation or distributed storage on the parsed data.

[0080] The specific implementation of a real-time analysis device for heterogeneous power data and the same real-time analysis method for heterogeneous power data will not be described again in this embodiment.

[0081] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 3 As shown, the electronic device in this embodiment includes a processor 301 and a memory 302; wherein the memory 302 is used to store computer execution instructions; and the processor 301 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0082] Alternatively, the memory 302 can be either standalone or integrated with the processor 301.

[0083] When the memory 302 is set up independently, the electronic device also includes a bus 303 for connecting the memory 302 and the processor 301.

[0084] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 301, implement the above method.

[0085] This invention also provides a computer program product, including a computer program that, when executed by a processor 301, implements the above-described method.

[0086] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0087] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0088] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0089] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 301 to execute some steps of the methods of the various embodiments of this application.

[0090] It should be understood that the processor 301 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 301 can be any conventional processor 301. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 301 executing the steps, or as a combination of hardware and software modules within the processor 301 executing the steps.

[0091] The memory 302 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0092] Bus 303 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 303 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 303 in the accompanying drawings of this application is not limited to only one bus 303 or one type of bus 303.

[0093] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0094] An exemplary storage medium is coupled to processor 301, enabling processor 301 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of processor 301. Processor 301 and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, processor 301 and storage medium can exist as discrete components in an electronic device or host device.

[0095] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0096] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A method for real-time analysis of heterogeneous power data, characterized in that, Includes the following steps: S1: Obtain real-time heterogeneous power data from several power grid measurement data sources; use Kafka distributed message queue to connect the real-time heterogeneous power data to the stream computing engine; S2 uses a stream computing engine to route, process stateful events, and clean heterogeneous power data, as follows: Routing routes heterogeneous power data to the corresponding resolvers according to their business domains; Events are handled in a state. The parser identifies the event type in the heterogeneous power data, and the state machine triggers the state handling process corresponding to the event type to obtain the parsed standard format data and update the corresponding state in the state machine. The state machine states include data mapping, parsing progress, and business context; the data mapping stores data format mapping rules; the parsing progress records the data parsing progress; the business context records the topological relationship between the device and the ledger and / or the event dependencies of continuous measurement data. The data is cleaned by using a window manager to segment the parsed standard format data at intervals, and then filtering and deduplicating the segmented data to obtain the parsed data. S3 outputs the parsed data, and performs model transformation or distributed storage on the parsed data.

2. The real-time parsing method for heterogeneous power data according to claim 1, characterized in that, The S2 further includes: using distributed snapshot fault tolerance for fault recovery; specifically: using a fault tolerance controller to periodically generate consistent snapshots among distributed nodes, storing the consistent snapshots in a distributed file system, and using the most recent consistent snapshot stored in the file system to recover data when a node failure occurs.

3. The real-time analysis method for heterogeneous power data according to claim 2, characterized in that, The frequency of periodically generating consistent snapshots is 10-15 seconds.

4. The real-time parsing method for heterogeneous power data according to claim 1, characterized in that, S1 further includes: using a priority resource allocation mechanism to allocate computing resources, allocating more computing resources and bandwidth to high-priority data.

5. The method for real-time parsing of heterogeneous power data according to claim 1, characterized in that, The step of using a window manager to segment the parsed standard format data at intervals involves: segmenting the parsed standard format data at intervals using time windows; the time windows include a first-level window for processing high-frequency data and a second-level window for aggregating event data; the first time window is segmented at second-level intervals; the second time window is segmented at minute-level intervals; when segmenting at intervals, the first-level window or the second-level window is dynamically selected based on the business scenario of the heterogeneous power data.

6. The real-time parsing method for heterogeneous power data according to claim 1, characterized in that, The data format mapping rule is the conversion logic from JSON to SG-CIM.

7. A real-time analysis device for heterogeneous power data, characterized in that, Including the following: The access module is used to obtain real-time heterogeneous power data from several power grid measurement data sources; and uses the Kafka distributed message queue to connect the real-time heterogeneous power data to the stream computing engine. The parsing module is used to route, process stateful events, and clean heterogeneous power data using the stream computing engine, as follows: Routing routes heterogeneous power data to the corresponding resolvers according to their business domains; Events are handled in a state. The parser identifies the event type in the heterogeneous power data, and the state machine triggers the state handling process corresponding to the event type to obtain the parsed standard format data and update the corresponding state in the state machine. The state machine states include data mapping, parsing progress, and business context; the data mapping stores data format mapping rules; the parsing progress records the data parsing progress; the business context records the topological relationship between the device and the ledger and / or the event dependencies of continuous measurement data. The data is cleaned by using a window manager to segment the parsed standard format data at intervals, and then filtering and deduplicating the segmented data to obtain the parsed data. The output module is used to output the parsed data, and to perform model transformation or distributed storage on the parsed data.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, cause a computer to perform the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images