Distributed real-time data processing method and device and storage medium

Abstract

The invention discloses a distributed real-time data processing method and device and a storage medium, and belongs to the field of electric power Internet of Things data, and the method comprises the steps: collecting multi-source data from the electric power Internet of Things in real time, packaging the multi-source data into standard format data, and asynchronously pushing the standard format data to a distributed message queue; asynchronously pulling the standard format data from the distributed message queue, distributing the standard format data to a predefined real-time data processing task, splitting the real-time data processing task into a plurality of sub-tasks, executing each sub-task in parallel through a plurality of processing nodes, and generating a processing result; and allocating a unique identifier to the processing result, performing de-duplication verification, pushing the processing result passing the verification to a subscription terminal or a service system in real time, and asynchronously writing the processing result into a database or discarding the processing result according to the service type. Therefore, the timeliness and the resource utilization rate of data processing of the power internet of things can be improved by implementing the method and the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of power internet of things data, and in particular to a distributed real-time data processing method and device and storage medium. BACKGROUND

[0002] In the field of power internet of things data processing, the traditional data processing method is mainly based on batch processing architecture, and the specific steps are as follows: a large amount of measurement data such as current, voltage, power data and device state data is collected by a data acquisition terminal, and then written into a relational database, and the data needs to be completely stored before triggering subsequent data processing events; after a period of storage, a batch of data is processed within a certain time period, and the "accumulated complete batch of data" is read from the database, and data cleaning, statistical analysis and report generation are performed; after the operation is completed, the results are written back to the database or static reports are generated, and operation and maintenance personnel and dispatch personnel can view the results offline.

[0003] The traditional data batch processing method has the following defects: the real-time performance is seriously insufficient, and cannot support dynamic monitoring and fault disposal of the power grid; the dependence on the database is too high, and there is a waste of resources and performance bottleneck; the adaptability to the scene is poor, and for the power internet of things, it cannot meet the dynamic index calculation and real-time interactive response. SUMMARY

[0004] The present application provides a distributed real-time data processing method, device and storage medium, which can improve the timeliness and resource utilization of power internet of things data processing.

[0005] The present application provides a distributed real-time data processing method, device and storage medium, which can improve the timeliness and resource utilization of power internet of things data processing. The present application provides a distributed real-time data processing method, device and storage medium, which can improve the timeliness and resource utilization of power internet of things data processing. The present application provides a distributed real-time data processing method, device and storage medium, which can improve the timeliness and resource utilization of power internet of things data processing. The present application provides a distributed real-time data processing method, device and storage medium, which can improve the timeliness and resource utilization of power internet of things data processing.

[0006] This invention achieves efficient data access by asynchronously pushing multi-source data collected in real time from the power Internet of Things (IoT) into a standard format and then encapsulating it. It also improves data processing efficiency and system resource utilization by asynchronously pulling data from the message queue and distributing it to multiple processing nodes for parallel execution of subtasks. Furthermore, it avoids duplicate data output and resource waste by deduplicating and storing the processing results as needed. Compared to existing technologies that suffer from insufficient real-time performance and resource waste, this application improves the timeliness and resource utilization of power IoT data processing.

[0007] Furthermore, the encapsulation of the multi-source data into standard format data includes: The collected multi-source data is parsed and cleaned, and then encapsulated into standard format data according to the business type; wherein the standard format data includes the business identifier field corresponding to the multi-source data.

[0008] The embodiments of the present invention can improve the standardization and consistency of data by parsing, cleaning and encapsulating multi-source data into standard format data.

[0009] Furthermore, after collecting multi-source data from the power Internet of Things in real time, encapsulating the multi-source data into standard format data, and asynchronously pushing the standard format data to a distributed message queue, the method further includes: Based on the business identifier field, each of the standard format data is divided into a corresponding business theme, and each business theme is split into multiple partitions; wherein each partition is configured with multiple copies.

[0010] This invention, by dividing data into different topics and storing partitioned copies based on the business identifier field, can avoid data loss caused by single-node failure.

[0011] Furthermore, the step of dividing the real-time data processing task into multiple sub-tasks and executing each sub-task in parallel through multiple processing nodes includes: Based on the regional correlation of power Internet of Things data, each subtask is scheduled to the corresponding regional processing node for calculation.

[0012] The embodiments of the present invention can reduce data transmission latency and improve data processing efficiency and localization capabilities by scheduling subtasks to corresponding processing nodes based on regional correlation.

[0013] Furthermore, the deduplication verification includes: Retrieve the unique identifier corresponding to the processing result from the memory cache; If the unique identifier is found in the memory cache, the verification result is a failure and the processing result is discarded; If the unique identifier is not found in the memory cache, then a query is performed in the deduplication database; If the unique identifier is found in the deduplication database, the verification result is failed and the processing result is discarded. At the same time, the unique identifier is recorded in the memory cache. If the unique identifier is not found in the deduplication database, the verification result is passed, and the unique identifier is recorded in the memory cache and the deduplication database.

[0014] The embodiments of the present invention utilize a dual verification mechanism of memory caching and deduplication database to efficiently identify duplicate data, avoid duplicate output of results, and improve the accuracy of data processing.

[0015] Furthermore, during the data push and task processing process, real-time monitoring is performed from the data link dimension, node dimension, and business task dimension. When any monitoring indicator in any dimension is abnormal, a corresponding early warning signal is generated, and control strategies are dynamically executed according to the early warning level of the early warning signal.

[0016] The embodiments of the present invention perform multi-dimensional real-time monitoring and dynamic control during data push and task processing, which can promptly detect and respond to system anomalies and ensure the stable operation of the processing link.

[0017] Furthermore, the system performs real-time monitoring from the data link dimension, node dimension, and business task dimension. When any monitoring indicator in any dimension shows an anomaly, a corresponding early warning signal is generated, including: The data accumulation amount and data input rate of the distributed message queue are compared with a preset first threshold and a second threshold, respectively, to generate a first comparison result and a second comparison result; if the first comparison result does not meet the first condition or the second comparison result does not meet the first condition, a data link dimension warning signal is generated. The CPU utilization, memory utilization, and task execution latency of the processing node are compared with preset third, fourth, and fifth thresholds, respectively, to generate third comparison results, fourth comparison results, and fifth comparison results; if the third comparison result does not meet the third condition, the fourth comparison result does not meet the fourth condition, or the fifth comparison result does not meet the fifth condition, a node-level warning signal is generated. The task priority weight of the real-time data processing task is compared with a preset sixth threshold to generate a sixth comparison result; if the sixth comparison result does not meet the sixth condition, a business task dimension warning signal is generated. The warning levels for the warning signals include mild, moderate, and severe warnings.

[0018] The embodiments of the present invention can achieve refined monitoring and anomaly identification of system status through multi-dimensional threshold comparison and early warning signal generation mechanism.

[0019] Furthermore, the step of dynamically executing the control strategy based on the warning level of the warning signal includes: When the warning level of the warning signal is a mild warning, the mild warning control strategy is executed first; if the warning signal is not lifted after the mild warning control strategy is executed, the moderate warning control strategy is executed; if the warning signal is not lifted after the moderate warning control strategy is executed, the severe warning control strategy is executed. When the warning level of the warning signal is moderate, the moderate warning control strategy shall be executed first; if the warning signal is not lifted after the moderate warning control strategy is executed, the severe warning control strategy shall continue to be executed. When the warning level of the warning signal is a severe warning, the severe warning control strategy shall be implemented. When the warning signal is lifted, each control strategy is lifted in the reverse order of execution of the previously executed control strategies.

[0020] The embodiments of the present invention can achieve elastic resource scheduling by dynamically executing and gradually releasing control strategies according to the warning level.

[0021] Another embodiment of the present invention provides a distributed real-time data processing device, including: a data acquisition module, a data processing module, and a result deduplication module; The data acquisition module is used to collect multi-source data from the power Internet of Things in real time, encapsulate the multi-source data into standard format data, and asynchronously push the standard format data to a distributed message queue. The data processing module is used to asynchronously pull the standard format data from the distributed message queue, allocate the standard format data to a predefined real-time data processing task, and split the real-time data processing task into multiple sub-tasks. The sub-tasks are executed in parallel by multiple processing nodes to generate processing results. The result deduplication module is used to assign a unique identifier to the processing result and perform deduplication verification. The processing result that passes the verification is pushed to the subscription terminal or business system in real time, and the processing result is asynchronously written to the database or discarded according to the business type.

[0022] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of a distributed real-time data processing method as described in the present invention. Attached Figure Description

[0023] Figure 1 A flowchart illustrating an embodiment of the distributed real-time data processing method provided by the present invention; Figure 2 A flowchart illustrating one embodiment of data accumulation monitoring provided by the present invention; Figure 3 A flowchart illustrating an embodiment of data input rate monitoring provided by the present invention; Figure 4 A flowchart illustrating one embodiment of the deduplication verification provided by the present invention; Figure 5 A schematic diagram of one embodiment of the distributed streaming data processing architecture provided by the present invention; Figure 6 A flowchart illustrating an embodiment of the distributed streaming data processing architecture provided by the present invention for performing real-time data processing tasks; Figure 7 A flowchart illustrating another embodiment of the distributed streaming data processing architecture provided by the present invention for performing real-time data processing tasks; Figure 8 A flowchart illustrating another embodiment of the distributed streaming data processing architecture provided by the present invention for performing real-time data processing tasks; Figure 9 This is a schematic diagram of one embodiment of the distributed real-time data processing device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0026] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0029] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0030] See Figure 1 To address the issues of insufficient real-time performance and resource waste in existing technologies, an embodiment of the present invention provides a distributed real-time data processing method, including steps S101 to S103: Step S101: Collect multi-source data in real time from the power Internet of Things, encapsulate the multi-source data into standard format data, and asynchronously push the standard format data to a distributed message queue.

[0031] It should be noted that the process of collecting multi-source data in real time from the power Internet of Things (IoT), encapsulating the multi-source data into standard format data, and asynchronously pushing the standard format data to a distributed message queue refers to the continuous acquisition of various types of raw data, such as current, voltage, power, and equipment status, from data-generating nodes in the power IoT, such as substation sensors, various distribution smart terminals, or user-side smart meters, through the data acquisition layer. Subsequently, the collected raw data is parsed, cleaned, and format-converted, and a unified business identifier field is added according to preset business rules, thereby encapsulating it into standard structured data that can be identified and processed internally by the system. Finally, without blocking the data acquisition link, the encapsulated standard format data is sent to the message transmission layer (such as a distributed message queue like Kafka) for buffering and storage, achieving decoupling between the data production and consumption ends and providing a stable and reliable data source for subsequent asynchronous parallel processing.

[0032] Preferably, the step of encapsulating the multi-source data into standard format data includes: The collected multi-source data is parsed and cleaned, and then encapsulated into standard format data according to the business type; wherein the standard format data includes the business identifier field corresponding to the multi-source data.

[0033] Specifically, multi-source data can be measurement data such as current, voltage, or power; equipment status data such as switch status or equipment temperature; or fault signal data such as short-circuit warnings or overload indicator lights. In particular, during multi-source data acquisition, the acquisition agent does not store the data; it simply performs standard format conversion and immediately pushes the data to a distributed message queue.

[0034] Preferably, after collecting multi-source data from the power Internet of Things in real time, encapsulating the multi-source data into standard format data, and asynchronously pushing the standard format data to a distributed message queue, the method further includes: Based on the business identifier field, each of the standard format data is divided into a corresponding business theme, and each business theme is split into multiple partitions; wherein each partition is configured with multiple copies.

[0035] Specifically, business topics can be divided into measurement data, fault signal data, or equipment status data, etc. Each business topic can be further divided into multiple partitions to achieve parallel data transmission. Simultaneously, each partition is configured with 3-5 replicas, distributed across physical nodes of the backend support layer hardware devices in different power IoT architectures. This avoids data loss due to single-node failures, meeting the core requirement of "no data loss" in the power system, thus realizing the partitioning and replication design. In particular, after pushing data to a distributed message queue, there is no need to wait for confirmation before collecting the next batch of data.

[0036] Step S102: Asynchronously pull the standard format data from the distributed message queue, allocate the standard format data to a predefined real-time data processing task, and split the real-time data processing task into multiple sub-tasks. Execute each sub-task in parallel through multiple processing nodes to generate processing results.

[0037] It should be noted that asynchronously pulling the standard format data from the distributed message queue and allocating the standard format data to predefined real-time data processing tasks means that the distributed processing layer actively retrieves data from the distributed message queue according to the business identifier and assigns it to the corresponding real-time computing tasks defined in the system according to the data characteristics for execution.

[0038] It should be noted that breaking down the real-time data processing task into multiple subtasks and executing each subtask in parallel through multiple processing nodes means decomposing a complete real-time processing task into multiple independent computing units based on data partitioning or computing logic, dynamically scheduling these units to different processing nodes in the cluster for simultaneous execution, and finally aggregating the outputs of each node to form a complete processing result.

[0039] Preferably, the step of dividing the real-time data processing task into multiple sub-tasks and executing each sub-task in parallel through multiple processing nodes includes: Based on the regional correlation of power Internet of Things data, each subtask is scheduled to the corresponding regional processing node for calculation.

[0040] Specifically, real-time data processing tasks are broken down into multiple subtasks and allocated to different nodes according to a "data partition-processing node" mapping rule to ensure load balancing. Simultaneously, based on the regional correlation of power data, data partitions are bound to processing nodes geographically to reduce cross-regional data transmission and lower network latency. In particular, the distributed processing layer asynchronously pulls data from the message queue based on its own processing capacity, reducing inter-node dependencies. Even if a node in the processing layer experiences a temporary failure, the message queue can still cache data, allowing processing to continue once the node recovers, thus avoiding data link interruptions.

[0041] Step S103: Assign a unique identifier to the processing result and perform deduplication verification. Push the verified processing result to the subscription terminal or business system in real time, and write the processing result to the database asynchronously or discard it according to the business type.

[0042] It should be noted that assigning a unique identifier to the processing result and performing deduplication verification means generating a unique identifier (such as a structured UUID) for each generated processing result, and building a multi-level verification mechanism based on memory cache and persistent database. By querying whether the identifier already exists, duplicate results are accurately judged and filtered to ensure the uniqueness and accuracy of the final output data.

[0043] It should be noted that pushing the verified processing results to the subscribed terminal or business system in real time, and asynchronously writing the processing results to the database or discarding them according to the business type, means that the deduplicated and verified valid processing results are immediately distributed to the pre-subscribed terminal users or downstream business applications. At the same time, based on the business attributes and value of the result data, data that needs to be retained for a long time is arranged to be asynchronously stored in the database, while intermediate or low-value data that does not need to be retained is directly cleared, thereby achieving efficient use of resources.

[0044] Preferably, the deduplication verification includes: Retrieve the unique identifier corresponding to the processing result from the memory cache; If the unique identifier is found in the memory cache, the verification result is a failure and the processing result is discarded; If the unique identifier is not found in the memory cache, then a query is performed in the deduplication database; If the unique identifier is found in the deduplication database, the verification result is failed and the processing result is discarded. At the same time, the unique identifier is recorded in the memory cache. If the unique identifier is not found in the deduplication database, the verification result is passed, and the unique identifier is recorded in the memory cache and the deduplication database.

[0045] In one embodiment, a complete workflow design of "unique identifier (UUID) generation - deduplication verification - duplicate message processing" is used to solve the problem of "duplicate critical messages" caused by issues such as "data retransmission due to network jitter," "retrying to send data at processing nodes," or "duplicate subscriptions by business systems." The detailed implementation process is as follows... Figure 4 As shown, it includes: Step 1: Define the scope of "critical information" requiring deduplication, specifically including "fault alarms," ​​"compliance verifications," and "user-customized results." Fault alarms include short-circuit alarms, over-temperature alarms, and abnormal switch status alarms. These messages directly trigger fault handling processes (such as work order generation and maintenance dispatch), and duplicate pushes will lead to duplicate dispatches. Compliance verifications include records of measurement data inaccuracy and abnormal records of business data not conforming to the DL / T1500-2016 protocol specifications. These messages need to be archived for subsequent auditing; duplicate writing will consume additional storage resources. User-customized results include "regional peak-valley load difference" results customized by dispatchers and "average daily operating efficiency of equipment" results customized by the maintenance team. These messages need to be pushed accurately; duplicate pushes will interfere with user judgment.

[0046] Step Two: Formulating Rules for Generating Unique Identifiers (UUIDs). Based on the business attributes of critical messages, a "structured UUID" is designed to ensure that the UUID for the same message is unique, and that UUIDs for different messages are not duplicated. It also includes "traceability information" to facilitate subsequent troubleshooting and message tracing. The UUID format adopts a "fixed-length 32-bit character + business identifier segment" design, with the following structure: Business type code (2 bits) - Message source identifier (8 bits) - Timestamp (13 bits) - Random sequence (7 bits) - Check digit (2 bits), for a total length of 32 bits, for example, 01-0A3B7D9C-1698765432109-8F2E4A1-3C. After critical messages are computed in the distributed processing layer and before being sent to the message transmission layer, the "UUID generation module" of the processing node generates a UUID in real time. After generating the UUID, the processing node temporarily stores the "UUID-message content-generation time" in the local "generated message cache table" (in-memory type, valid for 1 hour) and sends the message (including the UUID) to the message transmission layer. If message transmission fails (such as network interruption), the processing node can resend the message through the cache table to avoid message loss.

[0047] Step 3: Deduplication Validation Mechanism. A "Deduplication Validation Center" is set up in the result output layer as the "only gate" before key messages are pushed to the business system / terminal. Through "multi-level caching + database validation," it ensures that duplicate messages are accurately identified. The specific process is as follows: First, fast matching via memory cache. The Deduplication Validation Center maintains a "real-time deduplication cache" (implemented based on Redis, memory-based, with an expiration time of 30 minutes). The cache key is "UUID," and the cache value is "message processing status" (processed / pending). When the result output layer receives a key message (containing the UUID), it first queries the real-time deduplication cache: if the UUID exists in the cache and the status is "processed," it is determined to be a duplicate message, and the "duplicate message processing" process is directly executed; if the UUID does not exist in the cache, or the status is "pending," it enters the second-level validation. Second, persistent database verification is implemented. The system deploys a "deduplication database" (using MySQL with a table partitioning strategy, partitioned by "business type code") to store information such as "UUID-message content-processing time-processing status" for long-term verification and traceability. After entering the second-level verification, the result output layer sends a query request to the deduplication database (SQL statement: SELECT processing status FROM deduplication table_business type WHERE EUUID=?). If the database query result is "processed", it is determined to be a duplicate message, and the real-time deduplication cache is updated (set). If the UUID status is "processed", the "duplicate message processing" process is executed. If the database query result is "no record", it is determined to be a new message, and the "new message processing" process (updating the real-time deduplication cache and database) is executed. Table partitioning strategy: Divide into 3 physical tables according to "business type code" (01-fault alarm, 02-compliance verification, 03-user customization), and each table is partitioned according to "time range" (e.g., one sub-table per month) to improve query efficiency. For example, fault alarm messages in October 2024 are stored in "deduplication table_01_202410". Third, exception handling: if the real-time deduplication cache query times out, it directly enters the second-level verification to ensure that the verification is not interrupted. If the deduplication database query times out, the message is temporarily stored in the "verification pending queue" (implemented based on Kafka), and the verification is re-initiated after the database is restored to avoid message loss. If the UUID checksum error is found during the verification process, it is marked as "invalid message", discarded and written to the error log, and an alarm notification is sent to the technical contact.

[0048] Step 4: Based on the deduplication verification results, execute the "New Message Processing" and "Duplicate Information Processing" processes respectively. The processing flow for new messages or new data is as follows: First, update the real-time deduplication cache, setting the UUID status to "Processed" with an expiration time of 30 minutes. Second, write to the deduplication database by executing the INSERT statement (INSERTINTO Deduplication Table_Business Type (UUID, Message Content, Processing Time, Processing Status) VALUES (?,?,NOW(),'Processed')), simultaneously starting a database transaction to ensure the atomicity of "cache update - database write" and avoid data inconsistency. Third, push messages to target terminals / business systems according to message type: fault alarm messages are pushed to the dispatch center dashboard and maintenance personnel's mobile terminals (such as mobile apps) via WebSocket, while simultaneously calling the fault handling system API interface to trigger work order generation; compliance verification messages are pushed to the compliance audit system via HTTP POST requests, while asynchronously writing to the time-series database (for long-term archiving); user-customized result messages are pushed according to the user-configured push method (email / SMS / UI interface), for example: sending a "regional peak-valley load difference" result email to dispatchers, while simultaneously updating the data in the user-customized interface. Fourth, record a "message processing log", including UUID, processing time, push target, and push result (success / failure), for easy follow-up. The duplicate message handling process is as follows: First, record a "duplicate message log," including the UUID, reception time, and number of times the message was received (the cumulative number of times the UUID was received repeatedly), to avoid meaningless log redundancy; Second, determine the number of times the message was received: if the number of times the message was received is ≤3, no push is performed, only the duplicate message log is updated to avoid duplicate pushes caused by slight network jitter; if the number of times the message was received is >3, it is determined to be "abnormal duplicate," and an alarm notification (SMS + email) is sent to the technical contact person, prompting them to investigate the cause (such as message transmission link failure, duplicate subscription in the business system); Third, do not update the real-time deduplication cache and database (to avoid overwriting the original records), and directly discard the duplicate message, without affecting the processing of subsequent new messages.

[0049] Specifically, data with long-term analytical value, such as fault handling records and monthly equipment operation statistics, can be written to a time-series database adapted to the characteristics of power time-series data. This type of data can be used for subsequent fault tracing, equipment lifecycle analysis, and other scenarios. For real-time measurement data with low value density, such as current and voltage, it is only temporarily stored in memory during distributed task processing. After the indicator calculation is completed, it is discarded if it does not need to be retained long-term, significantly reducing the consumption of storage resources and avoiding the performance loss caused by writing meaningless data to the database. In particular, data to be stored is transmitted to the database asynchronously, allowing processing nodes to continue processing the next batch of data without waiting for database write confirmation.

[0050] Preferably, during the data push and task processing process, real-time monitoring is performed from the data link dimension, node dimension, and business task dimension. When any monitoring indicator in any dimension is abnormal, a corresponding early warning signal is generated, and a control strategy is dynamically executed according to the early warning level of the early warning signal.

[0051] It should be noted that real-time monitoring from the perspectives of data link, node, and business task during data push and task processing refers to continuously tracking and collecting indicators of data flow status, computing node resource status, and business task execution status simultaneously, so as to achieve perception of the operational status of the entire data processing link.

[0052] It should be noted that when any monitoring indicator in any dimension is abnormal, a corresponding early warning signal is generated. Based on the warning level of the early warning signal, the dynamic execution of the control strategy means comparing the real-time collected monitoring indicators with preset thresholds. Once anomalies such as data accumulation, node overload, or task timeout are detected, a mild, moderate, or severe early warning is automatically generated. Subsequently, based on the severity of the early warning, corresponding control measures are dynamically triggered and executed, such as task migration, upstream rate limiting, or resource expansion, to achieve precise and adaptive system load management.

[0053] Preferably, the real-time monitoring is performed from the data link dimension, node dimension, and business task dimension. When any monitoring indicator in any dimension is abnormal, a corresponding early warning signal is generated, including: The data accumulation amount and data input rate of the distributed message queue are compared with a preset first threshold and a second threshold, respectively, to generate a first comparison result and a second comparison result; if the first comparison result does not meet the first condition or the second comparison result does not meet the first condition, a data link dimension warning signal is generated. The CPU utilization, memory utilization, and task execution latency of the processing node are compared with preset third, fourth, and fifth thresholds, respectively, to generate third comparison results, fourth comparison results, and fifth comparison results; if the third comparison result does not meet the third condition, the fourth comparison result does not meet the fourth condition, or the fifth comparison result does not meet the fifth condition, a node-level warning signal is generated. The task priority weight of the real-time data processing task is compared with a preset sixth threshold to generate a sixth comparison result; if the sixth comparison result does not meet the sixth condition, a business task dimension warning signal is generated. The warning levels for the warning signals include mild, moderate, and severe warnings.

[0054] Specifically, based on the business characteristics of power Internet of Things data processing, key indicators such as the data accumulation volume and data input rate of the distributed message queue of the data link, the CPU utilization, memory utilization and task execution latency of each processing node in the link are monitored, covering the entire process of "data link-computing node-business task", which can effectively solve problems such as "mismatch between data input rate and node processing capacity", "competition for resources between different business tasks" and "sudden data surge".

[0055] In one embodiment, the parameters at the data link dimension mainly focus on the operational status of the distributed message queue (Kafka). Data backlog refers to the difference between the offset (message sequence number) written by the producer of a Kafka partition and the offset consumed by the consumer; a larger difference indicates more severe backlog. By acquiring the offsets from both ends in real time and calculating the difference, combined with the partition's preset capacity, a warning signal is determined to be triggered. In this embodiment, a warning is triggered when the data backlog exceeds 60% of the partition capacity; that is, the first threshold is 60% of the partition capacity, and the first condition is that the data backlog is less than or equal to 60% of the partition capacity. The data backlog monitoring process is as follows: Figure 2 As shown, it includes: Step 1: Deploy the "KafkaOffset Collector". Employ a "master-slave collector backup mechanism" by deploying two collectors. When the master collector is working normally, the slave collector only synchronizes data and does not send alerts. When the master collector fails, the slave collector automatically switches to master mode after a certain period to ensure uninterrupted offset collection.

[0056] Step 2: Establish a connection with the Kafka cluster via the Kafka client API. On the production side, the `listOffsets()` method is called to obtain the "latest write offset" (i.e., `logEndOffset`) for each partition of each business topic (measuring data, fault signals). On the consumer side, the `listConsumerGroupOffsets()` method is called to obtain the "consumed offset" (i.e., `committedOffset`) for each partition by the distributed processing layer consumer group.

[0057] Step 3: Calculation and Threshold Comparison of Backlogged Data. For each partition, the real-time backlogged data volume is calculated using the formula: Backlogged Data = Latest Write Offset - Consumed Offset. Simultaneously, the partition's preset capacity is obtained via the Kafka API, with a preset maximum message storage capacity of 100,000 messages per partition. The backlogged rate is calculated as: Backlogged Data / Partition Capacity × 100%. If the backlogged rate exceeds 60% (the warning threshold), the partition is immediately marked as "warning status," and the warning information (partition ID, Topic name, and backlogged rate) is pushed to the automated flow control module.

[0058] Step 4: In a certain partition, due to network interruption, the consumer node failure causes "the consumer offset to not update for a long time". The time threshold is preset to 500ns, but it can be redefined according to the actual situation. Even if the current backlog rate does not reach the threshold, the monitoring module will trigger "potential backlog warning" to avoid large-scale backlog caused by consumption stagnation.

[0059] In one embodiment, the data link dimension parameters also include the data input rate. The data input rate, based on acquisition agent log statistics and KafkaTopic traffic analysis, refers to the number of valid messages pushed by the data acquisition layer to the Kafka message queue per unit time. Data is measured separately by business type and fault signals to avoid data confusion between different businesses that could lead to inaccurate rate judgments. The data input rate monitoring process is as follows: Figure 3 As shown, it includes: Step 1: Real-time counting by the data acquisition agent. A "data transmission counter" is embedded in substation sensors and distribution smart terminals. Each time the data acquisition agent pushes a message to Kafka, the counter accumulates the count based on the "business identifier field" in the message header, distinguishing different business types. The counter uploads the "business type - cumulative number of messages sent" to the "rate statistics center" every 50ms (consistent with the monitoring frequency), and resets the count within the specified period after uploading to ensure statistical accuracy.

[0060] Step Two: Kafka-side Traffic Verification. The rate statistics center simultaneously uses Kafka's native `kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec` metric (Kafka's built-in JMX monitoring metric) to obtain the "number of messages written per second" for each topic and compares it with the rate data uploaded by the data collection agent. If the deviation exceeds 5% (data collection agent statistics measure a rate of 1000 messages / second, while Kafka's statistics show 940 messages / second), it is determined to be "data loss or statistical anomaly," and an alert is immediately sent to the technical contact person.

[0061] Step 3: Rate Anomaly Identification. The rate statistics center calculates the "average input rate within 50ms" (number of messages sent within 50ms × 20, converted to messages / second) according to service type. If the rate of a certain service type suddenly exceeds twice the historical average (the measured data shows a normal rate of 500 messages / second, but a sudden increase to 1200 messages / second), it is marked as "abnormal rate fluctuation" and pushed to the flow control module to assess whether rate limiting is required.

[0062] In one embodiment, node-level monitoring is achieved by monitoring CPU utilization, memory utilization, and task execution latency. CPU utilization, collected from operating system monitoring tools and node agents, refers to the activity level of a computing node's CPU cores per unit of time. It is necessary to monitor both the overall utilization of a single node and the utilization of the stream computing engine processes to avoid misjudgments caused by non-business processes consuming resources. The CPU utilization monitoring process includes: Step 1: Node Agent Deployment and Data Collection. Deploy a "Node Monitoring Agent" on each distributed processing node (physical machine or cloud server). The agent collects CPU data in the following ways: calling operating system commands; in Linux systems, execute the command `top-b-n1-p[stream computing engine process ID]` to obtain the "real-time CPU usage percentage" of the stream computing engine process; simultaneously execute the command `mpstat` to obtain the overall CPU utilization of the node (excluding idle processes); the collection frequency is 200ms / time, consistent with the metric monitoring frequency, and after each collection, the "Node ID - Process CPU Utilization - Overall CPU Utilization" is uploaded to the "Resource Monitoring Center".

[0063] Step Two: High Load Determination and Early Warning. The resource monitoring center sets the following determination rules: if the CPU utilization of a single node's stream computing engine process exceeds 85% for 10 consecutive seconds (or the overall CPU utilization of the node exceeds 90% for 10 consecutive seconds), it is determined to be a "high load state". The "10 consecutive seconds" is achieved through a "sliding window counting": data is collected once every 200ms. If the collected values ​​exceed the threshold for 50 consecutive times (50 × 200ms = 10 seconds), an early warning is triggered to avoid false warnings caused by instantaneous peak values.

[0064] Step 3: Cross-node comparative analysis. The resource monitoring center simultaneously calculates the average CPU utilization of all processing nodes in the same region. If a node's utilization exceeds the average by 30% (e.g., if the regional average is 60%, and this node is at 95%), even if it does not reach the 85% threshold, it is marked as "relatively high load," prompting the flow control module to pay attention to the node's load balancing. For Windows Server nodes that may exist in the power IoT, the node monitoring agent obtains the process CPU utilization using the command `tasklist / FI "IMAGENAMEeq [stream computing engine process name]"` to ensure consistent monitoring logic across different operating system environments.

[0065] In one embodiment, memory utilization refers to the heap memory usage of the stream computing engine process (heap memory is the core memory area for running stream computing tasks), combined with the overall physical memory utilization of the node, to prevent heap memory overflow or system memory exhaustion that could lead to node crashes. The memory utilization monitoring process includes: Step 1: Heap Memory Data Collection. When the stream computing engine starts, the JMX monitoring port is opened. The node monitoring agent connects to this port via the javax.management API to obtain core JVM heap memory metrics: used heap memory (HeapMemoryUsage.used); maximum heap memory (HeapMemoryUsage.max, set by the engine startup parameter -Xmx, e.g., -Xmx8g represents a maximum heap memory of 8GB). At a collection frequency of 200ms / time, the heap memory utilization rate is calculated as: used heap memory / maximum heap memory × 100%.

[0066] Step Two: System Memory Verification. The node monitoring agent simultaneously collects overall memory data via operating system commands: `free -m` on Linux systems and `systeminfo` on Windows systems, obtaining "Node Physical Memory Usage" and "Total Physical Memory." The system memory utilization rate is calculated as: (Used Memory / Total Physical Memory) × 100%. If the heap memory utilization rate exceeds 90% (warning threshold), or the system memory utilization rate exceeds 95%, a "Memory Overflow Warning" is triggered and pushed to the flow control module.

[0067] Step 3: Memory Leak Monitoring. The resource monitoring center records the "10-minute moving average" of heap memory usage for each node. If the average continuously increases (e.g., increases by 5% every 5 minutes) without a downward trend, it is determined to be a "potential memory leak," and an alert is sent to the technical contact, prompting them to analyze the stream computing task code (whether there are any unreleased cached objects). When the node monitoring agent collects heap memory data, it avoids the "garbage collection (GC) period" of the stream computing engine: it listens for GC events through JMXAPI, and if GC is in progress when data is collected, it delays for 50ms before collecting, avoiding the impact of memory data fluctuations caused by GC on usage calculation.

[0068] In one embodiment, task execution latency is based on a comparison of task lifecycle logs and timestamps, referring to the total time consumed by a stream computing task from receiving data to outputting results. This latency needs to be statistically analyzed separately for each sub-task type (e.g., anomaly detection, compliance verification). Because the computational complexity of different tasks varies significantly, differentiated business thresholds need to be set. The task execution latency monitoring process includes: Step 1: Task Timestamp Tracking. Timestamp tracking is embedded in the stream processing tasks of the distributed processing layer to record key node times: Data Reception Time (t1): Immediately after the task pulls data from Kafka, the current system time (accurate to milliseconds) is recorded; Result Output Time (t2): When the task completes the computation and pushes the result to the result output layer, the current system time is recorded; Each task instance writes "Task ID - Subtask Type - t1 - t2" to a local "Task Log File," and the log is generated on a 100ms cycle (consistent with the monitoring frequency).

[0069] Step 2: Latency Calculation and Threshold Comparison. Deploy a "Task Latency Analyzer" to read the task logs of all processing nodes at 100ms intervals and calculate the execution latency of each subtask: Latency = t2 - t1. The analyzer has a built-in "Subtask Threshold Dictionary" (e.g., fault detection latency threshold 500ms, compliance verification latency threshold 200ms). If the latency of a subtask exceeds the corresponding threshold, it is marked as "latency exceeded," and the "number of times the threshold is exceeded" for that subtask is counted. If there are 3 consecutive instances of exceeding the threshold, it is pushed to the flow control module.

[0070] Step 3: Preliminary Root Cause Identification of Latency. If a certain type of subtask (such as anomaly detection) generally exceeds latency limits, the analyzer further collects the "intermediate step time" of this type of task (such as model loading time, data preprocessing time) to determine whether the latency exceedance is due to "complex computational logic" or "insufficient resources," providing optimization directions for the flow control module (such as task migration, resource expansion). Task logs adopt a dual mechanism of "local storage + remote backup": local logs are stored on the node disk (retained for 24 hours), and simultaneously synchronized to a remote log server in real time via FTP to avoid log loss due to node failure, which would affect latency analysis.

[0071] In one embodiment, monitoring at the business task level refers to the dynamic management of task priorities. The core monitoring focuses on "task priority weights," which are precisely controlled and updated in real-time through a "static configuration library + dynamic adjustment rule engine," ensuring that core business tasks receive resources preferentially. The task priority weight monitoring process includes: Step 1: Deploy the "Task Configuration Center" and maintain a "Static Weight Configuration Library" based on MySQL. The library stores information such as "Task Type - Basic Weight - Applicable Scenarios" (e.g., a basic weight of 10 for fault alarm tasks and 3 for regular statistics tasks). The configuration center provides an HTTP API interface. All task instances in the distributed processing layer call the API every 1 minute (the static weight update frequency is low, so high-frequency monitoring is not required) to read their corresponding basic weights, ensuring consistent weights across all task instances.

[0072] Step 2: Deploy the "Dynamic Rule Engine," which has a built-in "Grid Status - Adjustment Rule" mapping table and receives "Grid Operation Status Data" from the power Internet of Things in real time. The "Grid Status - Adjustment Rule" mapping table is shown in Table 1.

[0073] Table 1 - Mapping Table of "Power Grid Status - Adjustment Rules" Step 3: After receiving the dynamically adjusted increment, the task instance calculates the final weight (final weight = base weight + dynamically adjusted increment) and reports the "task ID - final weight - calculation time" to the "task weight monitoring center". The monitoring center verifies the weight's rationality in real time: if the final weight of a task exceeds the preset maximum value or falls below the minimum value, it is determined to be "weight abnormal" and immediately sends a "weight reset request" to the task configuration center to restore the task's base weight, preventing abnormal weights from causing resource allocation imbalance. A "version number control" mechanism is adopted: each weight configuration (static + dynamic) is associated with a unique version number, and the task instance carries the version number when reporting weights; if the monitoring center finds that the version numbers of the same task on different instances are inconsistent, it triggers a "weight synchronization instruction", requiring instances with lower versions to re-obtain the latest weights from the configuration center and rule engine to ensure that the weights of all instances are consistent.

[0074] Preferably, the step of dynamically executing the control strategy based on the warning level of the warning signal includes: When the warning level of the warning signal is a mild warning, the mild warning control strategy is executed first; if the warning signal is not lifted after the mild warning control strategy is executed, the moderate warning control strategy is executed; if the warning signal is not lifted after the moderate warning control strategy is executed, the severe warning control strategy is executed. When the warning level of the warning signal is moderate, the moderate warning control strategy shall be executed first; if the warning signal is not lifted after the moderate warning control strategy is executed, the severe warning control strategy shall continue to be executed. When the warning level of the warning signal is a severe warning, the severe warning control strategy shall be implemented. When the warning signal is lifted, each control strategy is lifted in the reverse order of execution of the previously executed control strategies.

[0075] In one embodiment, a corresponding control strategy is triggered based on the warning level of the warning signal to achieve "precise flow control with minimal impact." Specifically, when the warning level is mild, the control strategy is local resource scheduling targeting high single-node load / local queue backlog. This embodiment uses a single processing node CPU utilization of 85%-90%, a single topic partition data backlog of 60%-70%, and a single task execution latency exceeding the threshold by 10%-20% as trigger conditions. The specific control strategy is as follows: the flow control module migrates subtasks with lower loads on the current node to idle nodes (CPU utilization below 60%) according to the "data partition-processing node" mapping rule. During the migration, the task context is synchronized via a message queue to ensure uninterrupted data transmission. If the backlog of a single topic partition exceeds the limit, 1-2 new partitions are automatically added to the topic, and these new partitions are allocated to idle Kafkabroker nodes to achieve parallel data transmission and reduce the backlog pressure on a single partition. For example, when the CPU utilization of the processing node of substation A reaches 88% (mild warning), the system automatically migrates the "Voltage Statistics Task of Substation A" on that node to the idle node of the adjacent substation B. After the migration is completed, the CPU utilization of the substation A node drops to 75%, restoring normal processing capacity.

[0076] In one embodiment, when the warning level is moderate, the control strategy is upstream rate limiting plus temporary caching for high load on multiple nodes / global queue backlog. This embodiment uses the following triggering conditions: CPU utilization of processing nodes exceeding 30% (90%-95%), data backlog of more than two Topic partitions exceeding 70%-80%, and execution latency of core tasks (such as fault detection) exceeding a threshold by 20%-50%. The specific control strategy is as follows: the flow control module sends a "tiered rate limiting instruction" to the data acquisition layer, adjusting the acquisition frequency according to service type: the acquisition frequency of core data (fault signals, switch status) remains unchanged, while the acquisition frequency of non-core data (user-side ordinary power load) is reduced from 100ms / time to 500ms / time, while retaining the "emergency recovery mechanism". If the queue backlog subsequently drops below 50%, a "current limiting release command" is immediately sent to restore the original sampling frequency. A "disk cache pool" (with a preset capacity of twice the node's memory) is enabled on each processing node to temporarily store real-time data exceeding processing capacity on disk, using a "first-in, first-out" (FIFO) strategy to manage the cached data. Once the node load returns to a normal level (CPU utilization below 80%), cached data is read and processed first to avoid data loss. For example, during peak summer demand, a surge in the input rate of power grid measurement data in a certain area caused the CPU utilization of three processing nodes to reach 92% (moderate warning). The system immediately sent a current limiting command to the user-side data acquisition agent in that area, reducing the sampling frequency of ordinary electricity load from 100ms / time to 500ms / time. Simultaneously, each processing node enabled disk caching to temporarily store data exceeding processing capacity. After 10 minutes, the queue backlog decreased from 75% to 45%, the system lifted the current limiting, and the normal sampling frequency was restored.

[0077] In one embodiment, when the warning level is severe, the control strategy is task priority scheduling + resource expansion (for situations where the overall system load is too high / core business is affected). This embodiment uses the following triggering conditions: CPU utilization of more than 50% of processing nodes ≥95%, data accumulation of more than 50% of Topic partitions ≥80%, and core task execution latency exceeding the threshold ≥50%. The specific control strategy is as follows: Activate the "Core Business Assurance Mode," sort tasks by priority weight, suspend non-core tasks with a weight below 5 (monthly equipment operation statistics), and release CPU and memory resources to prioritize core tasks with a weight of ≥8 (such as fault alarms and line overload detection); at the same time, temporarily store the status (processed data volume, calculation progress) of low-priority tasks currently being executed to the storage layer, and resume execution from the breakpoint after the system recovers; if the system is deployed in a cloud environment (power private cloud), the flow control module automatically sends a "resource expansion request" to the cloud platform to add 2-3 processing node instances. After the nodes start, they are automatically registered to the distributed processing layer, and the flow control module allocates some sub-tasks to the new nodes to quickly reduce the load on the original nodes; if the system is deployed on physical machines, send an "emergency expansion notification" (SMS + email) to the administrator, prompting them to manually add physical nodes. For example, when a region experiences heavy rain, multiple line fault signals are reported simultaneously, causing 60% of the processing nodes to have a CPU utilization rate of 96% (severe warning). The system immediately suspends non-core tasks such as "user electricity consumption habit analysis" (weight 3) and prioritizes resources for the "line fault detection" (weight 12) task. At the same time, it requests 3 additional processing nodes from the power private cloud. The new nodes come online after 1 minute, sharing 30% of the fault detection task. After 3 minutes, the overall CPU utilization rate of the system drops to 78%, and the latency of core tasks returns to normal.

[0078] Specifically, mild alert control strategies take precedence over moderate alert control strategies, and moderate alert control strategies take precedence over severe alert control strategies, in order to avoid "excessive flow control". For example, if a mild alert is triggered due to high load on a single node, task migration is executed first. If the load does not decrease after migration and the conditions for a moderate alert are met, then upstream rate limiting and temporary caching are triggered.

[0079] Specifically, the flow control module synchronizes data with the message transmission layer and the distributed processing layer through "heartbeat packets" (100ms / time) to ensure the real-time performance of monitoring indicators. After the control strategy is executed, the flow control module writes "strategy type-execution time-effect data" (changes in input rate after rate limiting, changes in node load after task migration) into the audit log for subsequent optimization.

[0080] Specifically, when the factors that triggered the warning are eliminated (fault signal reporting ends, data input rate drops), the flow control module releases the control strategy in a "reverse order": first, it stops resource expansion / task priority scheduling, then it releases upstream flow limiting / temporary caching, and finally it restores the node task allocation before task migration, ensuring that the system smoothly transitions to a normal state.

[0081] This invention enables efficient data access by asynchronously pushing multi-source data collected in real time from the power Internet of Things into a standard format and then encapsulating it. By asynchronously pulling data from the message queue and distributing it to multiple processing nodes for parallel execution of subtasks, data processing efficiency and system resource utilization can be improved. By deduplicating and storing the processing results as needed, duplicate data output and resource waste can be avoided.

[0082] Optionally, in this embodiment of the invention, encapsulating the multi-source data into standard format data includes: The collected multi-source data is parsed and cleaned, and then encapsulated into standard format data according to the business type; wherein the standard format data includes the business identifier field corresponding to the multi-source data.

[0083] The embodiments of the present invention can improve the standardization and consistency of data by parsing, cleaning and encapsulating multi-source data into standard format data.

[0084] Optionally, in this embodiment of the invention, encapsulating the multi-source data into standard format data includes: The collected multi-source data is parsed and cleaned, and then encapsulated into standard format data according to the business type; wherein the standard format data includes the business identifier field corresponding to the multi-source data.

[0085] The embodiments of the present invention can improve the standardization and consistency of data by parsing, cleaning and encapsulating multi-source data into standard format data.

[0086] Optionally, in this embodiment of the invention, after collecting multi-source data from the power Internet of Things in real time, encapsulating the multi-source data into standard format data, and asynchronously pushing the standard format data to a distributed message queue, the method further includes: Based on the business identifier field, each of the standard format data is divided into a corresponding business theme, and each business theme is split into multiple partitions; wherein each partition is configured with multiple copies.

[0087] This invention, by dividing data into different topics and storing partitioned copies based on the business identifier field, can avoid data loss caused by single-node failure.

[0088] Optionally, in this embodiment of the invention, the step of dividing the real-time data processing task into multiple sub-tasks and executing each sub-task in parallel through multiple processing nodes includes: Based on the regional correlation of power Internet of Things data, each subtask is scheduled to the corresponding regional processing node for calculation.

[0089] The embodiments of the present invention can reduce data transmission latency and improve data processing efficiency and localization capabilities by scheduling subtasks to corresponding processing nodes based on regional correlation.

[0090] Optionally, in this embodiment of the invention, the deduplication verification includes: Retrieve the unique identifier corresponding to the processing result from the memory cache; If the unique identifier is found in the memory cache, the verification result is a failure and the processing result is discarded; If the unique identifier is not found in the memory cache, then a query is performed in the deduplication database; If the unique identifier is found in the deduplication database, the verification result is failed and the processing result is discarded. At the same time, the unique identifier is recorded in the memory cache. If the unique identifier is not found in the deduplication database, the verification result is passed, and the unique identifier is recorded in the memory cache and the deduplication database.

[0091] The embodiments of the present invention utilize a dual verification mechanism of memory caching and deduplication database to efficiently identify duplicate data, avoid duplicate output of results, and improve the accuracy of data processing.

[0092] Optionally, in this embodiment of the invention, during the data push and task processing, real-time monitoring is performed from the data link dimension, node dimension and business task dimension. When any monitoring indicator in any dimension is abnormal, a corresponding early warning signal is generated, and a control strategy is dynamically executed according to the early warning level of the early warning signal.

[0093] The embodiments of the present invention perform multi-dimensional real-time monitoring and dynamic control during data push and task processing, which can promptly detect and respond to system anomalies and ensure the stable operation of the processing link.

[0094] Optionally, in this embodiment of the invention, the real-time monitoring from the data link dimension, node dimension, and business task dimension, and the generation of a corresponding early warning signal when any monitoring indicator in any dimension is abnormal, includes: The data accumulation amount and data input rate of the distributed message queue are compared with a preset first threshold and a second threshold, respectively, to generate a first comparison result and a second comparison result; if the first comparison result does not meet the first condition or the second comparison result does not meet the first condition, a data link dimension warning signal is generated. The CPU utilization, memory utilization, and task execution latency of the processing node are compared with preset third, fourth, and fifth thresholds, respectively, to generate third comparison results, fourth comparison results, and fifth comparison results; if the third comparison result does not meet the third condition, the fourth comparison result does not meet the fourth condition, or the fifth comparison result does not meet the fifth condition, a node-level warning signal is generated. The task priority weight of the real-time data processing task is compared with a preset sixth threshold to generate a sixth comparison result; if the sixth comparison result does not meet the sixth condition, a business task dimension warning signal is generated. The warning levels for the warning signals include mild, moderate, and severe warnings.

[0095] The embodiments of the present invention can achieve refined monitoring and anomaly identification of system status through multi-dimensional threshold comparison and early warning signal generation mechanism.

[0096] Optionally, in this embodiment of the invention, the step of dynamically executing a control strategy based on the warning level of the warning signal includes: When the warning level of the warning signal is a mild warning, the mild warning control strategy is executed first; if the warning signal is not lifted after the mild warning control strategy is executed, the moderate warning control strategy is executed; if the warning signal is not lifted after the moderate warning control strategy is executed, the severe warning control strategy is executed. When the warning level of the warning signal is moderate, the moderate warning control strategy shall be executed first; if the warning signal is not lifted after the moderate warning control strategy is executed, the severe warning control strategy shall continue to be executed. When the warning level of the warning signal is a severe warning, the severe warning control strategy shall be implemented. When the warning signal is lifted, each control strategy is lifted in the reverse order of execution of the previously executed control strategies.

[0097] The embodiments of the present invention can achieve elastic resource scheduling by dynamically executing and gradually releasing control strategies according to the warning level.

[0098] like Figure 5As shown, based on the above-described method embodiments, a distributed streaming data processing architecture is provided, which can implement the distributed real-time data processing method provided by any of the above-described method embodiments of the present invention. The distributed streaming data processing architecture includes: a data acquisition layer, a message passing layer, a distributed processing layer, a result output layer, and a persistent storage layer; The data acquisition layer is used for real-time access to multi-source streaming data for streaming processing. It is deployed at various data-generating nodes in the power Internet of Things, such as substation sensors, various distribution smart terminals, and user-side smart meters. Employing a "lightweight acquisition agent" model, it supports multi-protocol adaptation and can acquire in real-time measurement data such as current, voltage, or power; equipment status data such as switch status or equipment temperature; and fault signal data such as short-circuit warnings or overload indicator lights. During the acquisition process, the acquisition agent does not store the data; it only completes the standard format conversion of the data and immediately pushes it to the message transmission layer. Specifically, the data acquisition layer can execute the action in step S101.

[0099] The message passing layer serves as a distributed asynchronous message hub. The message transmission layer uses the distributed message queue Kafka as its core component, undertaking the functions of "data buffering and asynchronous distribution." The message transmission layer divides power data into multiple message topics (Topics) based on business type, such as measurement data, fault signal data, and equipment status data. Each topic is further divided into multiple partitions (Partitions) to achieve parallel data transmission. Simultaneously, each partition is configured with 3-5 replicas, distributed across physical nodes of the backend support layer hardware devices in different power IoT architectures. This avoids data loss due to single-node failures, meeting the core requirement of "no data loss" in the power system, thus realizing the partitioning and replication design. To achieve asynchronous communication, after the data acquisition layer pushes data to the message queue, it can continue collecting the next batch of data without waiting for confirmation from the distributed processing layer. The distributed processing layer asynchronously pulls data from the message queue according to its own processing capabilities, reducing inter-node dependencies. Even if a node in the processing layer temporarily fails, the message queue can still cache data and continue processing after the node recovers, avoiding data link interruption. Specifically, the message passing layer is equivalent to the "distributed message queue" described in executable step S101.

[0100] The distributed processing layer is used to achieve parallel real-time computing. This layer has multiple "processing node clusters," each deploying a stream computing engine. Nodes interact with each other through a message transmission layer. The core design of this layer includes task sharding, load balancing, and localized computing optimization. Task sharding and load balancing break down real-time data processing tasks, including anomaly checking and indicator calculation, into multiple subtasks, which are then distributed to different nodes according to a "data partition-processing node" mapping rule to ensure load balancing. A load balancing monitoring mechanism is also configured to monitor the CPU and memory usage of nodes in real time. When a node's load is too high, the subtasks are automatically migrated to a less loaded node to avoid single-point overload. Localized computing optimization addresses the "regional correlation" of power data by binding data partitions to processing nodes according to their geographical location, reducing cross-regional data transmission and lowering network latency. Specifically, the distributed processing layer can execute the action in step S102.

[0101] The result output layer supports real-time data push and interaction across multiple scenarios. Breaking away from the traditional "static report" model for batch data, this layer adopts a "subscription-push" mechanism, supporting real-time data output across multiple terminals and business scenarios. It achieves push and interaction through real-time integration with business systems via a UI interface. Regarding UI interface subscription and push, it primarily targets the dispatch center's large screen and maintenance personnel's mobile terminals, providing data service interfaces. The user interface can subscribe to specific indicators such as real-time load rate and equipment health. After the processing layer completes its calculations, the results are immediately pushed to all subscribed terminals, ensuring "second-level updates" of the interface data to meet real-time monitoring needs. For real-time integration with business systems, it connects with systems such as the power grid fault handling system and the equipment maintenance management system. Real-time analysis results such as fault alarms and abnormal equipment signals are pushed to the business systems via API interfaces, triggering subsequent automated processes such as fault work order generation and maintenance personnel dispatch, achieving a closed loop of "data processing-business response." Specifically, the result output layer can execute the action in step S103.

[0102] The persistent storage layer enables on-demand storage, reducing database dependency. It employs differentiated storage strategies and asynchronous storage mechanisms to archive data, avoiding the resource waste caused by "full storage" in traditional batch processing architectures. For data with long-term analytical value, such as fault handling records and monthly equipment operation statistics, the system writes it to a time-series database adapted to the characteristics of power time-series data. This type of data can be used for subsequent fault tracing, equipment lifecycle analysis, and other scenarios. For real-time measurement data with low value density, such as current and voltage, it is only temporarily stored in memory during distributed processing layer calculations. After the indicator calculation is completed, it is discarded if long-term retention is not required, significantly reducing storage resource consumption and avoiding performance degradation caused by writing meaningless data to the database. Once the distributed processing layer completes real-time computation, the data to be stored is transmitted to the database asynchronously. Processing nodes can continue processing the next batch of data without waiting for database write confirmation, completely eliminating the drag on processing speed caused by "synchronous storage". If the database is temporarily unavailable due to maintenance, failure or other reasons, the system will temporarily store the data to be stored in a dedicated queue in the message transmission layer. After the database is restored to normal, the data will be read from the queue and stored in the database, ensuring that the storage link does not affect the continuity of the real-time processing link and does not cause the loss of the data to be stored.

[0103] like Figure 6 As shown, based on the above-described distributed streaming data processing architecture embodiment, an embodiment of the distributed streaming data processing architecture performing real-time data processing tasks is provided, wherein the real-time data processing task is to perform real-time data processing through real-time model training.

[0104] Specifically, to adapt to the dynamic anomaly detection environment of the power Internet of Things, and considering the differences between the fault characteristics of new equipment and those of traditional equipment, real-time model training is integrated into the streaming data processing process to achieve a linkage between "automatic model update and real-time data analysis." The detailed implementation process includes: First, the data is preprocessed and features are extracted. After receiving real-time data in the distributed processing layer, data cleaning is performed to extract sensor outliers and fill in missing values. Then, features such as current frequency and voltage peak value are extracted to generate the input features of the model.

[0105] Secondly, the training and updating of the real-time model are completed. The basic anomaly detection model trained based on historical data is deployed to the processing nodes (physical servers or cloud servers deployed in the processing layer). The processing layer reads the latest fault data from the persistent storage layer at regular intervals, merges it with the real-time data, performs incremental training on the basic model, updates the model parameters, and ensures that the model can identify new anomaly patterns. After the training is completed, the model can be synchronized with all processing nodes through the message passing layer to replace the old model, realize "model update", and perform data processing without interruption.

[0106] Finally, the processing node uses the updated model to perform anomaly detection on real-time data and simultaneously activates an automated flow control mechanism to monitor the model's computation rate and data input rate in real time. When a sudden surge in sensor data occurs, exceeding the model's processing rate, the flow control module is automatically triggered. The flow control mechanism is designed as follows: upstream flow limiting sends a signal to the data acquisition layer to temporarily reduce the sensor data acquisition frequency in that area (from milliseconds to 10 milliseconds to ensure no core data is lost); temporary caching enables disk caching on the processing node to temporarily store data exceeding processing capacity. Once the model's computation rate recovers, cached data is processed first to avoid memory overflow. When the model detects a device anomaly, it generates an alarm message and assigns a unique UUID (containing device ID, alarm time, and anomaly type), sending it to the message transmission layer. Upon receiving the message, the result output layer first queries the "processed alarm UUID list." If it's a duplicate message, it's discarded; if it's a new message, it's pushed to the scheduling center and maintenance terminal to ensure that alarms are "not duplicated and not missed."

[0107] like Figure 7 As shown, based on the above-described distributed streaming data processing architecture embodiment, another embodiment of the distributed streaming data processing architecture for executing real-time data processing tasks is provided. The real-time data processing tasks are real-time data processing through the industry-standardized operating procedures and protocols of the power grid (adapting to compliance monitoring scenarios).

[0108] Specifically, this is achieved through several steps: constructing a standardized rule base, real-time rule matching and processing, automated control and rule adaptation, and deduplication and compliant archiving of critical messages. To meet the power grid's requirement of "strictly adhering to industry standards" (such as DL / T1500-2016 "Application Protocol for Real-time Data Communication in Power Systems"), fixed processing rules are built based on standardized operating procedures (SOPs) and protocols to ensure data processing complies with industry standards. The implementation process includes: First, establish a rule base. Organize data processing specifications and standardized business processes, such as the accuracy requirements for power industry measurement data and alarm level classification standards, and transform them into executable rule scripts, storing them in the "rule base node" to support online rule modification and updates.

[0109] Secondly, real-time rule matching and processing. After the distributed processing layer pulls data from the message transmission layer, it first requests the corresponding business rules from the rule base node, matching the transmission line data with "line overload judgment rules," and then executes the processing according to the rules, including: data compliance verification, checking the accuracy of measurement data (such as retaining two decimal places for current data to meet industry requirements), if it does not meet the requirements, it is marked as "invalid data," only recorded and not included in subsequent analysis; business logic execution, such as judging the real-time current data according to the "overload judgment rule" (if the current value exceeds 120% of the rated current for 5 seconds, an overload alarm is triggered), and generating an overload alarm if the condition is met.

[0110] Secondly, automated flow control and rule adaptation. To address the differences in computational complexity among different rules (e.g., simple precision verification takes less time, while complex fault location rules take longer), the flow control module monitors in real time: if the node processing rate for "complex rules" lags behind, it automatically migrates some "simple rule" processing tasks to idle nodes, releasing resources from complex rule nodes; if the data volume in a certain area suddenly increases (e.g., heavy rain causing frequent data transmission from multiple line sensors), a "rule priority" mechanism is temporarily activated, prioritizing "fault alarm rules" and temporarily suspending "routine statistical rules" to ensure core business operations are not delayed.

[0111] Finally, key messages are deduplicated and archived for compliance. Generated key messages (such as compliance alerts and data verification anomaly records) are assigned unique UUIDs, and the results are deduplicated at the output layer before being pushed to the business system. At the same time, all compliance processing records are asynchronously written to the database to meet the industry's "data traceability" requirements and facilitate subsequent auditing and compliance checks.

[0112] like Figure 8 As shown, based on the above-described distributed streaming data processing architecture embodiment, another embodiment of the distributed streaming data processing architecture for executing real-time data processing tasks is provided, wherein the real-time data processing tasks are based on the results required by the user for final output.

[0113] Specifically, real-time data processing is achieved through several steps, including demand analysis and task configuration, real-time data targeted retrieval, automated flow control and demand priority scheduling, and customized result push and deduplication. It primarily targets the "customized indicator requirements" of power grid users (dispatchers, maintenance teams) (e.g., "peak-valley load difference" and "average daily operating efficiency of equipment" in a specific regional power grid), supporting user-defined output results and realizing "demand-driven" real-time data processing. The specific steps include: First, user requirement analysis and task configuration. A "user requirement configuration interface" is provided, where users can input customized indicator definitions: "peak-valley load difference = maximum load of the day - minimum load of the day", design the calculation cycle according to actual needs, and the data output format can be set according to actual needs, such as outputting data, data tables or curves; the system will parse the requirements into stream computing tasks, assign a unique task ID, and store it in the "task configuration center".

[0114] Secondly, real-time data is retrieved and calculated in a targeted manner. The distributed processing layer obtains user-customized tasks from the task configuration center and retrieves the required data from the message transmission layer according to the task requirements. To calculate the "regional peak-valley load difference," only load data for that region is retrieved, and real-time calculations are performed according to user-defined logic: incremental calculation optimization—for periodic indicators, incremental calculations are performed only based on "the result of the previous period + new data," eliminating the need to reread historical data and reducing the amount of computation; result precision control—the calculation precision is adjusted according to user requirements (such as retaining one decimal place or an integer) to ensure that the output results meet user expectations.

[0115] Secondly, automated flow control and demand priority scheduling. When multiple user-customized tasks run simultaneously, the flow control module combines "task priority" with the scheduling center's "overall network load index" having higher priority than the operations and maintenance team's "single device efficiency index" and node load, dynamically adjusting the task execution order: high-priority tasks are allocated computing resources first to ensure timely output of results; low-priority tasks can be temporarily delayed in computation if the node load is too high, and the delay can be adjusted according to actual needs, with computation resumed after the load decreases to avoid affecting high-priority tasks.

[0116] Finally, customized result push and deduplication are performed. After the calculation is completed, a message containing "task ID, calculation time, and customized result" is generated and assigned a unique UUID. The result output layer pushes the result according to the user-configured push method: email, SMS, UI interface, etc. At the same time, it deduplicates based on UUID to avoid duplicate pushes caused by task retries, ensuring that users obtain "accurate and non-duplicate" customized data.

[0117] like Figure 9 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; One embodiment of the present invention provides a distributed real-time data processing device, including: a data acquisition module 901, a data processing module 902, and a result deduplication module 903; The data acquisition module 901 is used to collect multi-source data from the power Internet of Things in real time, encapsulate the multi-source data into standard format data, and asynchronously push the standard format data to a distributed message queue. The data processing module 902 is used to asynchronously pull the standard format data from the distributed message queue, allocate the standard format data to a predefined real-time data processing task, and split the real-time data processing task into multiple sub-tasks, execute each sub-task in parallel through multiple processing nodes, and generate processing results. The result deduplication module 903 is used to assign a unique identifier to the processing result and perform deduplication verification, push the verified processing result to the subscription terminal or business system in real time, and asynchronously write the processing result to the database or discard it according to the business type.

[0118] Optionally, in this embodiment of the invention, the data acquisition module 901 includes: a data encapsulation submodule; The data encapsulation submodule is used to parse and clean the collected multi-source data, and encapsulate the multi-source data into standard format data according to the business type; wherein the standard format data includes the business identifier field corresponding to the multi-source data.

[0119] The embodiments of the present invention can improve the standardization and consistency of data by parsing, cleaning and encapsulating multi-source data into standard format data.

[0120] Optionally, in this embodiment of the invention, a data partitioning submodule is further included after the data acquisition module 901; The data partitioning submodule is used to divide each of the standard format data into corresponding business themes according to the business identifier field, and to split each of the business themes into multiple partitions; wherein each partition is configured with multiple copies.

[0121] This invention, by dividing data into different topics and storing partitioned copies based on the business identifier field, can avoid data loss caused by single-node failure.

[0122] Optionally, in this embodiment of the invention, the data processing module 902 includes: a task scheduling submodule; The task scheduling submodule is used to schedule each subtask to the corresponding processing node for calculation based on the regional correlation of power Internet of Things data.

[0123] The embodiments of the present invention can reduce data transmission latency and improve data processing efficiency and localization capabilities by scheduling subtasks to corresponding processing nodes based on regional correlation.

[0124] Optionally, in this embodiment of the invention, the result deduplication module 903 includes: a memory query submodule, a first query result submodule, a database query submodule, a second query result submodule, and a third query result submodule; The memory query submodule is used to query the unique identifier corresponding to the processing result in the memory cache; The first query result submodule is configured to, if the unique identifier is found in the memory cache, set the verification result to fail and discard the processing result; The database query submodule is used to perform a query in the deduplication database if the unique identifier is not found in the memory cache. The second query result submodule is used to, if the unique identifier is found in the deduplication database, set the verification result as failing and discard the processing result, and at the same time record the unique identifier in the memory cache; The third query result submodule is used to verify the unique identifier if it is not found in the deduplication database, and record the unique identifier in the memory cache and the deduplication database.

[0125] The embodiments of the present invention utilize a dual verification mechanism of memory caching and deduplication database to efficiently identify duplicate data, avoid duplicate output of results, and improve the accuracy of data processing.

[0126] Optionally, in this embodiment of the invention, the distributed real-time data processing device further includes: a real-time monitoring module; The real-time monitoring module is used to perform real-time monitoring from the data link dimension, node dimension and business task dimension during data push and task processing. When any monitoring indicator in any dimension is abnormal, a corresponding early warning signal is generated, and a control strategy is dynamically executed according to the early warning level of the early warning signal.

[0127] The embodiments of the present invention perform multi-dimensional real-time monitoring and dynamic control during data push and task processing, which can promptly detect and respond to system anomalies and ensure the stable operation of the processing link.

[0128] Optionally, in this embodiment of the invention, the real-time monitoring module includes: a data link dimension submodule, a node dimension submodule, a business task dimension submodule, and an early warning level submodule; The data link dimension submodule is used to compare the data accumulation amount and data input rate of the distributed message queue with a preset first threshold and a second threshold, respectively, and generate a first comparison result and a second comparison result; if the first comparison result does not meet the first condition or the second comparison result does not meet the first condition, a data link dimension early warning signal is generated. The node dimension submodule is used to compare the CPU utilization, memory utilization, and task execution latency of the processing node with preset third, fourth, and fifth thresholds, respectively, and generate third comparison results, fourth comparison results, and fifth comparison results; if the third comparison result does not meet the third condition, the fourth comparison result does not meet the fourth condition, or the fifth comparison result does not meet the fifth condition, then a node dimension warning signal is generated. The business task dimension submodule is used to compare the task priority weight of the real-time data processing task with a preset sixth threshold and generate a sixth comparison result; if the sixth comparison result does not meet the sixth condition, a business task dimension warning signal is generated. The warning level submodule is used to classify the warning level of the warning signal into mild warning, moderate warning and severe warning.

[0129] The embodiments of the present invention can achieve refined monitoring and anomaly identification of system status through multi-dimensional threshold comparison and early warning signal generation mechanism.

[0130] Optionally, in this embodiment of the invention, the real-time monitoring module further includes: a mild warning submodule, a moderate warning submodule, a severe warning submodule, and a control cancellation submodule; The mild warning submodule is used to prioritize the execution of the mild warning control strategy when the warning level of the warning signal is mild; if the warning signal is not lifted after the execution of the mild warning control strategy, then the moderate warning control strategy is executed; if the warning signal is not lifted after the execution of the moderate warning control strategy, then the severe warning control strategy is executed. The moderate warning submodule is used to prioritize the execution of the moderate warning control strategy when the warning level of the warning signal is moderate; if the warning signal is not lifted after the execution of the moderate warning control strategy, the severe warning control strategy will continue to be executed. The severe warning submodule is used to execute a severe warning control strategy when the warning level of the warning signal is severe warning. The control deactivation submodule is used to deactivate each control strategy step by step in the reverse order of execution of the previously executed control strategies after the warning signal is deactivated.

[0131] The embodiments of the present invention can achieve elastic resource scheduling by dynamically executing and gradually releasing control strategies according to the warning level.

[0132] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement a distributed real-time data processing method provided by any of the above-described method embodiments of the present invention.

[0133] In this embodiment of the invention, the data acquisition module 901 collects multi-source data from the power Internet of Things in real time, encapsulates it into a standard format, and then asynchronously pushes it to a distributed message queue, which can achieve efficient data access. The data processing module 902 asynchronously pulls data from the message queue and distributes it to multiple processing nodes to execute sub-tasks in parallel, which can improve data processing efficiency and system resource utilization. The result deduplication module 903 performs deduplication verification on the processing results and stores them as needed, which can avoid duplicate data output and resource waste.

[0134] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0135] Based on the above-described embodiment of a distributed real-time data processing method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a distributed real-time data processing method according to any embodiment of the present invention.

[0136] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0137] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0138] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0139] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute a distributed real-time data processing method as described in any of the above-described method embodiments of the present invention.

[0140] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0141] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A distributed real-time data processing method, characterized in that, include: The system collects multi-source data in real time from the power Internet of Things, encapsulates the multi-source data into standard format data, and asynchronously pushes the standard format data to a distributed message queue. The standard format data is asynchronously pulled from the distributed message queue, the standard format data is allocated to a predefined real-time data processing task, and the real-time data processing task is split into multiple sub-tasks. Each sub-task is executed in parallel by multiple processing nodes to generate processing results. Assign a unique identifier to the processing result and perform deduplication verification. Push the verified processing result to the subscription terminal or business system in real time, and write the processing result to the database asynchronously or discard it according to the business type.

2. The distributed real-time data processing method as described in claim 1, characterized in that, The step of encapsulating the multi-source data into standard format data includes: The collected multi-source data is parsed and cleaned, and then encapsulated into standard format data according to the business type; wherein the standard format data includes the business identifier field corresponding to the multi-source data.

3. The distributed real-time data processing method as described in claim 2, characterized in that, After collecting multi-source data in real time from the power Internet of Things, encapsulating the multi-source data into standard format data, and asynchronously pushing the standard format data to a distributed message queue, the process also includes: Based on the business identifier field, each of the standard format data is divided into a corresponding business theme, and each business theme is split into multiple partitions; wherein each partition is configured with multiple copies.

4. The distributed real-time data processing method as described in claim 1, characterized in that, The step of dividing the real-time data processing task into multiple sub-tasks and executing each sub-task in parallel through multiple processing nodes includes: Based on the regional correlation of power Internet of Things data, each sub-task is scheduled to the corresponding regional processing node for calculation.

5. The distributed real-time data processing method as described in claim 1, characterized in that, The deduplication verification includes: Retrieve the unique identifier corresponding to the processing result from the memory cache; If the unique identifier is found in the memory cache, the verification result is a failure and the processing result is discarded; If the unique identifier is not found in the memory cache, then a query is performed in the deduplication database; If the unique identifier is found in the deduplication database, the verification result is failed and the processing result is discarded. At the same time, the unique identifier is recorded in the memory cache. If the unique identifier is not found in the deduplication database, the verification result is passed, and the unique identifier is recorded in the memory cache and the deduplication database.

6. The distributed real-time data processing method as described in claim 1, characterized in that, During data push and task processing, real-time monitoring is performed from the dimensions of data link, node, and business task. When any monitoring indicator in any dimension is abnormal, a corresponding early warning signal is generated, and control strategies are dynamically executed according to the warning level of the early warning signal.

7. A distributed real-time data processing method as described in claim 6, characterized in that, The system performs real-time monitoring from the dimensions of data link, node, and business task. When any monitoring indicator in any dimension shows an anomaly, a corresponding early warning signal is generated, including: The data accumulation amount and data input rate of the distributed message queue are compared with a preset first threshold and a second threshold, respectively, to generate a first comparison result and a second comparison result; if the first comparison result does not meet the first condition or the second comparison result does not meet the first condition, a data link dimension warning signal is generated. The CPU utilization, memory utilization, and task execution latency of the processing node are compared with preset third, fourth, and fifth thresholds, respectively, to generate third comparison results, fourth comparison results, and fifth comparison results; if the third comparison result does not meet the third condition, the fourth comparison result does not meet the fourth condition, or the fifth comparison result does not meet the fifth condition, a node-level warning signal is generated. The task priority weight of the real-time data processing task is compared with a preset sixth threshold to generate a sixth comparison result; if the sixth comparison result does not meet the sixth condition, a business task dimension warning signal is generated. The warning levels for the warning signals include mild, moderate, and severe warnings.

8. The distributed real-time data processing method as described in claim 7, characterized in that, The step of dynamically executing a control strategy based on the warning level of the warning signal includes: When the warning level of the warning signal is a mild warning, the mild warning control strategy is executed first; if the warning signal is not lifted after the mild warning control strategy is executed, the moderate warning control strategy is executed; if the warning signal is not lifted after the moderate warning control strategy is executed, the severe warning control strategy is executed. When the warning level of the warning signal is moderate, the moderate warning control strategy shall be executed first; if the warning signal is not lifted after the moderate warning control strategy is executed, the severe warning control strategy shall continue to be executed. When the warning level of the warning signal is a severe warning, the severe warning control strategy shall be implemented. When the warning signal is lifted, each control strategy is lifted in the reverse order of execution of the previously executed control strategies.

9. A distributed real-time data processing device, characterized in that, include: Data acquisition module, data processing module, and result deduplication module; The data acquisition module is used to collect multi-source data from the power Internet of Things in real time, encapsulate the multi-source data into standard format data, and asynchronously push the standard format data to a distributed message queue. The data processing module is used to asynchronously pull the standard format data from the distributed message queue, allocate the standard format data to a predefined real-time data processing task, and split the real-time data processing task into multiple sub-tasks. The sub-tasks are executed in parallel by multiple processing nodes to generate processing results. The result deduplication module is used to assign a unique identifier to the processing result and perform deduplication verification. The processing result that passes the verification is pushed to the subscription terminal or business system in real time, and the processing result is asynchronously written to the database or discarded according to the business type.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform a distributed real-time data processing method as described in any one of claims 1-7.

Images