Centralized system full data automatic synchronization method based on kafka message queue

CN122933625APending Publication Date: 2026-10-16HUANENG LANCANG RIVER HYDROPOWER CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610924642.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-06-25
Publication Date
2026-10-16

AI Technical Summary

Benefits of technology

[0036] 1. This invention partitions Kafka topics, splits incremental real-time data and static full data for dual-channel transmission, and delivers and consumes incremental working data in milliseconds, no longer subject to fixed synchronization cycles, breaking the minute-level and hour-level data delays. Dynamically changed data during the synchronization window can be synchronized in real time, meeting the second-level scheduling requirements for centralized control failure emergency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122933625A_ABST
    Figure CN122933625A_ABST
Patent Text Reader

Abstract

The application discloses a centralized control system full-amount data automatic synchronization method based on a Kafka message queue, relates to the technical field of power station centralized control systems, and comprises the following steps: pre-dividing full-amount data of a substation of a power station according to data types; marking dynamic changes of real-time working condition incremental data by means of a station CDC change capture component, marking period hash fingerprints of configuration static files and historical stock data tables, and synchronously binding a power PTP unified time sequence stamp; dividing the partitions of a Kafka subject according to station networking bandwidth and data volume, hierarchically delivering messages according to classified delivery rules, splitting dual-channel transmission of incremental real-time data and static full-amount data through the partition architecture of the Kafka subject, delivering and consuming incremental condition data at a millisecond level, no longer being subject to a fixed synchronization period, breaking minute-level and hour-level data delay, and synchronously changing data in a window period to instantly satisfy the needs of centralized control fault emergency second-level scheduling.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the technical field of power plant centralized control system, specifically relating to a method for automatic synchronization of all data in a centralized control system based on Kafka message queue. Background Technology

[0002] The power station centralized control system is used to manage and coordinate all data of real-time measurement points, historical operating conditions, configuration screens, equipment configurations, and ledger parameters of multiple distributed power stations, so as to realize unmanned operation of the power stations and unified scheduling and control by the centralized control center.

[0003] Currently, the existing automatic synchronization methods for all centralized control data are mainly of three types: timed batch full export and overwrite synchronization, database native master-slave log synchronization, and file-level Rsync periodic replication. The data source is the substation business database and configuration file server. According to a preset fixed time period, the full data tables, configuration models, and configuration files of the substation are transmitted in batches to the target storage end of the centralized control center through dedicated line transmission, database dump scripts, and file transfer protocols to complete the periodic automatic overwrite synchronization of all data of the entire station. This synchronization mode is simple to deploy and adaptable to old centralized control software and hardware systems, and is the mainstream synchronization solution for current new energy, hydropower, and power grid substation centralized control projects.

[0004] However, the traditional full-volume batch synchronization method has extremely poor real-time performance in large-scale multi-site centralized control network scenarios. Relying on fixed-period triggering of full-volume synchronization, the data delay remains at the minute to hour level. During the synchronization window period, dynamic incremental data of station remote signaling changes, load telemetry, and fault alarms cannot be synchronized. In the event of a fault, the data of the centralized control center is lagging behind and cannot support the second-level emergency dispatch of the power grid.

[0005] To address this issue, we propose a method for automatic full data synchronization in centralized control systems based on Kafka message queues. Summary of the Invention

[0006] The purpose of this invention is to provide a method for automatic full data synchronization of a centralized control system based on Kafka message queues. This method is no longer subject to fixed synchronization cycles, breaks through minute-level and hour-level data delays, and allows for real-time synchronization of dynamically changed data during the synchronization window to meet the second-level scheduling requirements of centralized control system fault emergency response, thereby solving the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for automatic full data synchronization in a centralized control system based on Kafka message queues, applied to a data interaction network architecture between multiple substations and the power centralized control center, includes the following steps:

[0009] S1. Pre-classify the data types of the full data of the substations of the power plant to obtain classified data. The classified data includes real-time incremental data, static full files of equipment configuration, and historical archive full data tables.

[0010] S2. Dynamically mark the incremental data of real-time operating conditions through the CDC change capture component of the station, perform periodic hash fingerprint marking on the configuration static file and historical stock data table, and bind the unified time stamp of power PTP in sync to obtain the marked and classified data.

[0011] S3. Based on the network bandwidth and data volume of the site, divide the Kafka topic into partitions, and deliver the tagged and classified data according to the classification and delivery rules. The Kafka message cluster performs persistent storage, transmission peak shaping, and breakpoint marking on the delivered messages.

[0012] S4: The consumer end of the central control center is bound to the Kafka topic of the corresponding partition, and a multi-threaded parallel consumption mode is used for differentiated consumption parsing.

[0013] S5. Retrieve the classification data and the labeled classification data through the full-domain time-series verification module, and perform a comparison and verification with the full data after consumption restoration.

[0014] If the fingerprint matches the timing verification, the data will be written to the target database and configuration server of the central control center in compliance with regulations.

[0015] If the verification is inconsistent or a transmission interruption is detected, the breakpoint information marked by the Kafka message cluster is retrieved, and a resend request is initiated only for abnormal shard data to synchronously correct the time-series misaligned data, thus completing the closed-loop self-healing of data synchronization.

[0016] Preferably, in step S1, when classifying data types, data that undergoes short-cycle millisecond-level dynamic changes and cannot be synchronized with delay is classified as real-time operating condition incremental data; visual configuration files with fixed file formats, rarely subject to independent changes, and only updated by manual modification are classified as equipment configuration static full-volume files; and historical ledger data that has been solidified and archived, used only for daily verification, and has a large volume is classified as historical archived full-volume data tables.

[0017] Preferably, in step S2, the site CDC change capture component includes a database log listening interface unit, a file read / write sniffing unit, a time-series binding processing unit, and a local cache forwarding unit connected in series. The database log listening interface unit is connected to the site's local real-time business database kernel port, the file read / write sniffing unit is connected to the site's configuration file server read / write port, and the time-series binding processing unit is bidirectionally connected to the database log listening interface unit and the file read / write sniffing unit. The time-series binding processing unit has a built-in PTP timing chip and a hash computing chip. The local cache forwarding unit is connected to the site's Kafka front-end gateway.

[0018] Preferably, the following process is performed on the real-time incremental operating data to dynamically mark changes:

[0019] A1. Real-time capture of the transaction logs of the site business database through the database log listening interface unit, filter the committed valid add, delete and modify transactions, and filter the invalid transactions that have been rolled back.

[0020] A2. Extract the original field data of telemetry, teleindication, and alarm points within the transaction, call the differential change judgment algorithm to determine whether the data has undergone a valid change, and remove invalid data with numerical jitter, interference, or false triggering.

[0021] A3. Bind the valid change data to the unique ID of the equipment in the field, generate the change tag code, and complete the dynamic change marking;

[0022] A4. Push to the timing binding processing unit for delivery.

[0023] Preferably, when performing periodic hash fingerprint marking on static configuration files and historical data tables, a fixed period of 1 hour is used as the fingerprint collection period. The file read / write sniffing unit periodically retrieves the full-domain field information of the target file and the existing data table, calls the SHA-256 encrypted hash fingerprint generation algorithm, and performs full-domain encryption operations on the file byte stream, data table field values, and field quantities to generate a unique and irreversible hash fingerprint code. Each file / table is bound to a set of exclusive fingerprints. Any byte change in the file or data table will automatically change the hash fingerprint.

[0024] The preferred synchronization and binding process for the unified time stamp of the power PTP is as follows:

[0025] B1. The timing binding processing unit has a built-in PTPv2 power dedicated timing module, which connects to the regional power timing master clock to complete the full network time synchronization of the substation's local clock, with a clock synchronization error control of ≤1ms.

[0026] B2. Upon completion of data marking, extract the absolute time sequence value of the entire network of the master clock standard.

[0027] B3. Use the time series value as a suffix field and concatenate and bind it to the end of the incremental change label and hash fingerprint code message;

[0028] B4. Use a clock skew calibration algorithm to verify the compliance of the timing stamp format. Unqualified timing stamps will be directly rebound.

[0029] Preferably, in step S3, when partitioning the Kafka topic, a joint partitioning algorithm combining bandwidth weight and data volume entropy is used to complete the topic partitioning. The partitions of the Kafka topic include high-speed incremental partitions and full synchronization dedicated partitions. The high-speed incremental partitions belong to the high-speed incremental topic, have a large number of partitions, a small single partition cache capacity, and the highest send / receive priority. The full synchronization dedicated partitions belong to the full synchronization topic, have a small number of partitions, a large single partition cache capacity, and a lower send / receive priority than the incremental partitions.

[0030] Preferably, the classification and delivery rules specifically include:

[0031] For real-time incremental data: After completing the change tagging and time-series binding, the CDC change capture component immediately pushes the data to the Kafka front-end gateway without time restrictions, and delivers it to the high-speed incremental partition in real time without any conditions. It preempts the transmission priority and ensures millisecond-level uplink.

[0032] For static configuration files and historical data tables: data is not delivered in real time. The Kafka-Zookeeper node determines the business period of the site. The daily grid load valley value from 00:00 to 05:00 is defined as the low-peak business window. During the window, data is evenly divided and packaged according to the partition byte threshold. The size of a single fragment is uniformly controlled to 64MB. The fragment number is traced and bound to the source station fingerprint time sequence information. After packaging, the data is delivered to the dedicated partition for full synchronization. During peak business periods, the delivery of full fragment data is prohibited.

[0033] Preferably, in step S5, the global timing verification module uses an encrypted point-to-point retrieval method to actively retrieve the original hash fingerprint, source PTP timing stamp, and segment number ledger stored in the power station CDC. At the same time, it retrieves the files, data tables, and incremental measurement point data restored by the centralized control terminal to perform fingerprint consistency verification and global timing verification. The fingerprint consistency verification uses a fingerprint consistency judgment algorithm to compare whether the source hash fingerprint and the secondary hash fingerprint of the restored data at the consumer end are consistent. The global timing verification is used to compare whether the clock deviation of the PTP timing stamps at both ends is within the allowable power deviation threshold.

[0034] Preferably, when correcting time-series misaligned data, the time-series stamp of the data suffix at the central control terminal is rewritten based on the time-series compensation value to unify the time sequence across the entire network; if the fingerprint and time sequence are all verified as qualified, an inbound permission instruction is issued, and compliant data is inbound and archived; then, the Kafka offset point for this transmission is updated synchronously, and the abnormal ledger for this transmission is cleared; this is superimposed with daily low-peak baseline full-domain comparison and correction, forming a closed loop from data collection, transmission, verification, correction, inbound, and ledger update.

[0035] The automatic full data synchronization method for centralized control systems based on Kafka message queues proposed in this invention has the following advantages compared with existing technologies:

[0036] 1. This invention partitions Kafka topics, splits incremental real-time data and static full data for dual-channel transmission, and delivers and consumes incremental working data in milliseconds, no longer subject to fixed synchronization cycles, breaking the minute-level and hour-level data delays. Dynamically changed data during the synchronization window can be synchronized in real time, meeting the second-level scheduling requirements for centralized control failure emergency.

[0037] 2. This invention utilizes the peak-shaving and decoupling capabilities of the Kafka message cluster to avoid transmitting large volumes of configuration and historical full data during peak power business periods, and replaces the full database transmission with fragmented and packaged transmission; at the same time, it decouples the source end of the power station from the centralized control write business, and the power station is only responsible for data marking and delivery, without having to occupy local computing power to perform full packaging and comparison operations, which greatly reduces the read and write load of the power station database and avoids the full transmission occupying the dedicated line bandwidth of remote control and telemetry business;

[0038] 3. This invention retrieves classified data and labeled classified data through a global timing verification module, and performs comparison and verification with the full data after consumption restoration. It removes dirty data during the synchronization window, automatically corrects timing misalignment and version deviation data, solves the problems of conflict between field station and centralized control data and configuration version mismatch, and avoids remote control security risks caused by manual data supplementation errors. Attached Figure Description

[0039] Figure 1 A flowchart according to an embodiment of the present invention is shown;

[0040] Figure 2 A structural block diagram of a site CDC change capture component according to an embodiment of the present invention is shown;

[0041] Figure 3 A flowchart illustrating the dynamic change marking process for real-time incremental operating data according to an embodiment of the present invention is shown.

[0042] Figure 4 A flowchart illustrating the synchronization binding process of the unified time stamp of the power PTP according to an embodiment of the present invention is shown. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] This invention provides, for example Figure 1-4 The method for automatic full data synchronization in a centralized control system based on Kafka message queues is applied to a data interaction network architecture between multiple power plant substations and the power centralized control center. This architecture includes a source data acquisition module at the power plant level, a distributed Kafka message cluster, a consumption and writing module at the central control center, and a global time-series verification module. The source data acquisition module includes a power plant CDC change capture component, deployed at each power plant substation. This component is used to classify the full data types at the power plant, perform incremental data change capture, static full data hashing, and bind a unified global PTP time-series stamp. The Afka message cluster has independent high-speed incremental topics and sharded full topics for receiving tiered data messages from the site, enabling message persistence, transmission peak smoothing, breakpoint retention, and partitioned traffic control. The central control center's consumption and writing module connects to each partition topic of the Kafka message cluster, configured with multi-threaded parallel consumption units to perform differentiated real-time incremental data storage and sharded full data restoration and assembly. The full-domain time-series verification module connects to both the site's source data acquisition module and the central control center's consumption and writing module, for bidirectional hash fingerprint comparison, time-series verification, automatic resending of abnormal shards, and automatic correction of deviation data.

[0045] The method includes the following steps:

[0046] S1. Pre-classify the data types of the full data of the substations of the power plant to obtain classified data. The classified data includes real-time incremental data, static full files of equipment configuration, and historical archive full data tables.

[0047] Substations of power stations include independent local control stations for photovoltaic power stations, hydropower plants, substations, wind farms, and energy storage power stations. They have independent local measurement and control, data storage, equipment configuration, and local alarm functions. They are remotely controlled and managed by the superior regional power control center. They are different from the main control station of the control center and are the data acquisition source stations.

[0048] The full data of substations in power plants includes real-time measurement and control data, static configuration data, existing historical archive data, and basic equipment configuration data. This data supports a closed-loop system for station equipment operation and maintenance, grid dispatching, fault analysis, and record archiving. Real-time measurement and control data includes station-wide telemetry values, remote signaling changes, remote control execution receipts, equipment fault alarms, protection action points, and load fluctuation parameters. Static configuration data includes primary equipment wiring configuration diagrams, secondary protection configuration drawings, measurement and control point configuration configurations, communication protocol configuration parameters, and equipment topology binding files. Existing historical archive data includes hourly / daily / monthly equipment operation history data tables, fault history ledgers, operation and maintenance archive data tables, and historical power consumption statistics tables. Basic equipment configuration data includes equipment number parameters, communication IP configurations, permission configurations, and local linkage setting parameters.

[0049] When classifying data types, data that undergoes short-cycle, millisecond-level dynamic changes and cannot be synchronized with delay is classified as real-time incremental operating condition data; visual configuration files with fixed file formats, rarely subject to independent changes, and only updated by manual modification are classified as static full-volume equipment configuration files; and historical ledger data that has been solidified and archived, used only for daily verification, and has a large volume is classified as historical archived full-volume data tables.

[0050] S2. Dynamically mark the incremental data of real-time operating conditions through the CDC change capture component of the station, perform periodic hash fingerprint marking on the configuration static file and historical stock data table, and bind the unified time stamp of power PTP in sync to obtain the marked and classified data.

[0051] The site CDC change capture component is a locally embedded integrated acquisition component, such as... Figure 2 As shown, the system includes a database log monitoring interface unit, a file read / write sniffing unit, a time-series binding processing unit, and a local cache forwarding unit connected in series. The database log monitoring interface unit connects to the local real-time business database kernel port of the site, specifically capturing database redo and binlog change logs; the file read / write sniffing unit connects to the read / write port of the site's configuration file server, monitoring configuration file addition, modification, and deletion operations; the time-series binding processing unit bidirectionally connects to the database log monitoring interface unit and the file read / write sniffing unit, and has a built-in PTP timing chip and hash computing chip; the local cache forwarding unit connects to the site's Kafka front-end gateway, completing the pre-caching and forwarding of tagged data.

[0052] like Figure 3 As shown, the following process is performed on the real-time incremental operating data to dynamically mark changes:

[0053] A1. Real-time capture of the transaction logs of the site business database through the database log listening interface unit, filter the committed valid add, delete and modify transactions, and filter the invalid transactions that have been rolled back.

[0054] A2. Extract the original field data of telemetry, telesignaling, and alarm points within the transaction, and call the differential change determination algorithm to determine whether the data has undergone a valid change, eliminating invalid data due to numerical jitter, interference, or false triggering. The formula expression for the differential change determination algorithm is as follows:

[0055] ,

[0056] In the formula, For a change to be considered valid, a change is defined as a difference greater than 0. This is the real-time data of the measurement point at time n. Data is retained for the measurement points in the previous sampling period. The threshold value for interference immunity of power measurement points is typically 0.02 for electrical measurement points and 0 for switch measurement points.

[0057] A3. Bind the valid change data to the unique ID of the equipment in the field, generate the change tag code, and complete the dynamic change marking;

[0058] A4. Push to the timing binding processing unit for delivery.

[0059] When performing periodic hash fingerprinting on static configuration files and historical data tables, a fixed 1-hour fingerprint collection cycle is used. The file read / write sniffing unit periodically retrieves the full-domain field information of the target file and the existing data table, calls the SHA-256 encrypted hash fingerprint generation algorithm, and performs full-domain encryption operations on the file byte stream, data table field values, and field counts to generate a unique and irreversible hash fingerprint. Each file / table is bound to a unique fingerprint. Any byte change in the file or data table will automatically change the hash fingerprint. The formula expression for the encrypted hash fingerprint generation algorithm is as follows:

[0060] ,

[0061] In the formula, To ultimately generate the hash fingerprint, This is the complete raw byte sequence of the file / data table. This is a unique code for a station substation. For periodic sampling timestamps, This is the bitwise XOR operator.

[0062] like Figure 4 As shown, the synchronization binding process for the unified time stamp of the power PTP is as follows:

[0063] B1. The timing binding processing unit has a built-in PTPv2 power dedicated timing module, which connects to the regional power timing master clock to complete the full network time synchronization of the substation's local clock, with a clock synchronization error control of ≤1ms.

[0064] B2. Upon completion of data marking, extract the absolute time sequence value of the entire network of the master clock standard.

[0065] B3. Use the time sequence value as a suffix field and concatenate and bind it to the end of the incremental change label and hash fingerprint code message to achieve one source and one stamp binding.

[0066] B4. A clock skew calibration algorithm is used to verify the compliance of the timing stamp format. Unqualified timing stamps are directly rebound. The formula expression for the clock skew calibration algorithm is:

[0067] ,

[0068] In the formula, To ultimately bind a compliant PTP unified time stamp, For the absolute time of the master clock, This is the local clock offset compensation value for the substation.

[0069] S3. Divide the Kafka topic into partitions based on the network bandwidth and data volume of the site. Pack the tagged and classified data into hierarchical packages according to the classification and delivery rules and deliver the messages. Use the Kafka message cluster to perform persistent storage, transmission peak shaping, and breakpoint marking on the delivered messages.

[0070] Kafka topics are logical data storage classification units within a Kafka message cluster, used to isolate power synchronization messages of different business types, achieving data classification and isolation for sending and receiving, and avoiding congestion caused by mixed incremental and full data messages. The Kafka message cluster is a multi-node Broker network distributed power-specific message cluster, including three types of network nodes: Broker service nodes, Zookeeper scheduling and control nodes, and replica storage nodes. It connects to the power plant CDC change capture component, the central control center consumption and writing module, and the full-domain time sequence verification module, serving as the core hub for end-to-end data relay. It undertakes the entire relay work of data reception, message scheduling, caching and retention, and partitioned distribution for substations, and is an intermediate decoupling module connecting the source substations and the target central control center.

[0071] The Kafka topic partitioning algorithm uses a combination of bandwidth weight and data volume entropy value to complete the topic partitioning. The formula expression of the joint partitioning algorithm is as follows:

[0072] ,

[0073] In the formula, N represents the number of partitions for a single topic, rounded up. The safety factor for power line transmission is 0.5; E is the entropy value of the data volume, the larger the volume, the higher the entropy value; and W is the real-time available uplink bandwidth of the station.

[0074] The Kafka topic is divided into partitions including high-speed incremental partitions and full synchronization dedicated partitions. The high-speed incremental partitions belong to the high-speed incremental topic, have a large number of partitions, a small single partition cache capacity, and the highest send / receive priority. The full synchronization dedicated partitions belong to the full synchronization topic, have a small number of partitions, a large single partition cache capacity, and a lower send / receive priority than the incremental partitions.

[0075] The specific classification and delivery rules include:

[0076] For real-time incremental data: After completing the change tagging and time-series binding, the CDC change capture component immediately pushes the data to the Kafka front-end gateway without time restrictions, and delivers it to the high-speed incremental partition in real time without any conditions. It preempts the transmission priority and ensures millisecond-level uplink.

[0077] For static configuration files and historical data tables: data is not delivered in real time. The Kafka-Zookeeper node determines the business period of the site. The daily grid load valley value from 00:00 to 05:00 is defined as the low-peak business window. During the window, data is evenly divided and packaged according to the partition byte threshold. The size of a single fragment is uniformly controlled to 64MB. The fragment number is traced and bound to the source station fingerprint time sequence information. After packaging, the data is delivered to the dedicated partition for full synchronization. During peak business periods, the delivery of full fragment data is prohibited.

[0078] The Kafka message cluster performs the following processes to persistently store delivered messages, smooth out transmission spikes, and mark breakpoints:

[0079] After receiving the partition message, C1 and Broker nodes perform dual-copy disk persistence according to the power data retention specifications, and retain the message using a replica redundancy storage algorithm. The replica redundancy determination formula is as follows:

[0080] ,

[0081] In the formula, For storage security ratio, a value greater than 0.9 is considered compliant with persistence rules. This represents the number of copies already copied to disk. The preset total number of copies;

[0082] C2, when the instantaneous uplink traffic of partitioned messages in the Kafka message cluster exceeds the dedicated line bandwidth threshold, activates the token bucket traffic shaping algorithm to limit and temporarily suspend the delivery of all fragmented messages, prioritizing the passage of incremental messages and stabilizing link transmission traffic. The formula for the token bucket traffic shaping algorithm is:

[0083] ,

[0084] In the formula, For the actual flow of traffic, For instantaneous uplink request traffic, Maximum bandwidth traffic allowed for the token bucket;

[0085] C3 and Kafka message clusters automatically record the offset point after each shard or incremental message is transmitted. When the network link is interrupted or the message is lost, the current offset breakpoint value is locked, and the breakpoint ledger is generated by binding the site ID and partition number and permanently stored in the Zookeeper node. Breakpoint tracing can directly retrieve the offset value without the need for full-domain data backtracking.

[0086] S4: The consumer end of the central control center is bound to the Kafka topic of the corresponding partition, and a multi-threaded parallel consumption mode is used for differentiated consumption parsing;

[0087] The consumer end of the control center is deployed in multiple consumer server clusters on the control center's intranet and connected to the Kafka message cluster's intranet gateway. It is the target execution terminal for receiving, parsing, restoring, and writing data down to the database. The control center consumer end is divided into incremental dedicated consumer server groups and full sharded consumer server groups. It uses a static ID-based binding method for partition groups to connect to Kafka topics. The high-speed incremental topic ID, full synchronization topic ID, and corresponding partition offset access permissions are pre-entered in the consumer end configuration file to complete the group-to-group binding. The incremental dedicated consumer server group only allows high-speed partition messages, and the full sharded consumer group only allows dedicated partition sharded messages, with bidirectional isolation and no crosstalk.

[0088] The multi-threaded parallel consumption mode splits the thread pool into equal consumption sub-threads, with the number of threads matching the number of topic partitions. Each thread has exclusive consumption rights for a single partition, while multiple threads in the same group pull packets independently in parallel. Threads share a time-series verification cache pool and do not compete for resources or interfere with each other's parsing jobs. This is different from single-threaded serial consumption and significantly improves the efficiency of full-shard parsing.

[0089] Differentiated consumption analysis includes high-speed partition incremental consumption analysis and full-volume partitioned segmented consumption analysis. High-speed partition incremental consumption analysis uses an incremental consumption thread to pull incremental messages with change tags and PTP time stamps, disassembles the tag, time sequence, and measurement point value fields, calls the measurement point normalization and restoration algorithm, formats and restores the power standard measurement point data, and directly writes it into the centralized control real-time database. The formula expression for the normalization and restoration algorithm is:

[0090] ,

[0091] In the formula, Y is the standard measurement point value for entry into the warehouse, X is the original data collected in the message, and k and b are the fixed conversion coefficients for the measurement points at the station.

[0092] The full-volume segmented consumption analysis uses multi-threaded parallel processing to pull segmented data packets, sorts and reassembles them according to their segment numbers, calls a segment concatenation integrity verification algorithm to check the number of missing segments, decompresses and restores the original configuration file and historical data table of the complete segments, and writes them to the corresponding storage directory after time-series verification. The formula expression of the segment concatenation integrity verification algorithm is:

[0093] ,

[0094] In the formula, To concatenate the total number of bytes, If the number of bytes in the i-th fragment is consistent with the total number of bytes in the source packet, the splicing is deemed compliant, where d is the total number of fragments.

[0095] S5. Retrieve the classification data and the labeled classification data through the full-domain time-series verification module, and perform a comparison and verification with the full data after consumption restoration.

[0096] If the fingerprint matches the timing verification, the data will be written to the target database and configuration server in the centralized control center in compliance with regulations.

[0097] If the verification fails or a transmission interruption is detected, the breakpoint information marked by the Kafka message cluster is retrieved. Only abnormal shard data is resent as a retransmission request, synchronously correcting misaligned data and completing a closed-loop self-healing data synchronization process. The formula for correcting misaligned data is:

[0098] ,

[0099] In the formula, This is the timing compensation value. For source-side standard PTP timings, For the local timing of the central control terminal;

[0100] When correcting time-series misaligned data, the time-series stamp of the data suffix at the central control terminal is rewritten based on the time-series compensation value to unify the time sequence across the entire network. If the fingerprint and time sequence are all verified as qualified, an entry permission instruction is issued, and compliant data is entered into the database and archived. Then, the Kafka offset point for this transmission is updated synchronously, and the abnormal ledger for this transmission is cleared. Overlaying this with daily low-peak baseline full-domain comparison and correction, a closed loop is formed from the entire chain of data collection, transmission, verification, correction, entry into the database, and ledger update, achieving full data synchronization and unmanned self-healing, and routinely ensuring that the data of the substations and the central control station are of the same source.

[0101] The full-domain time-series verification module includes a cross-domain data retrieval unit, a fingerprint time-series comparison unit, a breakpoint information parsing unit, a deviation correction calculation unit, and a closed-loop instruction issuance unit. The cross-domain data retrieval unit is electrically connected to the site CDC change capture component and the Kafka message cluster, and is used to connect to the site CDC cache library and the centralized control consumption parsing cache library to realize bidirectional data retrieval from the source and target ends. The fingerprint time-series comparison unit is electrically connected to the cross-domain data retrieval unit and has a built-in hash comparison computing chip. The breakpoint information parsing unit is electrically connected to the Kafka message cluster and is used to connect to the Kafka Zookeeper breakpoint ledger library. The deviation correction calculation unit is electrically connected to the centralized control center consumption write module and is used to connect to the centralized control write database. The closed-loop instruction issuance unit is electrically connected to the Kafka message cluster and is used to reverse-link the Kafka cluster to issue reissue instructions.

[0102] The global timing verification module employs an encrypted point-to-point retrieval method to actively retrieve the original hash fingerprint, source PTP timing stamp, and segment number ledger stored in the power station's CDC. Simultaneously, it retrieves the restored files, data tables, and incremental measurement point data from the centralized control terminal for fingerprint consistency verification and global timing verification. The fingerprint consistency verification uses a fingerprint consistency judgment algorithm to compare whether the source hash fingerprint and the secondary hash fingerprint of the restored data from the consumer end are consistent. The global timing verification is used to compare whether the clock deviation of the PTP timing stamps at both ends is within the allowable power deviation threshold.

[0103] The formula for the fingerprint consistency determination algorithm is as follows:

[0104] ,

[0105] In the formula, This is a verification indicator; 1 indicates successful verification, and 0 indicates failed verification. The original hash fingerprint from the source end. The consumption data is restored by a second hash fingerprint.

[0106] This invention partitions Kafka topics, splitting incremental real-time data and static full data for dual-channel transmission. Incremental operational data is delivered and consumed in milliseconds, no longer constrained by fixed synchronization cycles, breaking the minute-level and hour-level data latency. Dynamically changed data during the synchronization window can be synchronized instantly, meeting the second-level scheduling requirements of centralized control fault emergency response. Furthermore, through the peak-shaving and decoupling capabilities of the Kafka message cluster, large-volume configuration and historical full data are transmitted out of peak power business periods, and fragmented and packaged transmission is used instead of full database transmission. At the same time, the source end of the power station is decoupled from the centralized control write business. The power station is only responsible for data marking and delivery, without occupying local computing power to perform full packaging and comparison operations, which greatly reduces the read and write load of the power station database and avoids the full transmission crowding out the dedicated bandwidth of remote control and telemetry business.

[0107] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatic full data synchronization in a centralized control system based on Kafka message queues, applied to a data interaction network architecture between multiple substations and the power centralized control center, characterized by: Includes the following steps: S1. Pre-classify the data types of the full data of the substations of the power plant to obtain classified data. The classified data includes real-time incremental data, static full files of equipment configuration, and historical archive full data tables. S2. Dynamically mark the incremental data of real-time operating conditions through the CDC change capture component of the station, perform periodic hash fingerprint marking on the configuration static file and historical stock data table, and bind the unified time stamp of power PTP simultaneously to obtain the marked and classified data. S3. Based on the network bandwidth and data volume of the site, divide the Kafka topic into partitions, and deliver the tagged and classified data according to the classification and delivery rules. The Kafka message cluster performs persistent storage, transmission peak shaping, and breakpoint marking on the delivered messages. S4: The consumer end of the central control center is bound to the Kafka topic of the corresponding partition, and a multi-threaded parallel consumption mode is used for differentiated consumption parsing; S5. Retrieve the classification data and the labeled classification data through the full-domain time-series verification module, and perform a comparison and verification with the full data after consumption restoration. If the fingerprint matches the timing verification, the data will be written to the target database and configuration server in the centralized control center in compliance with regulations. If the verification is inconsistent or a transmission interruption is detected, the breakpoint information marked by the Kafka message cluster is retrieved, and a resend request is initiated only for abnormal shard data to synchronously correct the time-series misaligned data, thus completing the closed-loop self-healing of data synchronization.

2. The method for automatic full data synchronization in a centralized control system based on a Kafka message queue as described in claim 1, characterized in that: In step S1, when classifying data types, data that undergoes short-cycle millisecond-level dynamic changes and cannot be synchronized with delay is classified as real-time operating condition incremental data; visual configuration files with fixed file formats, rarely undergoing independent changes, and only updated when manually modified are classified as equipment configuration static full files. Historical data that has been archived, used only for daily verification, and is of a large volume is designated as a full-volume historical archive data table.

3. The method for automatic full data synchronization of a centralized control system based on a Kafka message queue according to claim 1, characterized in that: In step S2, the site CDC change capture component includes a database log listening interface unit, a file read / write sniffing unit, a time-series binding processing unit, and a local cache forwarding unit connected in series. The database log listening interface unit is connected to the site's local real-time business database kernel port. The file read / write sniffing unit is connected to the site's configuration file server read / write port. The time-series binding processing unit is bidirectionally connected to the database log listening interface unit and the file read / write sniffing unit, and the time-series binding processing unit has a built-in PTP timing chip and a hash computing chip. The local cache forwarding unit is connected to the site's Kafka front-end gateway.

4. The method for automatic full data synchronization of a centralized control system based on a Kafka message queue as described in claim 3, characterized in that: The following process is performed to dynamically mark changes to the real-time incremental operating data: A1. Real-time capture of the transaction logs of the site business database through the database log listening interface unit, filter the committed valid add, delete and modify transactions, and filter the invalid transactions that have been rolled back. A2. Extract the original field data of telemetry, teleindication, and alarm points within the transaction, call the differential change judgment algorithm to determine whether the data has undergone a valid change, and remove invalid data with numerical jitter, interference, or false triggering. A3. Bind the valid change data to the unique ID of the equipment in the field, generate the change tag code, and complete the dynamic change marking; A4. Push to the timing binding processing unit for delivery.

5. The method for automatic full data synchronization in a centralized control system based on a Kafka message queue as described in claim 4, characterized in that: When performing periodic hash fingerprinting on static configuration files and historical data tables, a fixed period of 1 hour is used as the fingerprint collection period. The file read / write sniffing unit periodically retrieves the full-domain field information of the target file and the existing data table, calls the SHA-256 encrypted hash fingerprint generation algorithm, and performs full-domain encryption operations on the file byte stream, data table field values, and field counts to generate a unique and irreversible hash fingerprint code. Each file / table is bound to a unique set of fingerprints. Any byte change in the file or data table will automatically change the hash fingerprint.

6. The method for automatic full data synchronization of a centralized control system based on a Kafka message queue according to claim 5, characterized in that: The synchronization and binding process for the unified time stamp of the power PTP is as follows: B1. The timing binding processing unit has a built-in PTPv2 power dedicated timing module, which connects to the regional power timing master clock to complete the full network time synchronization of the substation's local clock, with a clock synchronization error control of ≤1ms. B2. Upon completion of data marking, extract the absolute time sequence value of the entire network of the master clock standard. B3. Use the time series value as a suffix field and concatenate and bind it to the end of the incremental change label and hash fingerprint code message; B4. Use a clock skew calibration algorithm to verify the compliance of the timing stamp format. Unqualified timing stamps will be directly rebound.

7. The method for automatic full data synchronization of a centralized control system based on a Kafka message queue according to claim 1, characterized in that: In step S3, when partitioning the Kafka topic, a joint partitioning algorithm combining bandwidth weight and data volume entropy is used to complete the topic partitioning. The partitions of the Kafka topic include high-speed incremental partitions and full synchronization dedicated partitions. The high-speed incremental partitions belong to the high-speed incremental topic, have a large number of partitions, a small single partition cache capacity, and the highest send / receive priority. The full synchronization dedicated partitions belong to the full synchronization topic, have a small number of partitions, a large single partition cache capacity, and a lower send / receive priority than the incremental partitions.

8. The method for automatic full data synchronization of a centralized control system based on a Kafka message queue according to claim 7, characterized in that: The specific classification and delivery rules include: For real-time incremental data: After completing the change tagging and time-series binding, the CDC change capture component immediately pushes the data to the Kafka front-end gateway without time restrictions, and delivers it to the high-speed incremental partition in real time without any conditions. It preempts the transmission priority and ensures millisecond-level uplink. For static configuration files and historical data tables: data is not delivered in real time. The Kafka-Zookeeper node determines the business period of the site. The daily grid load valley value from 00:00 to 05:00 is defined as the low-peak business window. During the window, data is evenly divided and packaged according to the partition byte threshold. The size of a single fragment is uniformly controlled to 64MB. The fragment number is traced and bound to the source station fingerprint time sequence information. After packaging, the data is delivered to the dedicated partition for full synchronization. During peak business periods, the delivery of full fragment data is prohibited.

9. The method for automatic full data synchronization of a centralized control system based on a Kafka message queue according to claim 1, characterized in that: In step S5, the global timing verification module uses an encrypted point-to-point retrieval method to actively retrieve the original hash fingerprint, source PTP timing stamp, and segment number ledger stored in the power station CDC. At the same time, it retrieves the restored files, data tables, and incremental measurement point data from the centralized control terminal for fingerprint consistency verification and global timing verification. The fingerprint consistency verification uses a fingerprint consistency judgment algorithm to compare whether the source hash fingerprint and the secondary hash fingerprint of the restored data from the consumer terminal are from the same source. The global timing verification is used to compare whether the clock deviation of the PTP timing stamps at both ends is within the allowable power deviation threshold.

10. The method for automatic full data synchronization of a centralized control system based on a Kafka message queue according to claim 9, characterized in that: When correcting misaligned data, the time stamp of the data suffix on the central control terminal is rewritten based on the time compensation value to unify the time sequence across the entire network. If the fingerprint and time sequence are all verified as qualified, an inbound permission instruction is issued and the compliant data is entered into the database and archived. Then, the offset point of this transmission is updated synchronously in Kafka and the abnormal ledger is cleared. By superimposing daily low-peak baseline full-domain comparison and correction, a closed loop is formed from the entire chain of data collection, transmission, verification, correction, storage, and ledger update.