Electric energy metering data management system based on distributed storage

Abstract

The present application relates to the technical field of distributed storage, and specifically to an electric energy metering data management system based on distributed storage, which comprises a metering synchronization state module, which collects the disk IO queue depth value of a distributed storage slave node and the memory water level value at the current time in real time, generates a received available capacity, acquires a heartbeat confirmation identifier issued by a superior master node, merges the received available capacity and the heartbeat confirmation identifier, and acquires a back pressure state load. In the present application, the concurrent distribution load is disassembled, sorted in ascending order, reconstructed into a continuous log stream, written into a disk sector, sector write completion state code extracted, data feature code extracted, and the sector write completion state code and the data feature code are spliced into a node data synchronization voucher on the slave node side, which can make the disk writing state and content consistency form a synchronous closed loop verification, and enhance the checkability, traceability and settlement credibility of the electric energy metering data in the cross-node writing process.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to an energy metering data management system based on distributed storage. Background Technology

[0002] Distributed storage technology refers to the technology of storing data in a distributed manner, managing it collaboratively, accessing it uniformly, and protecting it with redundancy through multiple physical or logical nodes.

[0003] While existing technologies provide fundamental support for multi-node distributed storage and unified access, their practical application leans towards static organization of storage resources. They lack comprehensive constraints on the instantaneous load state of nodes, differences in log services, time-varying characteristics of transmission paths, and post-disk content verification, thus exposing shortcomings in electricity metering data scenarios. On one hand, node write pressure often only becomes apparent after data enters the replication link. When the master node lacks prior monitoring of slave node queuing and memory usage, continuous write operations can easily lead to buffer congestion at the receiving end. This can cause prolonged log residency time or, in severe cases, disrupt synchronization rhythm. For example, during peak metering collection periods, a large number of logs flood into the same slave node, leading to continuous disk queuing and potentially delaying subsequent accounting records. On the other hand, existing technologies often process different types of logs using similar strategies. Ordinary telemetry logs and billing settlement logs compete for the same resources in the synchronization link, affecting settlement timeliness and accounting continuity. On the other hand, while existing technologies possess multi-node redundancy capabilities, they lack fine-grained utilization of the real-time differences between multiple parallel transmission paths. When the round-trip delay distribution between paths changes, data blocks may still flow along fixed or coarse-grained paths, resulting in some links being idle and others congested, making it difficult to stably control the overall transmission delay. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a distributed storage-based power metering data management system.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a power metering data management system based on distributed storage includes: The metering synchronization status module collects the disk I / O queue depth value and the memory level value of the distributed storage slave node in real time to generate the available capacity for receiving; it obtains the heartbeat confirmation identifier issued by the superior master node, merges the available capacity for receiving with the heartbeat confirmation identifier, and obtains the back pressure status load. The metering log degradation module disassembles the back pressure state load to extract the available receiving capacity, compares it with a preset load warning threshold, and generates an asynchronous dump record; it locks the billing and settlement log package in a high-speed memory pool to generate a strongly synchronized log set. The multipath concurrent distribution module performs byte slicing on the strongly synchronized log set and obtains the real-time network round-trip time values ​​of multiple parallel transmission paths between master and slave nodes to generate multipath transmission data blocks; it retrieves the global log sequence number carried by each multipath transmission data block and the source end feature code extracted by the master node to generate concurrent distribution payload. The node disk feedback module imports the concurrently distributed load into the slave node's memory buffer pool and decomposes it to generate a reconstructed continuous log stream. It writes the reconstructed continuous log stream into the distributed database disk sector and obtains the sector write completion status code. It performs hash calculation on the reconstructed continuous log stream to extract the local data feature code. It concatenates and binds the sector write completion status code with the consistency verification status of the local data feature code and the source end feature code to obtain the node data synchronization certificate.

[0006] Preferably, the step of obtaining the back pressure load is as follows: The system reads the number of disk I / O queue entries of the distributed storage slave nodes in real time, records the disk I / O queue depth value at the corresponding time, and synchronously reads the proportion of occupied memory capacity to total memory capacity at the same time, records the memory level value at the corresponding time, and obtains the disk I / O queue depth value and memory level value. The remaining memory capacity at the current moment is calculated based on the memory level and the total memory capacity. A preset queue congestion attenuation constant is retrieved. The back pressure penalty coefficient is calculated by using the natural constant as the base and the negative of the product of the disk I / O queue depth and the queue congestion attenuation constant as the exponent. The remaining memory capacity is multiplied by the back pressure penalty coefficient to obtain the available receiving capacity. Obtain the heartbeat confirmation identifier issued by the superior master node, verify the node identifier content, issuance timing content and confirmation status content corresponding to the heartbeat confirmation identifier, write the received available capacity into the carrying capacity field, write the heartbeat confirmation identifier into the confirmation identifier field, merge them according to the fixed field arrangement order, and generate the back pressure state load.

[0007] Preferably, the step of obtaining the asynchronous dump record is as follows: The capacity field, node identifier field, and heartbeat confirmation field in the back pressure state load are analyzed. The receiveable capacity corresponding to the capacity field is extracted. The preset load warning threshold is retrieved. The receiveable capacity is compared with the preset load warning threshold bit by bit to determine the threshold. When the receiveable capacity is less than or equal to the preset load warning threshold, a degradation trigger flag is written and a load degradation determination result is generated. Read the degradation trigger flag in the load degradation judgment result, and after confirming the valid status corresponding to the degradation trigger flag, scan the log category identifier, log sequence number and queue entry order in the master node's queue to be allocated one by one, extract the queue entries of ordinary telemetry log packets corresponding to the log category identifier, extract the queue entries of billing and settlement log packets corresponding to the log category identifier, write the ordinary telemetry log packets to the storage location allocated by the object storage pool, record the storage address, writing batch identifier and log sequence number returned by the object storage pool, perform address mapping on the storage address, writing batch identifier and log sequence number, and generate billing and settlement log packets and asynchronous dump records.

[0008] Preferably, the steps for obtaining the strongly synchronized log set are as follows: Based on the billing and settlement log package and the asynchronous dump record, the billing and settlement log package is written into the locking area of ​​the high-speed memory pool, and the locking area address, log sequence number and locking status identifier are registered. The log sequence number, object storage address and dump completion identifier are extracted from the asynchronous dump record. Sequence association is performed according to the same log sequence number. The locked billing and settlement log package and the asynchronous dump record with completed sequence association are written into the synchronization combination area in a unified encapsulation order to generate a strongly synchronized log set.

[0009] Preferably, the step of obtaining the multipath transmission data block is as follows: Read the global log sequence number, log start byte position, and log end byte position from the strong synchronization log set. Cut the log byte interval into segments according to the preset slice length. Record the start byte position, end byte position, and slice sequence number corresponding to each slice. Bind each slice to its global log sequence number and write it into the slice registration area to generate the sliced ​​strong synchronization log set. Based on the sliced ​​strong synchronization log set, the real-time network round-trip time value corresponding to each parallel transmission path between the master and slave nodes is collected, and the real-time network round-trip time value is sorted in ascending order to form a parallel transmission path sorting sequence. A preset slice allocation ratio sequence corresponding to the parallel transmission path sorting sequence is retrieved one by one according to the sequence number. According to the preset slice allocation ratio sequence, the corresponding number of slice contents are extracted sequentially from the sliced ​​strong synchronization log set, and written into the parallel transmission path with the corresponding sequence number. The parallel transmission path identifier, slice sequence number, and global log sequence number corresponding to each slice content are recorded to generate a multipath transmission data block.

[0010] Preferably, the step of obtaining the concurrently distributed load is as follows: The global log sequence number registered in each multipath transmission data block is retrieved. The slice content, parallel transmission path identifier, and slice sequence number in each multipath transmission data block are extracted. The original complete data belonging to the same global log sequence number are hashed to obtain the source end feature code. The global log sequence number, slice sequence number, parallel transmission path identifier, slice content, and source end feature code are written in a fixed encapsulation field order. The registration relationship between the global log sequence number and the record to which the slice content belongs is verified. The multipath transmission data block is encapsulated block by block to generate a concurrently distributed payload.

[0011] Preferably, the step of obtaining the reconstructed continuous log stream is as follows: The concurrently distributed payload is written into the receiving area of ​​the slave node's memory buffer pool. The global log sequence number field, slice order number field, parallel transmission path identifier field, data content field, and source end feature code field are split according to the encapsulation field order. The global log sequence number, slice order number, and data content are extracted from each record. First, the data content is divided into homogeneous clusters according to the global log sequence number. Then, within the group of the same global log sequence number, the data content is arranged in ascending order according to the slice order number. The data content is then concatenated according to the arrangement result to form a reconstructed continuous log stream.

[0012] Preferably, the step of obtaining the node data synchronization certificate is as follows: According to the reconstructed continuous log stream, read the target sector number from the distributed database disk sector allocation table, write the reconstructed continuous log stream segment by segment into the distributed database disk sector according to the target sector number, record the sector start position, sector end position and write commit identifier of each segment of the written content, poll the write feedback content returned by the distributed database disk sector, extract the completion identifier field in the write feedback content, and form a sector write completion status code. The reconstructed continuous log stream associated with the sector write completion status code is retrieved. All character content in the reconstructed continuous log stream is scanned in byte order. A fixed-length summary result of the character content is extracted as a local data feature code. The local data feature code is compared with the split source end feature code. If the comparison is consistent, a valid verification status is generated. The sector write completion status code is written to the front of the voucher, and the valid verification status is written to the back of the voucher. Binding processing is performed according to a preset concatenation order to generate a node data synchronization voucher.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, the available receiving capacity is constructed based on the disk I / O queue depth and memory level of the distributed storage slave nodes. This available receiving capacity is then combined with the heartbeat confirmation identifier into a backpressure state load, allowing the master node to grasp the real-time carrying capacity boundary of the slave nodes before data is sent, preventing write pressure from accumulating in the later stages of the link transmission. A comparison is performed between the available receiving capacity and the load warning threshold. Ordinary telemetry log packets are dumped to the object storage pool, while billing and settlement log packets are locked in a high-speed memory pool. Simultaneously, a sequence association is established between asynchronous dump records and billing and settlement log packets, enabling data at different value levels to receive differentiated processing paths. This reduces the proportion of non-critical writes in the master-synchronous link while maintaining the priority residency and continuous traceability of billing and settlement log packets. Furthermore, a strong... Synchronous log sets are byte-sliced, and slice allocation is completed by combining the real-time network round-trip time values ​​of multiple parallel transmission paths. The data content and global log sequence number are then encapsulated into a concurrent distribution payload, which allows the transmission load to be dynamically distributed according to the link status, alleviating queuing backlog on a single path and ensuring that the sequence order remains recoverable under parallel transmission conditions. Around the concurrent distribution payload, the process of disassembling, ascending order reorganization, reconstructing the continuous log stream, writing disk sectors, extracting sector write completion status codes, and extracting data feature codes is completed on the slave node side. The sector write completion status code and data feature code are concatenated into a node data synchronization certificate, which enables synchronous closed-loop verification of disk status and content consistency, enhancing the verifiability, traceability, and settlement credibility of electricity metering data during cross-node writing. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the principle of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] Please see Figure 1 The present invention provides a technical solution: a power metering data management system based on distributed storage includes: The metering synchronization status module collects the disk I / O queue depth value and the memory level value of the distributed storage slave node in real time to generate the available receiving capacity; it obtains the heartbeat confirmation identifier issued by the superior master node, merges the available receiving capacity with the heartbeat confirmation identifier, and obtains the back pressure status load. The metering log degradation module disassembles the back pressure state load to extract the available receiving capacity, compares it with the preset load warning threshold, and generates asynchronous dump records; it locks the billing and settlement log package in a high-speed memory pool to generate a strongly synchronized log set. The multipath concurrent distribution module performs byte slicing on the strongly synchronized log set and obtains the real-time network round-trip time values ​​of multiple parallel transmission paths between master and slave nodes to generate multipath transmission data blocks; it retrieves the global log sequence number carried by each multipath transmission data block and the source end feature code extracted by the master node to generate concurrent distribution payload. The node disk feedback module imports the concurrently distributed load into the node's memory buffer pool and decomposes it to generate a reconstructed continuous log stream. It writes the reconstructed continuous log stream to the distributed database disk sector and obtains the sector write completion status code. It performs hash calculation on the reconstructed continuous log stream to extract the local data feature code. It concatenates and binds the sector write completion status code with the consistency verification status of the local data feature code and the source end feature code to obtain the node data synchronization certificate.

[0017] The steps for obtaining the back pressure load are as follows: The system reads the number of disk I / O queue entries of the distributed storage slave nodes in real time, records the disk I / O queue depth value at the corresponding time, and synchronously reads the proportion of occupied memory capacity to total memory capacity at the same time, records the memory level value at the corresponding time, and obtains the disk I / O queue depth value and memory level value. The remaining memory capacity at the current moment is calculated based on the memory level and the total memory capacity. The preset queue congestion attenuation constant is retrieved. The back pressure penalty coefficient is calculated by using the natural constant as the base and the negative of the product of the disk I / O queue depth and the queue congestion attenuation constant as the exponent. The remaining memory capacity is multiplied by the back pressure penalty coefficient to obtain the available receiving capacity. Obtain the heartbeat confirmation identifier issued by the superior master node, verify the node identifier content, issuance timing content and confirmation status content corresponding to the heartbeat confirmation identifier, write the received available capacity into the carrying capacity field, write the heartbeat confirmation identifier into the confirmation identifier field, merge them according to the fixed field arrangement order, and generate the back pressure state load.

[0018] Specifically, by calling interfaces provided by the operating system kernel, such as periodically reading the ` / sys / block / {device} / stat` file in a Linux environment or using the low-level system call of the `iostat` command, the total number of currently incomplete requests in the I / O request queue of a specific disk device (e.g., ` / dev / sda`) is obtained at a fixed sampling interval of 100 milliseconds. This value is recorded as the disk I / O queue depth at the corresponding sampling time. Simultaneously, to ensure the synchronization of data collection, within the same code execution cycle that reads the disk I / O queue depth, the system's memory usage information is immediately queried, for example, by parsing the ` / proc / meminfo` file to extract... We take two key metrics, MemTotal and MemAvailable (or a combination of MemFree, Buffers, and Cached), and calculate the proportion of occupied memory to total memory. Specifically, the calculation method is to divide the difference between (MemTotal and MemAvailable) by MemTotal. This calculated proportion is recorded as the memory level at the current moment. By assigning the same high-precision timestamp to each disk I / O queue depth value and memory level value, we ensure the temporal correspondence between these two metrics, ultimately obtaining a series of paired data of disk I / O queue depth values ​​and memory level values ​​indexed by timestamps.

[0019] Based on the memory level value obtained in the previous step and the pre-configured total node memory capacity, the remaining memory capacity at the current moment is first calculated. For example, if the total memory capacity is 64GB and the memory level value is 0.75, then the remaining memory capacity is 64*(1-0.75)=16GB. Next, a preset queue congestion attenuation constant is retrieved. This constant is an empirical value obtained through regression analysis of disk I / O queue depth values ​​and system throughput data under different loads over a historical month. Its goal is to ensure that the penalty coefficient significantly decreases when the queue depth reaches a performance inflection point. For example, analysis shows that when the disk I / O queue depth reaches 20, the system response time increases sharply. Therefore, the attenuation constant is set to 0.05. Thus, at a depth of 20, the exponent is -1, and the penalty coefficient is e to the power of -1, approximately equal to 0.368, forming an effective backpressure signal. Subsequently, the backpressure penalty coefficient is calculated using this constant. The calculation formula is as follows: ,in, This represents the final calculated back pressure penalty coefficient, with a value range of (0,1]. This represents the real-time disk I / O queue depth value obtained in the previous step. The predefined queue congestion attenuation constant represents the value of the queue. The base of the natural logarithm is used. Finally, the previously calculated remaining memory capacity is multiplied by this back pressure penalty coefficient. For example, multiplying 16GB of remaining memory capacity by the calculated back pressure penalty coefficient of 0.368 yields approximately 5.89GB. This result is the available receiving capacity that takes into account disk write pressure.

[0020] After receiving the heartbeat acknowledgment identifier sent by the superior master node over the network, the data structure of this identifier is first parsed. This identifier is a fixed-length data packet containing multiple fields, such as a 16-byte globally unique identifier (UUID) for the node, an 8-byte UNIX timestamp, and a 1-byte acknowledgment status code. Then, a strict verification process is executed, comparing the globally unique identifier in the identifier with the slave node's own identifier byte by byte to ensure the message is correctly attributed. Simultaneously, the timestamp in the identifier is read and compared with the current node's system time to determine if the delay is within a preset 500-millisecond window; if it exceeds this window, it is considered invalid. Next, the acknowledgment status code is checked; for example, 0x01 indicates that the master node has successfully acknowledged the previous heartbeat, and 0x0F indicates that the master node has requested a status update. The state is reset, and subsequent processing continues only when the status code is 0x01. After the verification passes, the back pressure state payload is constructed. This is a new data structure with a predefined format, such as a structure with a total length of 48 bytes. The received available capacity (e.g., 5.89GB) calculated in the previous step is converted into a 64-bit unsigned integer representation of the number of bytes and written into the first 8 bytes of the structure as the carrying capacity field. Then, the received and verified heartbeat acknowledgment identifier (e.g., 25 bytes) is directly copied to the position after the carrying capacity field as the acknowledgment identifier field. Finally, these byte streams are merged into a continuous data block according to the predefined fixed arrangement order of carrying capacity field first and acknowledgment identifier field last, generating the final back pressure state payload.

[0021] The steps for obtaining asynchronous dump records are as follows: The capacity field, node identifier field, and heartbeat confirmation field in the back pressure state load are parsed. The receiveable capacity corresponding to the capacity field is extracted, the preset load warning threshold is retrieved, and the receiveable capacity is compared with the preset load warning threshold bit by bit to determine the threshold. When the receiveable capacity is less than or equal to the preset load warning threshold, a degradation trigger flag is written, and a load degradation determination result is generated. Read the degradation trigger flag in the load degradation judgment result, and after confirming the valid status corresponding to the degradation trigger flag, scan the log category identifier, log sequence number and queue entry order in the master node's queue to be allocated one by one, extract the queue entries of ordinary telemetry log packets corresponding to the log category identifier, extract the queue entries of billing and settlement log packets corresponding to the log category identifier, write the ordinary telemetry log packets to the storage location allocated by the object storage pool, record the storage address, write batch identifier and log sequence number returned by the object storage pool, perform address mapping on the storage address, write batch identifier and log sequence number, and generate billing and settlement log packets and asynchronous dump records.

[0022] Specifically, based on the predefined backpressure state load data structure, such as a fixed format with a total length of 48 bytes, the load is first parsed in a structured manner. The first 8 bytes of its header are read and interpreted as a 64-bit unsigned integer, thereby extracting the available receiving capacity in bytes. Next, a load warning threshold preset in the system configuration file is retrieved. This threshold is set based on the minimum memory buffer required to ensure that at least one complete batch of billing and settlement logs can be processed without delay under full-load synchronization conditions. The specific calculation method is to calculate the maximum volume of billing and settlement log packets to be synchronized per unit time (e.g., 1 second) over the past week, and then multiply it by a safety factor, for example... If the maximum size is 512MB and the security factor is set to 1.5, then the load warning threshold is 768MB. Subsequently, the extracted available receive capacity is compared with this 768MB load warning threshold. This comparison is a direct integer comparison, not a bitwise comparison. When the result is that the available receive capacity is less than or equal to the load warning threshold, a boolean field named degradation_flag is set to true in a newly created memory data structure; otherwise, it is set to false. This data structure, which contains the degradation_flag field and the associated node identifier, generates the load degradation determination result.

[0023] First, the `degradation_flag` field in the load degradation judgment result generated in the previous step is read. Once its value is confirmed to be true, indicating that the degradation condition has been triggered, the log distribution processing flow is immediately initiated. This flow follows a first-in, first-out (FIFO) order, starting from the head of the allocation queue in the master node's memory and scanning each log entry in the queue. Each log entry contains an 8-bit log category identifier to distinguish the log type, a globally unique 64-bit log sequence number, and the log data itself. During the scanning process, log entries are sorted according to their category identifier. For example, when the category identifier is 0x01, it is determined to be a normal telemetry log packet; when the category identifier is 0x02, it is determined to be a billing and settlement log packet. For all entries determined to be normal telemetry log packets, their data content is processed through a pre-configured... Object storage client interfaces, such as the PUTObjectAPI compliant with the Amazon S3 protocol, upload data to a specified bucket in a highly available object storage pool. After the upload operation is complete, the object storage service returns information containing the storage address (e.g., a URI) and an ETag as a write batch identifier. At this point, the returned storage address and write batch identifier are bound to the original log sequence number of the log packet and encapsulated into a structured record. Meanwhile, all entries identified as billing and settlement log packets are retrieved from the allocation queue and stored in memory. Finally, these stored billing and settlement log packets are combined with the structured records containing address mapping information generated for all ordinary telemetry log packets to form the billing and settlement log packet and the asynchronous dump record.

[0024] The steps to obtain a strongly synchronized log set are as follows: Based on the billing and settlement log package and the asynchronous dump record, the billing and settlement log package is written into the locking area of ​​the high-speed memory pool, and the locking area address, log sequence number and locking status identifier are registered. The log sequence number, object storage address and dump completion identifier are extracted from the asynchronous dump record. Sequence association is performed according to the same log sequence number. The locked billing and settlement log package and the asynchronous dump record with completed sequence association are written into the synchronization combination area in a unified encapsulation order to generate a strongly synchronized log set.

[0025] Specifically, based on the billing and settlement log packets filtered and retained in memory in the previous step, and the asynchronous dump records generated for ordinary telemetry log packets, the complete data content of each billing and settlement log packet is first copied to a specific area of ​​a high-speed memory pool locked via the mlock() system call. This memory pool is a pre-allocated physical memory area that is prohibited from being swapped to virtual memory by the operating system. After the copy operation is completed, the memory pool manager returns a stable memory address pointing to the data block. Subsequently, in a global registry table implemented as a hash table, an entry containing the locked area address and a lock status identifier with a value of 1 is created using the log sequence number of the log packet as the key. Then, all asynchronous dump records are traversed from... Each record has its log sequence number extracted and used as a key to search the global registry table to locate the locked billing and settlement log packet with the same log sequence number. Once a matching entry is found, a new composite data structure is created according to a predefined uniform encapsulation order. This structure first contains the complete binary data of the locked billing and settlement log packet, followed by the complete contents of the associated asynchronous dump record (including the object storage address and dump completion identifier). This new composite data structure is written as an independent entry into a memory buffer called the synchronization composite area. This process is repeated until all billing and settlement log packets have been associated, labeled, and encapsulated, ultimately forming a strongly synchronized log set in the synchronization composite area.

[0026] The steps for obtaining multipath transmission data blocks are as follows: Read the global log sequence number, log start byte position, and log end byte position from the strong synchronization log set. Cut the log byte range into segments according to the preset slice length. Record the start byte position, end byte position, and slice sequence number corresponding to each slice. Bind each slice to its global log sequence number and write it into the slice registration area to generate the sliced ​​strong synchronization log set. Based on the sliced ​​strong synchronization log set, the real-time network round-trip time value corresponding to each parallel transmission path between master and slave nodes is collected and sorted in ascending order according to the real-time network round-trip time value to form a parallel transmission path sorting sequence. The preset slice allocation ratio sequence corresponding to the parallel transmission path sorting sequence is retrieved one by one according to the sequence number. According to the preset slice allocation ratio sequence, the corresponding number of slice contents are extracted sequentially from the sliced ​​strong synchronization log set and written into the parallel transmission path with the corresponding sequence number. The parallel transmission path identifier, slice sequence number and global log sequence number corresponding to each slice content are recorded to generate multipath transmission data blocks.

[0027] Specifically, for each entry in the strongly synchronized log set, its metadata is first parsed to extract the global log sequence number, the start byte position, and the end byte position of the log data. Then, based on a preset slice length, which takes into account the maximum transmission unit (MTU) of the network transmission protocol and the need to reduce packet overhead (e.g., 1400 bytes), the entire byte interval from the start byte position to the end byte position of the log is truncated segment by segment. If the length of the last segment is less than 1400 bytes, it is truncated according to its actual remaining length. Each time a truncation is completed, a data slice is generated, and the content corresponding to that slice is immediately recorded. The start and end byte positions of the log data are determined, and a slice sequence number starting from 0 is assigned to each slice. For example, a 5000-byte log will be divided into four slices, numbered 0, 1, 2, and 3, with lengths of 1400, 1400, 1400, and 800 bytes respectively. Finally, the content of each slice is bound to its global log sequence number and written as a new record to a slice registry area implemented with a hash table structure for temporary storage. The key of this registry area is the global log sequence number, and the value is a list containing information about all slices. After this processing, a strongly synchronized log set after slices is obtained.

[0028] Based on the strongly synchronized log set generated in the previous step after slicing, a parallel network probing task is first initiated. For all parallel transmission paths established between the master and slave nodes, such as three independent TCP connections, an ICMPEcho request message with a high-precision timestamp is sent per second through each path. Upon receiving a response, the difference between the sending and receiving times is calculated to obtain the real-time network round-trip time (RTT) for each path. Subsequently, these three paths are sorted in ascending order according to their corresponding RTT values, for example, path 2 (5ms), path 1 (8ms), and path 3 (12ms), forming a dynamic parallel transmission path sorting sequence. Next, a preset slice allocation ratio sequence corresponding to the number of parallel transmission paths is retrieved. This ratio sequence is derived from long-term operational data statistics and aims to allocate more data to higher-quality slices. For example, for three paths, the sequence might be set to {0.5, 0.3, 0.2}. This means that the path with the lowest RTT will be allocated 50% of the slices, the second lowest 30%, and the highest 20%. Then, based on this slice allocation ratio sequence, the corresponding number of slice contents are extracted sequentially from the strongly synchronized log set after the slices, according to the slice order number. For example, if there are a total of 100 slices to be sent, then path 2 will allocate 50, path 1 will allocate 30, and path 3 will allocate 20. These allocated slice contents are written into the send buffer of the corresponding parallel transmission path. At the same time as sending, the parallel transmission path identifier (e.g., path ID), slice order number, and global log sequence number corresponding to each slice content are recorded. This set of information containing slice data and its metadata is used to generate a multipath transmission data block.

[0029] The steps for obtaining concurrently distributed payloads are as follows: Retrieve the global log sequence number registered in each multipath transmission data block, extract the slice content, parallel transmission path identifier, and slice sequence number from each multipath transmission data block, perform hash calculation on the original complete data belonging to the same global log sequence number to obtain the source end feature code, write the global log sequence number, slice sequence number, parallel transmission path identifier, slice content, and source end feature code in a fixed encapsulation field order, verify the registration relationship between the global log sequence number and the record to which the slice content belongs, complete the block-by-block encapsulation of the multipath transmission data block, and generate concurrently distributed payloads.

[0030] Specifically, for each multipath transmission data block to be sent via a specific parallel transmission path, the internally registered global log sequence number is first retrieved, and the slice content, parallel transmission path identifier, and slice sequence number are extracted. Then, before data transmission, a hash calculation is performed on the original complete data belonging to the same global log sequence number. This original complete data is obtained by reassembling all slices in order according to the global log sequence number from the slice registration area. The hash algorithm used is SHA-256, and the result is a 32-byte digest, which serves as the source-end signature. Then, the data is processed according to a fixed encapsulation field. The data packets to be sent are constructed in the following order: first, an 8-byte global log sequence number; then, a 4-byte slice sequence number; next, a 1-byte parallel transmission path identifier; then, the variable-length slice content (maximum 1400 bytes); and finally, a 32-byte source signature. During the encapsulation process, a verification operation is performed to confirm that the global log sequence number of the currently encapsulated slice content matches the global log sequence number in the encapsulation field. In this way, all multipath transmission data blocks are encapsulated one by one, ultimately forming a series of uniformly structured concurrently distributed payloads to be sent concurrently.

[0031] The steps for reconstructing a continuous log stream are as follows: The concurrently distributed payload is written to the receiving area of ​​the slave node's memory buffer pool. The global log sequence number field, slice order number field, parallel transmission path identifier field, data content field, and source end feature code field are split according to the encapsulation field order. The global log sequence number, slice order number, and data content are extracted from each record. First, the data content is divided into homogeneous clusters according to the global log sequence number. Then, within the group of the same global log sequence number, the data content is arranged in ascending order according to the slice order number. The data content is then concatenated according to the arrangement result to form a reconstructed continuous log stream.

[0032] Specifically, the concurrently distributed payloads received from the network interface are written one by one into a pre-allocated 128MB circular receiving area in the node's memory buffer pool. Then, according to the fixed encapsulation field order of the concurrently distributed payload, the byte stream of each payload is split. Specifically, byte offsets 0 to 7 are read as a 64-bit global log sequence number field, byte offsets 8 to 11 are read as a 32-bit slice sequence number field, byte offset 12 is read as an 8-bit parallel transmission path identifier field, byte offset 45 to the end is read as a 32-byte source signature field, and the variable-length portion between the two is read as the data content field. After splitting, the global log sequence number, slice sequence number, and data content are extracted from each record. The extracted information is temporarily stored in a hash mapping table with the global log sequence number as the key and the slice information list as the value. This operation realizes the homogeneous clustering of data slices from different parallel paths. When all expected slices of a specific global log sequence number (based on the total number of slices negotiated with the master node) arrive, or after the preset 2-second reorganization timeout, the reconstruction process of the group is triggered. Within the group of the same global log sequence number, the insertion sort algorithm is used to sort all slice information in ascending order according to the slice sequence number. Finally, according to the sorted result, the data content of each slice is concatenated end to end in a new memory buffer to form a byte-continuous reconstructed continuous log stream that is completely consistent with the source log.

[0033] The steps to obtain the node data synchronization certificate are as follows: Based on the reconstructed continuous log stream, read the target sector number from the distributed database disk sector allocation table, write the reconstructed continuous log stream segment by segment into the distributed database disk sector according to the target sector number, record the sector start position, sector end position and write commit flag of each segment of the written content, poll the write feedback content returned by the distributed database disk sector, extract the completion flag field in the write feedback content, and form a sector write completion status code. Retrieve the reconstructed continuous log stream associated with the sector write completion status code, scan all character content in the reconstructed continuous log stream in byte order, extract a fixed-length summary result of the character content as the local data feature code, compare the local data feature code with the split source end feature code for consistency, if the comparison is consistent, generate a valid verification status, write the sector write completion status code to the front of the voucher, write the valid verification status to the back of the voucher, perform binding processing according to the preset splicing order, and generate node data synchronization voucher.

[0034] Specifically, based on the reconstructed continuous log stream generated in the previous step, the internal space allocation interface of the distributed database storage engine is first called, passing in the total byte length of the reconstructed continuous log stream. This interface queries the internally maintained B+ tree structure disk sector allocation table, which records information on all free disk blocks, and returns a set of target sector numbers that are sufficient to accommodate the log stream and may be physically discontinuous. Then, according to this set of target sector numbers, the reconstructed continuous log stream is divided into multiple data segments equal to the disk sector size (e.g., 4KB), and an asynchronous write request containing the target sector number and data content is generated for each data segment. This is then batch-processed through the underlying io_uring interface. The system submits these write requests to the operating system kernel. At the same time, it generates a unique write commit identifier for this entire batch of write operations. It then associates the start and end positions of the sectors corresponding to each data segment with this identifier and records them in an in-transit write trace table. Subsequently, it continuously polls the io_uring completion queue at 10-millisecond intervals to check the write feedback content related to the write commit identifier. When all data segment write operations return a completion status, it extracts the specific integer value (e.g., 0 for success) returned by the disk driver from the feedback content, representing whether the write was successful or failed. These values ​​are then combined to form the sector write completion status code.

[0035] Based on the memory copy of the reconstructed continuous log stream associated with the sector write completion status code, the data verification process is initiated. This process first processes all characters in the reconstructed continuous log stream as a continuous byte array, inputting it into a SHA-256 hash algorithm computation instance, byte-by-byte in byte order from beginning to end. The algorithm outputs a 32-byte fixed-length digest, which is the local data signature. Then, this newly calculated local data signature is compared byte-by-byte in memory with the source signature, which was split and saved from the concurrently distributed payload. If the two 32-byte hashes are compared... If the feature codes of the segments are completely identical, the comparison function returns 0, and a single-byte valid verification status with a value of 0x01 is generated. If they are inconsistent, an invalid verification status with a value of 0x00 is generated. Finally, according to a preset concatenation order, a node data synchronization certificate with a total length of 9 bytes is created. The 8-byte sector writing completion status code obtained in the previous step (if the status code itself is less than 8 bytes, the high bits are padded with zeros) is written into the first 8 bytes of the certificate as the front part of the certificate. Then, this single-byte valid verification status is written into the 9th byte of the certificate as the back part of the certificate. Through this binding process, the node data synchronization certificate is finally generated.

Claims

1. A power metering data management system based on distributed storage, characterized in that, The system includes: The metering synchronization status module collects the disk I / O queue depth value and the memory level value of the distributed storage slave node in real time to generate the available capacity for receiving; it obtains the heartbeat confirmation identifier issued by the superior master node, merges the available capacity for receiving with the heartbeat confirmation identifier, and obtains the back pressure status load. The metering log degradation module disassembles the back pressure state load to extract the available receiving capacity, compares it with a preset load warning threshold, and generates an asynchronous dump record; it locks the billing and settlement log package in a high-speed memory pool to generate a strongly synchronized log set. The multipath concurrent distribution module performs byte slicing on the strongly synchronized log set and obtains the real-time network round-trip time values ​​of multiple parallel transmission paths between master and slave nodes to generate multipath transmission data blocks; it retrieves the global log sequence number carried by each multipath transmission data block and the source end feature code extracted by the master node to generate concurrent distribution payload. The node disk feedback module imports the concurrently distributed load into the slave node's memory buffer pool and decomposes it to generate a reconstructed continuous log stream. It writes the reconstructed continuous log stream into the distributed database disk sector and obtains the sector write completion status code. It performs hash calculation on the reconstructed continuous log stream to extract the local data feature code. It concatenates and binds the sector write completion status code with the consistency verification status of the local data feature code and the source end feature code to obtain the node data synchronization certificate.

2. The distributed storage based electric energy metering data management system of claim 1, wherein, The steps for obtaining the back pressure load are as follows: The system reads the number of disk I / O queue entries of the distributed storage slave nodes in real time, records the disk I / O queue depth value at the corresponding time, and synchronously reads the proportion of occupied memory capacity to total memory capacity at the same time, records the memory level value at the corresponding time, and obtains the disk I / O queue depth value and memory level value. The remaining memory capacity at the current moment is calculated based on the memory level and the total memory capacity. A preset queue congestion attenuation constant is retrieved. The back pressure penalty coefficient is calculated by using the natural constant as the base and the negative of the product of the disk I / O queue depth and the queue congestion attenuation constant as the exponent. The remaining memory capacity is multiplied by the back pressure penalty coefficient to obtain the available receiving capacity. Obtain the heartbeat confirmation identifier issued by the superior master node, verify the node identifier content, issuance timing content and confirmation status content corresponding to the heartbeat confirmation identifier, write the received available capacity into the carrying capacity field, write the heartbeat confirmation identifier into the confirmation identifier field, merge them according to the fixed field arrangement order, and generate the back pressure state load.

3. The distributed storage based electric energy metering data management system of claim 1, wherein, The steps for obtaining the asynchronous dump record are as follows: The capacity field, node identifier field, and heartbeat confirmation field in the back pressure state load are analyzed. The receiveable capacity corresponding to the capacity field is extracted. The preset load warning threshold is retrieved. The receiveable capacity is compared with the preset load warning threshold bit by bit to determine the threshold. When the receiveable capacity is less than or equal to the preset load warning threshold, a degradation trigger flag is written and a load degradation determination result is generated. Read the degradation trigger flag in the load degradation judgment result, and after confirming the valid status corresponding to the degradation trigger flag, scan the log category identifier, log sequence number and queue entry order in the master node's queue to be allocated one by one, extract the queue entries of ordinary telemetry log packets corresponding to the log category identifier, extract the queue entries of billing and settlement log packets corresponding to the log category identifier, write the ordinary telemetry log packets to the storage location allocated by the object storage pool, record the storage address, writing batch identifier and log sequence number returned by the object storage pool, perform address mapping on the storage address, writing batch identifier and log sequence number, and generate billing and settlement log packets and asynchronous dump records.

4. The distributed storage based electric energy metering data management system of claim 1, wherein, The steps for obtaining the strongly synchronized log set are as follows: Based on the billing and settlement log package and the asynchronous dump record, the billing and settlement log package is written into the locking area of ​​the high-speed memory pool, and the locking area address, log sequence number and locking status identifier are registered. The log sequence number, object storage address and dump completion identifier are extracted from the asynchronous dump record. Sequence association is performed according to the same log sequence number. The locked billing and settlement log package and the asynchronous dump record with completed sequence association are written into the synchronization combination area in a unified encapsulation order to generate a strongly synchronized log set.

5. The distributed storage based electric energy metering data management system of claim 1, wherein, The steps for obtaining the multipath transmission data block are as follows: Read the global log sequence number, log start byte position, and log end byte position from the strong synchronization log set. Cut the log byte interval into segments according to the preset slice length. Record the start byte position, end byte position, and slice sequence number corresponding to each slice. Bind each slice to its global log sequence number and write it into the slice registration area to generate the sliced ​​strong synchronization log set. Based on the sliced ​​strong synchronization log set, the real-time network round-trip time value corresponding to each parallel transmission path between the master and slave nodes is collected, and the real-time network round-trip time value is sorted in ascending order to form a parallel transmission path sorting sequence. A preset slice allocation ratio sequence corresponding to the parallel transmission path sorting sequence is retrieved one by one according to the sequence number. According to the preset slice allocation ratio sequence, the corresponding number of slice contents are extracted sequentially from the sliced ​​strong synchronization log set, and written into the parallel transmission path with the corresponding sequence number. The parallel transmission path identifier, slice sequence number, and global log sequence number corresponding to each slice content are recorded to generate a multipath transmission data block.

6. The distributed storage based electric energy metering data management system of claim 1, wherein, The steps for obtaining the concurrently distributed load are as follows: The global log sequence number registered in each multipath transmission data block is retrieved. The slice content, parallel transmission path identifier, and slice sequence number in each multipath transmission data block are extracted. The original complete data belonging to the same global log sequence number are hashed to obtain the source end feature code. The global log sequence number, slice sequence number, parallel transmission path identifier, slice content, and source end feature code are written in a fixed encapsulation field order. The registration relationship between the global log sequence number and the record to which the slice content belongs is verified. The multipath transmission data block is encapsulated block by block to generate a concurrently distributed payload.

7. The distributed storage based electric energy metering data management system of claim 1, wherein, The steps for obtaining the reconstructed continuous log stream are as follows: The concurrently distributed payload is written into the receiving area of ​​the slave node's memory buffer pool. The global log sequence number field, slice order number field, parallel transmission path identifier field, data content field, and source end feature code field are split according to the encapsulation field order. The global log sequence number, slice order number, and data content are extracted from each record. First, the data content is divided into homogeneous clusters according to the global log sequence number. Then, within the group of the same global log sequence number, the data content is arranged in ascending order according to the slice order number. The data content is then concatenated according to the arrangement result to form a reconstructed continuous log stream.

8. The distributed storage based electric energy metering data management system of claim 1, wherein, The steps for obtaining the node data synchronization certificate are as follows: According to the reconstructed continuous log stream, read the target sector number from the distributed database disk sector allocation table, write the reconstructed continuous log stream segment by segment into the distributed database disk sector according to the target sector number, record the sector start position, sector end position and write commit identifier of each segment of the written content, poll the write feedback content returned by the distributed database disk sector, extract the completion identifier field in the write feedback content, and form a sector write completion status code. The reconstructed continuous log stream associated with the sector write completion status code is retrieved. All character content in the reconstructed continuous log stream is scanned in byte order. A fixed-length summary result of the character content is extracted as a local data feature code. The local data feature code is compared with the split source end feature code. If the comparison is consistent, a valid verification status is generated. The sector write completion status code is written to the front of the voucher, and the valid verification status is written to the back of the voucher. Binding processing is performed according to a preset concatenation order to generate a node data synchronization voucher.

Images