Distributed persistent log storage system and method based on storage-level memory

Abstract

The invention relates to the technical field of log storage, and discloses a distributed persistent log storage system and method based on a storage-level memory, and the method comprises the following steps: monitoring network connection states of three AZs in real time, when a partition event is detected, automatically switching a write-in operation to a weak consistency mode by the system, temporarily storing log data by using an SCM, and storing the log data by using the SCM; and during the network partitioning period, dynamically adjusting the quorum number of the RAFT consensus protocol, recalculating the weight according to the communication state of each AZ, automatically degrading to a weak consistent write-in mode, and continuously monitoring a partition recovery signal at the same time. According to the method, in high-sensitivity scenes such as power grid control instructions and the like, the low-delay transmission advantage of RDMA is maintained, the double-master data conflict risk caused by network isolation is eliminated, the global uniqueness of power operation instructions and the safety and stability of the system are guaranteed, and the requirements for millisecond-level fault isolation and zero data conflict are met.

Description

Technical Field

[0001] This invention relates to the field of log storage, and more specifically, to a distributed persistent log storage system and method based on storage-level memory. Background Technology

[0002] In an architecture that integrates cloud-native and edge computing, traditional log storage systems face challenges such as throughput bottlenecks, high latency, and low fault recovery efficiency.

[0003] In particular, in high-performance distributed storage based on RDMA, when a network partition occurs, the traditional TCP / IP layer can detect the partition through a timeout mechanism. However, the zero-copy kernel bypass feature of RDMA makes it impossible for the application layer to directly perceive the link interruption, causing multiple partitions to simultaneously believe that they are the primary partition, thus leading to data conflicts. Summary of the Invention

[0004] This invention provides a distributed persistent log storage system and method based on storage-level memory, which solves the technical problem of protocol split-brain under RDMA network partitioning in traditional technology, which cannot achieve sub-millisecond persistent writing and fault self-healing capabilities.

[0005] This invention provides a distributed persistent log storage method based on storage-level memory, comprising the following steps:

[0006] S100, Partition Status Detection and Weak Consistency Write: Real-time monitoring of network connectivity status in three AZs. When a partition event is detected, the system automatically switches the write operation to weak consistency mode, uses SCM to temporarily store log data, and records hybrid HLC timestamps and service priorities.

[0007] S200, Dynamic Quorum Adjustment: During network partitioning, the quorum of the RAFT consensus protocol is dynamically adjusted, the weights are recalculated based on the connectivity status of each Availability Zone (AZ), and the system automatically downgrades to weak consistency write mode while continuously monitoring partition recovery signals.

[0008] S300, conflict merging after partition recovery: When the network partition is recovered, logs to be merged are collected from the weak consistency groups of each AZ, sorted by HLC and time-series conflicts are detected, concurrent write conflicts are resolved through business priority arbitration, the merged logs are submitted to the strong consistency group in batches, the global vector clock state is updated and weak consistency data is cleaned up.

[0009] S400, Spatiotemporal Sharding Hot and Cold Tiering: Calculates heat values ​​based on the spatiotemporal characteristics and access frequency of logs, dynamically determines storage tiering thresholds, migrates data to storage media with different performance levels, performs compression on cold data to optimize storage costs, and updates metadata indexes to achieve efficient query routing.

[0010] Furthermore, the specific steps for partition status detection and weakly consistent writing are as follows:

[0011] S110, Partition Status Detection: Calls the partition detector to calculate cross-AZ network connectivity via UDP heartbeat packets;

[0012] S120, Business Priority Verification: Parse the business tags in the logs to determine whether strong consistency is required;

[0013] S130, Routing Decision and Writing: Dynamically select the writing protocol based on the system partition status and business priority, and perform SCM persistence operations;

[0014] S140, Hybrid Logic Clock Tag: Generates globally sortable timing tags for weakly consistent logs;

[0015] S150, Status Flags and Response: Returns the write result to the client, with attached status flags.

[0016] Furthermore, routing decisions and writes include strong consistency group writes and weak consistency group writes;

[0017] Strong consistency group write: When the service priority is greater than or equal to 7 and the system is in a normal state, the RAFT and DC protocols are used for cross-availability zone replication;

[0018] Weak consistency group write: When the business priority is less than 7 or the system is in isolation, local SCM direct write is performed.

[0019] Furthermore, the specific steps for adjusting the dynamic quorum are as follows:

[0020] S210, Survival Node Statistics: Based on the output of the partition detector, count the number of currently surviving nodes;

[0021] S220, Correction Factor Generation: Calculate the correction factor based on the severity of the partition to ensure that the quorum is strictly greater than the partition boundary;

[0022] S230, Dynamic quorum calculation: A dynamic quorum is generated by combining the number of surviving nodes and the correction factor;

[0023] S240, Brain Split Immune Validation: Validation that dynamic quorum can prevent brain split in any partitioned scenario;

[0024] S250, Write permission decision: Before writing to a strong consistency group, check whether the current partition meets the dynamic quorum requirement.

[0025] Furthermore, the specific verification steps for the brain split immune verification are as follows:

[0026] Verification is performed by contradiction; assume that two network partitions P1 and P2 exist in the system, containing k and m nodes respectively;

[0027] For a split-brain scenario to occur, both partitions must meet the condition that the number of nodes is greater than or equal to the dynamic quorum. At the same time, the total number of nodes in the two partitions must be less than or equal to the total number of surviving nodes in the system. Substituting these conditions into the calculation reveals that the minimum total number of nodes required for the two partitions is greater than the actual total number of surviving nodes in the system, which contradicts the reality.

[0028] The final dynamic quorum must be greater than or equal to 1 and less than or equal to the current number of surviving nodes.

[0029] Furthermore, the specific content of the write permission decision is as follows: Before performing a write operation, the system compares the current number of live nodes with the dynamic quorum.

[0030] When the number of surviving nodes is greater than or equal to the dynamic quorum, it means that the current partition meets the write conditions, the system returns a commit allowed status, and ensures that the response delay of the permission check does not exceed 100 microseconds;

[0031] If the number of surviving nodes is less than the dynamic quorum, the write freeze (FREEZE) state is returned.

[0032] For frozen write requests, the system will schedule retry, with the retry interval employing an exponential backoff strategy.

[0033] Furthermore, the specific steps for merging conflicts after partition recovery are as follows:

[0034] S310, Recovery Event Listener: Listens to the partition detector and captures partition recovery events;

[0035] S320, Weak Consistency Log Collection: Pulls logs written during the isolation period from weak consistency groups in all availability zones;

[0036] S330, Timing Conflict Detection: Sort all logs by mixed logic clock values ​​to detect concurrent conflicts;

[0037] S340, Business Priority Arbitration: Arbitrate concurrent conflict logs according to business priority;

[0038] S350, Vector Clock Synchronization: Updates the global vector clock status and marks the timing position of merged logs;

[0039] S360, Strong Consistency Group Commit: Batch commits merged logs to a strong consistency group, using dynamic quorum confirmation;

[0040] S370, State Cleanup and Notification: Clean up weakly consistent group data and update client log status.

[0041] Furthermore, the specific steps for spatiotemporal segmentation and hot / cold stratification are as follows:

[0042] S410, Spatiotemporal Feature Extraction: Analyze the spatiotemporal attributes of logs and calculate the time decay factor and spatial locality weight;

[0043] S420, Access Popularity Calculation: Statistical log access frequency is combined with spatiotemporal characteristics to generate a comprehensive heat value;

[0044] S430, tiered threshold decision: Determine the storage tiered threshold based on thermal value distribution;

[0045] S440, Data Sharding Migration: Migrate logs to the corresponding storage layer based on heat value;

[0046] S450, Cold Data Compression: Performs lossless compression on cold data;

[0047] S460, Metadata Update: Update storage location metadata;

[0048] S470, Layered monitoring feedback: Dynamically adjusts the weighting parameters of thermal values.

[0049] Furthermore, the steps for updating metadata are as follows:

[0050] The unique identifier of the log is hashed to obtain a fixed-length hash value. Then, the hash value is XORed with the storage level identifier to obtain the final storage location index.

[0051] Storage tier identifiers include SCM, QLC-NVMe, and HDD.

[0052] This invention also proposes a distributed persistent log storage system based on storage-level memory, which performs the steps of a distributed persistent log storage method based on storage-level memory as described above, including:

[0053] Partition Status Detection and Weak Consistency Write Module: This module monitors the network connectivity status of the three Availability Zones (AZs) in real time. When a network partition event is detected, the write operation is automatically switched to weak consistency mode. Log data is temporarily written to local storage-level memory, while recording a hybrid logical clock timestamp and a business priority tag.

[0054] Dynamic quorum adjustment module: During the network partitioning period, this module dynamically adjusts the quorum weight of the RAFT consensus protocol, recalculates the arbitration threshold based on the real-time connectivity status of each AZ, maintains partial system availability by automatically downgrading to weak consistency write mode, and continuously monitors partition recovery signals.

[0055] Partition recovery conflict merging module: When the network recovers, this module collects logs to be merged from the weakly consistent storage area of ​​each AZ, sorts them by mixed logical clock timestamps to identify timing conflicts, resolves concurrent write conflicts through the business priority arbitration mechanism, submits the merged logs to the strong consistency group in batches, updates the global vector clock state and cleans up the weakly consistent storage area.

[0056] Spatiotemporal sharding hot and cold stratification module: This module analyzes the spatiotemporal characteristics and access frequency of logs, dynamically calculates heat values ​​and determines stratification thresholds, migrates data to three-tier storage media based on heat values, performs compression on cold data to optimize storage costs, and updates metadata indexes to achieve efficient query routing across storage tiers;

[0057] Global Metadata Coordination Module: Unifies the management of log storage location index, vector clock status, and hierarchical strategy parameters, provides metadata query services for all modules, accelerates metadata access through distributed caching, and synchronizes the status changes of each module in real time;

[0058] Layered Compression and Cost Optimization Module: For the cold data storage layer, it implements lossless compression algorithms to reduce storage overhead, dynamically adjusts compression strategies based on data characteristics and access patterns, monitors the capacity and performance indicators of each storage layer, and automatically triggers data reorganization or migration to balance storage costs and access latency.

[0059] The beneficial effects of this invention are as follows:

[0060] This invention utilizes the synergistic effect of the RDMA link layer active liveness detection mechanism and the storage layer distributed lock arbitration mechanism to quickly detect link interruptions and trigger master node switching when physical network partitions occur, forcing single-partition writes; at the same time, combined with a global data version verification strategy, it automatically resolves cross-partition write conflicts during the recovery phase.

[0061] In highly sensitive scenarios such as power grid control commands, this design maintains the low-latency transmission advantage of RDMA while completely eliminating the risk of dual-master data conflict caused by network isolation. It ensures the global uniqueness of power operation commands and the security and stability of the system, meeting the power grid's requirements for millisecond-level fault isolation and zero data conflict in extreme fault scenarios. Attached Figure Description

[0062] Figure 1 This is a flowchart of a distributed persistent log storage method based on storage-level memory according to the present invention;

[0063] Figure 2 This is a structural block diagram of a distributed persistent log storage system based on storage-level memory according to the present invention.

[0064] In the diagram: 101. Partition status detection and weak consistency writing module; 102. Dynamic quorum adjustment module; 103. Partition recovery conflict merging module; 104. Spatiotemporal sharding cold and hot layering module; 105. Global metadata coordination module; 106. Layered compression and cost optimization module. Detailed Implementation

[0065] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0066] like Figure 1 As shown, a distributed persistent log storage method based on storage-level memory includes the following steps:

[0067] S100, Partition Status Detection and Weak Consistency Write: Real-time monitoring of network connectivity status in three Availability Zones (AZs). When a partition event is detected, the system automatically switches the write operation to weak consistency mode, uses storage-class memory (SCM) to temporarily store log data, and records hybrid logical clock (HLC) timestamps and business priorities to ensure that low-latency local write capabilities are still provided during partitioning.

[0068] In one embodiment of the present invention, the following steps are specifically included:

[0069] S110, Partition Status Detection: Call the Partition Detector to calculate cross-AZ network connectivity via UDP heartbeat packets.

[0070] Implementation method:

[0071] Construct a 3×3 network connectivity matrix, where each element represents the connectivity status between two availability zones, with 0 indicating connectivity and 1 indicating disconnection. The row and column indices of the matrix correspond to the availability zone numbers.

[0072] ;

[0073] in, Let n be the network connectivity matrix. The number of available zones is fixed at 3;

[0074] The current system partition status is determined by iterating through all elements of the matrix and summing them. If the sum of the matrix elements is 0, it means that all availability zones are connected, and the system status is marked as normal; otherwise, it means that there is a network partition, and the system status is marked as isolated.

[0075] The system sends a heartbeat packet every 100 milliseconds to check connectivity. If no response is received within 300 milliseconds, the connection is considered broken.

[0076] Calculate partition status:

[0077] ;

[0078] in, For partition status identification, The matrix row and column indices represent the AZ numbers. This is a normal label. For isolation marking, This represents the summation of all elements in the matrix;

[0079] S120, Business Priority Verification: Parse the business tags in the logs to determine whether strong consistency is required.

[0080] Implementation method:

[0081] The system assigns a business priority tag to each log entry, with a value ranging from 0 to 10. When the tag value is greater than or equal to 7, it indicates that this is a financial-grade critical business log entry, requiring strong consistency writing; when the tag value is less than 7, weak consistency writing can be used.

[0082] Strong consistency condition:

[0083] ;

[0084] in, This is a business priority label, with a value range of [0,10], where 7 is the financial-grade business threshold.

[0085] The specific priority classifications are as follows: values ​​between 0 and 3 (excluding 3) are low-priority services, values ​​between 3 and 7 (excluding 7) are medium-priority services, and values ​​between 7 and 10 are high-priority services that require strong consistency guarantees.

[0086] Business priority classification:

[0087] Low priority;

[0088] Medium priority;

[0089] It has high priority (strong consistency).

[0090] S130, Routing Decision and Writing: Dynamically select the writing protocol based on the system partition status and service priority, and perform SCM persistence operations.

[0091] Implementation method:

[0092] 1. Strong Consistency Group Write: When the service priority is greater than or equal to 7 and the system is in a normal state, RAFT, DC protocol is used for cross-availability zone replication. The total write latency consists of two parts: twice the cross-availability zone round-trip latency (not exceeding 100 microseconds per transaction) and SCM local write latency (not exceeding 500 nanoseconds). If the write operation is not completed within 500 milliseconds, it is considered a timeout. Dynamic quorum confirmation is required during write operations.

[0093] Strongly consistent group write (when) and NRMAL):

[0094] Cross-AZ replication (RAFT-DC protocol):

[0095] ;

[0096] in, To ensure strong consistency write latency, Round-trip latency across AZ (actual measurement) ), SCM local write latency (actual measurement) Write timeout A dynamic quorum;

[0097] 2. Weak Consistency Group Write: When the business priority is less than 7 or the system is in isolation, local SCM direct write is performed. Each log to be written contains the following fields: log data content, mixed logical clock value, business priority, physical timestamp, and node identifier. The total write latency consists of SCM write time and system overhead time, where the system overhead does not exceed 100 microseconds.

[0098] Weakly consistent group writes (when) or ISLATED):

[0099] Local SCM direct writing:

[0100] ;

[0101] in, For log data content, For mixed logic clock values ​​(S140). For business priority, Physical timestamp For node identifiers;

[0102] Write latency:

[0103] ;

[0104] in, For weakly consistent write total latency, System overhead time;

[0105] S140, Hybrid Logic Clock (HLC) Tagging: Generates globally sortable timing tags for weakly consistent logs.

[0106] Implementation method:

[0107] The system obtains the current physical clock value (which includes the maximum 2-millisecond NTP clock offset between servers), and reads the HLC value (0 on the first write) and logical counter value (0 on the first write) of the previous log.

[0108] enter:

[0109] The current physical clock (including NTP offset) ), For NTP clock offset;

[0110] The HLC value of the preceding log (initially 0);

[0111] This is a logic counter (initially set to 0).

[0112] When the physical clock value equals the HLC value of the previous log, the larger of the physical clock value and the previous HLC value is added to the current logic counter value to obtain the new log's HLC value; otherwise, the larger of the physical clock value and the previous HLC value is added by 1 to obtain the new log's HLC value.

[0113] calculate:

[0114] ;

[0115] in, The mixed logical clock value of the current log. This is the maximum value between the physical clock and the preceding HLC. This is the current physical clock value. This is the value of the logic counter;

[0116] Simultaneously, the logical counter is updated: if the physical clock value equals the HLC value of the preceding log, the counter value is incremented by 1; otherwise, the counter value is reset to 0. This ensures that strictly increasing timing markers are generated for concurrent writes, even under clock synchronization.

[0117] Counter update:

[0118] ;

[0119] in, It is a monotonically increasing logic counter;

[0120] S150, Status Flags and Response: Returns the write result to the client, with attached status flags.

[0121] Implementation method:

[0122] The system returns different status flags based on the write results: A successful strong consistency commit returns the COMMITTED flag, with a response latency of no more than 201 microseconds, requiring no retries; a successful weak consistency or isolation write returns the ISOLATED_PENDING flag, with a response latency of no more than 101 microseconds, requiring retries after partition merging; a strong consistency group frozen during partitioning returns the REJECT flag, with a response latency of no more than 10 microseconds, employing an exponential backoff strategy for retries.

[0123] For operations requiring retries, the retry interval adopts an exponential growth strategy: the initial retry interval is 100 milliseconds, and the interval doubles for each subsequent retry, but not exceeding 5 seconds. The specific calculation method for the retry interval is as follows: use the number of retries as the exponent, perform an exponential operation on 100 milliseconds, and then take the smaller value between this number and 5 seconds as the actual retry interval.

[0124] Retry interval:

[0125] ;

[0126] in This represents the number of retries. Based on the retry interval, This is the maximum retry interval limit. This is the retry interval;

[0127] S200, Dynamic Quorum Adjustment: During network partitioning, the quorum of the RAFT consensus protocol is dynamically adjusted, the weights are recalculated based on the connectivity status of each Availability Zone (AZ), and the system is automatically downgraded to a weak consistency write mode. At the same time, the system continuously monitors partition recovery signals to ensure that the system maintains partial availability during partitioning.

[0128] In one embodiment of the present invention, the following steps are specifically included:

[0129] S210, Survival Node Statistics: Based on the output of the Partition Detector, count the number of currently surviving nodes.

[0130] Implementation method:

[0131] The network state matrix is ​​taken as input, reflecting the connectivity between availability zones. The matrix is ​​3×3, corresponding to 3 availability zones. For each element in the matrix, the node's liveness status is checked by traversing the matrix: a live node is marked as 1, and a faulty node as 0. The counts of all liveness statuses are summed to obtain the total number of live nodes in the current system. This value must be greater than or equal to 1 and less than or equal to the total number of availability zones, 3.

[0132] Input: Network state matrix (From S110);

[0133] in, Let n be the network connectivity matrix. The number of availability zones is fixed at 3; the total number of nodes is... (Fixed number of AZs);

[0134] Calculate the number of surviving nodes:

[0135] ;

[0136] in, For indicator functions, live = 1, fault = 0. This represents the number of currently alive nodes. For node indexing, ;

[0137] Output: (Integer, ≥1);

[0138] S220, Correction Factor Generation: Calculate the correction factor based on the severity of the partition to ensure that the quorum is strictly greater than the partition boundary.

[0139] Implementation method:

[0140] Obtain the number of currently connected availability zones, an integer between 1 and 3. When the number of connected availability zones is greater than or equal to 2, the system is in a normal state, and the correction factor is set to 0. When the number of connected availability zones is less than 2, the system is in an isolated state, and the correction factor needs to be calculated. The calculation method is: take the logarithm of the current number of surviving nodes to the base 2, and round up. The result is the correction factor. This calculation method can appropriately increase the quorum threshold when partitioning occurs, thereby effectively preventing the risk of system split-brain.

[0141] Input: Number of connected AZs (From S110);

[0142] calculate:

[0143] ;

[0144] in, As a correction factor, This represents the number of currently connected Availability Zones (AZs). It is the logarithm to the base 2. This is a rounding up operation;

[0145] in, The project adds a quorum threshold when partitioning to eliminate the risk of boundary split-brain.

[0146] S230, Dynamic quorum calculation: A dynamic quorum is generated by combining the number of surviving nodes and the correction factor.

[0147] Implementation method:

[0148] Calculate the base quorum: Divide the current number of surviving nodes by 2 and round down, then add 1. This base value ensures consensus can be reached under normal circumstances. Next, add the base quorum to the correction factor to obtain a temporary value. Finally, compare this temporary value with the current number of surviving nodes, and take the smaller of the two as the final dynamic quorum. This ensures that the quorum will neither be less than the minimum consensus requirement nor exceed the actual number of available nodes. The final dynamic quorum must be greater than or equal to 1 and less than or equal to the current number of surviving nodes.

[0149] Basic quorum:

[0150] ;

[0151] in, Based on the quorum, This is a round-down operation;

[0152] Final value:

[0153] ;

[0154] in, For the final dynamic quorum, To perform the minimum value operation, the range of values ​​is: ;

[0155] S240, Brain Split Immune Validation: Validates that dynamic quorum can prevent brain split in any partitioned scenario.

[0156] Implementation method:

[0157] Verification is performed by contradiction. Assume the system has two network partitions, P1 and P2, containing k and m nodes respectively. For a split-brain scenario to occur, both partitions must satisfy the condition that the number of nodes is greater than or equal to the dynamic quorum. Simultaneously, the total number of nodes in both partitions must be less than or equal to the total number of surviving nodes in the system. Substituting these conditions into the calculation reveals that the minimum total number of nodes required for the two partitions is greater than the actual total number of surviving nodes in the system, which contradicts reality. Therefore, it can be proven that in any partitioning scenario, it is impossible for two partitions to simultaneously meet the quorum requirement, thus ensuring system consistency.

[0158] Assume there are two partitions (Number of nodes) )and (Number of nodes) ):

[0159] Must meet ;

[0160] Must meet ;

[0161] Constraints ;

[0162] in, For two network partitions, This represents the number of nodes in partition P1. The number of nodes in partition P2

[0163] Proof by contradiction:

[0164] ;

[0165] ;

[0166] ;

[0167] ;

[0168] Conclusion: It is impossible for two partitions to exist simultaneously that satisfy the quorum requirement.

[0169] S250, Write permission decision: Before writing to a strong consistency group, check whether the current partition meets the dynamic quorum requirement.

[0170] Implementation method:

[0171] Before performing a write operation, the system compares the current number of live nodes with the dynamic quorum. If the number of live nodes is greater than or equal to the dynamic quorum, the current partition meets the write conditions, and the system returns an ALLOW_COMMIT state, ensuring that the response latency for the permission check does not exceed 100 microseconds. If the number of live nodes is less than the dynamic quorum, the system returns a FREEZE state. For frozen write requests, the system will schedule retries. The retry interval uses an exponential backoff strategy: the first retry waits 100 milliseconds, and each subsequent retry waits twice as long, but not exceeding 5 seconds. This mechanism avoids the additional burden on the system caused by frequent retries.

[0172] Logic and Output:

[0173] like :

[0174] Commit allowed (returns ALLW CMMIT, associated with a strongly consistent group of S150), response latency is... ;

[0175] in, For permission check response delays, ALLW_CMMIT is a status indicator indicating that submission is allowed.

[0176] Otherwise: Freeze strong consistency writes (return FREEZE, associated with S150 REJECT), retry interval is... ;

[0177] in, This is the waiting time for retrying. This represents the number of retries, and FREEZE is the status indicator for freezing writes.

[0178] S300, Conflict Merging After Partition Recovery: After network partition recovery, logs to be merged are collected from the weak consistency groups of each AZ, sorted by Hybrid Logical Clock (HLC) and timing conflicts are detected. Concurrent write conflicts are resolved through service priority arbitration. The merged logs are submitted in batches to the strong consistency group (RAFT, DC), the global vector clock state is updated and weak consistency data is cleaned up.

[0179] In one embodiment of the present invention, the following steps are specifically included:

[0180] S310, Recovery Event Listener: Listens to the Partition Detector and captures partition recovery events (when the availability zone status changes from isolated to normal).

[0181] Implementation method:

[0182] The system detects status changes by comparing the current availability zone status with the previous status. Specifically, it obtains the current availability zone status and the previous availability zone status, and determines a partition recovery event has occurred only if the previous status was isolated and the current status is normal. The availability zone status can only be either normal or isolated. The system requires that the response time for status change detection not exceed 1 millisecond to ensure timely capture of recovery events.

[0183] Recovery conditions:

[0184] ;

[0185] in, The result of the state change detection at time t. Let be the state of AZ at time t. The state of AZ at time t-1. For logical AND operation, For logical NOT operation;

[0186] Response latency:

[0187] ;

[0188] in, For state change detection response time, Maximum allowable response latency;

[0189] S320, Weak Consistency Log Collection: Pulls logs written during the isolation period from the weak consistency groups (ST, VClock storage area) of all availability zones.

[0190] Implementation method:

[0191] The system will collect pending logs from all availability zones. A set is defined to store all pending logs. Then, each availability zone (a fixed total of 3) is traversed, and the pending logs from that availability zone are added to the set. The total time for the entire collection process consists of two parts: twice the cross-availability zone round-trip latency (100 microseconds per trip) and the maximum write latency to storage-class memory (500 nanoseconds). The system uses the sum of these two parts as the upper limit of performance constraints.

[0192] Log collection definition:

[0193] ;

[0194] in, For the set of all pending logs, Let i be the set of logs to be processed for the i-th AZ. The total number of AZs (fixed at 3). This is a set union operation;

[0195] Performance constraints:

[0196] ;

[0197] in, Total time spent collecting logs, The round-trip time across AZ is 100μs. SCM write latency (500ns). This is for calculating the maximum value.

[0198] S330, Timing Conflict Detection (HLC Sorting): Sorts all logs by mixed logic clock values ​​to detect concurrent conflicts.

[0199] Implementation method:

[0200] Define a set of conflicting log pairs. When two logs have the same mixed logical clock timestamp, add them as a pair to the conflict set. For all logs to be processed, perform a global sort by comparing their mixed logical clock timestamps: if one log's timestamp is less than another's, the former is considered to be earlier in time. This sorting method ensures that a clear chronological order can be established for all logs.

[0201] Set of conflicting pairs:

[0202] ;

[0203] in, For a set of log pairs with time-series conflicts, For the log pairs to be compared, A hybrid logical clock timestamp for the log;

[0204] Global sorting:

[0205] ;

[0206] in, This is the total order relation symbol. For if and only if, It can be any element;

[0207] S340, Business Priority Arbitration: Arbitrate concurrent conflict logs according to business priority.

[0208] Implementation method:

[0209] The system adopts a three-level arbitration strategy: compare the difference in business priority between two conflicting logs. If the priority of the first log is 3 or more higher than that of the second log, the first log is selected. If the priority of the second log is 3 or more higher than that of the first log, the second log is selected. If the priority difference between the two logs is less than 3, a log merging operation (such as dictionary merging) is performed to preserve the information of the two logs.

[0210] Arbitration Rules:

[0211] ;

[0212] in, For arbitration selection function, For log pairs pending arbitration, Prioritize log entries for business purposes. Priority difference threshold ( =3), This is the function for log merging operations;

[0213] Example of a merge operator:

[0214] ;

[0215] in, For custom merge operators;

[0216] S350, Vector Clock Synchronization: Updates the global vector clock state and marks the timing position of merged logs.

[0217] Implementation method:

[0218] For each availability zone (numbered 1 to 3), the system updates the clock value for that availability zone in the global vector clock array. The update rule is as follows: in the sorted log set, find all logs originating from that availability zone, take the maximum value of the mixed logical clock timestamps of these logs, and use this maximum value as the new value for that availability zone in the vector clock array. This update method ensures that the vector clock accurately reflects the latest write timings of each availability zone.

[0219] Update rules (for each AZ) k ):

[0220] ;

[0221] in, For global vector clock array, AZ number ( ), For the sorted log collection, The log source is AZ. For mixed logic clock timestamps;

[0222] S360, Strong Consistency Group Commit: Commits merged logs in bulk to a strong consistency group (RAFT, DC) using dynamic quorum confirmation.

[0223] Implementation method:

[0224] The system must meet the following conditions upon submission: the number of confirmation responses must be greater than or equal to the dynamic quorum (this value is calculated from S230). The total time taken for the submission process consists of three parts: twice the cross-availability zone round-trip latency (for consensus protocol communication), the product of the total number of merged logs and the storage-level memory write latency (for data persistence), and a 10-millisecond merge operation timeout threshold.

[0225] Submission requirements:

[0226] ;

[0227] in, To confirm the response count, For dynamic quorum (from S230);

[0228] Delay model:

[0229] ;

[0230] in, To submit the total time spent, For cross-AZ round trip delay, To merge the total number of logs, For SCM write latency, The merge operation timeout threshold (10ms);

[0231] S370, State Cleanup and Notification: Clean up weakly consistent group data and update client log status.

[0232] Implementation method:

[0233] The system updates the final status flag for each log entry based on the processing result: when logs are successfully merged, the status is marked as "Merged"; when logs fail during arbitration, the status is marked as "Arbitration Failed". These status flags are used to notify clients of the final processing result of the logs and guide subsequent cleanup operations.

[0234] Status flag update:

[0235] ;

[0236] in, This represents the final state of log L. This indicates a successful merge. This indicates an arbitration failure status.

[0237] S400, Spatiotemporal Sharding Hot and Cold Tiering: Calculates heat values ​​based on the spatiotemporal characteristics of logs (timestamp, source partition) and access frequency, dynamically determines storage tiering thresholds, migrates data to storage media with different performance (SCM / QLC, NVMe / HDD), performs compression on cold data to optimize storage costs, and updates metadata indexes to achieve efficient query routing.

[0238] In one embodiment of the present invention, the following steps are specifically included:

[0239] S410, Spatiotemporal Feature Extraction: Analyze the spatiotemporal attributes of logs (timestamp and source partition), and calculate the time decay factor and spatial locality weight.

[0240] Implementation method:

[0241] The time decay factor is calculated as follows: Obtain the current system timestamp (in milliseconds) and the log's mixed logical clock timestamp (in milliseconds), and calculate the time difference between them. Then, multiply this difference by a decay coefficient (0.001 per millisecond), negate the result, and perform an exponential operation to obtain a decay factor between 0 and 1. The smaller the value, the cooler the data. This calculation method allows the data's popularity to decay exponentially over time.

[0242] The calculation process for spatial locality weight is as follows: Obtain the vector clock value of the availability zone from which the logs originate, which represents the cumulative write volume of that availability zone. Then, divide this value by the sum of the vector clock values ​​of all availability zones (iterate through the three availability zones and sum them) to obtain a weight value between 0 and 1, which reflects the proportion of that availability zone in the total write volume. This calculation method can reflect the spatial distribution characteristics of the data.

[0243] Time decay factor:

[0244] ;

[0245] in, This is the time decay factor, with a value range of [0,1]. The smaller the value, the colder the data. This is the current system timestamp. For logs, a hybrid logical clock timestamp. The attenuation coefficient is 0.001 / ms, which controls the attenuation rate. The base of the natural logarithm is approximately 2.71828;

[0246] Spatial locality weights:

[0247] ;

[0248] in, This is the spatial locality weight, with a value range of [0,1], representing the proportion of data in that AZ. Let be the vector clock value of the k-th AZ, representing the cumulative write volume of that AZ. The total number of AZs is fixed at 3. For AZ index, the value range is [1,3]. The AZ number representing the log source;

[0249] S420, Access Popularity Calculation: Statistical log access frequency is used to generate a comprehensive heat value by combining spatiotemporal characteristics.

[0250] Implementation method:

[0251] The calculation process for normalized access frequency is as follows: Obtain the actual access frequency of the log (in times / second) and the historical maximum access frequency. Compare the actual access frequency with the historical maximum frequency, take the smaller value, and then divide it by the historical maximum frequency to obtain a normalized frequency value between 0 and 1. This normalization process can unify access frequencies of different magnitudes to the same scale.

[0252] Normalized access frequency:

[0253] ;

[0254] in, This represents the normalized access frequency, with a value range of [0,1]. This is a log access frequency count, measured in times per second. This represents the highest historical access frequency, expressed in times per second. The function is for finding the minimum value;

[0255] The calculation process for the comprehensive heat value is as follows: multiply the time decay factor by a weight of 0.4, the spatial locality weight by a weight of 0.3, and the normalized access frequency by a weight of 0.3. Add these three weighted results to obtain a comprehensive heat value between 0 and 1. This weighted calculation method comprehensively considers the temporal, spatial, and access characteristics of the data, and the sum of the three weights being 1 ensures the normalization of the calculation result.

[0256] Calculation of heat value:

[0257] ;

[0258] in, The comprehensive thermal value ranges from [0,1]. For time decay weight, For spatial locality weights, Weighted by access frequency. The time decay factor, For spatial locality weights, To normalize the access frequency, the constraint is: ;

[0259] S430, tiered threshold decision: Determine the storage tiered threshold based on the thermal value distribution.

[0260] Implementation method:

[0261] The dynamic threshold calculation process is as follows: Sort all data by their heat values, find the first quantile (values ​​at the 25th percentile) and the third quantile (values ​​at the 75th percentile). The difference between these two is the interquartile range, used to measure the dispersion of the data. Then, add 1.5 times the interquartile range to the third quantile to obtain the hot data threshold; subtract 1.5 times the interquartile range from the first quantile to obtain the cold data threshold. Data above the hot data threshold is stored in the SCM (Supply Chain Management System), data below the cold data threshold is stored in the HDD (High-Demand Storage System), and data in between is stored in the QLC (Quadrant Library) or NVMe (Variable Residual Storage System). This quartile-based threshold calculation method can adapt to changes in data distribution.

[0262] Dynamic threshold calculation:

[0263] ;

[0264] ;

[0265] in, This is a threshold for hot data; data exceeding this value is stored in the SCM (Search Engine Management System). This is a threshold for cold data; data smaller than this value is stored in the HDD. This represents the first quantile of the thermal value distribution, indicating the value at the 25th percentile. This represents the third quantile of the thermal value distribution, indicating the value at the 75th percentile. The interquartile range (IQR) is used to measure the dispersion of data. This is the outlier coefficient, used to expand the threshold range;

[0266] S440, Data Sharding Migration: Migrate logs to the corresponding storage layer based on heat value.

[0267] Implementation method:

[0268] The process for selecting a storage layer is as follows: The heat value of the data is obtained and compared with hot data thresholds and cold data thresholds. When the heat value is greater than or equal to the hot data threshold, storage-class memory (SCM) is selected as the storage layer; when the heat value is less than the hot data threshold but greater than or equal to the cold data threshold, QLC flash memory (QLC, NVMe) is selected as the storage layer; when the heat value is less than the cold data threshold, hard disk drive (HDD) is selected as the storage layer.

[0269] The calculation process for migration batch control is as follows: The total number of logs meeting the migration conditions is counted, and this number is compared with the maximum migration limit for a single batch (1024 logs). The smaller of the two values ​​is taken as the actual amount of data migrated in this batch. This batch control mechanism can prevent the system from being overwhelmed by an excessively large amount of data migrated in a single batch.

[0270] Migration rules:

[0271] ;

[0272] in, Select the result for the storage level. Storage-class memory, the highest-performance storage tier. It is a QLC flash memory, a medium-performance storage tier. For mechanical hard drives, the lowest performance storage tier. For data heat values, For hot data threshold, Cold data threshold

[0273] Migration batch control:

[0274] ;

[0275] in, The amount of data migrated in a single operation. This is a limit on the maximum number of migrations per batch. For log entries to be migrated, The total number of logs required to meet the migration criteria. The function is for finding the minimum value;

[0276] S450, Cold Data Compression: Performs lossless compression on cold data.

[0277] Implementation method:

[0278] The compression ratio prediction calculation process is as follows: Obtain the size of the log data (in bytes), calculate the logarithm to base 10, multiply this value by a coefficient of 0.4, and then add 0.6 to obtain the predicted compression ratio. This logarithmic function calculation method reflects the characteristic that the larger the data volume, the better the compression effect can usually be obtained.

[0279] The compression trigger condition is determined as follows: the total size of the data to be compressed is accumulated, and compression is triggered when the total size reaches or exceeds 1 TiB. This threshold-based triggering mechanism allows batch compression to be performed only when the data volume accumulates to a certain scale, thus improving compression efficiency.

[0280] Compression ratio prediction:

[0281] ;

[0282] in, To predict the compression ratio, it represents the ratio of the compressed size to the original size. For log data size, It is a logarithmic function with base 10. This is the compression ratio coefficient. Based on the compression ratio;

[0283] Compression trigger conditions:

[0284] ;

[0285] in, For the cumulative data size, The compression trigger threshold is set to a size of 1 TiB.

[0286] S460, Metadata Update: Update storage location metadata.

[0287] Implementation method:

[0288] The location index is calculated as follows: A hash operation is performed on the unique identifier of the log file to obtain a fixed-length hash value. Then, this hash value is XORed with the storage tier identifier (SCM, QLC-NVMe, or HDD) to obtain the final storage location index. This hash-and-XOR-based index calculation method can achieve uniform data distribution and supports fast location lookup.

[0289] action:

[0290] Modify metadata tags:

[0291] ;

[0292] in, For storage tier identifier, A collection of enumerations at the storage level, containing (Storage Class Memory) (QLC flash memory) and (Mechanical hard drives) Three types;

[0293] Location index calculation:

[0294] ;

[0295] in, This is the index for the storage location of log item Li. A unique identifier for the log file, used to locate the log file. For hash functions, Mapped to a fixed-length value. This is a bitwise XOR operation used to combine hash values ​​and storage layer information. Storage tier identifier;

[0296] S470, Layered monitoring feedback: Dynamically adjusts the weighting parameters of thermal values.

[0297] Implementation method:

[0298] The adaptive weight calculation process is as follows: Obtain the current cache hit rate and the target hit rate (both values ​​between 0 and 1), and calculate the difference between them. Then, multiply this difference by the learning rate (0.05), and add the result to the current time decay weight to obtain the updated weight value. Simultaneously, other weights need to be adjusted accordingly to ensure that the sum of all weights remains 1. This feedback-based weight adjustment mechanism can dynamically optimize the layering strategy based on the actual operating performance of the system.

[0299] Weight adaptive:

[0300] ;

[0301] in, For the updated time decay weights, The decay weight is the weight at the current time. This represents the current cache hit rate, with a value in the range [0,1]. The target hit rate is defined, with a value range of [0,1]. The learning rate controls the step size for weight adjustment, with the following constraints: Ensure that the sum of the weights is 1;

[0302] like Figure 2 As shown, in one embodiment, a distributed persistent log storage system based on storage-level memory is proposed, comprising the following modules:

[0303] Partition Status Detection and Weak Consistency Write Module 101: This module monitors the network connectivity status of the three Availability Zones (AZs) in real time. When a network partition event is detected, it automatically switches the write operation to weak consistency mode. Log data is temporarily written to local storage-level memory (SCM), while recording a hybrid logical clock timestamp and business priority tag to ensure low-latency local write capability is maintained during partitioning and to retain complete spatiotemporal and semantic information for subsequent conflict merging.

[0304] Dynamic Quorum Adjustment Module 102: During network partitioning, this module dynamically adjusts the quorum weights of the RAFT consensus protocol, recalculating the arbitration threshold based on the real-time connectivity status of each Availability Zone (AZ). It maintains partial system availability by automatically downgrading to a weak consistency write mode, while continuously monitoring partition recovery signals to ensure real-time synchronization between weight adjustments and network state changes.

[0305] Partition Recovery Conflict Merging Module 103: When the network recovers, this module collects logs to be merged from the weakly consistent storage areas of each Availability Zone (AZ), sorts them by mixed logical clock timestamps to identify timing conflicts, resolves concurrent write conflicts (such as high-priority overwrite or data fusion) through a business priority arbitration mechanism, submits the merged logs in batches to the strong consistency group (RAFT multi-datacenter cluster), updates the global vector clock state, and cleans up the weakly consistent storage areas.

[0306] Spatiotemporal sharding hot / cold tiering module 104: This module analyzes the spatiotemporal characteristics (timestamp, source partition) and access frequency of logs, dynamically calculates heat values, and determines tiering thresholds. Based on the heat values, data is migrated to three storage media tiers: frequently accessed hot data is stored in SCM, moderately accessed warm data is stored in QLC NVMe SSDs, and infrequently accessed cold data is stored in HDDs. Cold data is compressed to optimize storage costs, and metadata indexes are updated to achieve efficient query routing across storage tiers.

[0307] Global Metadata Coordination Module 105: Unifies the management of log storage location indexes, vector clock status, and hierarchical strategy parameters, providing metadata query services for all modules. It accelerates metadata access through distributed caching and synchronizes state changes across modules in real time, ensuring strong metadata consistency during partitioning, merging, and hierarchical operations.

[0308] Tiered Compression and Cost Optimization Module 106: For the cold data storage layer, it implements lossless compression algorithms (such as ZSTD) to reduce storage overhead and dynamically adjusts the compression strategy according to data characteristics and access patterns. It monitors the capacity and performance indicators of each storage layer and automatically triggers data reorganization or migration to balance storage costs and access latency.

[0309] In one embodiment, the above-described distributed persistent log storage system and method based on storage-level memory is applied to the following example:

[0310] The following is a background:

[0311] System: Provincial Power Grid Monitoring Center Log System

[0312] Partitioning scenario: Main data center (AZ1), backup center (AZ2), edge substation (AZ3);

[0313] Business Log:

[0314] High priority: Transaction operation log (TX, such as on / off commands);

[0315] Medium priority: Configuration change log (CFG, such as parameter adjustments);

[0316] Low priority: Monitoring logs (MON, such as temperature reports);

[0317] S100, Partition Detection:

[0318] S110 (Status Monitoring): Network jitter caused the edge substation (AZ3) to disconnect from the main center, and the partition detector marked the AZ3 status as ISOLATED.

[0319] S120, (Write Category):

[0320] AZ3 received a switch command (TX log): Due to high priority, writing to the weak consistency group is allowed.

[0321] AZ3 received a temperature report (MON log): Due to low priority, it is temporarily stored in the local buffer.

[0322] S130 (Weakly Consistent Write): The TX log is written to the ST-VClock storage area of ​​AZ3, and the status is marked as ISOLATED_PENDING.

[0323] S140 (Clock Synchronization): Adds a hybrid logic timestamp (HLC_t) to the TX log to record the local clock of AZ3.

[0324] S150 (Dynamic Routing): The configuration change request (CFG log) was routed to AZ2 (Backup Center) due to the partition.

[0325] S200, Strong Consensus Group Dynamic Arbitration:

[0326] S210 (Partition Topology Update): AZ3 was detected to be offline. The updated topology is: {AZ1, AZ2} online, AZ3 offline.

[0327] S220, (Statutory number calculation): Dynamic calculation of the number of arbitrators: at least 2 AZs are required for confirmation (originally 2 out of 3 AZs were required).

[0328] S230 (Log Commit): AZ2 receives the configuration change (CFG log), broadcasts it to AZ1 and AZ2 (itself), and commits it after receiving two ACKs.

[0329] S240 (State Synchronization): Committed CFG logs are synchronized in the RAFT-DC storage areas of AZ1 and AZ2.

[0330] S250 (Client Response): The configuration change was successfully responded to the client, but the AZ3 logs are still marked as pending.

[0331] S300, conflict merging after partition recovery:

[0332] S310 (Recovery Detection): Network recovery, AZ3 status switches from ISOLATED back to NORMAL, triggering merge.

[0333] S320 (Log Collection): Pulls the TX logs (switch commands) from the AZ3 during the isolation period and merges them with the CFG logs (parameter adjustment) from the main center.

[0334] S330 (Timing Conflict Detection): The TX log and CFG log timestamps are found to be the same (concurrent operation).

[0335] S340, (Business Arbitration):

[0336] TX log priority = 9 (highest), CFG log priority = 5.

[0337] If the priority difference is ≥3, the TX log will overwrite the CFG log.

[0338] S350 (Clock Synchronization): Updates the global vector clock and records the last timestamp of AZ3.

[0339] S360 (Strong Consistent Commit): The merged logs are committed to the strong consistency group in the main center (AZ1).

[0340] S370, (Status Cleanup): Deletes pending logs from AZ3 and marks them as MERGED.

[0341] S400, spatiotemporal segmentation for hot and cold stratification:

[0342] S410 (Spatiotemporal Feature Extraction):

[0343] Switch command (TX): New log, low time decay; from edge station, high spatial weight.

[0344] Temperature Report (MON): Log from 3 days ago, high time decay.

[0345] S420 (Access Popularity Calculation): TX log: 50 accesses within 1 minute is considered high-frequency hot data; MON log: 1 access within 1 day is considered low-frequency cold data.

[0346] S430 (Tiered Decision): TX logs are assigned to the hot tier (SCM storage); MON logs are assigned to the cold tier (HDD storage).

[0347] S440 (Data Migration): Batch migration of MON logs to HDD, while TX logs remain in SCM.

[0348] S450 (Cold Data Compression): Performs ZSTD compression on MON logs in HDD, saving 70% of space.

[0349] S460 (Metadata Update): Update indexes: TX logs are located in SCM, MON logs are located in HDD.

[0350] S470 (Monitoring Feedback): SCM layer hit rate of 95% detected, maintaining weight parameters.

[0351] Example results:

[0352] Partition tolerance: During the outage of the edge station (AZ3), high-priority operations are executed normally; after recovery, conflicting commands are overridden according to priority to ensure grid security.

[0353] Storage optimization: Millisecond-level response for hot commands (TX) and 60% reduction in storage costs with cold monitoring (MON) compression.

[0354] Global consistency: All terminals query the same switch status and configuration parameters.

[0355] In summary, this solution achieves "uninterrupted partitioning, conflict-free recovery, and adaptive storage" in power grid operation and maintenance, meeting the high reliability and low cost requirements of the power system.

[0356] The embodiments of the present invention have been described above, but the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention, all of which are within the protection scope of the present invention.

Claims

1. A distributed persistent log storage method based on storage-level memory, characterized in that, Includes the following steps: S100, Partition Status Detection and Weak Consistency Write: Real-time monitoring of network connectivity status in three AZs. When a partition event is detected, the system automatically switches the write operation to weak consistency mode, uses SCM to temporarily store log data, and records hybrid HLC timestamps and service priorities. S200, Dynamic Quorum Adjustment: During network partitioning, the quorum of the RAFT consensus protocol is dynamically adjusted, the weights are recalculated based on the connectivity status of each Availability Zone (AZ), and the system automatically downgrades to weak consistency write mode while continuously monitoring partition recovery signals. S300, conflict merging after partition recovery: When the network partition is recovered, logs to be merged are collected from the weak consistency groups of each AZ, sorted by HLC and time-series conflicts are detected, concurrent write conflicts are resolved through business priority arbitration, the merged logs are submitted to the strong consistency group in batches, the global vector clock state is updated and weak consistency data is cleaned up. S400, Spatiotemporal Sharding Hot and Cold Tiering: Calculates heat values ​​based on the spatiotemporal characteristics and access frequency of logs, dynamically determines storage tiering thresholds, migrates data to storage media with different performance levels, performs compression on cold data to optimize storage costs, and updates metadata indexes to achieve efficient query routing.

2. The distributed persistent log storage method based on storage-level memory according to claim 1, characterized in that, The specific steps for partition status detection and weakly consistent writes are as follows: S110, Partition Status Detection: Calls the partition detector to calculate cross-AZ network connectivity via UDP heartbeat packets; S120, Business Priority Verification: Parse the business tags in the logs to determine whether strong consistency is required; S130, Routing Decision and Writing: Dynamically select the writing protocol based on the system partition status and business priority, and perform SCM persistence operations; S140, Hybrid Logic Clock Tag: Generates globally sortable timing tags for weakly consistent logs; S150, Status Flags and Response: Returns the write result to the client, with attached status flags.

3. The distributed persistent log storage method based on storage-level memory according to claim 1, characterized in that, Routing decisions and writes include strong consistency group writes and weak consistency group writes; Strong consistency group write: When the service priority is greater than or equal to 7 and the system is in a normal state, the RAFT and DC protocols are used for cross-availability zone replication; Weak consistency group write: When the business priority is less than 7 or the system is in isolation, local SCM direct write is performed.

4. The distributed persistent log storage method based on storage-level memory according to claim 1, characterized in that, The specific steps for adjusting the dynamic quorum are as follows: S210, Survival Node Statistics: Based on the output of the partition detector, count the number of currently surviving nodes; S220, Correction Factor Generation: Calculate the correction factor based on the severity of the partition to ensure that the quorum is strictly greater than the partition boundary; S230, Dynamic quorum calculation: A dynamic quorum is generated by combining the number of surviving nodes and the correction factor; S240, Brain Split Immune Validation: Validation that dynamic quorum can prevent brain split in any partitioned scenario; S250, Write permission decision: Before writing to a strong consistency group, check whether the current partition meets the dynamic quorum requirement.

5. The distributed persistent log storage method based on storage-level memory according to claim 1, characterized in that, The specific verification steps for immune validation of brain split are as follows: Verification is performed by contradiction; assume that two network partitions P1 and P2 exist in the system, containing k and m nodes respectively; For a split-brain scenario to occur, both partitions must meet the condition that the number of nodes is greater than or equal to the dynamic quorum. At the same time, the total number of nodes in the two partitions must be less than or equal to the total number of surviving nodes in the system. Substituting these conditions into the calculation reveals that the minimum total number of nodes required for the two partitions is greater than the actual total number of surviving nodes in the system, which contradicts the reality. The final dynamic quorum must be greater than or equal to 1 and less than or equal to the current number of surviving nodes.

6. The distributed persistent log storage method based on storage-level memory according to claim 1, characterized in that, The specific content of the write permission decision is as follows: Before performing a write operation, the system compares the current number of live nodes with the dynamic quorum. When the number of surviving nodes is greater than or equal to the dynamic quorum, it means that the current partition meets the write conditions, the system returns a commit allowed status, and ensures that the response delay of the permission check does not exceed 100 microseconds; If the number of surviving nodes is less than the dynamic quorum, the write freeze (FREEZE) state is returned. For frozen write requests, the system will schedule retry, with the retry interval employing an exponential backoff strategy.

7. The distributed persistent log storage method based on storage-level memory according to claim 1, characterized in that, The specific steps for merging conflicts after partition recovery are as follows: S310, Recovery Event Listener: Listens to the partition detector and captures partition recovery events; S320, Weak Consistency Log Collection: Pulls logs written during the isolation period from weak consistency groups in all availability zones; S330, Timing Conflict Detection: Sort all logs by mixed logic clock values ​​to detect concurrent conflicts; S340, Business Priority Arbitration: Arbitrate concurrent conflict logs according to business priority; S350, Vector Clock Synchronization: Updates the global vector clock status and marks the timing position of merged logs; S360, Strong Consistency Group Commit: Batch commits merged logs to a strong consistency group, using dynamic quorum confirmation; S370, State Cleanup and Notification: Clean up weakly consistent group data and update client log status.

8. A distributed persistent log storage method based on storage-level memory according to claim 1, characterized in that, The specific steps for spatiotemporal partitioning and hot / cold stratification are as follows: S410, Spatiotemporal Feature Extraction: Analyze the spatiotemporal attributes of logs and calculate the time decay factor and spatial locality weight; S420, Access Popularity Calculation: Statistical log access frequency is combined with spatiotemporal characteristics to generate a comprehensive heat value; S430, tiered threshold decision: Determine the storage tiered threshold based on thermal value distribution; S440, Data Sharding Migration: Migrate logs to the corresponding storage layer based on heat value; S450, Cold Data Compression: Performs lossless compression on cold data; S460, Metadata Update: Update storage location metadata; S470, Layered monitoring feedback: Dynamically adjusts the weighting parameters of thermal values.

9. A distributed persistent log storage method based on storage-level memory according to claim 1, characterized in that, The steps for updating metadata are as follows: The unique identifier of the log is hashed to obtain a fixed-length hash value. Then, the hash value is XORed with the storage level identifier to obtain the final storage location index. Storage tier identifiers include SCM, QLC-NVMe, and HDD.

10. A distributed persistent log storage system based on storage-level memory, characterized in that, The steps of performing a distributed persistent log storage method based on storage-level memory as described in any one of claims 1-9 include: Partition status detection and weak consistency write module: Real-time monitoring of network connectivity status of three AZs. When a network partition event is detected, the write operation is automatically switched to weak consistency mode. Log data is temporarily written to local storage-level memory, and a hybrid logical clock timestamp and business priority label are recorded at the same time. Dynamic quorum adjustment module: During the network partitioning period, the quorum weight of the RAFT consensus protocol is dynamically adjusted, the arbitration threshold is recalculated according to the real-time connectivity status of each AZ, the system maintains partial availability by automatically downgrading to weak consistency write mode, and the partition recovery signal is continuously monitored. Partition recovery conflict merging module: When the network recovers, it collects logs to be merged from the weak consistency storage area of ​​each AZ, sorts them by mixed logical clock timestamps to identify timing conflicts, resolves concurrent write conflicts through the business priority arbitration mechanism, submits the merged logs to the strong consistency group in batches, updates the global vector clock state and cleans up the weak consistency storage area. Spatiotemporal sharding hot and cold stratification module: Analyzes the spatiotemporal characteristics and access frequency of logs, dynamically calculates heat values ​​and determines stratification thresholds, migrates data to three-tier storage media based on heat values, performs compression on cold data to optimize storage costs, and updates metadata indexes to achieve efficient query routing across storage tiers; Global Metadata Coordination Module: Unifies the management of log storage location index, vector clock status, and hierarchical strategy parameters, provides metadata query services for all modules, accelerates metadata access through distributed caching, and synchronizes the status changes of each module in real time; Layered Compression and Cost Optimization Module: For the cold data storage layer, it implements lossless compression algorithms to reduce storage overhead, dynamically adjusts compression strategies based on data characteristics and access patterns, monitors the capacity and performance indicators of each storage layer, and automatically triggers data reorganization or migration to balance storage costs and access latency.

Images