A storage class memory based distributed persistent log storage system and method

Abstract

The application relates to the technical field of log storage, and discloses a distributed persistent log storage system and method based on storage class memory, which comprises the following steps: real-time monitoring of the network connection state of three AZs; when a partition event is detected, the system automatically switches the write operation to a weak consistency mode, uses SCM to temporarily store log data, and records a hybrid HLC timestamp and a business priority; during network partition, the quorum of a RAFT consensus protocol is dynamically adjusted, the weight is recalculated according to the connection state of each AZ, the system is automatically degraded to a weak consistency write mode, and a partition recovery signal is continuously monitored. In a high-sensitive scene such as power grid control instruction, the application not only maintains the low-delay transmission advantage of RDMA, but also eliminates the double-main data conflict risk caused by network isolation, guarantees the global uniqueness and system safety and stability of power operation instructions, and meets the requirements of millisecond-level fault isolation and zero data conflict.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of log storage, more particularly, to a distributed persistent log storage system and method based on storage class memory. BACKGROUND

[0002] Under the architecture of cloud native and edge computing, traditional log storage systems face challenges such as throughput bottleneck, high latency, and low fault recovery efficiency.

[0003] In particular, in high-performance distributed storage based on RDMA, when network partitioning occurs, the traditional TCP / IP layer can detect the partitioning through a timeout mechanism, but the zero-copy kernel bypass feature of RDMA makes the application layer unable to directly perceive link interruption, resulting in multiple partitions simultaneously considering themselves as the main partition, and thus causing data conflicts. SUMMARY

[0004] The present application provides a distributed persistent log storage system and method based on storage class memory, which solves the protocol split-brain under RDMA network partitioning in traditional technology and cannot achieve sub-millisecond persistent write and fault self-healing capabilities.

[0005] The present application provides a distributed persistent log storage method based on storage class memory, comprising the following steps:

[0006] S100, partition state detection and weakly consistent write: real-time monitoring of the network connectivity state of the three AZs, when a partition event is detected, the system automatically switches the write operation to a weakly consistent mode, uses SCM to temporarily store log data, and records a hybrid HLC timestamp and business priority;

[0007] S200, dynamic quorum adjustment: during network partitioning, dynamically adjusting the quorum of the RAFT consensus protocol, recalculating the weight according to the connectivity state of each AZ, automatically downgrading to a weakly consistent write mode, and continuously listening for partition recovery signals;

[0008] S300, conflict merging after partition recovery: when the network partitioning is restored, collect the logs to be merged from the weakly consistent groups of each AZ, sort them by HLC and detect timing conflicts, solve concurrent write conflicts through business priority arbitration, batch submit the merged logs to the strongly consistent group, update the global vector clock state and clean up the weakly consistent data;

[0009] S400, spatiotemporal sharding cold and hot layering: calculating the heat value according to the spatiotemporal characteristics and access frequency of the logs, dynamically deciding the storage layering threshold, migrating data to different performance storage media, performing compression optimization on cold data to reduce storage cost, and updating the metadata index to achieve efficient query routing.

[0010] Further, the specific steps of partition state detection and weak consistent write are as follows:

[0011] S110, partition state detection: call partition detector to calculate cross-AZ network connectivity through UDP heartbeat packet;

[0012] S120, service priority verification: parse the service label of the log to determine whether strong consistency is required;

[0013] S130, routing decision and write: dynamically select the write protocol according to the system partition state and service priority, and execute the SCM persistence operation;

[0014] S140, hybrid logical clock marking: generate globally orderable time sequence marking for weak consistent log;

[0015] S150, state marking and response: return the write result to the client with the state marking.

[0016] Further, the routing decision and write include strong consistent group write and weak consistent group write;

[0017] Strong consistent group write: when the service priority is greater than or equal to 7 and the system is in normal state, use RAFT, DC protocol for cross availability zone replication;

[0018] Weak consistent group write: when the service priority is less than 7 or the system is in isolation state, execute local SCM direct write.

[0019] Further, the specific steps of dynamic quorum adjustment are as follows:

[0020] S210, live node statistics: based on the output of the partition detector, count the number of currently live nodes;

[0021] S220, bias factor generation: calculate the bias factor according to the partition severity to ensure that the quorum is strictly greater than the partition boundary;

[0022] S230, dynamic quorum calculation: combine the number of live nodes and the bias factor to generate the dynamic quorum;

[0023] S240, brain split immunity verification: verify that the dynamic quorum can avoid brain split in any partition scenario;

[0024] S250, write permission decision: check whether the current partition meets the dynamic quorum requirement before strong consistent group write.

[0025] Further, the specific verification steps of brain split immunity verification are as follows:

[0026] The verification is carried out by reductio ad absurdum; suppose that two network partitions P1 and P2 appear in the system, containing k and m nodes respectively;

[0027] If a brain split is to be formed, both partitions must satisfy the condition that the number of nodes is greater than or equal to the dynamic quorum, and at the same time, the total number of nodes in the two partitions must be less than or equal to the total number of surviving nodes in the system. Substituting these conditions into the calculation shows that the minimum total number of nodes required for the two partitions will be greater than the actual total number of surviving nodes in the system, which is inconsistent with the actual situation;

[0028] The finally generated dynamic quorum must be greater than or equal to 1 and less than or equal to the current number of surviving nodes.

[0029] Further, the specific content of the write permission decision is as follows: before the system performs a write operation, the current number of surviving nodes is compared with the dynamic quorum;

[0030] When the number of surviving nodes is greater than or equal to the dynamic quorum, it indicates that the current partition satisfies the write condition, and the system returns an allowed commit state, and ensures that the response delay of the permission check does not exceed 100 microseconds;

[0031] If the number of surviving nodes is less than the dynamic quorum, a FREEZE state is returned;

[0032] For the frozen write request, the system will arrange a retry, and the retry interval uses an exponential backoff strategy.

[0033] Further, the specific steps of conflict merging after partition recovery are as follows:

[0034] S310, recovery event listening: listening to the partition detector to capture the partition recovery event;

[0035] S320, weakly consistent log collection: pulling the isolated period write log from the weakly consistent group of all available areas;

[0036] S330, timing conflict detection: sorting all logs according to the hybrid logical clock value, detecting concurrent conflicts;

[0037] S340, business priority arbitration: arbitrating concurrent conflict logs according to business priority;

[0038] S350, vector clock synchronization: updating the global vector clock state to mark the timing position of the merged log;

[0039] S360, strongly consistent group submission: batch submitting the merged log to the strongly consistent group and using the dynamic quorum for confirmation;

[0040] S370, state cleaning and notification: cleaning the weakly consistent group data and updating the client log state.

[0041] Further, the specific steps of the space-time slice cold-hot layering are as follows:

[0042] S410, space-time feature extraction: analyze the space-time attributes of the log, calculate the time decay factor and the spatial locality weight;

[0043] S420, access heat calculation: count the access frequency of the log, and generate a comprehensive heat value in combination with the space-time features;

[0044] S430, layering threshold decision: determine the storage layering threshold according to the heat value distribution;

[0045] S440, data slice migration: migrate the log to the corresponding storage layer according to the heat value;

[0046] S450, cold data compression: performing lossless compression on cold data;

[0047] S460, metadata update: update the storage location metadata;

[0048] S470, layering monitoring feedback: dynamically adjust the heat value weight parameter.

[0049] Further, the steps of metadata update are as follows:

[0050] Hashing the unique identifier of the log to obtain a fixed-length hash value, and then performing a bit XOR operation on the hash value and the storage level identifier to obtain the final storage location index;

[0051] The storage level identifier includes SCM, QLC-NVMe and HDD.

[0052] The application also proposes a distributed persistent log storage system based on storage class memory, which performs the steps of a distributed persistent log storage method based on storage class memory as described above, comprising:

[0053] Partition state detection and weakly consistent write module: this module monitors the network connectivity state of the three AZs in real time, and when a network partition event is detected, it automatically switches the write operation to a weakly consistent mode, temporarily writes the log data to the local storage class memory, and records the hybrid logical clock timestamp and business priority label;

[0054] Dynamic quorum adjustment module: during the network partition duration, this module dynamically adjusts the quorum weight of the RAFT consensus protocol, recalculates the arbitration threshold according to the real-time connectivity state of each AZ, maintains partial availability of the system by automatically downgrading to a weakly consistent write mode, and continuously listens for partition recovery signals;

[0055] Partition recovery conflict merging module: when the network recovers, the module collects logs to be merged from the weakly consistent storage area of each AZ, sorts them according to the hybrid logical clock timestamp to identify timing conflicts, solves concurrent write conflicts through a business priority arbitration mechanism, and batch submits the merged logs to the strongly consistent group, updates the global vector clock state and cleans up the weakly consistent storage area;

[0056] Spacetime slice cold-hot layering module: the module analyzes the spacetime characteristics and access frequency of logs, dynamically calculates the heat value and decides the layering threshold, migrates data to the three-layer storage medium according to the heat value, performs compression optimization on cold data to reduce storage cost, and updates the metadata index to realize efficient query routing across storage layers;

[0057] Global metadata coordination module: unified management of log storage location index, vector clock state and layering strategy parameters, provides metadata query service for all modules, accelerates metadata access through distributed cache, and synchronizes the state changes of each module in real time;

[0058] Layered compression and cost optimization module: for cold data storage layer, implement lossless compression algorithm to reduce storage overhead, dynamically adjust compression strategy according to data characteristics and access mode, monitor the capacity and performance indicators of each storage layer, automatically trigger data reorganization or migration, balance storage cost and access delay.

[0059] The beneficial effects of the present application are:

[0060] The present application, through the RDMA link layer active probing mechanism and the storage layer distributed lock arbitration mechanism, cooperates to quickly perceive link interruption and trigger master node switching when physical network partition occurs, and forces single-partition write; at the same time, combined with the global data version checking strategy, automatically solves the cross-partition write conflict in the recovery stage.

[0061] The design not only maintains the low-delay transmission advantage of RDMA in high-sensitive scenarios such as power grid control instructions, but also completely eliminates the risk of double-master data conflict caused by network isolation, guarantees the global uniqueness of power operation instructions and the safety and stability of the system, and meets the requirements of millisecond-level fault isolation and zero data conflict in extreme fault scenarios of power grid. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a flowchart of a distributed persistent log storage method based on storage-level memory of the present application;

[0063] Figure 2 is a structural block diagram of a distributed persistent log storage system based on storage-level memory of the present application.

[0064] In the figure: 101, partition state detection and weak consistent write module; 102, dynamic quorum adjustment module; 103, partition recovery conflict merging module; 104, spatiotemporal sharding cold-hot layering module; 105, global metadata coordination module; 106, hierarchical compression and cost optimization module. DETAILED DESCRIPTION

[0065] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that discussions of these implementations are merely provided to enable those skilled in the art to better understand so as to implement the subject matter described herein, and variations of elements discussed can be made by one skilled in the art, without departing from the scope of the present description. Various examples can omit, substitute, or add various procedures or components as appropriate, and the methods described can not require the particular order described, where appropriate. Also, features described with respect to some examples can be combined in other examples.

[0066] As shown in Figure 1 a storage class memory based distributed persistent log storage method, comprising the following steps:

[0067] S100, partition state detection and weak consistent write: real-time monitoring of the network connectivity state of three availability zones (AZ), when a partition event is detected, the system automatically switches the write operation to a weak consistent mode, uses the storage class memory (SCM) to temporarily store the log data, and records the hybrid logical clock (HLC) timestamp and business priority, ensuring that low-latency local write capability can still be provided during the partition period.

[0068] In an embodiment of the present application, the following steps are specifically included:

[0069] S110, partition state detection: call the partition detector (Partition Detector) to calculate the cross-AZ network connectivity through the UDP heartbeat packet.

[0070] Implementation:

[0071] A 3x3 network connectivity matrix is constructed, and each element in the matrix represents the connectivity state between two availability zones, 0 indicating connectivity and 1 indicating disconnection. The row and column indices of the matrix correspond to the numbers of the availability zones, respectively.

[0072] ;

[0073] wherein, is an n x n network connectivity matrix, is the number of availability zones, fixed at 3;

[0074] The partition state of the current system is determined by traversing all elements of the matrix and summing. If the sum of the matrix elements is 0, it indicates that all available areas are in a connected state, and the system state is marked as normal (NORMAL) at this time; otherwise, it indicates that there is network partitioning, and the system state is marked as isolated (ISOLATED).

[0075] The system sends a heartbeat packet every 100 milliseconds for connectivity detection. If no response is received for more than 300 milliseconds, it is determined that the connection is disconnected.

[0076] Calculate the partition state:

[0077] ;

[0078] wherein, is the partition state identifier, is the matrix row and column index, representing the AZ number, is the normal mark, is the isolated mark, represents the sum of all elements of the matrix;

[0079] S120, business priority verification: analyze the business label of the log to determine whether strong consistency is required.

[0080] Implementation:

[0081] The system assigns a business priority label to each log, with a value range of 0 to 10. When the label value is greater than or equal to 7, it indicates that this is a financial-level critical business log that requires strong consistency writing; when the label value is less than 7, weak consistency writing can be used.

[0082] Strong consistency condition:

[0083] ;

[0084] wherein, is the business priority label, with a value range of [0, 10], and 7 is the financial-level business threshold;

[0085] The specific priority classification is as follows: values between 0 and 3 (not including 3) are low-priority business, values between 3 and 7 (not including 7) are medium-priority business, and values between 7 and 10 are high-priority business, which requires strong consistency guarantee;

[0086] Business priority classification:

[0087] is low priority;

[0088] is medium priority;

[0089] High priority (strong consistency).

[0090] S130, Routing decision and write: dynamically select write protocol according to system partition state and service priority, and execute SCM persistence operation.

[0091] Implementation:

[0092] 1. Strong consistency group write: when the service priority is greater than or equal to 7 and the system is in normal state, use RAFT, DC protocol for cross availability zone replication. The total write delay consists of two parts: twice the cross availability zone round trip delay (each no more than 100 microseconds) and the SCM local write delay (no more than 500 nanoseconds). If the write operation is not completed within 500 milliseconds, it is determined to be timeout. The write needs to meet the confirmation requirement of dynamic quorum.

[0093] Strong consistency group write (when and NRMAL):

[0094] Cross AZ replication (RAFT-DC protocol):

[0095] ;

[0096] Wherein, is the total delay of strong consistency write, is the cross AZ round trip delay (measured ), is the SCM local write delay (measured ), write timeout, is the dynamic quorum;

[0097] 2. Weak consistency group write: when the service priority is less than 7 or the system is in isolation state, execute local SCM direct write. Each log to be written contains the following fields: log data content, hybrid logical clock value, service priority, physical timestamp and node identifier. The total write delay consists of SCM write time and system overhead time, wherein the system overhead is not more than 100 microseconds.

[0098] Weak consistency group write (when or ISLATED):

[0099] Local SCM direct write:

[0100] ;

[0101] Wherein, is the log data content, is the hybrid logical clock value (S140), For business priority, For physical timestamp, For node identifier;

[0102] Write latency:

[0103] ;

[0104] Wherein, Total weak-consistent write latency, System overhead time;

[0105] S140, hybrid logical clock (HLC) marking: generate globally orderable timing marks for weak-consistent logs.

[0106] Implementation:

[0107] The system obtains the current physical clock value (including the maximum 2 ms NTP clock offset between servers), and reads the HLC value (0 for the first write) and the logical counter value (0 for the first write) of the previous log.

[0108] Input:

[0109] Current physical clock (including NTP offset ), NTP clock offset;

[0110] HLC value of the previous log (initially 0);

[0111] Logical counter (initially 0);

[0112] When the physical clock value is equal to the HLC value of the previous log, add the larger of the physical clock value and the previous HLC value to the current logical counter value as the HLC value of the new log; otherwise, add 1 to the larger of the physical clock value and the previous HLC value as the HLC value of the new log.

[0113] Calculation:

[0114] ;

[0115] Wherein, Hybrid logical clock value of the current log, Maximum value of the physical clock and the previous HLC, Current physical clock value, Logical counter value;

[0116] Simultaneously update the logical counter: if the physical clock value is equal to the HLC value of the previous log, then add 1 to the counter value; otherwise, reset the counter value to 0. This ensures that even in the case of clock synchronization, strictly increasing time markers are generated for concurrent writes.

[0117] Counter update:

[0118] ;

[0119] wherein, is a monotonically increasing logical counter;

[0120] S150, state marker and response: return the write result to the client, and attach the state marker.

[0121] Implementation:

[0122] The system returns different state markers according to the write result: the COMMITTED marker is returned when the strong consistency is successfully submitted, the response delay is not more than 201 microseconds, and no retry is needed; the ISOLATED_PENDING marker is returned when the weak consistency or isolated write is successfully written, the response delay is not more than 101 microseconds, and the retry needs to wait for the partition to be merged; the REJECT marker is returned when the strong consistency group is frozen during the partition period, the response delay is not more than 10 microseconds, and the exponential backoff strategy is used for retry

[0123] For operations that need to be retried, the retry interval uses an exponential growth strategy: the first retry interval is 100 milliseconds, and then each retry interval is doubled, but the maximum is not more than 5 seconds. The specific retry interval calculation method is: use the retry number as the index, perform exponential operation on 100 milliseconds, and then take the smaller value of 5 seconds as the actual retry interval.

[0124] Retry interval:

[0125] ;

[0126] wherein is the number of retries, is the base retry interval, is the maximum retry interval upper limit, is the retry interval time;

[0127] S200, dynamic quorum adjustment: during network partition, dynamically adjust the quorum of the RAFT consensus protocol, recalculate the weight according to the connectivity state of each AZ, automatically downgrade to the weak consistency write mode, and continuously listen to the partition recovery signal to ensure that the system maintains partial availability during the partition period.

[0128] In an embodiment of the present application, the following steps are specifically included:

[0129] S210, Live Node Counting: Based on the output of the Partition Detector, count the number of live nodes.

[0130] Implementation:

[0131] Get the network state matrix as input, which reflects the connectivity state between each availability zone. The dimension of the matrix is 3x3, corresponding to 3 availability zones. For each element in the matrix, check the live state of the node by traversing: 1 for live state, 0 for failure state. Accumulate the count value of all live states to get the total number of live nodes in the current system. This value must be greater than or equal to 1 and less than or equal to the total number of availability zones, which is 3.

[0132] Input: Network State Matrix (from S110);

[0133] where, is the network connectivity matrix of n x n, is the number of availability zones, fixed at 3; the total number of nodes is (fixed AZ number);

[0134] Calculate the number of live nodes:

[0135] ;

[0136] where, is the indicator function, live = 1, failure = 0, is the number of currently live nodes, is the node index, ;

[0137] Output: (int, ≥ 1);

[0138] S220, Correction Factor Generation: Calculate the correction factor based on the partition severity to ensure that the quorum is strictly greater than the partition boundary.

[0139] Implementation:

[0140] Get the number of currently connected availability zones, which is an integer ranging from 1 to 3. When the number of connected availability zones is greater than or equal to 2, the system is in a normal state, and the correction factor is set to 0; when the number of connected availability zones is less than 2, the system is in an isolated state, and the correction factor needs to be calculated. The calculation method is: take the logarithm of the current number of live nodes with base 2, and round up to get the correction factor. This calculation method can appropriately increase the quorum threshold when a partition occurs, effectively preventing the risk of split-brain in the system.

[0141] Input: Connected AZs number (from S110);

[0142] Calculation:

[0143] ;

[0144] wherein, is the correction factor, is the current connected AZs number, is the base-2 logarithm, is the ceiling operation;

[0145] wherein, The term increases the quorum threshold when partitioning, eliminating the risk of boundary split-brain.

[0146] S230, Dynamic Quorum Calculation: Combine the number of live nodes and the correction factor to generate the dynamic quorum.

[0147] Implementation:

[0148] Calculate the base quorum: divide the current number of live nodes by 2 and take the floor, then add 1. This base value ensures that consensus can be reached under normal circumstances. Then add the base quorum to the correction factor to get a temporary value. Finally, compare this temporary value with the current number of live nodes, and take the smaller value as the final dynamic quorum. This ensures that the quorum is neither smaller than the minimum consensus requirement nor exceeds the actual available node number. The final dynamic quorum generated must be greater than or equal to 1 and less than or equal to the current number of live nodes.

[0149] Base Quorum:

[0150] ;

[0151] wherein, is the base quorum, is the floor operation;

[0152] Final Value:

[0153] ;

[0154] wherein, is the final dynamic quorum, is the minimum operation, taking values in the range ;

[0155] S240, Split-Brain Immunity Verification: Verify that the dynamic quorum can avoid any split-brain scenario in partitioning.

[0156] Implementation:

[0157] By reductio ad absurdum. Assume there are two network partitions P1 and P2 in the system, containing k and m nodes respectively. If a split brain is to be formed, both partitions must satisfy the condition that the number of nodes is greater than or equal to the dynamic quorum. At the same time, the total number of nodes in both partitions must be less than or equal to the total number of live nodes in the system. Substituting these conditions into the calculation can find that the minimum total number of nodes required by the two partitions will be greater than the actual total number of live nodes in the system, which contradicts the reality. Therefore, it can be proved that in any partition scenario, it is impossible for two partitions to satisfy the quorum requirement at the same time, thus ensuring the consistency of the system.

[0158] Assume there are two partitions (node number ) and (node number ):

[0159] Must satisfy ;

[0160] Must satisfy ;

[0161] Constraint ;

[0162] Where, is the two network partitions, is the number of nodes in partition P1, is the number of nodes in partition P2

[0163] Reductio ad absurdum:

[0164] ;

[0165] ;

[0166] ;

[0167] ;

[0168] Conclusion: It is impossible for two partitions to satisfy the quorum at the same time.

[0169] S250, write permission decision: Before writing to the strong consistency group, check whether the current partition meets the dynamic quorum requirement.

[0170] Implementation:

[0171] The system compares the current number of live nodes with the dynamic quorum before performing a write operation. When the number of live nodes is greater than or equal to the dynamic quorum, it indicates that the current partition meets the write condition, and the system returns an ALLOW_COMMIT state and ensures that the response delay of the permission check does not exceed 100 microseconds. If the number of live nodes is less than the dynamic quorum, the system returns a FREEZE state. For a frozen write request, the system schedules a retry. The retry interval uses an exponential backoff strategy: the first retry waits for 100 milliseconds, and each subsequent retry waits for twice the time of the previous one, but not more than 5 seconds. This mechanism can avoid additional burden on the system caused by frequent retries.

[0172] Logical AND output:

[0173] If :

[0174] ALLOW_COMMIT (return ALLW_COMMIT, associated with the strongly consistent group of S150), the response delay is ;

[0175] wherein is the permission check response delay, and ALLW_COMMIT is the state identifier of the ALLOW_COMMIT

[0176] Otherwise: freeze the strongly consistent write (return FREEZE, associated with the REJECT of S150), and the retry interval is ;

[0177] wherein is the retry waiting time, is the number of retries, and FREEZE is the state identifier of the frozen write;

[0178] S300, conflict merging after partition recovery: After the network partition is recovered, collect the logs to be merged from the weakly consistent groups of each AZ, sort them according to the hybrid logical clock (HLC), and detect the timing conflicts. Resolve concurrent write conflicts through business priority arbitration, batch commit the merged logs to the strongly consistent group (RAFT, DC), update the global vector clock state, and clean up the weakly consistent data.

[0179] In an embodiment of the present application, the following steps are specifically included:

[0180] S310, restore event listening: listen to the partition detector (Partition Detector) to capture the partition recovery event (the availability zone state changes from isolation to normal).

[0181] Implementation:

[0182] The system detects state changes by comparing the available zone state at the current time and the previous time. Specifically, the available zone state at the current time and the available zone state at the previous time are obtained, and when and only when the previous time state is isolated and the current time state is normal, it is determined that a partition recovery event has occurred. The available zone state can only be normal or isolated. The system requires that the response time of state change detection be no more than 1 millisecond to ensure timely capture of recovery events.

[0183] Recovery condition:

[0184] ;

[0185] wherein, is the state change detection result at time t, is the AZ state at time t, is the AZ state at time t-1, is the logical AND operation, is the logical NOT operation;

[0186] Response delay:

[0187] ;

[0188] wherein, is the state change detection response time, is the maximum allowed response delay;

[0189] S320, weakly consistent log collection: pull the logs written during the isolated period from the weakly consistent group (ST, VClock storage area) of all available zones.

[0190] Implementation:

[0191] The system will collect all the pending logs in all available zones. Define a set to store all pending logs, then iterate through each available zone (the total number is fixed at 3), and add the pending logs in that available zone to the set. The total time consumption of the entire collection process consists of two parts: twice the cross-zone round-trip latency (100 microseconds each time) and the maximum write delay of the storage level memory (500 nanoseconds). The system takes the sum of the two parts as the upper limit of the performance constraint.

[0192] Log set definition:

[0193] ;

[0194] wherein, is the set of all pending logs, is the set of pending logs in the i-th AZ, is the total number of AZs (fixed at 3), is the set union operation;

[0195] Performance constraints:

[0196] ;

[0197] where, is the total time spent on log collection, is the cross-AZ round-trip latency (100 μs), is the SCM write latency (500 ns), is the max operation;

[0198] S330, timing conflict detection (HLC ordering): all logs are sorted by hybrid logical clock values, and concurrent conflicts are detected.

[0199] Implementation:

[0200] A set of conflicting log pairs is defined. When the hybrid logical clock timestamps of two logs are the same, they are added to the conflict set as a pair. For all logs to be processed, a global ordering is established by comparing their hybrid logical clock timestamps: when the timestamp of a log is less than that of another log, the former is considered to be earlier in time than the latter. This ordering ensures that all logs can establish a clear precedence relationship.

[0201] Conflict pair set:

[0202] ;

[0203] where, is the set of log pairs with timing conflicts, is the log pair to be compared, is the hybrid logical clock timestamp of the log;

[0204] Global ordering:

[0205] ;

[0206] where, is the total order symbol, is if and only if, is any element;

[0207] S340, service priority arbitration: concurrent conflict logs are arbitrated according to service priority.

[0208] Implementation:

[0209] The system adopts a three-level arbitration strategy: compare the business priority difference of the two conflict logs, if the priority of the first log is 3 or more higher than the priority of the second log, select the first log; if the priority of the second log is 3 or more higher than the priority of the first log, select the second log; if the priority difference of the two logs is less than 3, perform a log merge operation (such as dictionary merge) to retain the information of the two logs.

[0210] Arbitration rules:

[0211] ;

[0212] wherein, is an arbitration selection function, is a pair of logs to be arbitrated, is the business priority of the log, is the priority difference threshold (T =3), is a log merge operation function;

[0213] Merge operator example:

[0214] ;

[0215] wherein, is a custom merge operator;

[0216] S350, vector clock synchronization: update the global vector clock state, mark the timing position of the merged log.

[0217] Implementation:

[0218] For each availability zone (numbered from 1 to 3), the system updates the clock value of the availability zone in the global vector clock array. The update rule is: in the sorted log set, find all logs originating from the availability zone, take the maximum value of the hybrid logical clock timestamp of these logs as the new value of the availability zone in the vector clock array. This update method ensures that the vector clock can accurately reflect the latest write timing of each availability zone.

[0219] Update rule (for each AZ k ):

[0220] ;

[0221] wherein, is the global vector clock array, is the AZ number (n ), is the sorted log set, is the log source AZ, For hybrid logical clock timestamp;

[0222] S360, strongly consistent group commit: commit the merged log batch to the strongly consistent group (RAFT, DC) using dynamic quorum validation.

[0223] Implementation:

[0224] The system needs to meet the following conditions when committing: the number of acknowledgment responses must be greater than or equal to the dynamic quorum (the value comes from the calculation result of S230). The total time consumption of the commit process consists of three parts: twice the cross-AZ round-trip delay (for communication of the consensus protocol), the product of the total number of merged logs and the storage class memory write delay (for data persistence), and a 10-millisecond timeout threshold for the merge operation.

[0225] Commit conditions:

[0226] ;

[0227] Where, is the acknowledgment response count, is the dynamic quorum (from S230);

[0228] Delay model:

[0229] ;

[0230] Where, is the total time consumption of the commit, is the cross-AZ round-trip delay, is the total number of merged logs, is the SCM write delay, is the merge operation timeout threshold (10ms);

[0231] S370, state cleanup and notification: clean up weakly consistent group data and update client log state.

[0232] Implementation:

[0233] The system updates the final state flag for each log according to the processing result: when the log is successfully completed, the state flag is marked as "merged"; when the log fails in the arbitration process, the state flag is marked as "arbitration failed". These state flags are used to notify the final processing result of the client log and guide the subsequent cleanup operation.

[0234] State flag update:

[0235] ;

[0236] Where, is the final state of log L, To merge a success state identifier, To arbitrate a failure state identifier;

[0237] S400, spatiotemporal slice cold and hot layering: according to the spatiotemporal characteristics (timestamp, source partition) and access frequency of the log, the heat value is calculated, the storage layering threshold is dynamically decided, the data is migrated to different performance storage media (SCM / QLC, NVMe / HDD), the compression optimization is performed on the cold data to save the storage cost, and the metadata index is updated to realize efficient query routing.

[0238] In an embodiment of the present application, the following steps are specifically included:

[0239] S410, spatiotemporal feature extraction: analyze the spatiotemporal attributes (timestamp and source partition) of the log, calculate the time decay factor and the spatial locality weight.

[0240] Implementation:

[0241] The calculation process of the time decay factor is as follows: the current system timestamp (unit: millisecond) and the hybrid logical clock timestamp (unit: millisecond) of the log are obtained, and the time difference value of the two is calculated. Then multiply the difference value by the decay coefficient (0.001 per millisecond), take the negative of the result and perform exponential operation to obtain a decay factor between 0 and 1, the smaller the value, the colder the data. This calculation method can make the heat of data decay exponentially over time.

[0242] The calculation process of the spatial locality weight is as follows: the vector clock value of the log source available area is obtained, which represents the cumulative write amount of the available area. Then divide the value by the sum of all available area vector clock values (traverse 3 available areas and sum), to get a weight value between 0 and 1, which reflects the proportion of the available area in the total write amount. This calculation method can reflect the spatial distribution characteristics of data.

[0243] Time decay factor:

[0244] ;

[0245] Wherein, is the time decay factor, the value range is [0, 1], the smaller the value, the colder the data, is the current system timestamp, is the hybrid logical clock timestamp of the log, is the decay coefficient, 0.001 / ms, controlling the decay rate, is the base number of natural logarithm, about equal to 2.71828;

[0246] Spatial locality weight:

[0247] ;

[0248] wherein, is the spatial locality weight, with a value range of [0, 1], representing the data proportion of the AZ, is the vector clock value of the kth AZ, representing the cumulative write amount of the AZ, is the total number of AZs, fixed at 3, is the AZ index, with a value range of [1, 3], is the number of the log source AZ;

[0249] S420, access heat calculation: count the log access frequency and generate a comprehensive heat value combining the space-time features.

[0250] Implementation:

[0251] The calculation process of the normalized access frequency is as follows: obtain the actual access frequency (unit: times / second) and the historical maximum access frequency of the log. Compare the actual access frequency with the historical maximum frequency, take the smaller value of the two, and then divide by the historical maximum frequency to obtain a normalized frequency value between 0 and 1. This normalization process can unify access frequencies of different magnitudes to the same scale.

[0252] Normalized access frequency:

[0253] ;

[0254] wherein, is the normalized access frequency, with a value range of [0, 1], is the log access frequency count, with a unit of times / second, is the historical maximum access frequency, with a unit of times / second, is the minimum value function;

[0255] The calculation process of the comprehensive heat value is as follows: multiply the time decay factor by a weight of 0.4, multiply the spatial locality weight by a weight of 0.3, multiply the normalized access frequency by a weight of 0.3, and add the three weighted results to obtain a comprehensive heat value between 0 and 1. This weighted calculation method considers the time feature, space feature and access feature of the data, and the sum of the three weights is 1 to ensure the normalization of the calculation result.

[0256] Heat value calculation:

[0257] ;

[0258] wherein, is the comprehensive heat value, with a value range of [0, 1], is the time decay weight, a spatial locality weight, an access frequency weight, a time decay factor, a spatial locality weight, a normalized access frequency, constrained by ;

[0259] S430, hierarchical threshold decision: determine the storage hierarchical threshold according to the heat value distribution.

[0260] Implementation:

[0261] The calculation process of the dynamic threshold is as follows: sort the heat values of all data, find the first quantile (value at 25% position) and the third quantile (value at 75% position), and the difference between the two is the interquartile range, which is used to measure the dispersion degree of data. Then add 1.5 times the interquartile range to the third quantile to get the hot data threshold; subtract 1.5 times the interquartile range from the first quantile to get the cold data threshold. Data greater than the hot data threshold will be stored in SCM, data less than the cold data threshold will be stored in HDD, and data between the two will be stored in QLC, NVMe. This threshold calculation method based on quartiles can adapt to changes in data distribution.

[0262] Dynamic threshold calculation:

[0263] ;

[0264] ;

[0265] wherein, the hot data threshold, data greater than which is stored in SCM, the cold data threshold, data less than which is stored in HDD, the first quantile of the heat value distribution, representing the value at 25% position, the third quantile of the heat value distribution, representing the value at 75% position, the interquartile range, used to measure the dispersion degree of data, the outlier coefficient, used to expand the threshold range;

[0266] S440, data slice migration: migrate the log to the corresponding storage layer according to the heat value.

[0267] Implementation:

[0268] The judgment process of the storage layer selection is as follows: the heat value of the data is obtained, which is compared with the hot data threshold and the cold data threshold. When the heat value is greater than or equal to the hot data threshold, the storage class memory (SCM) is selected as the storage layer; when the heat value is less than the hot data threshold but greater than or equal to the cold data threshold, the QLC flash memory (QLC, NVMe) is selected as the storage layer; and when the heat value is less than the cold data threshold, the mechanical hard disk (HDD) is selected as the storage layer.

[0269] The calculation process of the migration batch control is as follows: the total number of logs meeting the migration condition is counted, and the smaller value between the total number and the maximum migration limit (1024) is taken as the actual migration data amount. This batch control mechanism can avoid the impact of too large migration data amount on the system.

[0270] Migration rules:

[0271] ;

[0272] Wherein, is the storage level selection result, is the storage class memory, the highest performance storage layer, is the QLC flash memory, the medium performance storage layer, is the mechanical hard disk, the lowest performance storage layer, is the data heat value, is the hot data threshold, is the cold data threshold

[0273] Migration batch control:

[0274] ;

[0275] Wherein, is the data amount of single migration, is the maximum migration number limit of single batch, is the log item to be migrated, is the total number of logs meeting the migration condition, is the minimum value function;

[0276] S450, cold data compression: lossless compression is performed on the cold data.

[0277] Implementation:

[0278] The calculation process of the compression ratio prediction is as follows: the size of the log data (in bytes) is obtained, the logarithm value with 10 as the base is calculated, the value is multiplied by a coefficient of 0.4, and then 0.6 is added to obtain the predicted compression ratio. The calculation method of this logarithm function reflects the feature that the larger the data amount, the better the compression effect that can be obtained.

[0279] The judgment process of the compression trigger condition is as follows: the total size of the data to be compressed is accumulated and counted, and the compression operation is triggered when the total size reaches or exceeds 1 TiB. This threshold-based triggering mechanism can perform batch compression when the data volume accumulates to a certain scale, improving the compression efficiency.

[0280] Compression ratio prediction:

[0281] ;

[0282] wherein, is the predicted compression ratio, representing the ratio of the size after compression to the original size, is the size of the log data, is the logarithmic function with base 10, is the compression ratio coefficient, is the base compression ratio;

[0283] Compression trigger condition:

[0284] ;

[0285] wherein, is the accumulated data size, is the compression trigger threshold, which is 1 TiB in size;

[0286] S460, metadata update: update the storage location metadata.

[0287] Implementation:

[0288] The calculation process of the location index is as follows: the unique identifier of the log is subjected to a hash operation to obtain a fixed-length hash value, and then the hash value is subjected to a bitwise XOR operation with the storage level identifier (SCM, QLC-NVMe or HDD) to obtain the final storage location index. This index calculation method based on hash and XOR can achieve uniform distribution of data and support fast location lookup.

[0289] Action:

[0290] Modify metadata tag:

[0291] ;

[0292] wherein, is the storage level identifier, is an enumerated set of storage levels, including (storage level memory), (QLC flash) and (mechanical hard disk) three types;

[0293] Location index calculation:

[0294] ;

[0295] wherein, is the storage location index of the log item Li, is the log unique identifier for locating the log, is a hash function mapping to a fixed length value, is a bit XOR operation for merging the hash value and the storage tier information, is the storage tier identification;

[0296] S470, hierarchical monitoring feedback: dynamically adjusting the heat value weight parameter.

[0297] Implementation:

[0298] The calculation process of weight adaptation is as follows: the current cache hit rate and the target hit rate (both are values between 0 and 1) are obtained, and the difference between the two is calculated. Then multiply the difference by the learning rate (0.05), and add the result to the current time decay weight to get the updated weight value. At the same time, the other weights need to be adjusted accordingly to ensure that the sum of all weights is 1. This feedback-based weight adjustment mechanism can dynamically optimize the hierarchical strategy according to the actual running effect of the system.

[0299] Weight adaptation:

[0300] ;

[0301] wherein, is the updated time decay weight, is the current time decay weight, is the current cache hit rate, with a value range of [0, 1], is the target hit rate, with a value range of [0, 1], is the learning rate, which controls the weight adjustment step size, and is constrained to to ensure that the sum of the weights is 1;

[0302] As shown in Figure 2 , in one embodiment, a distributed persistent log storage system based on storage class memory is proposed, which includes the following modules:

[0303] Partition state detection and weakly consistent write module 101: this module monitors the network connectivity state of the three available zones (AZ) in real time, and automatically switches the write operation to the weakly consistent mode when detecting the network partition event. The log data is temporarily written to the local storage class memory (SCM), and the hybrid logical clock timestamp and business priority label are recorded at the same time, ensuring that the low-latency local write capability is still maintained during the partition period, and the complete space-time and semantic information is reserved for subsequent conflict merging.

[0304] Dynamic quorum adjustment module 102: During network partition duration, this module dynamically adjusts the quorum weight of the RAFT consensus protocol, recalculates the arbitration threshold according to the real-time connectivity status of each AZ. By automatically downgrading to weakly consistent write mode, system partial availability is maintained, while continuously monitoring partition recovery signals to ensure that weight adjustment is synchronized with network state changes in real time.

[0305] Partition recovery conflict merging module 103: When the network recovers, this module collects logs to be merged from the weakly consistent storage area of each AZ, sorts them by hybrid logical clock timestamp to identify timing conflicts. Concurrent write conflicts are resolved through a business priority arbitration mechanism (such as high priority override or data fusion), and the merged logs are batched and submitted to the strongly consistent group (RAFT multi-data center cluster), updating the global vector clock state and cleaning up the weakly consistent storage area.

[0306] Spacetime slicing hot and cold layering module 104: This module analyzes the spacetime characteristics (timestamp, source partition) and access frequency of logs, dynamically calculates the heat value and decides the layering threshold. According to the heat value, data is migrated to three layers of storage media: hot data with high frequency access is stored in SCM, warm data with moderate access is stored in QLC NVMe solid state disk, and cold data with low frequency is stored in HDD mechanical disk. Cold data is compressed to optimize storage costs, and metadata indexes are updated to enable efficient query routing across storage layers.

[0307] Global metadata coordination module 105: Unified management of log storage location index, vector clock state and layering strategy parameters, providing metadata query services for all modules. Through distributed caching, metadata access is accelerated, and state changes of each module are synchronized in real time, ensuring that the system maintains strong consistency of metadata during partitioning, merging, layering, etc.

[0308] Layered compression and cost optimization module 106: For cold data storage layers, lossless compression algorithms (such as ZSTD) are implemented to reduce storage overhead, and compression strategies are dynamically adjusted according to data characteristics and access patterns. Monitor the capacity and performance indicators of each storage layer, automatically trigger data reorganization or migration, and balance storage costs and access latency.

[0309] In one embodiment, based on the above-mentioned distributed persistent log storage system and method based on storage-level memory, it is applied to the following examples:

[0310] Wherein the following background:

[0311] System: Provincial power grid monitoring center log system

[0312] Partition scenario: main data center (AZ1), backup center (AZ2), edge substation (AZ3);

[0313] Business Log:

[0314] High Priority: Transactional Operation Log (TX, e.g. switch instruction);

[0315] Medium Priority: Configuration Change Log (CFG, e.g. parameter adjustment);

[0316] Low Priority: Monitoring Log (MON, e.g. temperature report);

[0317] S100, Partition Detection:

[0318] S110, (State Monitoring): Network jitter causes edge substation (AZ3) to disconnect from main center, partition detector marks AZ3 state as ISOLATED.

[0319] S120, (Write Classification):

[0320] AZ3 receives switch instruction (TX log): due to high priority, allowed to write to weakly consistent group.

[0321] AZ3 receives temperature report (MON log): due to low priority, temporarily stored in local buffer.

[0322] S130, (Weakly Consistent Write): TX log written to AZ3's ST-VClock storage area, state marked as ISOLATED_PENDING.

[0323] S140, (Clock Synchronization): TX log stamped with hybrid logical timestamp (HLC_t), recording AZ3's local clock.

[0324] S150, (Dynamic Routing): Configuration change request (CFG log) routed to AZ2 (backup center) due to partition.

[0325] S200, Strongly Consistent Group Dynamic Arbitration:

[0326] S210, (Partition Topology Update): Detected AZ3 offline, updated topology to: {AZ1, AZ2} online, AZ3 offline.

[0327] S220, (Quorum Calculation): Dynamically calculated arbitration quorum: requires at least 2 AZ confirmations (originally required 2 out of 3 AZs).

[0328] S230, (Log Submission): AZ2 receives configuration change (CFG log), broadcast to AZ1 and AZ2 (itself), submitted after obtaining 2 ACKs.

[0329] S240, (State Synchronization): Submitted CFG log synchronized in AZ1 and AZ2's RAFT-DC storage area.

[0330] S250, (client response): configuration change success response client, AZ3 log still marked as pending.

[0331] S300, conflict merge after partition recovery:

[0332] S310, (recovery detection): network recovery, AZ3 state from ISOLATED to NORMAL, trigger merge.

[0333] S320, (log collection): pull TX log (switch instruction) from AZ3 during isolation period, merge with CFG log (parameter adjustment) of the main center.

[0334] S330, (time conflict detection): find TX log and CFG log timestamp same (concurrent operation).

[0335] S340, (service arbitration):

[0336] TX log priority = 9 (highest), CFG log priority = 5.

[0337] Priority difference ≥ 3, TX log covers CFG log.

[0338] S350, (clock synchronization): update global vector clock, record last timestamp of AZ3.

[0339] S360, (strongly consistent commit): merged log submitted to the strongly consistent group of the main center (AZ1).

[0340] S370, (state cleanup): delete pending log of AZ3, marked as MERGED.

[0341] S400, space-time slice hot and cold layering:

[0342] S410, (space-time feature extraction):

[0343] Switch instruction (TX): new log, low time decay; from edge station, high spatial weight.

[0344] Temperature reporting (MON): 3-day-old log, high time decay.

[0345] S420, (access frequency calculation): TX log: 50 accesses within 1 minute as high-frequency hot data; MON log: 1 access within 1 day as low-frequency cold data.

[0346] S430, (layering decision): TX log to hot layer (SCM storage); MON log to cold layer (HDD storage).

[0347] S440, (Data migration): Batch migrate MON logs to HDD, TX logs remain in SCM.

[0348] S450, (Cold data compression): ZSTD compression on MON logs in HDD, saving 70% space.

[0349] S460, (Metadata update): Update index: TX logs locate to SCM, MON logs locate to HDD.

[0350] S470, (Monitoring feedback): Monitor the SCM layer hit rate of 95%, keep the weight parameter.

[0351] Instance results:

[0352] Partition tolerance: During the network outage of the edge station (AZ3), high-priority operations are executed normally; after recovery, conflict instructions are overwritten according to priority, ensuring power grid safety.

[0353] Storage optimization: Hot instructions (TX) respond in milliseconds, and cold monitoring (MON) reduces storage costs by 60%.

[0354] Global consistency: All terminals query the final consistent switch state and configuration parameters.

[0355] Summary: This scheme realizes "partition without downtime, recovery without conflict, and adaptive storage" in power grid operation and maintenance, meeting the high reliability and low cost requirements of the power system.

[0356] The embodiments of the present application are described above, but the present application is not limited to the specific embodiments described above, which are only illustrative and not limiting. Those skilled in the art can make many forms under the inspiration of the present application, which are all within the protection scope of the present application.

Claims

1. A distributed persistent log storage method based on storage class memory, characterized in that, The method comprises the following steps: S100, partition state detection and weak consistent writing: real-time monitoring of network connectivity states of the three AZs, when a partition event is detected, the system automatically switches the writing operation to a weak consistent mode, uses an SCM to temporarily store log data, and records a hybrid HLC timestamp and a business priority; S200, dynamic quorum adjustment: during network partitioning, the quorum of the RAFT consensus protocol is dynamically adjusted, the weights are recalculated according to the connectivity states of the AZs, the system is automatically degraded to a weak consistent writing mode, and a partition recovery signal is continuously monitored; S300, conflict merging after partition recovery: after the network partition is recovered, the logs to be merged are collected from the weak consistent groups of the AZs, the HLC is sorted and the timing conflict is detected, the concurrent writing conflict is solved through business priority arbitration, the merged logs are batch-submitted to the strong consistent group, the global vector clock state is updated, and the weak consistent data is cleaned up; S400, space-time slice cold and hot layering: according to the space-time characteristics and access frequency of the logs, a heat value is calculated, a storage layering threshold is dynamically decided, data is migrated to different performance storage media, compression optimization is performed on cold data to reduce storage cost, and metadata index is updated to realize efficient query routing.

2. The method of claim 1, wherein, The specific steps of the partition state detection and weak consistent writing are as follows: S110, partition state detection: calling a partition detector, calculating cross-AZ network connectivity through a UDP heartbeat packet; S120, business priority verification: analyzing the business label of the log, and judging whether strong consistency is required; S130, routing decision and writing: dynamically selecting a writing protocol according to the system partition state and the business priority, and performing an SCM persistent operation; S140, hybrid logical clock labeling: generating a globally sortable timing label for the weak consistent log; S150, state labeling and response: returning the writing result to the client and attaching the state label.

3. The method of claim 1, wherein, The routing decision and writing include strong consistent group writing and weak consistent group writing; Strong consistent group writing: when the business priority is greater than or equal to 7 and the system is in a normal state, the RAFT and DC protocols are used for cross-AZ replication; Weak consistent group writing: when the business priority is less than 7 or the system is in an isolated state, a local SCM direct writing is performed.

4. The method of claim 1, wherein, The specific steps of the dynamic quorum adjustment are as follows: S210, live node statistics: based on the output of the partition detector, the number of currently live nodes is counted; S220, bias correction factor generation: calculating a bias correction factor according to the partition severity to ensure that the quorum is strictly greater than the partition boundary; S230, dynamic quorum calculation: combining the number of live nodes and the bias correction factor to generate a dynamic quorum; S240, brain split immunity verification: verifying that the dynamic quorum can avoid brain split in any partition scenario; S250, writing permission decision: checking whether the current partition meets the dynamic quorum requirement before strong consistent group writing.

5. The method of claim 1, wherein, The specific verification steps of the brain split immunity verification are as follows: Verification is performed by reductio ad absurdum; it is assumed that two network partitions P1 and P2 appear in the system, respectively containing k and m nodes; If a brain split is to be formed, both partitions must meet the condition that the number of nodes is greater than or equal to the dynamic quorum, and the total number of nodes in both partitions must be less than or equal to the total number of surviving nodes in the system. Substituting these conditions into the calculation shows that the minimum total number of nodes required for both partitions will be greater than the actual total number of surviving nodes in the system, which contradicts reality. The final dynamic quorum must be greater than or equal to 1 and less than or equal to the current number of surviving nodes.

6. The method of claim 1, wherein, The decision-making content of the write permission decision is as follows: before the system executes a write operation, it compares the current number of surviving nodes with the dynamic quorum; When the number of surviving nodes is greater than or equal to the dynamic quorum, it indicates that the current partition meets the write condition, the system returns the permission to commit state, and ensures that the response delay of the permission check does not exceed 100 microseconds; If the number of surviving nodes is less than the dynamic quorum, return the frozen write state; For the frozen write request, the system will arrange a retry, and the retry interval uses an exponential backoff strategy.

7. The method of claim 1, wherein, The specific steps of conflict merging after partition recovery are as follows: S310, recovery event listening: listening to the partition detector to capture the partition recovery event; S320, weak consistency log collection: pulling the isolated period write log from the weak consistency group of all available areas; S330, time sequence conflict detection: sorting all logs according to the hybrid logical clock value, detecting concurrent conflicts; S340, business priority arbitration: arbitrating concurrent conflict logs according to business priority; S350, vector clock synchronization: updating the global vector clock state to mark the time sequence position of the merged log; S360, strong consistency group submission: batch submitting the merged log to the strong consistency group and using the dynamic quorum to confirm; S370, state cleaning and notification: cleaning the weak consistency group data and updating the client log state.

8. The method of claim 1, wherein, The specific steps of time-space slice cold-hot layering are as follows: S410, time-space feature extraction: analyzing the time-space attributes of the log, calculating the time decay factor and the spatial locality weight; S420, access frequency calculation: calculating the access frequency of the log and generating a comprehensive heat value based on the time-space features; S430, layering threshold decision: determining the storage layering threshold according to the heat value distribution; S440, data slice migration: migrating the log to the corresponding storage layer according to the heat value; S450, cold data compression: performing lossless compression on cold data; S460, metadata update: updating the storage location metadata; S470, layering monitoring feedback: dynamically adjusting the heat value weight parameter.

9. The method of claim 1, wherein, The steps of metadata update are as follows: Hash the unique identifier of the log to obtain a fixed-length hash value, then perform a bitwise XOR operation between the hash value and the storage level identifier to obtain the final storage location index; The storage level identifier includes SCM, QLC-NVMe, and HDD.

10. A distributed persistent log storage system based on storage class memory, characterized in that, The steps of executing a storage-level memory-based distributed persistent log storage method according to any one of claims 1-9, comprising: Partition state detection and weak-consistency write module: Real-time monitoring of network connectivity status of three AZs, when detecting network partition event, automatically switch write operation to weak-consistency mode, log data temporary write to local storage level memory, while recording hybrid logical clock timestamp and business priority label; Dynamic quorum adjustment module: During the network partition duration, dynamically adjust the quorum weight of RAFT consensus protocol, recalculate the arbitration threshold according to the real-time connectivity status of each AZ, maintain the partial availability of the system by automatically downgrading to weak-consistency write mode, while continuously listening to partition recovery signals; Partition recovery conflict merging module: When the network recovers, collect logs to be merged from the weak-consistency storage area of each AZ, sort them by hybrid logical clock timestamp to identify timing conflicts, solve concurrent write conflicts through business priority arbitration mechanism, batch submit the merged logs to strong-consistency group, update global vector clock state and clean up weak-consistency storage area; Space-time slice hot and cold layering module: Analyze the space-time characteristics and access frequency of logs, dynamically calculate the heat value and decide the layering threshold, migrate data to three-layer storage media according to the heat value, perform compression optimization on cold data to reduce storage cost, and update metadata index to realize efficient query routing across storage layers; Global metadata coordination module: Unified management of log storage location index, vector clock state and layering strategy parameters, provides metadata query service for all modules, accelerates metadata access through distributed cache, and synchronizes the state changes of each module in real time; Layered compression and cost optimization module: For cold data storage layer, implement lossless compression algorithm to reduce storage overhead, dynamically adjust compression strategy according to data characteristics and access mode, monitor the capacity and performance indicators of each storage layer, automatically trigger data reorganization or migration, balance storage cost and access delay.

Images