Distributed cluster security backup method and system oriented to cloud-side environment

Abstract

The invention provides a distributed cluster security backup method and system oriented to a cloud-side environment, and relates to the technical field of distributed storage systems, and the method comprises the steps: obtaining cloud-side cluster data, and adaptively selecting a backup mode based on an erasure code and a multi-copy hybrid redundancy strategy; establishing a workload-aware incremental replication mechanism, and optimizing fault recovery by using idle resources; the problem of lock competition is solved by adopting a multi-copy consistency guarantee mechanism based on two-stage submission, and lock-free reading is realized based on MVCC. According to the invention, the data backup flexibility, the fault recovery efficiency and the system consistency guarantee capability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage systems, in particular to a distributed cluster security backup method and system for a cloud-edge environment. BACKGROUND

[0002] With the rapid development of cloud computing and edge computing technologies, cloud-edge integrated architecture is becoming an important mode of data management. This architecture combines the powerful computing and storage capabilities of the cloud with the real-time processing capabilities of the edge, effectively supporting various business scenarios. In this distributed environment, data security backup is a key technology to ensure reliable system operation. Traditional data backup mainly uses single redundancy strategies such as multiple copies or erasure codes. In the cloud-edge environment, data has distinct hot and cold attribute differences, diverse multi-tenant needs, and uneven resource distribution. At the same time, the unstable network connection in the cloud-edge environment and the limited resources of edge nodes make data consistency assurance and fault recovery more complex.

[0003] The main defects and deficiencies in the prior art include: traditional backup strategies usually use single multiple copies or erasure codes, which cannot be dynamically adjusted according to data characteristics and resource states, resulting in low storage efficiency or insufficient reliability. In the cloud-edge environment, data at different locations has different access frequencies and importance, and fixed backup strategies cannot meet this differentiated demand; existing incremental replication mechanisms lack workload awareness, and still perform a large number of backup operations when the system load is high, causing performance interference, and also cannot fully utilize system idle resources to improve recovery efficiency, especially in the case of limited edge node resources, this problem is more prominent; traditional distributed consistency mechanisms have problems of intense lock competition and high commit delay when processing multiple copies of data, especially when the same strategy is used for primary key and secondary key management, resulting in significant performance degradation of the system in high concurrency scenarios, which cannot meet the dual requirements of data consistency and high availability in the cloud-edge environment. SUMMARY

[0004] The present application provides a distributed cluster security backup method and system for a cloud-edge environment, which can solve the problems in the prior art.

[0005] In a first aspect of the present application, a distributed cluster security backup method for a cloud-edge environment is provided, comprising: obtaining data in the cloud-edge integrated cluster architecture, the cloud-edge integrated cluster architecture including a cloud cluster, multiple edge clusters, and an edge data collection gateway connecting the edge clusters; Based on the erasure code and the mixed redundancy backup strategy of multiple copies, the data backup mode is adaptively selected according to the hot and cold attributes of the data, the tenant demand and the storage resource state, and the data backup mode includes multiple copy backup and erasure code backup; wherein the data repair mechanism of the erasure code backup is implemented on the cluster node using the erasure code backup; the atomicity of the operation is ensured by the distributed transaction lock during the switching process of the data backup mode, and if any link fails, it is rolled back to the state before switching; A workload-aware incremental replication mechanism is established, a load-aware pre-parsing and pre-merging log block incremental log strategy is designed, and the efficiency of fault recovery is optimized by using idle resources; For lock competition and commit delay problems, a multi-copy consistency guarantee mechanism based on two-phase commit is designed, which is applied to a cluster composed of one coordination node and multiple data nodes; two-phase commit includes pre-commit phase and commit phase, and is based on primary key and secondary key separation management; wherein, the pre-commit phase belongs to the transaction start pre-write phase, mainly for conflict detection, locking and global commit timestamp allocation; the commit phase adopts the strategy of asynchronous processing secondary key and synchronous committing primary key; read operation is based on MVCC multi-version control, read pointer and write pointer of write event queue are maintained to realize lock-free reading of write completion and sequential advancement.

[0006] Adaptive selection of data backup mode according to the hot and cold attributes of the data, tenant demand and storage resource state, including: Based on the access characteristics of time series data, data with relatively frequent recent updates is defined as hot data, and multiple copy backup is adopted, and older data is defined as cold data, and erasure code backup is adopted; the definition of hot and cold data can be specified by a pre-set configuration item, that is, the data of the last one or two natural months can be hot data, or the data of the last one or two weeks can be hot data; for different tenant data differentiation needs, specify tenant data to use erasure code backup, and the rest of the data uses multiple copy replication backup by default; For the cluster node using the erasure code backup, periodically scan the data block checksum; if a damaged block is found, pull the missing check block or data block from other nodes, use idle CPU resources for parallel repair, and avoid affecting the front-end business; When the storage resource is lower than the pre-set resource threshold, a consistent snapshot is taken for the target data, erasure code shards are generated based on the snapshot, the shards are stored to the corresponding cluster node, and after confirming that all erasure code shards have been successfully stored, the original multiple copy data is deleted, completing the switching of the backup mode from multiple copy replication to erasure code; When the storage resource is expanded, the complete data is recovered through the related algorithm of erasure code, multiple copies are generated according to the multi-copy replication strategy, and the copies are stored in different nodes respectively, and after confirming that all the copies have been successfully stored, the original erasure code shard is deleted, and the switching from the erasure code backup mode to the multi-copy replication is completed.

[0007] The strategy of load-aware pre-resolving and pre-merging the incremental log of the log block is designed, and the efficiency of fault recovery is optimized by using idle resources, including: The data is divided into multiple logical segments according to natural time segments with an interval of one hour, and the incremental changes of each logical segment are recorded by prewriting log to form original incremental log data; the original incremental log data is asynchronously replicated from the master node to the backup server, and a compression algorithm is executed on the original incremental log data during the replication process and a CRC32 check code is added to generate compressed and checked incremental log data; A load monitoring module is deployed on the backup server, and the CPU utilization and I / O latency of the backup server are collected and counted in real time using the load monitoring module to generate real-time resource usage state data; based on the real-time resource usage state data, when the load monitoring module detects that the CPU utilization of the backup server is less than 20% and the I / O latency is less than 10 milliseconds, it is determined that the backup server is in a computing resource idle state and a pre-resolving task is triggered; In the pre-resolving task, the compressed and checked incremental log data is resolved to identify and generate corresponding operation vectors; according to the timestamp information of the operation vectors, the log data in adjacent time windows is identified and merged in time sequence, and the operation vectors of adjacent time windows are combined to generate pre-merged log blocks; the operation vectors in the pre-merged log blocks are serialized in Protobuf format to obtain serialized pre-merged log block data, and the data is stored in the cache of the backup server; When the master node in the distributed system fails, the serialized pre-merged log block data is directly read from the cache of the backup server.

[0008] The pre-commit phase belongs to the transaction start prewriting phase, mainly for conflict detection and locking and global commit timestamp allocation, including: In the transaction start prewriting phase, the coordination node assigns a globally unique and increasing timestamp as the snapshot version and transaction identifier of the transaction after receiving the transaction request, so that all read operations can obtain data based on this snapshot version; at the same time, one of all the measurement points modified by the transaction is selected as the primary key, and the remaining measurement points are set as secondary keys, so that the commit state of the primary key becomes the basis for determining the atomicity of the entire transaction; The coordination node sends a pre-commit request to all data nodes involved in the measurement point according to the transaction identifier and the measurement point information. After receiving the pre-commit request, the data node performs conflict detection on each measurement point to determine whether the measurement point has been modified by other transactions. If no conflict is detected, the data node locks the measurement point in memory and records the transaction identifier, and writes the relevant data changes to the pre-write log, and then returns a pre-commit success response to the coordination node. After the coordination node collects all pre-commit success responses from the data nodes, it assigns a globally increasing commit timestamp greater than all existing transaction identifiers and allocated commit timestamps as the final commit version marker for the transaction.

[0009] The commit phase adopts a strategy of asynchronous processing secondary keys and synchronous committing primary keys, including: In the commit phase, the coordination node first sends a commit request to the data node where the primary key is located. After receiving the commit request, the data node where the primary key is located updates the primary key data version to the commit timestamp and cleans up the pre-write lock. If the primary key commit fails, the coordination node triggers a rollback operation and sends a rollback request to all related data nodes to release all locks and clean up the pre-write data. If the primary key commit is successful, the coordination node processes the commit of the secondary keys in the background in an asynchronous manner, sends a commit request to the data nodes that manage these secondary keys, and the request carries the transaction identifier and the commit timestamp. After receiving the commit request, the data node finds the corresponding secondary key according to the transaction identifier, updates its data version to the commit timestamp, and releases the lock. When the coordination node detects that all modified measurement points belong to the same region, it skips the aforementioned asynchronous commit phase and directly sends a commit request containing the primary key and all secondary keys to the data node that manages the region through a remote procedure call, so that all measurement points complete the commit in the same operation, thereby reducing transaction latency. After the primary key commit is successful, the coordination node immediately returns a transaction success response to the client, and starts an asynchronous task in the background to clean up residual locks and log records.

[0010] By maintaining the read pointer and the write pointer of the write event queue, a lock-free read that allows reading only after writing is completed and advancing in order is implemented, including: The read pointer and the write pointer are set on the write event queue, the read pointer points to the first position of the write operation that has not been completed in the queue, and the write pointer points to the latest write position at the tail of the queue. The data units in the write event queue include readable data units that have been written, unreadable data units that have been written but not completed, and data units to be written. When a write operation request is received, a new write operation is added to the tail of the write event queue, the write pointer is moved to point to the newly added data unit, and the data unit is marked as written but not completed, indicating that the data unit has started to be written but has not been completed; When the write operation is completed, the corresponding data unit state is updated from written but not completed to written completed, indicating that the write of the data unit has been completed; When a read operation request arrives, the method checks the write status of the data unit pointed to by the read pointer and all data units in front of it, and if the write of a data unit has been completed but there are still data units in front of it with incomplete write operations, the data unit remains unreadable, the read pointer remains unchanged, and the read operation is blocked; When the data unit pointed to by the read pointer and all consecutive data units in front of it have completed the write operation, the read pointer is continuously moved until it encounters the position of the first data unit with an incomplete write operation, and the states of the consecutive data units with completed write operations are updated to readable, so that the read operation can access these data units in sequence.

[0011] In a second aspect of the embodiments of the present application, a distributed cluster security backup system for a cloud-edge environment is provided, comprising: A first unit is configured to obtain data in a cloud-edge integrated cluster architecture, wherein the cloud-edge integrated cluster architecture comprises a cloud cluster, a plurality of edge clusters, and an edge data acquisition gateway connected to the edge clusters. A second unit is configured to adaptively select a data backup mode according to the cold and hot attributes of the data, tenant requirements, and storage resource states based on a hybrid redundancy backup strategy of erasure coding and multiple copies, wherein the data backup mode comprises multiple copy backup and erasure coding backup; wherein a data repair mechanism of the erasure coding backup is implemented for a cluster node using the erasure coding backup; and wherein the atomicity of operations is ensured by a distributed transaction lock during the switching process of the data backup mode, and if any link fails, the state before switching is rolled back. A third unit is configured to establish a workload-aware incremental replication mechanism, design a strategy of workload-aware pre-analysis and pre-merge incremental logs, and optimize fault recovery efficiency by using idle resources. The fourth unit is used for solving the lock competition and commit delay problem, and a multi-copy consistency guarantee mechanism based on two-stage commit is designed, and the mechanism is applied to a cluster composed of one coordination node and multiple data nodes; the two-stage commit includes a pre-commit stage and a commit stage, and is managed based on separation of primary key and secondary key; wherein the pre-commit stage belongs to a transaction start pre-write stage, mainly performs conflict detection, locking and global commit timestamp allocation; the commit stage adopts an asynchronous processing secondary key and synchronous commit primary key strategy; a read operation is based on MVCC multi-version control, and through maintaining a read pointer and a write pointer of a write event queue, a lock-free read is realized, which allows reading only after writing is completed and sequentially advances.

[0012] The third aspect of the embodiment of the application, An electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0013] The fourth aspect of the embodiment of the application, A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0014] The beneficial effects of the present application are as follows: The present application can adaptively select the optimal backup mode according to the data hot and cold attributes, tenant requirements and storage resource states by implementing the erasure code and multi-copy hybrid redundancy backup strategy, significantly improves the data backup efficiency and storage space utilization rate under the cloud edge integrated cluster architecture, and guarantees the reliability of data recovery.

[0015] The workload-aware incremental replication mechanism designed by the present application effectively utilizes the idle resources of the system through the load-aware pre-analysis and pre-merge log block strategy, greatly reduces the performance overhead in the data synchronization and recovery process, optimizes the fault recovery efficiency, and enhances the elasticity and robustness of the system in complex network environments.

[0016] The multi-copy consistency guarantee mechanism based on two-stage commit proposed by the present application solves the lock competition and commit delay problem in the traditional distributed system, and through the lock-free read strategy of primary key and secondary key separation management and MVCC multi-version control, the concurrent processing capacity and transaction throughput of the system are significantly improved, the data consistency is guaranteed, and the read-write performance is optimized. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 It is a flowchart of the distributed cluster security backup method for the cloud edge environment of the embodiment of the present application. DETAILED DESCRIPTION

[0018] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the drawings in the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0019] The technical solutions of the present application will be described in detail in the following specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0020] Figure 1 The flowchart of the method for the cloud-edge environment-oriented distributed cluster security backup of the embodiments of the present application is shown in FIG. 1, which comprises the following steps. Figure 1 Obtaining data in the cloud-edge integrated cluster architecture, the cloud-edge integrated cluster architecture comprising a cloud cluster, a plurality of edge clusters and an edge data acquisition gateway connected to the edge clusters; Adaptive selection of a data backup mode according to the cold and hot attributes of the data, tenant requirements and storage resource states based on a hybrid redundancy backup strategy of erasure coding and multiple copies, the data backup mode comprising multiple copy backup and erasure coding backup; wherein a data repair mechanism of the erasure coding backup is implemented on a cluster node using the erasure coding backup; the atomicity of the operation is ensured by a distributed transaction lock during the switching process of the data backup mode, and if any link fails, the state before switching is rolled back; Establishing a workload-aware incremental replication mechanism, designing a strategy of workload-aware pre-analysis and pre-merging of an incremental log, and optimizing the fault recovery efficiency by using idle resources; For lock competition and commit delay problems, a multi-copy consistency guarantee mechanism based on two-phase commit is designed, the mechanism being applied to a cluster composed of one coordination node and a plurality of data nodes; the two-phase commit comprises a pre-commit phase and a commit phase, and is based on primary key and secondary key separation management; wherein the pre-commit phase belongs to a transaction start pre-write phase, mainly performing conflict detection, locking and global commit timestamp allocation; the commit phase adopts a strategy of asynchronous processing of secondary keys and synchronous commit of primary keys; read operation is based on MVCC multi-version control, and a read pointer and a write pointer of a write event queue are maintained to realize lock-free reading of write completion and sequential advancement.

[0021] In an optional embodiment, adaptive selection of a data backup mode according to the cold and hot attributes of the data, tenant requirements and storage resource states comprises:​ Based on the access characteristics of the time series data, the data with relatively frequent recent updates is defined as hot data, and a multi-copy backup is adopted, and the data with earlier updates is defined as cold data, and an erasure code backup is adopted; the definition of cold and hot data can be specified by a preset configuration item, that is, the data of the last one or two natural months can be hot data, or the data of the last one or two weeks can be hot data; for the differentiated needs of data of different tenants, the data of the specified tenant is backed up by using the erasure code, and the remaining data is backed up by default by using the multi-copy replication; The cluster nodes using the erasure code backup are periodically scanned for data block checksums; if a damaged block is found, the missing check block or data block is pulled from other nodes, and the idle CPU resources are used for parallel repair to avoid affecting the front-end business; When the storage resource is lower than the preset resource threshold, a consistent snapshot is taken for the target data, erasure code shards are generated based on the snapshot, the shards are stored to the corresponding cluster nodes, and after confirming that all erasure code shards have been successfully stored, the original multi-copy data is deleted, and the backup mode is switched from multi-copy replication to erasure code; When the storage resource is expanded, the complete data is recovered through the related algorithm of the erasure code, multiple copies are generated according to the multi-copy replication strategy, and the copies are stored to different nodes, and after confirming that all copies have been successfully stored, the original erasure code shards are deleted, and the backup mode is switched from erasure code to multi-copy replication.

[0022] The present application provides a method for adaptively selecting a data backup mode according to the cold and hot properties of data, tenant needs and storage resource states, intelligently judges data characteristics and system states, flexibly switches between multi-copy backup and erasure code backup, and realizes efficient use of storage resources.

[0023] In the present embodiment, the system first classifies data based on the access characteristics of time series data. The system classifies data as hot or cold by monitoring data update frequency. For example, for order data of a business system, the system records the update timestamp. If the data has been updated within the last 30 days, the data block is marked as hot data. Hot data is backed up using a multi-copy strategy, typically with three copies distributed across different physical nodes. If a node with a copy fails, the system can immediately read the complete data from other nodes, ensuring high data availability.

[0024] For data that exceeds one natural month and is only used for low-frequency queries, the system marks it as cold data. For example, the historical order data of a certain e-commerce platform has significantly reduced access frequency, with the number of daily queries not exceeding 0.1% of the total data volume. The system automatically marks this part of the data as cold data. Cold data uses an erasure code backup strategy, such as RS (Reed-Solomon) encoding, which divides the original data into multiple data blocks and generates check blocks. In specific implementation, a 10+4 encoding method can be used, that is, 10 original data blocks cooperate with 4 check blocks, which can tolerate the loss of any 4 blocks. This method significantly saves storage space compared to the multi-copy method, with redundancy reduced from 200% (3 copies) to 40% (10+4 encoding).

[0025] The system also supports differentiated backup strategies based on tenant needs. The system maintains a tenant configuration table that records the backup strategy preferences of specific tenants. For tenants designated for erasure code backup, regardless of their data access frequency, erasure code backup is used. For example, a certain data analysis tenant has a large amount of historical data but low access frequency, and the tenant configuration table sets its backup method as "EC_BACKUP" (erasure code backup). When processing data for this tenant, the system directly uses erasure code encoding, without the need to judge data cold and hot properties. For the remaining tenants without specified backup methods, the system defaults to multi-copy replication backup and dynamically adjusts based on data cold and hot properties.

[0026] For cluster nodes using erasure code backup, the system implements a regular data integrity check mechanism. The check program performs full data verification on each storage node of the cluster every 72 hours, discovers damaged blocks by calculating the CRC32 checksum of the data blocks and comparing it with the stored checksum. For example, in a certain storage cluster, the checksum of data block D5 stored on node N1 does not match, and the system determines that the block has been damaged. At this time, the system pulls enough data blocks and check blocks from other nodes N2, N3, N4, etc., and uses erasure code recovery algorithms to rebuild the damaged data block. During the rebuilding process, the system monitors the current node CPU utilization and only executes repair tasks when CPU utilization is less than 60%, and limits repair tasks to use at most 30% of CPU resources, avoiding affecting the performance of front-end business. After repair is complete, the system writes the rebuilt data block back to node N1 and updates the checksum record.

[0027] When the cluster storage resource is tight, the system triggers the backup mode switching from multiple copies to erasure code. The system regularly checks the storage resource usage, and when the available storage space is lower than the preset threshold (such as 20%), it identifies the multiple-copy hot data occupying a large space, and judges whether it has been converted into cold data. For the data set that meets the conversion conditions, the system first creates a consistent snapshot to ensure the integrity of the data during the conversion process. Taking a log data as an example, the system detects that the data set size is 100GB, there are 3 copies occupying 300GB of storage space, and it has not been updated for more than a month. The system creates a snapshot S1 for the data, generates erasure code shards based on S1, adopts a 10+4 coding mode, generates 14 shards, each about 10GB. The system stores these shards in different nodes of the cluster respectively, and records the shard location mapping table. After confirming that all shards have been successfully stored, the system deletes the original multiple-copy data, releasing about 160GB of storage space (from 300GB to 140GB).

[0028] When the cluster storage resource is expanded, the system can convert the cold data backed up by erasure code back to multiple-copy mode as needed. When the available storage space exceeds the preset rich threshold (such as 60%), the system identifies the data set converted to erasure code due to storage shortage, especially the data with increased recent access frequency. Taking the aforementioned log data as an example, if the recent query frequency of the data significantly increases, the system decides to convert it back to multiple-copy mode. The system reads at least 10 shards (under the 10+4 coding mode) from the erasure code shards distributed in different nodes, and reconstructs the complete data through the erasure code recovery algorithm. Subsequently, the system generates 3 complete copies according to the multiple-copy strategy, and stores them in 3 different nodes respectively. After confirming that all copies are successfully stored, the system deletes the original erasure code shards, completing the backup mode switching.

[0029] Through the above method, the system can dynamically adjust the data backup strategy according to the data characteristics, tenant demand and storage resource state, optimize the storage resource utilization efficiency while ensuring data reliability.

[0030] In an optional implementation, a strategy of load-aware pre-resolving and pre-merging incremental logs of log blocks is designed to optimize fault recovery efficiency through idle resource utilization, including: The data is divided into multiple logical segments according to natural time segments with an interval of one hour, and the incremental changes of each operation of adding, deleting or modifying in each logical segment are recorded by prewriting log to form original incremental log data; the original incremental log data is asynchronously replicated from the master node to the backup server, and a compression algorithm is executed on the original incremental log data during the replication process and a CRC32 check code is added to generate compressed and checked incremental log data; deploying a load monitoring module on the backup server, using the load monitoring module to collect and count the CPU utilization and I / O latency of the backup server in real time, and generating real-time resource usage state data; based on the real-time resource usage state data, when the load monitoring module detects that the CPU utilization of the backup server is less than 20% and the I / O latency is less than 10 milliseconds, it is determined that the backup server is in a computing resource idle state and a pre-analysis task is triggered; In the pre-analysis task, the compressed and checked incremental log data is analyzed, and the corresponding operation vector is identified and generated; according to the timestamp information of the operation vector, the log data in the adjacent time window is identified and merged in time sequence, and the operation vectors of the adjacent time window are combined to generate a pre-merged log block; the operation vectors in the pre-merged log block are serialized through the Protobuf format, and the serialized pre-merged log block data is obtained, and the data is stored in the cache of the backup server; When the master node in the distributed system fails, the serialized pre-merged log block data is directly read from the cache of the backup server.

[0031] The present application provides an incremental log strategy of load-aware pre-analysis and pre-merged log block, which realizes the technical scheme of optimizing fault recovery efficiency by idle resource utilization. In a distributed database system, in order to ensure data consistency and reliability, the data changes of the master node need to be copied to the backup server. This embodiment describes in detail how to use the idle resources of the backup server to process incremental log data in advance, thereby speeding up the fault recovery process.

[0032] In the data division stage, the system divides the data into multiple logical segments according to the natural time interval of each hour. For example, for a day of data, it can be divided into 00:00-01:00, 01:00-02:00, etc. 24 logical segments. For data operations in each logical segment, the system records incremental changes through the Write-Ahead Logging (WAL) mechanism. Specifically, when a user performs an INSERT, UPDATE, or DELETE operation, the system first records the operation content in the WAL file to form the original incremental log data. For example, when the "UPDATE users SET status='active' WHERE id=1234" operation is executed, the WAL record contains information such as table name (users), operation type (UPDATE), modified field (status), new value (active), condition field (id), and condition value (1234).

[0033] The log replication and processing phase, the system asynchronously replicates the original incremental log data from the master node to the backup server. During the replication process, compression processing is performed on the original incremental log data and a check code is added. The compression algorithm uses the LZ4 algorithm, which can achieve a compression ratio of 3:1 for highly repetitive log content. A 32-bit CRC32 check code is added at the end of each compressed log block for subsequent data integrity verification. For example, a log file containing 1000 operation records, with an original size of 5MB, may only occupy 1.7MB of storage space after compression, with 4 bytes of check code attached.

[0034] A load monitoring module is deployed on the backup server, which collects server resource usage every 5 seconds. The specific collection contents include CPU utilization (by reading / proc / stat file to calculate user state and system state CPU time ratio) and I / O latency (by iostat tool to collect disk operation average waiting time). The collected data is stored in a ring buffer, retaining the resource usage state data of the last 10 minutes. The load monitoring module analyzes the resource usage data in the last 1 minute, and when the CPU utilization of 6 consecutive samples is less than 20% and the I / O latency is less than 10 milliseconds, it is determined that the backup server is in a computing resource idle state, and the pre-analysis task is triggered.

[0035] When the pre-analysis task is executed, the system first decompresses the compressed and checked incremental log data, and ensures the data integrity through CRC32 check. The decompressed log is sorted according to transaction ID and operation timestamp, and the operation type (addition / deletion / change), affected table, affected row and specific field change of each log record are identified. The system converts the analysis result into an operation vector, each operation vector contains {operation_type, table_id, row_id, field_map, timestamp} five-tuple. For example, the aforementioned UPDATE operation can be converted into an operation vector of {2, 101, 1234, {status: "active"}, 1628156400}, where 2 represents the UPDATE operation type, 101 is the users table ID, and 1628156400 is the operation timestamp.

[0036] According to the timestamp information of the operation vector, the system identifies and merges the log data in adjacent time windows in chronological order. The size of the time window is set to 5 minutes, and the combination of operation vectors of two adjacent time windows generates a pre-merged log block. During the merging process, the system performs merging optimization on multiple operations on the same data row and only retains the final effective state. For example, if the same data row undergoes "insert->update->update" three times within 5 minutes, it can be merged into an insert operation with the final value; if it undergoes "insert->update->delete", it can be completely ignored. In this way, the log that originally contains hundreds of thousands of records can be optimized to tens of thousands of efficient operation vectors.

[0037] The operation vectors in the pre-merged log block are serialized in Protobuf format. Compared with JSON or XML format, the serialized data in Protobuf format has smaller volume and faster parsing speed. The serialization rule defines operation type (uint32), table ID (uint32), row ID (uint64), field mapping (map<string, bytes>), and timestamp (int64). The serialized pre-merged log block data is stored in the cache of the backup server, and the LRU (Least Recently Used) eviction policy is used to manage the cache space. By default, 1 GB of memory space is allocated for storing pre-processing results. For historical data exceeding the cache capacity, the system persists it to SSD storage and records the index information.

[0038] When the master node in the distributed system fails, the backup server detects that the heartbeat of the master node is interrupted (no heartbeat signal is received for more than 5 seconds for 3 times in a row) and starts the fault recovery process. The system directly reads the serialized pre-merged log block data from the cache of the backup server without the need to reparse the original log file. The operation vectors are deserialized through Protobuf, and the operations are applied in chronological order to quickly restore the data to the state before the failure. Compared with the traditional method of real-time parsing of original logs, this scheme can shorten the recovery time from minutes to seconds. For example, for a system containing 1 million transaction records, the traditional recovery may take 3-5 minutes, while using this scheme only takes 10-15 seconds to complete the recovery.

[0039] In an optional implementation, the pre-commit phase belongs to the transaction start pre-write phase, mainly performing conflict detection and locking and global commit timestamp allocation, including: In the transaction pre-write phase, the coordination node assigns a globally unique and increasing timestamp to the transaction as a snapshot version and transaction identification after receiving the transaction request, so that all read operations are based on the snapshot version to obtain data; meanwhile, it selects one point from all the points modified by the transaction as a primary key, and sets the remaining points as secondary keys, so that the commit state of the primary key becomes the basis for determining the atomicity of the entire transaction; According to the transaction identification and the point information, the coordination node sends a pre-commit request to all data nodes involved in the point, and after receiving the pre-commit request, the data node performs conflict detection on each point to determine whether the point has been modified by other transactions; if no conflict is detected, the data node locks the point in memory and records the transaction identification, and at the same time, writes the relevant data changes to the pre-write log, and then returns a pre-commit success response to the coordination node; After the coordination node collects all the pre-commit success responses from the data nodes, it assigns a globally increasing commit timestamp greater than all existing transaction identifications and allocated commit timestamps to the transaction as the final commit version mark of the transaction.

[0040] # Pre-commit phase implementation method of two-phase transaction commit in distributed time series database The embodiment provides a pre-commit phase implementation method of two-phase transaction commit in a distributed time series database, which is mainly applied in a distributed time series database system and is used for guaranteeing atomicity, consistency, isolation and durability (ACID characteristics) of transactions. The specific implementation process of the method is described in detail below.

[0041] In the distributed time series database system, the coordination node is responsible for receiving transaction requests initiated by the client and coordinating the execution process of the entire transaction. When the coordination node receives a transaction request, it will immediately assign a globally unique and increasing timestamp to the transaction, which is also used as the snapshot version and transaction identification of the transaction. For example, if the maximum allocated timestamp in the current system is TS_100, the coordination node will assign TS_101 as the transaction identification and snapshot version of the new transaction. This design enables the transaction to execute all read operations based on a consistent database snapshot, ensuring the consistency of read operations within the transaction.

[0042] The coordinating node analyzes all the measurement points involved in the transaction (i.e. data points in the time series database), and selects one of them as the primary key and the rest as secondary keys. The selection of the primary key can be based on various strategies, such as selecting the measurement point with the largest amount of modified data, selecting the measurement point located in the central node of the system, or selecting the measurement point with the highest historical access frequency. Suppose the transaction needs to modify measurement points P1, P2 and P3, the coordinating node can select P1 as the primary key and P2 and P3 as secondary keys. The setting of the primary key allows the system to quickly determine the commit state of the entire transaction by checking the commit state of the primary key, optimizing the efficiency of transaction state query.

[0043] The coordinating node sends a pre-commit request to all the data nodes involved in the measurement points according to the transaction identifier TS_101 and the measurement point information (P1, P2, P3). The pre-commit request contains the transaction identifier, the measurement point identifier, the new value, and the primary / secondary key flag. For example, the pre-commit request sent to the data node storing P1 contains: {transaction identifier: TS_101, measurement point: P1, new value: 37.5, type: primary key}; the pre-commit request sent to the data node storing P2 contains: {transaction identifier: TS_101, measurement point: P2, new value: 42.3, type: secondary key}.

[0044] After receiving the pre-commit request, the data node performs conflict detection for each measurement point. The purpose of conflict detection is to determine whether the measurement point has been locked or modified by other ongoing transactions. The data node completes this detection by checking the lock table in memory. The lock table is a hash table structure, with the measurement point identifier as the key and the transaction identifier currently holding the lock on the measurement point as the value (if it exists). For example, the lock table can be as follows: {P1: null, P2: TS_98, P3: null}, indicating that P2 is currently locked by transaction TS_98, and P1 and P3 are not locked.

[0045] If the data node detects that the measurement point has been locked by other transactions (e.g. P2 has been locked by TS_98), it returns a pre-commit failure response to the coordinating node, containing the conflicting transaction identifier information. If no conflict is detected, the data node locks the measurement point in memory and updates the value of the corresponding measurement point in the lock table to the current transaction identifier. For example, after locking P1, the lock table becomes: {P1: TS_101, P2: TS_98, P3: null}.

[0046] After the lock is successfully acquired, the data node writes the relevant data changes to the write-ahead log (WAL). The write-ahead log is a persistent storage structure that records all modification operations on the database for fault recovery. The format of the write-ahead log can be: {transaction identifier: TS_101, operation type: pre-commit, measurement point: P1, old value: 36.8, new value: 37.5, timestamp: 2023-09-15T10:15:30.000Z}. This step ensures that even if the system fails, the transaction state can be recovered through the log.

[0047] After the data node completes the lock and log writing, it returns a pre-commit success response to the coordinator node: {transaction identifier: TS_101, measurement point: P1, status: success}. The coordinator node collects all data node responses to determine whether the pre-commit phase is successful.

[0048] When the coordinator node collects all data node pre-commit success responses, it assigns a globally increasing commit timestamp to the transaction. The commit timestamp must be greater than all existing transaction identifiers and allocated commit timestamps. For example, if the maximum transaction identifier in the current system is TS_105 and the maximum allocated commit timestamp is CT_103, the commit timestamp of the new transaction should be CT_106. The commit timestamp serves as the final commit version marker for the transaction, which is used for subsequent visibility judgment and data query.

[0049] Through the above steps, the distributed time series database successfully completes the pre-commit phase of the two-phase commit, laying the foundation for the final commit of the transaction and ensuring the atomicity and consistency of the transaction in a distributed environment.

[0050] In an optional implementation, the commit phase adopts a strategy of asynchronous processing secondary keys and synchronous committing primary keys, including: In the commit phase, the coordinator node first sends a commit request to the data node where the primary key is located. After receiving the commit request, the data node where the primary key is located updates the primary key data version to the commit timestamp and cleans up the write-ahead lock. If the primary key commit fails, the coordinator node triggers a rollback operation and sends a rollback request to all related data nodes to release all locks and clean up the write-ahead data; If the primary key commit is successful, the coordinator node asynchronously processes the commit of the secondary keys in the background in parallel, sends a commit request to the data nodes that manage these secondary keys, and the request carries the transaction identifier and the commit timestamp. After receiving the commit request, the data node finds the corresponding secondary key according to the transaction identifier, updates its data version to the commit timestamp, and releases the lock; When the coordinator detects that all modified points belong to the same region, it skips the aforementioned asynchronous commit phase and directly sends a commit request containing the primary key and all secondary keys to the data node managing the region through a remote procedure call, making all points complete the commit in the same operation, thereby reducing transaction latency. After the primary key commit succeeds, the coordinator immediately returns a transaction success response to the client while starting an asynchronous task in the background to clean up residual locks and log records.

[0051] In a distributed database system, the optimization of the commit phase is crucial to transaction performance. The present embodiment provides a strategy of asynchronously processing secondary keys and synchronously committing primary keys to reduce transaction latency and improve system throughput.

[0052] When entering the commit phase, the coordinator first checks whether the data keys modified by the transaction belong to the same region. If all modified keys do not belong to the same region, the asynchronous commit strategy is implemented. The coordinator generates a commit timestamp, which is usually obtained based on the system's global clock service, such as a 64-bit integer generated by the HLC (Hybrid Logical Clock) mechanism, with the high 32 bits representing physical time and the low 32 bits representing a logical counter.

[0053] The coordinator first constructs a primary key commit request and sends a synchronous commit request to the data node where the primary key is located and waits for a response. The primary key commit request contains the transaction identifier and the commit timestamp. Assuming that the primary key K1 is located on node N1, the coordinator sends a {txid: T1, commit_ts: 100} request to N1.

[0054] After the primary key node receives the commit request, it performs the following operations: finds the local transaction record, checks whether the transaction state is in the prewrite state; updates the primary key data from the prewrite state to the committed state and modifies the version number to the commit timestamp; releases the write lock on the primary key; updates the transaction state record to committed; and returns a commit success response to the coordinator. This process is an atomic operation, ensuring that either all succeed or all fail.

[0055] If the primary key commit fails, for example, due to conflict detection failure or node failure, the coordinator needs to trigger the rollback operation. The coordinator constructs a rollback request containing the transaction identifier and sends a rollback instruction to all data nodes participating in the transaction. After the data node receives the rollback request, it marks the prewrite data as invalid, releases the related lock, and cleans up the transaction record. For example, if node N1 reports that the primary key K1 commit fails, the coordinator sends a {txid: T1, operation: rollback} request to N1, N2, N3, and N4.

[0056] If the primary key commit succeeds, the coordinator node asynchronously processes the secondary key commits in the background in parallel. For each secondary key modified by the transaction, the coordinator node constructs a commit request message containing the transaction identifier (e.g., a 128-bit unique identifier in UUID format) and the commit timestamp. The coordinator node sends these requests to the data nodes managing the secondary keys via internal RPC channels in a non-blocking manner without waiting for responses. For example, if transaction T1 modifies three secondary keys K2, K3, and K4 located on nodes N2, N3, and N4, respectively, the coordinator node sends asynchronous commit requests {txid: T1, commit_ts: 100} to these three nodes simultaneously.

[0057] Upon receiving a secondary key commit request, a data node looks up the corresponding transaction record in the local transaction table, updates the prewrite data version to the commit timestamp, and releases the write lock for the key. For example, upon receiving the commit request for K2, node N2 updates the prewrite version of K2 from {key: K2, value: "new_value", status: prepared} to {key: K2, value: "new_value", ts: 100, status: committed} and releases the exclusive lock on K2.

[0058] For the same-region optimization case, when the coordinator node detects that all keys (including primary and secondary keys) modified by a transaction are located in the same data partition or managed by the same data node, it can skip the asynchronous commit phase. The coordinator node constructs a single commit request containing information for all keys and the commit timestamp, and sends the request to the data node managing the region via a single RPC call. For example, if K1, K2, K3, and K4 are all located on node N1, the coordinator node sends a batch commit request {txid: T1, keys: [K1, K2, K3, K4], commit_ts: 100} directly to N1.

[0059] Upon receiving a batch commit request, a data node atomically updates the status of all keys from the prewrite state to the committed state in a single local transaction and releases all related locks. This same-region optimization significantly reduces cross-node communication overhead. In practical applications, for workloads with strong access locality, it can reduce commit latency by 50-70%.

[0060] After a transaction is completed, the coordinator node initiates a cleanup task in the background to delete the log records and metadata for completed transactions, freeing up system resources. The cleanup operation is usually performed with a certain delay (e.g., 5-10 seconds) to ensure that other components in the system (e.g., the replication subsystem) have sufficient time to process the committed changes.

[0061] In an alternative embodiment, a lock-free read that allows reading only after writing is completed and advancing in order is achieved by maintaining a read pointer and a write pointer of a write event queue, comprising: The read pointer and the write pointer are set on the write event queue, the read pointer points to the position of the first write operation not completed in the queue, and the write pointer points to the latest write position at the tail of the queue, and the data units in the write event queue include readable data units that have been written, unreadable data units that have been written but not completed, and data units to be written; When a write operation request is received, a new write operation is added to the tail of the write event queue, and the write pointer is moved to point to the newly added data unit, and the data unit is marked as having been written but not completed, indicating that the data unit has started to be written but has not been completed; When the write operation is completed, the state of the corresponding data unit is updated from having been written but not completed to having been written, indicating that the writing of the data unit has been completed; When a read operation request arrives, the method checks the writing state of the data unit pointed to by the read pointer and all data units in front of it, if the writing of a data unit has been completed but there are still data units in front of it with incomplete write operations, the data unit remains unreadable, the read pointer remains unchanged, and the read operation is blocked; When the data unit pointed to by the read pointer and all consecutive data units in front of it have completed the write operation, the read pointer is continuously moved until it encounters the position of the first data unit with an incomplete write operation, and the states of the consecutive data units that have completed the write operation are updated to readable, so that the read operation can access these data units in order.

[0062] The present application provides a lock-free read method that allows reading only after writing is completed and advancing in order by maintaining a read pointer and a write pointer of a write event queue. This method is suitable for concurrent processing scenarios and can ensure data consistency and reading order without using traditional lock mechanisms.

[0063] In implementation, the write event queue is designed as a circular buffer or a linked list structure, where each data unit contains at least two key information: data content and state flag. The state flag is used to identify the current state of the data unit, including "written", "written but not completed" and "to be written". The queue maintains two key pointers: read pointer and write pointer. The read pointer points to the position of the first write operation not completed in the queue, and the write pointer points to the latest write position at the tail of the queue.

[0064] In practice, the data structure of the write event queue can be defined as follows: each data unit contains a data field for storing the actual data content, a status field for marking the current status, and optional other metadata such as timestamp or operation ID. The status field can be updated using atomic operations to ensure consistency in a concurrent environment.

[0065] When the system receives a write operation request, it first checks whether the write event queue is full. If the queue is not full, a new data unit is allocated and added to the tail of the queue. The write process is as follows: the write pointer is moved one bit to point to the position of the new data unit; the data unit is marked as "written but not completed"; the data content of the write request is copied to the data field of the data unit; and the actual write operation is started. The write operation here can be performed asynchronously, such as writing data to disk or sending information to the network.

[0066] For example, assume that there are 5 data units in the current write event queue, and the write pointer points to position 5. At this time, a new write request is received, and the system moves the write pointer to position 6, creates a new data unit, marks the status as "written but not completed", and starts the write operation. At this time, the data unit at position 6 has been allocated but not completed.

[0067] When the write operation is completed, the system updates the status of the corresponding data unit. This process updates the status of the data unit from "written but not completed" to "written completed" through atomic operations. Atomic operations ensure the atomicity of status updates, avoiding inconsistent states. It should be noted that simply updating the status to "written completed" does not mean that the data unit is immediately readable, which also depends on whether there is an uncompleted write operation in front of it.

[0068] For example, assume that the write operation at position 6 is completed, and the system updates its status from "written but not completed" to "written completed". However, if the write operation at position 3 is not completed, the data unit at position 6 is still not readable, because the read operation must be performed in order.

[0069] When a read operation request arrives, the system first checks the status of the data unit pointed to by the read pointer. If the data unit is in the "written but not completed" state, the read operation is blocked and must wait for the completion of the write operation of the data unit. If the data unit pointed to by the read pointer is in the "written completed" state, the system will continue to check the consecutive data units until it encounters the first "written but not completed" data unit or reaches the position of the write pointer.

[0070] Specifically, the system starts from the read pointer and scans the states of data units sequentially along the queue. For each data unit with a state of "write completed", the system marks it as "readable" and moves the read pointer one bit. This process continues until the first data unit with an unfinished write operation is encountered, at which point the read pointer stops moving.

[0071] Assuming the current read pointer points to position 3, the data units at positions 3, 4, and 5 have states of "write completed", while the data unit at position 6 has a state of "write not completed". When a read operation request arrives, the system updates the states of the data units at positions 3, 4, and 5 to "readable" and moves the read pointer to position 6. The read operation can read the data at positions 3, 4, and 5 sequentially, but must wait for the write operation at position 6 to complete before continuing.

[0072] To further optimize performance, the method can also implement a batch state update mechanism. When multiple consecutive data units complete write operations simultaneously, the system can update the states of these data units to "readable" at once and move the read pointer accordingly, reducing the number of state updates.

[0073] In actual implementation, boundary conditions and exception handling also need to be considered. For example, when the write pointer catches up with the read pointer (the queue is full), new write requests should be blocked or rejected. Similarly, when the queue is empty (the read pointer equals the write pointer), read requests should be blocked or return empty results. In addition, the system should also be able to handle write operation failure cases, such as through rollback mechanisms or error markers.

[0074] Through this lock-free design, the method avoids the performance overhead and deadlock risk brought by traditional lock mechanisms, while ensuring the sequential consistency of data. It is particularly suitable for high-concurrency, low-latency application scenarios such as distributed databases, message queue systems, or real-time data processing platforms.

[0075] The second aspect of the embodiment of the application provides a distributed cluster security backup system for a cloud-edge environment, comprising: A first unit is configured to obtain data in the cloud-edge integrated cluster architecture, wherein the cloud-edge integrated cluster architecture comprises a cloud cluster, a plurality of edge clusters, and an edge data acquisition gateway connected to the edge clusters. A second unit is configured to adaptively select a data backup mode according to the cold and hot attributes of the data, tenant requirements, and storage resource states based on a hybrid redundancy backup strategy of erasure coding and multiple copies, wherein the data backup mode comprises multiple copy backup and erasure coding backup; wherein a data repair mechanism of the erasure coding backup is implemented on a cluster node using the erasure coding backup; and the atomicity of operations is ensured by a distributed transaction lock during a switching process of the data backup mode, and if any link fails, the system is rolled back to the state before switching. The third unit is configured to establish a workload-aware incremental replication mechanism, design a strategy of workload-aware pre-resolving and pre-merging an incremental log of a log block, and optimize fault recovery efficiency by using idle resources. The fourth unit is configured to design a two-phase commit-based multi-copy consistency guarantee mechanism for lock contention and commit delay problems, and the mechanism is applied to a cluster composed of one coordination node and multiple data nodes; the two-phase commit includes a pre-commit phase and a commit phase, and is based on primary key and secondary key separation management; the pre-commit phase belongs to a transaction start pre-write phase, mainly performs conflict detection and locking, and allocates a global commit timestamp; the commit phase adopts an asynchronous processing secondary key and synchronous committing primary key strategy; read operation is based on MVCC multi-version control, and a read pointer and a write pointer of a write event queue are maintained to realize a lock-free read that allows reading only after writing and sequentially advancing.

[0076] In a third aspect, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0077] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0078] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0079] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed cluster security backup method for cloud-edge environments, characterized in that, include: Data is acquired from the cloud-edge integrated cluster architecture, which includes a cloud cluster, multiple edge clusters, and an edge data acquisition gateway connecting the edge clusters. Based on a hybrid redundancy backup strategy combining erasure coding and multiple replicas, the data backup method is adaptively selected according to the hot / cold attributes of the data, tenant requirements, and storage resource status. The data backup method includes multiple replica backup and erasure coding backup. Among them, an erasure coding backup data repair mechanism is implemented for cluster nodes using erasure coding backup. During the switching process of the data backup method, the atomicity of the operation is ensured by distributed transaction locks. If any step fails, the system will roll back to the state before the switch. Establish a workload-aware incremental replication mechanism, design a strategy for load-aware pre-parsing and pre-merging log blocks for incremental logs, and optimize fault recovery efficiency by utilizing idle resources. To address lock contention and commit latency issues, a multi-replica consistency guarantee mechanism based on two-phase commit is designed. This mechanism is applied to a cluster consisting of one coordinating node and multiple data nodes. The two-phase commit includes a pre-commit phase and a commit phase, managed separately for primary and secondary keys. The pre-commit phase is the transaction initiation write phase, mainly performing conflict detection, locking, and global commit timestamp allocation. The commit phase adopts a strategy of asynchronously processing secondary keys and synchronously committing primary keys. Read operations are based on MVCC (Multi-Version Control), maintaining read and write pointers in the write event queue to achieve lock-free reading that allows reading only after write completion and proceeds sequentially.

2. The method according to claim 1, characterized in that, The data backup method is adaptively selected based on the hot / cold attributes of the data, tenant needs, and storage resource status, including: Based on the access characteristics of time-series data, data that has been updated relatively frequently recently is defined as hot data and is backed up using multiple replicas, while older data is defined as cold data and is backed up using erasure coding. The definition of hot and cold data can be specified through preset configuration items, that is, data from the most recent calendar month or two months can be designated as hot data, or data from the most recent week or two weeks can be designated as hot data. To meet the differentiated data needs of different tenants, data from specific tenants is backed up using erasure coding, while other data is backed up using multiple replicas by default. For cluster nodes backed up using erasure coding, data block checksums are periodically scanned; if corrupted blocks are found, missing checksum blocks or data blocks are pulled from other nodes and repaired in parallel using idle CPU resources to avoid affecting front-end services. When storage resources are below a preset resource threshold, a consistent snapshot of the target data is taken, erasure coding shards are generated based on the snapshot, and the shards are stored in the corresponding cluster nodes. After confirming that all erasure coding shards have been successfully stored, the original multi-replica data is deleted, and the backup method is switched from multi-replica to erasure coding. When storage resources are expanded, complete data is recovered using erasure coding algorithms. Multiple replicas are generated according to the multi-replica replication strategy, and these replicas are stored on different nodes. After confirming that all replicas have been successfully stored, the original erasure coding fragments are deleted, completing the switch from erasure coding to multi-replica replication for backup.

3. The method according to claim 1, characterized in that, Design a strategy for incremental logs with load-aware pre-parsing and pre-merging of log blocks, optimizing fault recovery efficiency through idle resource utilization, including: The data is divided into multiple logical segments according to natural time intervals of hour. For each logical segment, add, delete, and modify operations are recorded by writing logs to form the original incremental log data. The original incremental log data is asynchronously copied from the master node to the backup server. During the copying process, the original incremental log data is processed by a compression algorithm and a CRC32 checksum is added to generate compressed and checked incremental log data. A load monitoring module is deployed on the backup server. The load monitoring module collects and statistically analyzes the CPU utilization and I / O wait time of the backup server in real time to generate real-time resource usage status data. Based on the real-time resource usage status data, when the load monitoring module detects that the CPU utilization of the backup server is less than 20% and the I / O wait time is less than 10 milliseconds, it determines that the backup server is in a computing resource idle state and triggers a pre-parsing task. In the pre-parsing task, the compressed and verified incremental log data is parsed, and corresponding operation vectors are identified and generated. Based on the timestamp information of the operation vectors, log data in adjacent time windows are identified and merged in chronological order, and the operation vectors of adjacent time windows are combined to generate a pre-merged log block. The operation vectors in the pre-merged log block are serialized using Protobuf format to obtain serialized pre-merged log block data, and this data is stored in the cache of the backup server. When the master node in the distributed system fails, the serialized pre-merged log block data is read directly from the cache of the backup server.

4. The method according to claim 1, characterized in that, The pre-commit phase belongs to the transaction initiation write-ahead phase, and mainly involves conflict detection and locking, as well as global commit timestamp allocation, including: During the transaction initiation write-ahead phase, after receiving a transaction request, the coordinating node assigns a globally unique, incremental timestamp as the snapshot version and transaction identifier for the transaction, enabling all read operations to obtain data based on this snapshot version. Simultaneously, it selects one measurement point from all the measurement points modified by the transaction as the primary key and sets the remaining measurement points as secondary keys, making the commit status of the primary key the basis for determining the atomicity of the entire transaction. The coordinating node sends a pre-commit request to all data nodes involving the test points based on the transaction identifier and the test point information. After receiving the pre-commit request, the data node performs conflict detection on each test point to determine whether the test point has been modified by other transactions. If no conflict is detected, the data node locks the test point in memory and records the transaction identifier, where the lock information of the secondary key will point to the primary key. At the same time, it writes the relevant data changes to the write-ahead log and then returns a pre-commit success response to the coordinating node. After the coordinating node collects the pre-commit success responses from all data nodes, it assigns a globally incrementing commit timestamp greater than all existing transaction identifiers and the assigned commit timestamps to the transaction as the final commit version marker for the transaction.

5. The method according to claim 1, characterized in that, The commit phase employs a strategy of asynchronously processing secondary keys and synchronously committing primary keys, including: During the commit phase, the coordinating node first sends a commit request to the data node containing the primary key; after receiving the commit request, the data node containing the primary key updates the primary key data version to the commit timestamp and clears the write-ahead lock; if the primary key commit fails, the coordinating node triggers a rollback operation, sends a rollback request to all relevant data nodes, releases all locks and clears the write-ahead data; If the primary key is successfully committed, the coordinating node processes the secondary key's commit asynchronously in the background and sends a commit request to the data node that manages these secondary keys. The request carries a transaction identifier and a commit timestamp. After receiving the commit request, the data node finds the corresponding secondary key based on the transaction identifier, updates its data version to the commit timestamp, and releases the lock. When the coordinating node detects that all modified measurement points belong to the same region, it skips the aforementioned asynchronous commit phase and directly sends a commit request containing the primary key and all secondary keys to the data node managing the region through a single remote procedure call, so that all measurement points are committed in the same operation, thereby reducing transaction latency. After the primary key is successfully committed, the coordinating node immediately returns a transaction success response to the client, and simultaneously starts an asynchronous task in the background to clean up residual locks and log records.

6. The method according to claim 1, characterized in that, By maintaining read and write pointers in the write event queue, lock-free reading is implemented, allowing reading only after write completion and proceeding sequentially, including: The write event queue is configured with a read pointer and a write pointer. The read pointer points to the position of the first incomplete write operation in the queue, and the write pointer points to the latest write position at the tail of the queue. The data units in the write event queue include readable data units that have been written, unreadable data units that have been written but not completed, and data units to be written. When a write operation request is received, the new write operation is added to the tail of the write event queue, and the write pointer is moved forward to point to the newly added data unit. At the same time, the data unit is marked as written but not completed, indicating that the data unit has started writing but has not yet been completed. When the write operation is completed, the status of the corresponding data unit is updated from the written but not completed state to the written completed state, indicating that the writing of the data unit has been completed; When a read operation request arrives, the method checks the write status of the data unit pointed to by the read pointer and all data units in front of it. If the write of a data unit has been completed but there are still data units in front of it that have not completed the write operation, then the data unit remains unreadable, the read pointer remains unchanged, and the read operation is blocked. When the data unit pointed to by the read pointer and all the consecutive data units in front of it have completed the write operation, the read pointer moves forward continuously until it encounters the position of the first data unit that has not completed the write operation, and updates the state of the consecutive data units that have completed the write operation to the readable state, so that the read operation can access these data units in sequence.

7. A distributed cluster security backup system for cloud-edge environments, used to implement the method as described in any one of claims 1-6, characterized in that, include: The first unit is used to acquire data in the cloud-edge integrated cluster architecture, which includes a cloud cluster, multiple edge clusters, and an edge data acquisition gateway connecting the edge clusters. The second unit is used to adaptively select the data backup method based on the cold / hot attributes of the data, tenant needs, and storage resource status, according to the erasure coding and multi-replica hybrid redundancy backup strategy. The data backup method includes multi-replica backup and erasure coding backup. Among them, the erasure coding backup data repair mechanism is implemented for the cluster nodes using erasure coding backup. During the switching of the data backup method, the atomicity of the operation is ensured by distributed transaction lock. If any link fails, it will roll back to the state before the switch. The third unit is used to establish a workload-aware incremental replication mechanism, design a strategy for incremental logs with load-aware pre-parsing and pre-merging of log blocks, and optimize fault recovery efficiency by utilizing idle resources. The fourth unit addresses lock contention and commit latency issues by designing a two-phase commit-based multi-replica consistency guarantee mechanism. This mechanism is applied to a cluster consisting of a coordinating node and multiple data nodes. The two-phase commit includes a pre-commit phase and a commit phase, managed separately for primary and secondary keys. The pre-commit phase, belonging to the transaction initiation write phase, mainly performs conflict detection, locking, and global commit timestamp allocation. The commit phase employs a strategy of asynchronously processing secondary keys and synchronously committing primary keys. Read operations are based on MVCC (Multi-Version Control), maintaining read and write pointers in the write event queue to achieve lock-free reading that allows reading only after write completion and proceeds sequentially.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images