A method for off-site automatic failback and consistency recovery of a code repository

Abstract

The application discloses a kind of off-site automatic return of a kind of code repository and consistency recovery method, it is related to the technical field of software engineering system, the present application is aimed at solving the problem of slow recovery, the present application includes the following steps: S1: node recovery detection and role identification;S2: log abstract synchronization and difference detection: from current master node, obtain the submission abstract, read local log and compare, calculate difference interval ΔCSN, if there is intersection submission, then enter conflict processing procedure;The submission abstract obtained from current master node includes: submission serial number CSN, submission Hash, submission dependency ParentHash, submission vector clock VClock.The difference log dynamic recovery algorithm of the scheme, only play back missing part, do not do full amount synchronization.Vector clock causal relationship judgment mechanism is used, and multi-source submission concurrent conflict is automatically solved.

Description

Technical Field

[0001] This invention relates to the technical field of software engineering systems, specifically a method for automatic remote rollback and consistency recovery of code repositories. Background Technology

[0002] In modern software engineering, distributed code repositories (such as GitLab, Gitea, and Gerrit) are core infrastructure for continuous integration and software delivery. As enterprises increasingly demand high availability and cross-regional disaster recovery, more and more code repository systems are adopting cross-datacenter deployment, two-site three-center architectures, or primary / backup failover mechanisms to improve system reliability and disaster resilience.

[0003] Existing systems generally use a primary-backup architecture: when the data center where the primary node is located fails, the backup node takes over the service; when the primary node recovers, data synchronization, consistency verification, and role rollback need to be performed manually or semi-automatically. However, existing technologies typically employ full synchronization, involve a lot of manual intervention, and lack precise difference detection mechanisms, making them prone to data forks, commit conflicts, and inconsistencies during the recovery process.

[0004] The existing solutions have the following characteristics:

[0005] 1. Use full warehouse synchronization: After the master node recovers, the entire warehouse needs to be copied from the backup node back to the master node, which involves a huge amount of copying and takes a long time.

[0006] 2. Lack of differential log detection mechanism: It is usually impossible to accurately locate the missing commit range between the primary and backup, and can only perform a wide-range comparison.

[0007] 3. Conflict determination relies on manual intervention: When two nodes generate new commits, there is a lack of an automated causal relationship determination mechanism, which can easily lead to commit forks or erroneous overwriting.

[0008] 4. Lack of automated consistency verification system: Most only check the commit sequence or refs, without including object-level (Blob, Tree, Commit) consistency verification.

[0009] In summary, existing technologies generally suffer from the following drawbacks:

[0010] 1. Slow recovery speed: Full synchronization takes a long time, often several minutes to several hours.

[0011] 2. Differences cannot be accurately detected: There is a lack of incremental comparison mechanism at the level of submitted serial number (CSN).

[0012] 3. Difficulty in conflict decision-making: Lacking causal consistency tools such as vector clocks, it is impossible to automatically determine the order of submissions.

[0013] 4. Weak consistency guarantee: It only verifies the commit record, but does not verify the integrity at the object file level.

[0014] 5. The recovery process requires manual intervention: After the node is recovered, manual triggering of role switching, log checking and synchronization is required.

[0015] 6. Risk of data fork: During network splitting, the primary and backup nodes may generate new commits, leading to conflicts. Summary of the Invention

[0016] To address the aforementioned problems, specifically those raised in the background section, this invention proposes a method for automatic rollback and consistency recovery of code repositories deployed across different data centers, comprising the following steps: S1: Node recovery detection and role recognition; S2: Log summary synchronization and difference detection: Obtain the commit summary from the current master node, read and compare the local logs, calculate the difference interval ΔCSN, and if there are cross-commits, proceed to the conflict handling process; The process of obtaining the commit digest from the current master node includes: commit sequence number CSN, commit hash, commit dependency parenthash, and commit vector clock VClock; S3: Difference log replay and conflict resolution: Pull missing commits from the master node in CSN order and perform hash chain verification, vector clock causality verification and directory tree hash verification; S4: After the warehouse consistency check passes, the state is updated to synchronized; The repository consistency verification adopts a four-layer verification, including commit tree integrity, reference refs consistency, object file (Blob / Tree / Commit) hash verification, and repository snapshot check code comparison. S5: Automatic Registration and Regression: Nodes submit status and snapshot information to the cluster controller, the cluster controller updates the arbitration table, and the node officially becomes a standby node. If the master node experiences abnormal load, an active failback can be triggered.

[0017] A further provision of the present invention is that the node recovery detection and role recognition include: performing a system self-test after the original master node recovers and starts up; broadcasting a "node recovery" signal to the arbitration center; and obtaining the cluster member list and the current master node status.

[0018] A further feature of this invention is that if an active master node already exists, this node automatically degrades to recovering mode; if the cluster is unreachable, it enters a waiting arbitration mode.

[0019] A further provision of the present invention is that when fetching missing commits from the master node in CSN order, if a conflict is encountered, a vector clock is used to determine the causal order; if it cannot be determined, the commit from the master node shall prevail, and the local commit shall be written to the fork log ForkLog.

[0020] The beneficial technical effects of this invention are as follows: The dynamic recovery algorithm for difference logs in this solution only replays the missing parts and does not perform full synchronization. It automatically resolves multi-source commit concurrency conflicts through a vector clock causality judgment mechanism. Through automatic role degradation and automatic registration, the entire process is unattended after node recovery. It employs a full-link consistency verification mechanism, performing multi-layer verification from the commit tree to the object file. It utilizes a traceable recovery audit mechanism to fully record replay and conflict handling. With second-level rollback capability, typical recovery time is 5–10 seconds, far exceeding traditional solutions. Attached Figure Description

[0021] Figure 1 A schematic diagram of the overall structure of this solution is shown.

[0022] Figure 2 The sequence diagram of node recovery and role recognition is shown.

[0023] Figure 3 The consistency verification layer is shown.

[0024] Figure 4 The logic for resolving causal conflicts is shown. Detailed Implementation

[0025] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0026] This invention proposes a method for automatic off-site switchback and consistency recovery of code repositories, comprising the following steps: S1: Node recovery detection and role recognition: (1) After the original master node is restored and restarted, a system self-test is performed; (2) Broadcast a “node recovery” signal to the arbitration center; (3) Obtain the cluster member list and the current master node status; (4) If an active master node already exists, this node will automatically be downgraded to recovering; (5) If the cluster is unreachable, enter the waiting arbitration mode.

[0027] S2: Log summary synchronization and difference detection, obtaining the commit summary from the current master node: (1) Submit the serial number CSN (2) Submit hash (3) Submit dependency ParentHash (4) Vector clock VClock

[0028] Read and compare local logs, calculate the difference interval ΔCSN, and if there is cross-commit, proceed to the conflict handling process.

[0029] S3: Difference log replay and conflict resolution, pulling missing commits from the master node in CSN order and executing them: (1) Hash chain verification; (2) Vector clock causality check; (3) Directory tree hash verification; If a conflict occurs, the causal order is determined using a vector clock; if it cannot be determined, the commit of the master node takes precedence, and the local commit is written to the fork log.

[0030] S4: Warehouse consistency verification, including four layers of verification: (1) Submission tree integrity; (2) Reference consistency; (3) Hash verification of object files (Blob / Tree / Commit); (4) Comparison of warehouse snapshot verification codes; After all validations pass, the status is updated to synchronized.

[0031] S5: Automatic Registration and Regression (1) The node submits status and snapshot information to the cluster controller; (2) The controller updates the arbitration table; (3) The node officially becomes standby; (4) If the master node load is abnormal, an active switchback can be triggered.

[0032] The system modules of the above method include: ClusterMonitor: Cluster Detection and Role Recognition LogDiffer: Analysis of log differences between primary and standby systems DeltaReplayer: Difference Replay ConflictResolver: Vector Clock Conflict Resolution IntegrityVerifier: Multi-level consistency verification RejoinManager: Automatic Registration and Regression AuditLogger: Restore audit log records

[0033] Key data structures: (1) CommitLogEntry: A structure that records each commit operation, which contains key information about the commit. It is mainly used for: verifying the integrity of the committed data, comparing the differences between commits, and determining the causal relationship in the distributed system.

[0034] { CSN: uint64, Hash: string, ParentHash: string, Timestamp: int64, VectorClock: map[node_id]int64, Author: string, Message: string }

[0035] CSN: This is the sequence number in the commit log, representing the global order of a commit. In distributed systems, synchronization between master and slave nodes relies on this monotonically increasing number to ensure that commits are applied in the correct order.

[0036] Purpose: CSN can help locate missing commit intervals and determine the order of execution replays when recovering nodes.

[0037] It ensures that the order of commits will not conflict or become disordered after the node recovers.

[0038] Hash: This is a unique identifier for the submission, usually generated using a hash algorithm (such as SHA256), and is used to verify the integrity of the submitted content.

[0039] Purpose: Used to compare whether the contents of two submissions are the same.

[0040] As a measure to prevent tampering, any changes in the commit log will cause a change in the hash value, thus helping to detect data tampering or loss.

[0041] ParentHash: This refers to the hash value of the parent commit of this commit, used to build the chain relationship between commits. Through this field, the system can trace the history of each commit (i.e., which commit depends on which commit).

[0042] Purpose: To ensure the order of commits and dependencies.

[0043] During recovery, ParentHash is used to determine if there is a fork in the commit (two commits that have the same parent commit but different content).

[0044] ParentHash can be used to verify the integrity of the commit chain when replaying commits.

[0045] Timestamp: This is the timestamp of the commit, recording the creation time of the commit. It is usually in UNIX timestamp format (i.e., the number of seconds since 1970).

[0046] Purpose: Used to identify the time of submission.

[0047] When resolving conflicts, if the priority of two VectorClock commits cannot be clearly determined, the timestamp can be used as a reference for conflict resolution, and the earlier commit is usually chosen.

[0048] VectorClock: VectorClock is a vector clock used to record the commit timestamp of each node. In a distributed system, each node has its own logical time. VectorClock can record the timestamps of all nodes, ensuring the order of operations between nodes. `node_id` represents the unique identifier of a node. Each `node_id` corresponds to a timestamp indicating the commit order on that node.

[0049] Purpose: Used to determine the causal relationship between submissions.

[0050] If commit A and commit B occur simultaneously on different nodes, VectorClock can help determine whether commit A should precede commit B, or whether there is a conflict.

[0051] Author: Indicates the author of the submission, usually a user identifier (such as username, email, etc.). In cases of review, tracing, or branch conflicts, the Author field can be used to identify the specific person responsible for each submission.

[0052] Message: This is the description information for the submission, usually the descriptive text filled in by the developer when submitting.

[0053] (2) ClusterState: This data structure is used to record the status of each node in the cluster, ensuring that each node in the cluster is assigned a role (master node, standby node, etc.) according to specific rules. It provides key data support for the automatic recovery and master-slave switching of the cluster.

[0054] { Nodes: [NodeInfo], Primary: NodeID, Standbys: [NodeID], ArbitrationTerm: int64 }

[0055] Nodes: An array or list containing information about all nodes, representing all member nodes in the cluster.

[0056] NodeInfo: NodeID: A unique identifier for a node; Status: The current status of the node (e.g., online, offline, recovering, etc.); Role: The role of a node (master node, backup node, candidate node, etc.); LastHeartbeat: The last heartbeat time of a node, used to detect the health status of the node.

[0057] Primary: The current primary node.

[0058] Standbys: All standby nodes.

[0059] ArbitrationTerm: The arbitration version number or term number in the cluster. It is used to distinguish the state of the cluster at different time periods.

[0060] (3) RecoveryRecord: This data structure is used to store audit information of the recovery process, helping the system track the status and results of the recovery operation, as well as key events encountered during the recovery process (such as the number of difference logs, the number of conflicts, etc.). This is very important for subsequent auditing, performance analysis, and anomaly tracking.

[0061] { NodeID: string, StartTime: int64, DeltaCount: int, ConflictCount: int, Result: string }

[0062] NodeID: A unique identifier for the node being operated on during the recovery process. It is used to identify the specific node performing the recovery operation.

[0063] StartTime: The time when the recovery process began, usually recorded as a Unix timestamp (i.e., the number of seconds since January 1, 1970).

[0064] DeltaCount: The number of difference commits that need to be replayed during the recovery operation. Typically, it is the result obtained after difference detection (such as diffLogs), representing the difference region between the local logs and the master node logs.

[0065] ConflictCount: Records the number of conflicting commits encountered during the recovery process. A conflict occurs when two nodes produce different commits simultaneously during a primary / standby node recovery operation, and the system cannot directly determine which commit should take precedence. This helps operations personnel, developers, or auditors understand the frequency of conflicts and assess the effectiveness of existing conflict resolution methods.

[0066] Result: Indicates the outcome of the recovery operation. It typically includes one of the following possible states: success, failure, or partial. This is a key field for determining whether the recovery operation met expectations and whether further analysis or manual intervention is needed.

[0067] Pseudocode for the core algorithm: procedure AutoFailback(): detectCluster() if primaryExists(): role = "recovering" Δ = diffLogs(localLog, primaryLog) for commit in Δ ordered by CSN: if verifyHash(commit) and verifyClock(commit): applyCommit(commit) else: resolveConflict(commit) if verifyIntegrity(): registerAsStandby() writeAudit("rejoin success") else: writeAudit("integrity failed")

[0068] Overall code description:

[0069] (I) Role identification and control of off-site recovery nodes When the original master node recovers and starts up, it first performs a cluster status check to determine whether a valid master node exists in the current code repository cluster.

[0070] When a master node is detected in the cluster, the recovery node is automatically set to recovery status and prohibited from providing write services to the outside world, thereby avoiding data inconsistency issues caused by role conflicts.

[0071] (1) To prevent brain splitting; (2) Automatic character downgrade; (3) No human intervention is required.

[0072] (ii) Submission of discrepancy analysis mechanism In recovery mode, the recovery node performs difference analysis processing based on the commit log information stored locally and the commit log information of the master node.

[0073] The difference analysis process includes: Compare the serial numbers submitted by both parties; Determine the sequence number range corresponding to the missing submissions; Extract the commit logs within the specified interval as the set of commits to be recovered.

[0074] Using the above method, only the differences in commits can be recovered without performing a full copy of the repository.

[0075] (1) Fast recovery speed; (2) Reduce network and storage overhead; (3) It is significantly better than full synchronization.

[0076] (III) Submission Replay and Conflict Resolution Mechanism For the set of submissions to be recovered, the recovery nodes perform submission replay processing sequentially according to the submission sequence number.

[0077] Before replaying each commit, perform the following validation steps: Verify the integrity of the submission based on the submission digest value; Determine the causal relationship between submissions based on logical timestamp information.

[0078] When no causal conflict is detected between commits, a commit replay operation is performed. When concurrent or conflicting relationships are detected between commits, the commit version to be retained is determined according to the preset conflict handling rules, and the conflicting commit is recorded.

[0079] (1) Automated conflict handling; (2) It does not disrupt warehouse consistency; (3) Conflicts are auditable and traceable.

[0080] (iv) Multi-level consistency verification mechanism After completing the commit replay process, the recovery node performs a consistency check operation on the code repository, which includes at least the following: Submission relationship integrity verification; Reference pointer consistency check; Object file integrity verification; Warehouse overall status verification.

[0081] Only when all the consistency checks pass can the recovery node be allowed to rejoin the cluster and participate in subsequent synchronization.

[0082] Strong consistency; Prevent "half-recovery nodes" from coming online.

[0083] This solution compares with the traditional solution:

[0084] Although the invention has been described with reference to preferred embodiments, various modifications can be made thereto and components can be replaced with equivalents without departing from the scope of the invention. In particular, the technical features mentioned in the various embodiments can be combined in any manner, provided there is no structural conflict. The invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

[0085] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to those processes, articles, or apparatus / devices.

[0086] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for automatic off-site rollback and consistency recovery of a code repository, characterized in that: Includes the following steps: S1: Node recovery detection and role recognition; S2: Log summary synchronization and difference detection: Obtain the commit summary from the current master node, read and compare the local logs, calculate the difference interval ΔCSN, and if there are cross-commits, proceed to the conflict handling process; The process of obtaining the commit digest from the current master node includes: commit sequence number CSN, commit hash, commit dependency parenthash, and commit vector clock VClock; S3: Difference log replay and conflict resolution: Pull missing commits from the master node in CSN order and perform hash chain verification, vector clock causality verification and directory tree hash verification; S4: After the warehouse consistency check passes, the state is updated to synchronized; The repository consistency verification adopts a four-layer verification, including commit tree integrity, reference refs consistency, object file (Blob / Tree / Commit) hash verification, and repository snapshot check code comparison. S5: Automatic Registration and Regression: Nodes submit status and snapshot information to the cluster controller, the cluster controller updates the arbitration table, and the node officially becomes a standby node. If the master node experiences abnormal load, an active failback can be triggered.

2. The method for automatic off-site switchback and consistency recovery of a code repository according to claim 1, characterized in that: The node recovery detection and role recognition include: performing a system self-test after the original master node recovers and starts up; broadcasting a "node recovery" signal to the arbitration center; and obtaining the cluster member list and the current master node status.

3. The method for automatic off-site switchback and consistency recovery of a code repository according to claim 2, characterized in that: If an active master node already exists, this node will automatically be downgraded to recovering; if the cluster is unreachable, it will enter the waiting arbitration mode.

4. The method for automatic off-site switchback and consistency recovery of a code repository according to claim 1, characterized in that: When fetching missing commits from the master node in CSN order, if a conflict is encountered, a vector clock is used to determine the causal order. If it is impossible to determine, the commit made by the master node shall prevail, and the local commit shall be written to the fork log.

Images