A code repository backup method based on an interstellar file system

Abstract

The application discloses a code warehouse backup method based on an interstellar file system, and comprises the following steps: a backup manager receives a backup request sent by a code warehouse, and then selects a plurality of storage nodes in an IPFS network; the backup manager sends a backup command to backup agents on the selected storage nodes; the backup agents receive the backup command, and perform a fixing operation according to the backup command, so as to fix the files to be backed up on the selected storage nodes. The method can guarantee the reliability of backup, and can avoid using a dedicated backup device.

Description

Technical Field

[0001] The present invention relates to a code warehouse backup method, and in particular to a code warehouse backup method based on an interplanetary file system. Background Art

[0002] Source code is a core asset for software companies, and its management and storage are crucial. Currently, companies typically manage and store code in centralized software code repositories, with regular backups. Source code backups are typically performed on dedicated storage devices such as NAS and SAN. These devices typically feature high-availability features like disk redundancy to ensure code recovery in the event of a code repository failure or deliberate damage. However, dedicated storage devices are expensive. Using a distributed file system based on IPFS can significantly reduce reliance on dedicated storage devices.

[0003] However, using IPFS directly as a code repository storage has the following difficulties:

[0004] First, IPFS is only responsible for storage and does not provide a file system interface. When a file is added to IPFS, a unique hash value is generated. This value serves as the file's unique identifier on the IPFS network, and is also the file's "file name" on the IPFS network. This makes file management on IPFS somewhat difficult.

[0005] Secondly, IPFS nodes cannot guarantee that stored data will not be deleted. To ensure that data is not deleted, a pinning operation (PIN) must be performed on the node, which is less reliable. As a code storage backup, the storage node itself is unaware of the need to pin the data, requiring a remote pinning operation. However, IPFS does not officially provide such a remote PIN method. Adding dedicated backup equipment is expensive, and using redundant configurations further increases the cost. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a code repository backup method based on the InterPlanetary File System, which can ensure the reliability of the backup and avoid the use of dedicated backup equipment.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The present invention provides a code repository backup method based on the InterPlanetary File System, comprising:

[0009] The backup manager receives the backup request from the code repository and selects several storage nodes in the IPFS network;

[0010] The backup manager sends a backup command to the backup agent on each selected storage node;

[0011] The backup agent receives the backup command, performs a fixation operation according to the backup command, and fixes the files to be backed up on the selected storage nodes.

[0012] The code repository backup method based on the InterPlanetary File System described in the present invention is further improved in that:

[0013] The specific operations of selecting several storage nodes in the IPFS network are as follows:

[0014] Through the consistent hashing algorithm, several storage nodes are selected in the IPFS network according to the preset redundancy number.

[0015] The backup agent receives the backup command and performs a fixation operation according to the backup command to fix the files to be backed up on the selected storage nodes. The specific operation is:

[0016] The backup agent receives the backup command, calls the IPFS client program to perform a fix operation according to the backup command, and fixes the files to be backed up on the selected storage nodes.

[0017] It also includes: the backup manager establishes a correspondence table between directory tree nodes and IPFS hash values.

[0018] Before the backup manager receives the backup request sent by the code repository, the process further includes:

[0019] The code repository receives the submitted files and then calls the hook program to send a backup request to the backup manager.

[0020] It also includes: writing the file's absolute path name, file hash value, and IPFS hash value into the database.

[0021] Also includes: full backup of the code repository.

[0022] The specific operations for performing a full backup of the code repository are as follows:

[0023] The backup cycle is preset. When the backup time is reached, the backup manager packages the entire code repository and stores the packaged files in several storage nodes in the IPSF network. At the same time, the logs of the storage process, the IPSF hash value, and the name of the packaged file are written to the database.

[0024] Also includes: recovery of code repository.

[0025] The specific operations for restoring the code repository are as follows:

[0026] Get the ID submitted by the user, read the IPFS file ID of each file corresponding to the directory tree from the database based on the submitted ID, and then perform a Get operation to restore the file in the code repository.

[0027] The present invention has the following beneficial effects:

[0028] The code repository backup method based on the InterPlanetary File System described in the present invention uses the IPFS network to back up and store the code repository during specific operation, eliminating the need for special storage devices and reducing the cost of backup. At the same time, during the storage process, the present invention selects several storage nodes in the IPFS network and then sends a backup command to the backup agent on each storage node. The backup agent can then fix the files to be backed up on the selected storage nodes. It should be noted that the present invention uses multiple copies, which greatly enhances the reliability of the backup.

[0029] Furthermore, through the consistent hashing algorithm, several storage nodes are selected in the IPFS network according to the preset redundancy number to ensure that no large data migration occurs when the storage nodes are connected and exited. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0031] Figure 1 It is a schematic diagram of the present invention;

[0032] Figure 2 Flowchart for incremental backup. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] The Inter Planetary File System (IPFS) is a global, peer-to-peer distributed file system designed to complement (or even replace) the Hypertext Transfer Protocol (HTTP) that currently dominates the internet, connecting all computing devices with the same file system. It works by replacing domain-based addresses with content-based addresses, requiring only the content hash to be verified. This makes data storage and retrieval faster, more secure, more robust, and more durable.

[0036] The present invention is described in further detail below with reference to the accompanying drawings:

[0037] Example 1

[0038] refer to Figure 1 The IPFS-based code repository backup method of the present invention includes the following steps:

[0039] 1) The backup manager receives the backup request from the code repository and then uses the consistent hashing algorithm to select several storage nodes in the IPFS network based on the preset redundancy number;

[0040] Specifically, when a new submitted file exists in the code repository, a backup request is sent to the backup manager. Furthermore, it should be noted that the present invention uses a consistent hashing algorithm to select several storage nodes in the IPFS network based on a preset redundancy number to ensure that no large-scale data movement occurs when storage nodes join or exit the IPFS network.

[0041] 2) The backup manager sends a backup command to the backup agent on the selected storage node. The backup agent receives the backup command and performs a fix operation according to the backup command to fix the files to be backed up on the selected storage nodes.

[0042] It should be noted that the backup manager establishes a correspondence table between directory tree nodes and IPFS hash values. In addition, the backup agent calls the IPFS client program according to the backup command to perform a fixing operation, fixing the files to be backed up in the selected storage node.

[0043] Example 2

[0044] refer to Figure 2 This embodiment is used to perform incremental backup. The specific operation process is as follows:

[0045] 11) Use the post-receive hook of the git repository. When the submitted file is received, the git server calls the hook program and sends a backup request to the backup manager in the hook program.

[0046] 12) For each modified or newly added file, the backup manager calls the IPFS client to add the file to the IPFS network, and writes the file's absolute path name, file hash value, and IPFS hash value into the database.

[0047] Among them, the backup manager uses the consistent hashing algorithm to calculate the IP address of the IPFS node where the file will be pinned; then sends a pinning message to the backup agent at that IP address; and then writes it into the database using the submission ID as the key and the directory tree with the node file IPFS ID as the value.

[0048] Example 3

[0049] This embodiment is used to perform a full backup, and the specific process is as follows:

[0050] The backup manager sets the backup cycle. When the backup cycle is reached, the entire code base is packaged and added to the IPFS network. At the same time, the log, hash value and file name at the time of addition are written into the database.

[0051] Example 4

[0052] This embodiment is used to restore the code base. The specific process is as follows:

[0053] Get the ID submitted by the user, read the IPFS file ID of each file in the directory tree from the database according to the submitted ID, and then perform the Get operation to restore the data in the code repository

[0054] For example, the directory of the code repository is:

[0055]

[0056] It can be described in the following JSON format:

[0057] Key:dcf3279e9fbe73a60ebf17c1c721bfa15a01ccd3

[0058] Value:{".git / branches":QmagpDBG5u3rwcdx7csTRLQnvBVfXw1nixjTjdgZ7KyD4T,

[0059] ".git / config":QmNsHbmagmstawLn2poKr5ZQc35FB5Uvc5R9byQ3b3U2mk ...

[0061] ".git / hooks / applypatch-msg.sample":Qmb9xD84SuN6ZHQt4rQJFAfhrVZJsWvy2

[0062] }

[0063] It is guaranteed that all files can be restored based on the commit ID.

[0064] It should be noted that the present invention uses IPFS to back up source code, and avoids large-scale migration of data through consistent hashing, while establishing a correspondence between directory tree nodes and IPFS hash values, and recovering data through this correspondence.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A code repository backup method based on the InterPlanetary File System, characterized in that: include: The backup manager receives the backup request from the code repository and selects several storage nodes in the IPFS network; The backup manager sends a backup command to the backup agent on each selected storage node; The backup agent receives the backup command, performs a fixation operation according to the backup command, and fixes the files to be backed up on the selected storage nodes; It also includes: the backup manager establishes a corresponding relationship table between directory tree nodes and IPFS hash values; Also includes: full backup of the code repository; The specific operations for performing a full backup of the code repository are as follows: The backup cycle is preset. When the backup time is reached, the backup manager packages the entire code repository and stores the packaged files in several storage nodes in the IPSF network. At the same time, the logs of the storage process, the IPSF hash value, and the name of the packaged file are written to the database.

2. The code repository backup method based on the InterPlanetary File System according to claim 1, characterized in that: The specific operations of selecting several storage nodes in the IPFS network are as follows: Through the consistent hashing algorithm, several storage nodes are selected in the IPFS network according to the preset redundancy number.

3. The code repository backup method based on the InterPlanetary File System according to claim 1, characterized in that: The backup agent receives the backup command and performs a fixation operation according to the backup command to fix the files to be backed up on the selected storage nodes. The specific operation is: The backup agent receives the backup command, calls the IPFS client program to perform a fix operation according to the backup command, and fixes the files to be backed up on the selected storage nodes.

4. The code repository backup method based on the InterPlanetary File System according to claim 1, characterized in that: Before the backup manager receives the backup request sent by the code repository, the process further includes: The code repository receives the submitted files and then calls the hook program to send a backup request to the backup manager.

5. The code repository backup method based on the InterPlanetary File System according to claim 4, characterized in that: Also includes: Write the file's absolute path name, file hash value, and IPFS hash value into the database.

6. The code repository backup method based on the InterPlanetary File System according to claim 1, characterized in that: Also includes: Restore the code repository.

7. The code repository backup method based on the InterPlanetary File System according to claim 6, characterized in that: The specific operations for restoring the code repository are as follows: Get the ID submitted by the user, read the IPFS file ID of each file corresponding to the directory tree from the database based on the submitted ID, and then perform a Get operation to restore the file in the code repository.

Images