Large file version control method based on Git and S3 protocols

Abstract

The invention particularly relates to a large file version control method based on Git and S3 protocols, the method depends on a git service and an S3 cloud storage service, the git and S3 protocols are deeply integrated, a large file version control tool named as RVC is constructed, and the method specifically comprises the following steps that S1, the RVC brings a large file into version control and submits and uploads the large file to the cloud storage service; s2, pulling a large file from the cloud storage service through the protocol in S3; s3, carrying out difference comparison on the files of different versions; and S4, rolling back the version. According to the method, the mode that Git LFS is used for fragmented storage of large files such as data sets and models in current machine learning is abandoned, the large files are stored in the mode that the large files are uploaded to cloud storage through the S3 protocol, addresses and related information of the large files are stored in a Git warehouse for version management, and the problem that the large files are stored in the Git LFS fragmented storage mode, so that the large files cannot be stored in the Git LFS fragmented storage mode is solved. And the user needs to merge the files and then use the files.

Description

Technical Field

[0001] The present invention relates to the technical field of file version control, and in particular to a large file version control method based on Git and S3 protocols. Background Art

[0002] The current method for large file version control on the market mainly uses GitLFS, represented by the foreign platform huggingface, and domestic platforms such as Mota and Mole. The maximum transfer of each file using GitLFS is 5GB. If it exceeds, GitLFS will reject the file and display an error message. Many models generated by artificial intelligence training exceed 5GB. The above-mentioned domestic and foreign platforms are stored in the form of file fragments. Users must merge them before using them. It is not friendly to users who want to directly reference the stored large models for training. The merging process will consume a lot of time and machine performance.

[0003] In addition, the SDKs of the OBS cloud storage services of various cloud vendors or users themselves are not unified. If compatibility with multiple vendors is required, the management and interface horizontal migration will have very high costs.

[0004] Therefore, a large file version control method based on Git and S3 protocol is needed to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to propose a large file version control method based on Git and S3 protocols in order to solve the above problems.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A large file version control method based on Git and S3 protocol, which relies on git service and S3 cloud storage service, specifically includes the following steps:

[0008] S1: RVC puts large files into version control and submits them for upload to the cloud storage service;

[0009] S2: Pull large files from cloud storage services through the S3 protocol;

[0010] S3: Compare the differences between files of different versions;

[0011] S4: Version rollback.

[0012] Preferably, the S1: RVC includes the large file in version control and submits it for uploading to the cloud storage service, which specifically includes the following parts:

[0013] S11: using the initialization command RVC init of the present invention to generate a metadata file, the metadata is generated in a yaml file and recorded in a standard yaml format;

[0014] S12: Execute the add command RVC add of the present invention to mark the large file in the .ignore file of git, so that git ignores the large file and does not operate on the large file; execute gitadd to add the metadata file to the cache.

[0015] S13: Execute the commit command RVC commit, integrate the commit command of git, and record the metadata file in the .git local cache file;

[0016] S14: Execute the push command RVC push that integrates git push and S3 file transfer protocol. It integrates git push command and S3 protocol transfer command. By reading the address and authentication information of the S3 cloud storage service stored in the metadata file, the S3 transfer command is used to push the large file to the cloud storage service through the S3 protocol, and the metadata file is pushed to the git repository.

[0017] Preferably, the S2: pulling large files from a cloud storage service through the S3 protocol specifically includes:

[0018] After using the RVC clone command to clone the metadata file in the git repository, execute the pull command RVCpull locally to read the address and authentication information of the S3 cloud storage service in the metadata file, and pull the large file from the cloud storage service through the S3 protocol.

[0019] Preferably, the step S3: comparing the differences between files of different versions specifically includes:

[0020] Compare the differences between files of different versions. Use the command RVCdiffv1v2, which integrates the git difference comparison function, to compare the differences between the metadata of files of different versions. By reading the yaml-formatted metadata in the .git file, you can get the difference data between different versions and display it in a list in the window.

[0021] Preferably, in the step S4: version rollback, when the user wants to roll back to a version submitted before the current version, the rollback can be performed through the following steps:

[0022] S41: Use the RVC history command to view the version submission record first, and select the number of the submission record that needs to be rolled back;

[0023] S42: Execute RVC reset --number to roll back;

[0024] S43: Execute RVC pull to update the large file to the large file of the current rollback version.

[0025] Preferably, the metadata file in S11 includes the address of the S3 cloud storage service, authentication information, parameters, indicators, and description of the model file or data set file.

[0026] Preferably, the S3 protocol: Amazon Simple Storage Service is a public cloud storage service.

[0027] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0028] 1. The present invention abandons the current method of using Git LFS shards to store large files such as data sets and models in machine learning, and instead uses the S3 protocol to upload large files to cloud storage. The addresses and related information of large files are stored in the Git repository for version management, which solves the problem of using Git LFS shards to store large files and users needing to merge files before using them.

[0029] 2. The present invention adopts the S3 protocol to transfer large files, thus solving the problem of incompatibility of OBS cloud storage service SDKs of various cloud vendors or users themselves. Users only need to care about the cloud storage address, which greatly reduces the learning and management costs of configuring the OBS cloud storage service. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 is a flow chart of the present invention;

[0032] Figure 2 This is a flow chart of RVC version control and push of the present invention;

[0033] Figure 3 The flowchart of pulling large files for RVC of the present invention;

[0034] Figure 4 It is a flow chart of RVC difference comparison of the present invention;

[0035] Figure 5 This is a version rollback flow chart of the present invention; DETAILED DESCRIPTION

[0036] Several embodiments of the present application will be described in more detail below with reference to the accompanying drawings so that those skilled in the art can implement the present application. The present application can be embodied in many different forms and purposes and should not be limited to the embodiments described herein. These embodiments are provided to make the present application comprehensive and complete, and to fully convey the scope of the present application to those skilled in the art. The embodiments do not limit the present application.

[0037] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and / or this specification, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined as such herein.

[0038] See also Figure 1-Figure 5 As shown, the present invention provides a technical solution:

[0039] A large file version control method based on Git and S3 protocol, which relies on git service and S3 cloud storage service, deeply integrates git and S3 protocol, and builds a large file version control tool named RVC, wherein S3 protocol: Amazon Simple Storage Service is a public cloud storage service, specifically including the following steps:

[0040] S1: RVC puts large files into version control and submits them to the cloud storage service, which includes the following parts:

[0041] S11: Use the initialization command RVC init of the present invention to generate a metadata file, the metadata is generated in a yaml file, and is recorded in the standard format of yaml; wherein the metadata file includes the address of the S3 cloud storage service, authentication information, parameters, indicators, and description of the model file or the data set file;

[0042] S12: Execute the add command RVC add of the present invention to mark the large file in the .ignore file of git, so that git ignores the large file and does not operate on the large file; execute gitadd to add the metadata file to the cache.

[0043] S13: Execute the commit command RVC commit, integrate the commit command of git, and record the metadata file in the .git local cache file;

[0044] S14: Execute the push command RVC push that integrates git push and S3 file transfer protocol, which integrates git push command and S3 protocol transfer command, reads the address and authentication information of S3 cloud storage service stored in metadata file, uses S3 transfer command to push large file to cloud storage service through S3 protocol, and pushes metadata file to git repository;

[0045] S2: Pull large files from cloud storage services through the S3 protocol, including:

[0046] After using the RVC clone command to clone the metadata file in the git repository, execute the pull command RVCpull locally to read the address and authentication information of the S3 cloud storage service in the metadata file, and pull the large file from the cloud storage service through the S3 protocol;

[0047] S3: Compare the differences between files in different versions, including:

[0048] Compare the differences between files of different versions. Use the command RVCdiffv1v2, which integrates the git comparison command, to compare the differences between the metadata of different versions of files. By reading the yaml-formatted metadata in the .git file, you can get the difference data between different versions and display it in a list in the window.

[0049] S4: version rollback;

[0050] If the user wants to roll back to the version submitted before the current version, the rollback can be performed by following the steps below:

[0051] S41: Use the RVC history command to view the version submission record first, and select the number of the submission record that needs to be rolled back;

[0052] S42: Execute RVC reset --number to roll back;

[0053] S43: Execute RVC pull to update the large file to the large file of the current rollback version.

[0054] The Git used can be replaced by SVN;

[0055] Git is used to manage the version of metadata. The storage address of large files is written in the metadata for version control. This solves the problem that git cannot store large files and git LFS cannot directly transfer files. Files need to be split into multiple files of less than 5GB for upload, and they need to be merged before use.

[0056] Using the S3 protocol to transfer files, users do not need to read the SDK of the relevant cloud storage service again. They only need to configure the storage address, authentication information, etc., which greatly reduces the management and interface level migration. For ordinary users and developers, there is no need to study the complex cloud storage service SDK documents in depth, avoiding the tedious process of understanding and mastering the specific interfaces and operating specifications of different cloud vendors. Only the basic storage address and authentication information configuration is needed, making the operation more intuitive and simple, greatly reducing the technical difficulty of using cloud storage services for large file transfers.

[0057] Integrate Git and S3 into one tool (RVC). Users only need to use RVC according to the Git method, which can meet users' needs for file integrity of large file version control and low learning and management costs for using cloud storage services at one time.

[0058] The above formulas are obtained by collecting a large amount of data and performing software simulation, and a formula close to the actual value is selected. The influencing weight factor and specific coefficient value in the formula are set by technical personnel in this field according to actual conditions, and can be adjusted and modified later.

[0059] The above description of the embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large file version control method based on Git and S3 protocol, characterized in that: This method relies on the git service and the S3 cloud storage service and includes the following steps: S1: RVC puts large files into version control and submits them for upload to the cloud storage service; S2: Pull large files from cloud storage services through the S3 protocol; S3: Compare the differences between files of different versions; S4: Version rollback.

2. According to the large file version control method based on Git and S3 protocol according to claim 1, it is characterized in that: S1: RVC puts large files into version control and submits them to the cloud storage service, which specifically includes the following parts: S11: using the initialization command RVCinit of the present invention to generate a metadata file, the metadata is generated in a yaml file and recorded in a standard yaml format; S12: Execute the add command RVC add of the present invention to mark the large file in the .ignore file of git, so that git ignores the large file and does not operate on the large file; execute gitadd to add the metadata file to the cache. S13: Execute the commit command RVC commit, integrate the commit command of git, and record the metadata file in the .git local cache file; S14: Execute the push command RVC push that integrates git push and S3 file transfer protocol. It integrates git push command and S3 protocol transfer command. By reading the address and authentication information of the S3 cloud storage service stored in the metadata file, the S3 transfer command is used to push the large file to the cloud storage service through the S3 protocol, and the metadata file is pushed to the git repository.

3. A large file version control method based on Git and S3 protocol according to claim 1, characterized in that: S2: Pulling large files from the cloud storage service through the S3 protocol, specifically including: After using the RVC clone command to clone the metadata file in the git repository, execute the pull command RVC pull locally to read the address and authentication information of the S3 cloud storage service in the metadata file, and pull the large file from the cloud storage service through the S3 protocol.

4. A large file version control method based on Git and S3 protocol according to claim 1, characterized in that: S3: performing a difference comparison on files between different versions, specifically includes: Compare the differences between files of different versions. Use the RVC diffv1 v2 command that integrates the git difference comparison to compare the differences between the metadata of different versions of files. By reading the yaml-formatted metadata in the .git file, you can get the difference data between different versions and display it in a list in the window.

5. The large file version control method based on Git and S3 protocol according to claim 1, characterized in that: In the step S4: version rollback, when the user wants to roll back to the version submitted before the current version, the rollback can be performed through the following steps: S41: Use the RVC history command to view the version submission record first, and select the number of the submission record that needs to be rolled back; S42: Execute RVC reset --number to roll back; S43: Execute RVC pull to update the large file to the large file of the current rollback version.

6. A large file version control method based on Git and S3 protocol according to claim 1, characterized in that: The metadata file in S11 includes the address of the S3 cloud storage service, authentication information, parameters, indicators, and description of the model file or data set file.

7. A large file version control method based on Git and S3 protocol according to claim 1, characterized in that: The S3 protocol: Amazon Simple Storage Service is a public cloud storage service.

Images