A code warehouse site transfer method, system, device and storage medium

Abstract

The embodiment of the present invention discloses a code warehouse site transfer method, system, device and storage medium, wherein a code-backing platform is deployed on each regional site and a site transfer request is received; the code-backing platform authenticates the user, and if the authentication is passed, a code warehouse site transfer task is created; a transfer asynchronous task is formed and the state of the code warehouse / code warehouse group to be transferred is modified to being in operation and verified; if it exists, the code warehouse / code warehouse group to be transferred is imported into the target site cluster and the warehouse data is written to a file; the write operation is performed again, and the data with the transfer status of unsuccessful is written to another file; the database relationship is updated for all transferred code warehouses / code warehouse groups, and the code warehouse site transfer task is completed. When facing large-scale code warehouse transfers between different sites, it still has the asynchrony that multiple warehouses can be transferred together and has good accuracy.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data migration, and in particular to a code repository site transfer method, system, device and storage medium. Background Art

[0002] With the rapid development of software development technology and the continuous expansion of the development industry, code repositories have become an integral part of programmers' daily code hosting, standardized development processes, and collaborative development between programmers, greatly improving development efficiency. However, the demand for development is also growing, the scale of code repositories is constantly expanding, and the distribution of server sites is no longer limited to a single computer room, but is widely distributed in multiple sites in different regions. Code repositories are also facing the problem of transfer. The existing rsync technology method can transfer code repositories between different sites, which can support remote replication and synchronization with remote rsync daemons. Rsync technology compares the contents of the source files of the code repository site and the files of the target code repository site, and only copies the difference, thereby achieving incremental backup. Although rsync technology is transparent in operation, it is obvious that doing so has the problems of cumbersome operation procedures, technical barriers for users to use, and inability to operate multiple code repositories of multiple sites in batches at one time. In addition, it is difficult to ensure the timeliness and accuracy of the transferred code repositories when facing the need to transfer large-scale code repositories between different sites. When there is a failed transfer object during the transfer process, the user has no way of knowing the specific object of the failed transfer and the reason for the transfer failure. After the transfer operation fails, there is no rollback mechanism to remove dirty data after the failed operation. Summary of the invention

[0003] To this end, an embodiment of the present invention provides a code repository site transfer method, system, device and storage medium to solve the technical problems of the prior art of code repository transfer through rsync, such as cumbersome operation process, inability to perform batch operations, and inability to know the specific object and the cause of failure when there is an object that fails to be transferred.

[0004] In order to achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] According to a first aspect of an embodiment of the present invention, a code repository site transfer method is provided, the method comprising:

[0006] Deploy the Tuoma platform at each regional site and receive users' site transfer requests;

[0007] The Tuoma platform authenticates the user under the current tenant, and if the authentication is passed, a code repository site transfer task is created;

[0008] A transfer asynchronous task is formed and the state of the code repository / code repository group to be transferred is changed to being operated, and it is verified whether the code repository / code repository group to be transferred and the corresponding parent group member exist in the target site cluster;

[0009] If it exists, the code repository / code repository group to be transferred is imported into the target site cluster and the repository data is written into the successfully transferred data folder; if it does not exist, a member snapshot is generated;

[0010] Repeat the write operation once to write the data with unsuccessful transfer status into the failed transfer data folder;

[0011] Update the database relationships of all transferred code repositories / code repositories groups to complete the code repository site transfer task.

[0012] Furthermore, the Tuoma platform authenticates the user under the current tenant, including:

[0013] If the authentication is successful, a code repository site transfer task is created;

[0014] If the authentication fails, the page returns with a 403 error message, indicating that the user does not have sufficient permissions.

[0015] Furthermore, create a code repository site transfer task, including:

[0016] Select the code repository or code repository group to be transferred as the source object;

[0017] Get the code repository / code repository group specified in other sites or other sites as the target object;

[0018] If no code repository or code repository group is specified as the target object, the code is transferred to the root directory of the target site.

[0019] Furthermore, the method further comprises:

[0020] When executing the transfer asynchronous task, the specific timing data field is updated every preset period;

[0021] When the current task ends, stop updating the scheduled data field and start the scheduled inspection task;

[0022] The scheduled inspection tasks include:

[0023] Check the timing data field in the asynchronous task, if it is detected that there is a timing data field exceeding a preset threshold, change the state of the asynchronous task to interrupt and delete the corresponding file;

[0024] The code repository site transfer information is sent to the target site cluster, the repository currently performing the transfer operation is deleted, and the corresponding repository data is deleted.

[0025] Furthermore, the preset period is 1 minute, and the preset threshold is 5 minutes.

[0026] Furthermore, the method further comprises:

[0027] When an object fails to be transferred, the object and the reason for the failure are obtained;

[0028] After the code repository site transfer task is completed, the objects that failed to be transferred and the reasons for the failure are written into the operation log.

[0029] Furthermore, the method further comprises:

[0030] Receive failed rollback requests from users and roll back the warehouse where the transfer failed at the target site;

[0031] Delete the storage data ID and table data recorded in this transfer task.

[0032] According to a second aspect of an embodiment of the present invention, a code repository site transfer system is provided, the system comprising:

[0033] Deployment module, used to deploy the Tuoma platform on each regional site and receive users' site transfer requests;

[0034] An authentication module is used for the Tuoma platform to authenticate the user under the current tenant, and if the authentication is passed, a code repository site transfer task is created;

[0035] A state modification module is used to form a transfer asynchronous task and modify the state of the code repository / code repository group to be transferred to operating, and verify whether the code repository / code repository group to be transferred and the corresponding parent group member exist in the target site cluster;

[0036] An import module is used to import the code repository / code repository group to be transferred into the target site cluster if it exists and write the repository data into the successfully transferred data folder; if it does not exist, a member snapshot is generated; the write operation is repeated once, and the data with unsuccessful transfer status is written into the failed transfer data folder;

[0037] The database relationship update module is used to update the database relationships of all transferred code repositories / code repositories groups to complete the code repository site transfer task.

[0038] According to a third aspect of an embodiment of the present invention, a code repository site transfer device is provided, the device comprising: a processor and a memory;

[0039] The memory is used to store one or more program instructions;

[0040] The processor is used to run one or more program instructions to execute the steps of a code repository site transfer method as described in any one of the above items.

[0041] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a code repository site transfer method as described in any one of the above items are implemented.

[0042] The embodiments of the present invention have the following advantages:

[0043] The embodiment of the present invention enables users to quickly complete the operation of transferring code repositories to other sites only on the code hosting platform when they need to. In one operation, batches of repositories from different sites can be transferred to other sites, so that the repositories can be immediately retrieved and used in other sites after the transfer. When facing large-scale code repositories between different sites, it still has the asynchrony of multiple repositories being able to be transferred together and has good accuracy. At the same time, if an object fails to be transferred during the transfer process, the operating user can quickly perceive the specific object that failed to be transferred and the reason for the transfer failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0045] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0046] Figure 1 A schematic diagram of the logical structure of a code repository site transfer system provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a flow chart of a code repository site transfer method provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of regional site deployment of a code repository site transfer method provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of a code repository site transfer operation flow in a code repository site transfer method provided in an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of a code repository transfer process in a code repository site transfer method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Although rsync technology is transparent in operation, it obviously has the problems of cumbersome operation process, technical barriers for users, and inability to operate multiple code repositories of multiple sites in batches at one time. In addition, it is difficult to ensure the timeliness and accuracy of the transferred code repositories when faced with the need to transfer large-scale code repositories between different sites. When there is a failed transfer object during the transfer process, the user has no way of knowing the specific object that failed to be transferred and the reason for the transfer failure.

[0053] This is to solve the technical problems that the code repository transfer method using rsync has a cumbersome operation process, cannot perform batch operations, and cannot know the specific object and the reason for the failure when there is an object that fails to be transferred.

[0054] refer to Figure 1 The embodiment of the present invention discloses a code repository site transfer system, which includes: a deployment module 1; an authentication module 2; a state modification module 3; an import module 4; and a database relationship update module 5.

[0055] Corresponding to the above disclosed code warehouse site transfer system, the embodiment of the present invention further discloses a code warehouse site transfer method. The following describes in detail a code warehouse site transfer method disclosed in the embodiment of the present invention in combination with the above described code warehouse site transfer system.

[0056] refer to Figure 2The present invention discloses a code repository site transfer method, which includes: deploying a code repository platform on each regional site and receiving a user's site transfer request; the code repository platform authenticates the user under the current tenant, and creates a code repository site transfer task if the authentication is passed.

[0057] After the asynchronous task is formed, the site cluster will start to execute the transfer task according to the code repository transfer technology. After completing the transfer task, the database relationship will be rewritten to ensure that the transferred warehouse data is synchronized to the current site in an available state, so that the code repository can be used normally by users under the corresponding tenant of the transferred site.

[0058] After rewriting the database relationship, the code repository site transfer task created by the user operation is completed, and the source target repository is transferred to the target site.

[0059] Create a transfer asynchronous task and change the status of the code repository / code repository group to be transferred to "in operation", such as Figure 5 As shown, check whether the code repository / code repository group to be transferred and the corresponding parent group members exist in the target site cluster; if so, import the code repository / code repository group to be transferred into the target site cluster and write the repository data into the successful transfer data folder; if not, generate a member snapshot; repeat the write operation once, write the data with an unsuccessful transfer status into the failed transfer data folder; update the database relationship of all transferred code repositories / code repository groups to complete the code repository site transfer task.

[0060] During the transfer process, the status of all transferred warehouses will be set to "in operation". Then check whether the warehouse and all its parent group members exist in the target site cluster. If not, a member snapshot will be generated. The warehouse data transferred from the source site cluster is written to a file. This process will be repeated once, and the data with a transfer status of "unsuccessful" will be written to another file. Finally, all the warehouses transferred from the source site cluster are imported into the target site cluster.

[0061] Furthermore, the code-enabling platform authenticates the user under the current tenant, including: if the authentication is successful, creating a code repository site transfer task; if the authentication is not successful, the page returns and reports a 403 error, indicating that the user does not have sufficient authority.

[0062] refer to Figure 3 and Figure 4A code repository site transfer technology-based platform is deployed on the regional site. Users can perform operations to transfer the repository that needs to be transferred. During this process, the platform will authenticate the user under the current tenant. Only users who have passed the authentication under the current tenant can create code repository site transfer tasks. Users who have not passed the authentication will return a 403 error message on the page to remind users that they do not have sufficient permissions.

[0063] Furthermore, creating a code repository site transfer task includes: taking the code repository / code repository group to be transferred as a source object; and obtaining a code repository / code repository group specified in another site or another site as a target object.

[0064] If no code repository or code repository group is specified as the target object, the code is transferred to the root directory of the target site.

[0065] When a user with code repository site transfer permissions starts to create a transfer task, he or she can search for the code repository / code repository group that he or she wants to transfer by path or name as the source object, and then select another site or a specified code repository group in another site as the target object (if no code repository group is specified, it will be transferred to the root directory of the target site) to complete the creation of the transfer task.

[0066] The source object's code repository / code repository group can come from different sites. The code repository / code repository group selected in batches in the task can be transferred to the target site through this task.

[0067] After the user has selected the source object and target object of the task to be transferred, he can click Create to transfer the code repository site. The transfer will form an asynchronous task to ensure that tasks do not block each other in the case of multiple tasks, thus ensuring the efficiency of the transfer.

[0068] Furthermore, the method further comprises: when executing the transfer asynchronous task, updating the specific timing data field at every preset period; when the current task ends, stopping updating the timing data field and starting the timing check task.

[0069] The scheduled inspection task includes: checking the scheduled data field in the asynchronous task. If it is detected that there is a scheduled data field exceeding the preset threshold, the state of the asynchronous task is changed to interrupt and the corresponding file is deleted; the code warehouse site transfer information is sent to the target site cluster and the warehouse currently performing the transfer operation is deleted, and the corresponding warehouse data is deleted.

[0070] Furthermore, the preset period is 1 minute, and the preset threshold is 5 minutes.

[0071] When the asynchronous transfer task is executed, the specific timing data field will be updated every minute. When the current task ends, the update will be stopped, the timing task will be started, and the timing data field of the ongoing task will be checked. When the timing data field exceeds 5 minutes, the task status will be updated to interrupted, the corresponding file will be deleted, and the cluster of the target site will be notified that the transfer operation of the warehouse currently performing the transfer operation has failed. The warehouse currently performing the transfer operation will be deleted, and the corresponding warehouse data will be deleted.

[0072] Furthermore, the method further includes: when an object fails to be transferred, obtaining the object that fails to be transferred and the reason for the failure; after waiting for the code repository site transfer task to be completed, writing the object that fails to be transferred and the reason for the failure into the operation log.

[0073] When there are objects that fail to be transferred during the transfer process, this technology will not interrupt the transfer, but will notify the user of the objects that failed to be transferred and a brief reason for the failure. After the transfer is completed, the objects that failed to be transferred and the specific reasons will also be reflected in the log.

[0074] Furthermore, the method also includes: receiving a failed rollback request from a user, and performing a rollback operation on the warehouse that failed to be transferred in the target site; and deleting the storage data ID and table data recorded in this transfer task.

[0075] When the code repository fails to be transferred to another site, the user can perform a rollback operation. By rolling back the failed repository at the target site, all the storage data ID and table data recorded in the transfer task will be deleted to prevent dirty data from hindering the re-site transfer of the failed repository after the problem is solved.

[0076] The purpose of the present invention is to provide a more efficient and simple-to-operate code warehouse site transfer technology that can transfer multiple nodes and multiple warehouses at the same time. It allows users to quickly complete the operation when they need to transfer code warehouses to other sites on the code hosting platform. One operation can transfer warehouses of batches of different sites to other sites, so that the warehouses can be immediately retrieved and used in other sites after the transfer. When facing large-scale code warehouse transfers between different sites, it still has the asynchrony of multiple warehouses being able to transfer together and has good accuracy. At the same time, if there is an object transfer failure during the transfer process, it can meet the needs of the operating user to quickly perceive the specific object that failed to transfer and the reason for the transfer failure.

[0077] Compared with the existing rsync technology, the code repository site transfer technology of the present invention uses asynchronous queue technology to transfer warehouses to other sites in parallel in batches, which greatly improves the efficiency and operational convenience of code repository site transfer. Users can quickly get started with zero technical knowledge, and good performance can be maintained even in large-scale transfer operations.

[0078] At the same time, during the transfer operation, the user can easily check which specific warehouses failed to transfer successfully. Under the failed rollback operation supported by this technology, the user can also easily delete the failed operation data, ensuring that the next transfer operation after the user solves the problem will not fail due to dirty data problems, thereby improving the safety of user operations. The present invention has a significant improvement in efficiency, convenience, and security compared to the rsync technology for code warehouse site transfer.

[0079] The embodiments of the present invention have the following advantages:

[0080] 1) Using asynchronous task processing technology, it can process the transfer tasks of multiple warehouses in parallel in the background, improve the transfer efficiency, and maintain good performance even in large-scale transfer operations;

[0081] 2) Supports simultaneous transfer of multiple nodes and multiple warehouses. Users can select multiple warehouses for batch transfer in one operation;

[0082] 3) The site cluster provides status monitoring through this technology, and users can check the transfer progress and status at any time. When the transfer fails, the user can be notified immediately and the reason for the failure can be provided. At the same time, the error can be automatically detected, and retry or skip can be intelligently selected according to the error type to ensure the continuity of the transfer. Through data verification and integrity checks, the transferred code repository is ensured to be completely consistent with the original repository data, ensuring the accuracy of the transfer;

[0083] 4) It is not limited to a single code hosting platform, but can support a variety of different code hosting services, providing wide applicability;

[0084] 5) It also has a rollback mechanism. When a failure occurs, the user only needs to roll back the failed object in the target site, and continue to transfer the failed warehouse after the problem is solved, without any dirty data impact.

[0085] In addition, an embodiment of the present invention also provides a code repository site transfer device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a code repository site transfer method as described in any of the above items.

[0086] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a code repository site transfer method as described in any one of the above items are implemented.

[0087] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0088] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0089] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0090] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0091] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0092] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0093] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0094] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A code repository site transfer method, characterized in that: The method comprises: Deploy the Tuoma platform at each regional site and receive users' site transfer requests; The Tuoma platform authenticates the user under the current tenant, and if the authentication is passed, a code repository site transfer task is created; A transfer asynchronous task is formed and the status of the code repository / code repository group to be transferred is changed to being operated, and it is verified whether the code repository / code repository group to be transferred and the corresponding parent group member exist in the target site cluster; If it exists, the code repository / code repository group to be transferred is imported into the target site cluster and the repository data is written into the successfully transferred data folder; if it does not exist, a member snapshot is generated; Repeat the write operation once to write the data with unsuccessful transfer status into the failed transfer data folder; Update the database relationships of all transferred code repositories / code repositories groups to complete the code repository site transfer task; When executing the transfer asynchronous task, the specific timing data field is updated every preset period; When the current task ends, stop updating the scheduled data field and start the scheduled inspection task; The scheduled inspection tasks include: Check the timing data field in the asynchronous task, if it is detected that there is a timing data field exceeding a preset threshold, change the state of the asynchronous task to interrupt and delete the corresponding file; Send code repository site transfer information to the target site cluster and delete the repository currently performing the transfer operation, and delete the corresponding repository data; The preset period is 1 minute, and the preset threshold is 5 minutes; When an object fails to be transferred, the object and the reason for the failure are obtained; After the code repository site transfer task is completed, the objects that failed to be transferred and the reasons for the failure are written into the operation log; Receive a failed rollback request from the user and roll back the failed warehouse at the target site: Delete all storage data IDs and table data recorded in this transfer task.

2. A code repository site transfer method as claimed in claim 1, characterized in that: The Tuoma platform authenticates the user under the current tenant, including: If the authentication is successful, a code repository site transfer task is created; If the authentication fails, the page returns with a 403 error message, indicating that the user does not have sufficient permissions.

3. A code repository site transfer method as claimed in claim 2, characterized in that: Create a code repository site transfer task, including: Select the code repository or code repository group to be transferred as the source object; Get the code repository / code repository group specified in other sites or other sites as the target object; If no code repository or code repository group is specified as the target object, the code is transferred to the root directory of the target site.

4. A code repository site transfer system, characterized in that: The system comprises: Deployment module, used to deploy the Tuoma platform on each regional site and receive users' site transfer requests; An authentication module is used for the Tuoma platform to authenticate the user under the current tenant, and if the authentication is passed, a code repository site transfer task is created; A state modification module is used to form a transfer asynchronous task and modify the state of the code repository / code repository group to be transferred to operating, and verify whether the code repository / code repository group to be transferred and the corresponding parent group member exist in the target site cluster; An import module is used to import the code repository / code repository group to be transferred into the target site cluster if it exists and write the repository data into the successfully transferred data folder; if it does not exist, a member snapshot is generated; the write operation is repeated once, and the data with unsuccessful transfer status is written into the failed transfer data folder; The database relationship update module is used to update the database relationships of all transferred code repositories / code repositories groups to complete the code repository site transfer task; When executing the transfer asynchronous task, the specific timing data field is updated every preset period; When the current task ends, stop updating the scheduled data field and start the scheduled inspection task; The scheduled inspection tasks include: Check the timing data field in the asynchronous task, if it is detected that there is a timing data field exceeding a preset threshold, change the state of the asynchronous task to interrupt and delete the corresponding file; Send code repository site transfer information to the target site cluster and delete the repository currently performing the transfer operation, and delete the corresponding repository data; The preset period is 1 minute, and the preset threshold is 5 minutes; When an object fails to be transferred, the object and the reason for the failure are obtained; After the code repository site transfer task is completed, the objects that failed to be transferred and the reasons for the failure are written into the operation log; Receive a failed rollback request from the user and roll back the failed warehouse at the target site: Delete all storage data IDs and table data recorded in this transfer task.

5. A code repository site transfer device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a code repository site transfer method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a code repository site transfer method as described in any one of claims 1 to 3 are implemented.

Images