Data backup method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose a data backup method, device and equipment and a storage medium. The method is applied to a data backup system, the data backup system comprises a master library, the master library has a backup library, and the method comprises the following steps: connecting the master library and the backup library to enable the master library and the backup library to access each other; transmitting generated metadata to a message queue to send the metadata to the master library and the backup library through the message queue to synchronize; acquiring a code file and pushing the code file to the master library to enable the master library to synchronize the code file to the backup library through a synchronization script to perform backup. The technical scheme of the embodiments of the present application realizes hot backup of metadata and code in a message queue and script mode, does not need to stop, guarantees the safety of data assets, and does not affect users or developers to use data.

Description

Technical Field

[0001] This application relates to the field of computer technology, providing a data backup method, a data backup device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Currently, under the backdrop of digital transformation, enterprises are migrating their existing systems to the cloud, and a key step in this process is hosting data on the cloud. As cloud storage, it must not only ensure data security but also guarantee real-time performance. Real-time performance means not only supporting users to pull data for continuous integration at any time but also ensuring performance under high concurrency. Hot backup is necessary to guarantee the real-time nature of business operations. However, current hot backup methods involve system downtime for importing and exporting, impacting user and developer access and compromising data security. Summary of the Invention

[0003] The purpose of this application is to provide a data backup method, data backup device, electronic device, and computer-readable storage medium that uses message queues and scripts to achieve hot backup of metadata and code without downtime, ensuring the security of data assets and without affecting the use of data by users or developers.

[0004] This application proposes a data backup method applied to a data backup system, the data backup system including a primary database and a backup database. The method includes: connecting the primary database and the backup database to enable mutual access between the primary database and the backup database; transmitting generated metadata to a message queue to send the metadata to the primary database and the backup database for synchronization via the message queue; obtaining a code file and pushing the code file to the primary database to enable the primary database to synchronize the code file to the backup database for backup via a synchronization script.

[0005] Furthermore, the number of primary databases is at least two; the step of transmitting the generated metadata to a message queue to send the metadata to the primary database and the backup database for synchronization includes: creating an exchange in the message queue, as well as a sub-primary queue corresponding to each primary database and a backup queue corresponding to each backup database; binding the exchange to each queue using a routing key; transmitting the generated metadata to the exchange, so that the exchange sends the metadata to the target sub-primary queue and the backup queue corresponding to the backup database according to the routing key contained in the metadata, and then sending the metadata to the primary database and the backup database for synchronization through the target sub-primary queue and the backup queue.

[0006] Furthermore, the step of transmitting the generated metadata to a message queue, so as to send the metadata to the primary database and the backup database for synchronization via the message queue, includes: if the data type of the generated metadata is write data, then transmitting the metadata to the message queue, so as to send the metadata to the primary database and the backup database for synchronization via the message queue.

[0007] Furthermore, the method also includes: if the data type of the generated metadata is read data, then the metadata is routed to the master database by accessing a proxy server.

[0008] Furthermore, the step of connecting the primary database and the backup database to enable mutual access between the primary database and the backup database includes: generating key pairs for the primary database and the backup database respectively; establishing a connection channel between the primary database and the backup database; sending the public key of the primary database to the backup database and sending the public key of the backup database to the primary database, so as to enable passwordless access between the primary database and the backup database.

[0009] Furthermore, the step of pushing the code file to the master repository so that the master repository can synchronize the code file to the backup repository for backup via a synchronization script includes: creating a code repository in the master repository, and creating a synchronization script for triggering code synchronization and a hook function for associating with the backup repository in the directory of the code repository; pushing the code file from the client to the master repository so that the master repository can trigger the synchronization script to call the hook function to synchronize the code file to the backup repository.

[0010] Furthermore, the method also includes: when the primary database fails, modifying the proxy server configuration to replace the IP address of the primary database service with the IP address of the backup database; and modifying the code repository address of the primary database on the client to switch the backup database to the primary database.

[0011] This application also proposes a data backup device deployed in a data backup system, the data backup system including a primary database and a backup database. The device includes: a connection module for connecting the primary database and the backup database to enable mutual access between the primary database and the backup database; a metadata synchronization module for transmitting generated metadata to a message queue to send the metadata to the primary database and the backup database for synchronization via the message queue; and a code synchronization module for acquiring code files and pushing the code files to the primary database to enable the primary database to synchronize the code files to the backup database for backup via a synchronization script.

[0012] This application also proposes an electronic device comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method described above.

[0013] This application also proposes a computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a computer's processor, cause the computer to perform the method described above.

[0014] Compared with the prior art, this application has the following beneficial effects:

[0015] In the technical solution provided in this application, the generated metadata is synchronized to the primary database and the backup database through a message queue. By pushing the code file to the primary database, the primary database synchronizes the code file to the backup database for backup through a synchronization script. That is, hot backup of metadata and code is achieved by using message queues and scripts, without downtime, ensuring the security of data assets, and without affecting the use of data by users or developers. Attached Figure Description

[0016] Figure 1 A schematic diagram of one implementation environment involved in this application is shown;

[0017] Figure 2 A flowchart illustrating a data backup method according to an exemplary embodiment of this application is shown;

[0018] Figure 3 It shows Figure 2 A flowchart of step S120 in an exemplary embodiment shown in the illustrated example;

[0019] Figure 4 It shows Figure 2 A flowchart of step S110 in an exemplary embodiment shown in the illustrated example;

[0020] Figure 5 It shows Figure 2 A flowchart of step S130 in an exemplary embodiment shown in the illustrated example;

[0021] Figure 6 It shows Figure 2 The illustrated embodiment also includes a flowchart of primary / standby database switching;

[0022] Figure 7 A schematic diagram of a data backup system architecture is shown in an exemplary embodiment of this application;

[0023] Figure 8 It shows the basis Figure 7A schematic diagram of the metadata synchronization process in the system architecture;

[0024] Figure 9 A method based on is shown Figure 7 A schematic diagram of the system architecture code synchronization;

[0025] Figure 10 This illustration shows a schematic diagram of a main library code repository according to an exemplary embodiment of this application;

[0026] Figure 11 This illustration shows a schematic diagram of a backup code repository according to an exemplary embodiment of this application;

[0027] Figure 12 This application shows a schematic diagram illustrating a primary / backup switchover in an exemplary embodiment.

[0028] Figure 13 This application shows a schematic diagram illustrating the structure of a data backup device according to an exemplary embodiment.

[0029] Figure 14 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of this application is shown. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0034] Please see Figure 1 , Figure 1 This is a schematic diagram of an implementation environment involved in this application. The implementation environment includes a data backup system, which includes a primary database server 10 and a backup database server 20. The primary database server 10 has a primary database and a backup database, which is located on the backup database server 20. The primary database server 10 and the backup database server 20 can be connected via a network, which can be a wired network or a wireless network, and no restrictions are imposed here.

[0035] The data backup system also includes a client 30 and a data backup device 40. The client 30 is used to manage the data in the master database server and the backup database server. For example, the client 30 is a DevOps platform. The client 30 can also generate code files. For example, the client 30 can be a GitLab client or an IDE (Integrated Development Environment).

[0036] The data backup device 40 is used to connect the primary database and the backup database so that the primary database and the backup database can access each other, transmit the metadata generated by the client to the message queue so that the metadata can be sent to the primary database and the backup database for synchronization through the message queue, obtain the client code file and push the code file to the primary database so that the primary database can synchronize the code file to the backup database for backup through the synchronization script.

[0037] In some embodiments, the primary database server 10 and the standby database server 20 may be connected. The client 30 transmits the generated metadata to a message queue so that the metadata can be sent to the primary and standby databases for synchronization via the message queue. The client obtains the code file and pushes the code file to the primary database so that the primary database can synchronize the code file to the standby database for backup via a synchronization script.

[0038] The primary database server and the backup database server can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The data backup device can also be a server, and this is not restricted here.

[0039] The following is based on Figure 1 The implementation environment shown provides a detailed explanation of the data backup method:

[0040] Please see Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the present application, wherein the data backup method comprises... Figure 1 The data backup method, executed by the client or data backup device of the data backup system shown, includes steps S110 to S130, which are described in detail below:

[0041] S110. Connect the primary database and the standby database to enable them to access each other.

[0042] In this embodiment, the primary database is the main database, and the backup database is the standby database. The standby database is used to back up the data in the primary database. There can be one or more primary databases, and there can also be one or more backup databases. When there are multiple primary databases, different primary databases can be distinguished based on different services. For example, service 1 corresponds to primary database 1, and service 2 corresponds to primary database 2. This is not limited here.

[0043] In this embodiment of the application, the primary database and the backup database are connected, thereby enabling communication between the primary database and the backup database. The primary database can access the backup database, and the backup database can also access the primary database. The connection between the primary database and the backup database can be direct or indirect, and is not limited here.

[0044] In some embodiments, after the primary and standby databases are connected, since the primary and standby databases can access each other, the standby database can back up all the data in the primary database. At this point, the primary and standby databases are fully synchronized.

[0045] S120. The generated metadata is transmitted to a message queue so that the metadata can be sent to the primary and standby databases for synchronization.

[0046] It should be noted that the generated metadata refers to the metadata generated from the management of the provided services, such as the metadata generated from the management of users, permissions and code repositories. For example, if user A's contact information is modified, metadata about user A's modified contact information will be generated.

[0047] In this embodiment, the generated metadata is transmitted to the message queue in the order of data generation time. The message queue can be connected to the primary database and the backup database respectively. Then, the generated metadata is transmitted to the message queue, and the message queue sends the metadata to the primary database and the backup database. The primary database and the backup database process the metadata accordingly. For example, the primary database and the backup database will modify the contact information of user A based on the metadata of user A modifying contact information.

[0048] S130. Obtain the code file and push it to the primary database so that the primary database can synchronize the code file to the backup database via a synchronization script.

[0049] It should be noted that in this embodiment, the code files and the generated metadata are backed up in two different ways. The code files refer to the files that record the source code.

[0050] The code file can be obtained when a technician generates a new code file. The technician can submit the code file through a GitLab client or IDE development tool. Therefore, in this embodiment of the application, the code file is obtained through a GitLab client or IDE development tool.

[0051] After obtaining the code file, the code file is pushed to the primary database. The primary database stores a synchronization script, which is used to synchronize the code file in the primary database to the standby database. When the primary database receives the code file, it immediately triggers the synchronization script to synchronize the code file to the standby database for backup.

[0052] In this embodiment, the generated metadata is synchronized to the primary and backup databases via a message queue. The code file is pushed to the primary database, which then synchronizes the code file to the backup database via a synchronization script for backup. This achieves hot backup of metadata and code using message queues and scripts, without downtime, ensuring the security of data assets and without affecting the use of data by users or developers.

[0053] These steps S110 to S130 are described in detail below.

[0054] In this embodiment of the application, the number of master databases is at least two, such as... Figure 3 As shown, Figure 3 It shows Figure 2 The flowchart of step S120 in the illustrated embodiment includes:

[0055] S121. Create an exchange in the message queue, as well as a sub-master queue corresponding to each master database and a standby queue corresponding to each standby database.

[0056] In this embodiment of the application, an exchange, sub-queues, and backup queues are created in the message queue, wherein each sub-queue corresponds to a primary database, and each backup queue corresponds to a backup database; wherein the exchange is used to distribute data to different queues according to different rules, and the sub-primary queues and backup queues are used to transmit the received data to the corresponding primary database and backup database.

[0057] S122. Bind the switch to each queue using the routing key.

[0058] In this embodiment of the application, in order for the switch to accurately distribute data to the corresponding queue, it is necessary to bind the switch with each master queue and backup queue through a routing key.

[0059] In one example, the Binding Key of the switch and each master-sub queue is different. For example, the Binding Key of the switch and each sub queue is "Master n.Queue type". Master n indicates which queue it corresponds to, and queue type indicates the business type on which the metadata is generated. For example, Master 2.User indicates that it corresponds to the queue of Master 2, and the metadata is for the user business type.

[0060] S123. The generated metadata is transmitted to the switch, so that the switch sends the metadata to the target sub-master queue and the corresponding standby queue of the standby database according to the routing key contained in the metadata. The metadata is then sent to the master database and the standby database for synchronization through the target sub-master queue and the standby queue.

[0061] In this embodiment, when the generated metadata is transmitted to the switch, the metadata carries a specified Binding Key. The switch can then determine the target master / sub-queue corresponding to the metadata based on the Binding Key carried in the metadata and the Binding Key of all master / sub-queues. The metadata is then sent to the target master / sub-queue. Since the standby database has only one standby queue, the metadata is also sent directly to the standby queue corresponding to the standby database. The target master / sub-queue then sends the metadata to the master database, and the standby queue sends the metadata to the standby database for synchronization.

[0062] For example, if the generated metadata is a modification of the contact information of user A for the provided user service, and the user service corresponds to the main database 2, then the specified Binding Key carried by the metadata is the main database 2. The switch sends the metadata to the sub-main queue 2 corresponding to the main database 2, and at the same time sends the metadata to the backup queue.

[0063] In this embodiment, the switch is a topic switch. A topic-type switch matches the binding key based on wildcards. If a binding key meets the condition of a wildcard, it will be routed to the corresponding queue. At this time, the generated metadata is transmitted to the topic switch, so that the topic switch matches the routing key contained in the metadata based on wildcards. When the routing key meets the condition of the target wildcard, the metadata is sent to the target sub-master queue corresponding to the routing key and the backup queue corresponding to the backup database.

[0064] As mentioned earlier, the topic switch matches the Binding Key based on wildcards. The matching rules for wildcards are as follows: the Binding Key must be a string, with each word separated by a "."; the symbol "#" indicates matching one or more words; and the symbol "*" indicates matching a single word. Therefore, after receiving the metadata, the topic switch matches the routing key contained in the metadata based on wildcards.

[0065] In this embodiment of the application, the switch is bound to each queue by a routing key. After the metadata is transmitted to the switch, the switch can accurately and quickly transmit the metadata to the corresponding queue based on the routing key, so as to transmit it to the primary database and the backup database through the queue.

[0066] It should be noted that in the embodiments of this application, the read-write separation principle is adopted, that is, read data and write data are separated, and read data and write data are backed up in different ways. Therefore, before the generated metadata is transmitted to the message queue and sent to the master database and the standby database for synchronization through the message queue, it is also necessary to determine the data type of the generated metadata.

[0067] If the generated metadata is of write data type, the metadata is transferred to a message queue so that it can be sent to the primary and standby databases for synchronization.

[0068] Understandably, if the generated metadata is of write data type, it will inevitably require changes to the data in the primary and standby databases, such as adding or modifying data. Therefore, it is necessary to send the metadata to the primary and standby databases through a message queue so that the primary and standby databases can synchronously change the stored data based on the metadata.

[0069] In this embodiment, the read-write separation principle is adopted. If the generated metadata is of read data type, the metadata is routed to the primary database through an access proxy server. The proxy server is connected to both the primary and backup databases.

[0070] Since reading data does not change the data stored in the primary database, and the primary and standby databases are already synchronized, it is not necessary to synchronize metadata to the standby database. That is, when the generated metadata is of read data type, the metadata can be routed to the primary database via a proxy server, and the primary database will execute the metadata. For example, if the metadata is to read user B's contact information, the primary database will route user B's contact information to the proxy server so that the user can read it.

[0071] In this embodiment of the application, by separating read and write operations, write data is transmitted through a message queue and read data is routed through a proxy server. This ensures that the data is processed effectively while avoiding resource consumption and improving the performance of the main database.

[0072] It is worth noting that, such as Figure 4 As shown, Figure 4 It shows Figure 2 The flowchart of step S110 in the illustrated embodiment includes:

[0073] S111: Generate key pairs for the primary and backup databases respectively.

[0074] In this embodiment of the application, in order to ensure the data security in the primary database and the backup database, it is necessary to generate key pairs for the primary database and the backup database respectively, wherein the key pair includes a public key and a private key.

[0075] Optionally, you can generate key pairs for the primary and backup databases using a preset account. Understandably, although the preset account is the same, the key pairs in the primary and backup databases are different. In one example, the key pair is an SSH (Secure Shell) key pair.

[0076] S112. Establish a connection channel between the primary database and the standby database.

[0077] It is understandable that the primary and standby databases run on two separate servers, so the two servers can be connected to establish a connection channel between the primary and standby databases.

[0078] S113. Send the public key of the primary database to the backup database, and send the public key of the backup database to the primary database, so that the primary database and the backup database can access each other without password.

[0079] In this embodiment, the public key of the master database is sent to the backup database, and the public key of the backup database is sent to the master database. Thus, when the master database accesses the backup database, it can achieve passwordless access based on the public key of the backup database, and when the backup database accesses the master database, it can achieve passwordless access based on the public key of the master database.

[0080] In this embodiment of the application, the security of data in the primary and backup databases is ensured by using key pairs. By sending the corresponding public key to the other party, passwordless access is achieved while ensuring data security.

[0081] like Figure 5 As shown, Figure 5 It shows Figure 2 The flowchart of step S130 in the illustrated embodiment includes:

[0082] S131. Create a code repository in the primary repository, and create a synchronization script to trigger code synchronization and a hook function to associate with the backup repository in the directory of the code repository.

[0083] In this embodiment, after connecting the primary and backup databases, a mirror repository can be associated. First, a code repository is created on the primary database to store code files. After the code repository is successfully created, a synchronization script to trigger code synchronization and a hook function `custom_hooks` for associating with the backup database are automatically created in the corresponding directory of the code repository.

[0084] S132. Push the code file to the master database so that the master database triggers the synchronization script to call the hook function to synchronize the code file to the backup database.

[0085] Users submit code files on the client or development tools, which push the code files to the master database. The master database automatically triggers the execution of the synchronization script, which uses the hook function custom_hooks to synchronize the code files to the backup database.

[0086] It is understandable that the primary and standby databases should have the same structure. When a code file is created on the primary database, a code file also needs to be created on the standby database, and then the primary database will synchronize the code file to the standby database.

[0087] In this embodiment of the application, by creating synchronization scripts and hook functions in the directory of the code repository, the code files can be automatically synchronized to the backup repository, thus realizing hot backup of the code.

[0088] In this embodiment of the application, the data backup method further includes a primary / standby database switching process, such as... Figure 6 As shown, it also includes:

[0089] S140. When the primary database fails, modify the proxy server configuration to replace the primary database service's IP address with the backup database's IP address.

[0090] In this embodiment, the primary database may fail. To ensure continued operation, the standby database can be switched to the primary database in the event of a primary database failure. This requires two modifications: First, modifying the proxy server configuration. The proxy server contains configurations for both the primary and standby databases. Replace the standby database's IP address with the primary database's IP address. This allows direct access to the standby database via the proxy server for read-type metadata. For write-type metadata, after transferring the metadata to the message queue, if the primary database fails, the metadata will be synchronized to the standby database via the message queue.

[0091] S150, and modify the code repository address of the primary database on the client to switch the backup database to the primary database.

[0092] In this embodiment, step S140 involves modifications to the metadata, and step S150 involves modifications to the code files. As mentioned earlier, the user submits the code files to the client, and the client pushes the code files to the primary database. Therefore, it is necessary to modify the primary database's code repository address on the client to the backup database's code repository address, so that the client will push the code files to the backup database.

[0093] Once the primary database recovers, the primary and standby databases can access each other. The primary database can then access the standby database to retrieve the unsynchronized data and switch back.

[0094] In the embodiments of this application, the execution order of steps S140 and S150 can be S140 executed first, S150 executed first, or they can be executed simultaneously, and no limitation is made here.

[0095] In this embodiment, the backup database is only replaced when the primary database encounters a problem. Furthermore, the primary-backup switch can be achieved by modifying the configuration of the proxy server and the address of the code repository on the client. The cost of primary-backup switch is relatively small, the operation is simple, and the user experience is improved.

[0096] To facilitate understanding, this embodiment uses a concrete example to illustrate the data backup method. This data backup method is based on GitLab and uses a Topic message queue service to synchronize metadata such as repositories, users, and permissions within the GitLab service. It synchronizes code by triggering a custom script through code submission, thereby achieving hot backup of the entire code repository. This solves the problem of traditional code repository backups requiring system downtime for import and export, ensuring the security of code assets without affecting user or developer access to the code repository. Furthermore, its simple operation allows for switching between primary and backup repositories, greatly improving the user experience.

[0097] like Figure 7 As shown, Figure 7 This application provides a schematic diagram of a data backup system architecture. The system architecture includes a DevOps platform, a message queue service, a GitLab consumer microservice, a GitLab client or IDE development tool, a GitLab master database server, and a GitLab backup database server. The GitLab master database server includes multiple servers, each of which has the aforementioned master database. The backup database server has a backup database, that is, there are multiple master databases and one backup database.

[0098] based on Figure 7 The system architecture diagram shown illustrates that the DevOps platform generates messages (i.e., the aforementioned metadata) to the message queue service for managing users, permissions, and code repositories. The GitLab consumer microservice consumes these messages, which are then processed by a primary repository (GitLab primary server) and a backup repository (GitLab backup server) through the GitLab consumer microservice.

[0099] After a user commits code files in the GitLab client or IDE, the GitLab client or IDE will call the git_push function to push the updated code files locally to a primary repository and a backup repository.

[0100] It is understandable that the primary and standby databases can access each other, so the standby database can push relevant data to the primary database through custom_hooks.

[0101] like Figure 8 As shown, Figure 8 A method based on is shown Figure 7 A schematic diagram of the metadata synchronization process of the system architecture. Figure 7 The message queue service in this system is implemented by integrating the open-source component RabbitMQ. The message queue service includes Topic exchanges, queues corresponding to each master database, and queues corresponding to each standby database. The binding design between the topic exchange and each queue is based on the database name.queue class name; the binding key for master database queues is "master_n.queue_class"; and the binding key for standby database queues is "*.queue_class". This ensures that all messages are sent to the standby database queues; queues are then mapped one-to-one with consumers based on their names. The consumer microservices include consumers corresponding to standby databases and each master database, and these consumers then send messages to their respective master and standby databases.

[0102] For example, a metadata write operation on the DevOps platform might modify a user's phone number information on GitLab server 2 (primary database 2), generating a user modification message. This message, bound to "service2.user modification," will be sent to two message queues named "primary database 2.user modification" and "*.user modification" according to the topic matching rules. These queues are bound to the primary database 1 consumer and the standby database consumer, respectively. The consumer consumes the message and executes the service API calls on primary database 2 and the standby database to modify the user's phone number.

[0103] like Figure 9 As shown, Figure 9 A method based on is shown Figure 7 This is a diagram illustrating the code synchronization within the system architecture. After the GitLab master repository server completes the creation of the code repository, a synchronization script is automatically created in the code repository directory within the master repository. When a user commits code to the master repository using the GitLab client or IDE development tool, the script is automatically triggered, synchronizing the code commit to the backup repository service. In this way, GitLab's code repository hot backup function is implemented.

[0104] The implementation steps for one primary database and one standby database include:

[0105] 1. Deploy and install the primary GitLab code repository and the backup GitLab code repository on a Linux server using physical machines.

[0106] 2. Connect the two servers, add SSH permissions to each other, and enable passwordless access.

[0107] On the primary repository server, using the git account (which exists by default), generate the . / SSH directory and a key pair file. Add the contents of id_rsa.pub (public key) to the SSH keys settings on both the primary and standby repository servers.

[0108] 3. Associate with a mirror repository.

[0109] Once the GitLab master server has consumed the repository creation message and successfully created the repository, it will automatically generate a synchronization script in the corresponding repository path. The script content is as follows:

[0110] #! / bin / bash / / Script start line

[0111] cd / var / opt / Gitlab / git-data / repositories / $repositories_path / / / Navigate to the corresponding repository directory and perform the following operations.

[0112] git remote add --mirror=push $USER git@$Gitlab_IP:$repositories_path / / Add a remote repository mirror for backup

[0113] mkdir custom_hooks / / Create a folder with a custom name to store sub - scripts

[0114] cd custom_hooks / / Enter the created folder

[0115] touch post - receive / / Create the file post - receive

[0116] cat > post - receive << EOF / / Write the sub - script content into the file post - receive, the content is as follows:

[0117] #! / bin / bash / / Start line of the sub - script

[0118] echo "$USER" / / Output the current user

[0119] exec git push -u $USER & / / Use the current user to execute the action of pushing code to the mirror repository

[0120] EOF / / Finish writing the sub - script

[0121] chmod -R 755 post - receive / / Change the permission of the file post - receive to 755

[0122] Combined Figure 9 As shown, after the user submits a code file in the Gitlab client or IDE development tool, the Gitlab client or IDE development tool will call the git_push function to submit the code to the code repository 1 of the main repository. Since the main repository and the backup repository can access each other through SSH keys, the main repository automatically runs the above - mentioned synchronization script to call custom_hooks to push the code file path 1 to the code repository 1 of the backup repository.

[0123] As Figure 10 and Figure 11 shown Figure 10 shows a schematic diagram of the main repository code repository Figure 11 shows a schematic diagram of the backup repository code repository. The Gitlab client or IDE development tool adds a README.md file and submits the code file (test - push.tet) to the main repository. At the same time, the file (test - push.tet) can also be automatically available in the backup repository.

[0124] If the standby repository is replaced by the primary repository, after the Git client or IDE development tool commits the code file, the code file will be pushed to the code repository 2 of the standby repository. Similarly, the standby repository will automatically run the above synchronization script to call custom_hooks to push the code file path 2 to the code repository 2 of the primary repository.

[0125] As mentioned above, primary / standby switching is possible in the embodiments of this application. Figure 12 As shown, Figure 12 A schematic diagram of a master-slave failover is shown. The GitLab service in this application supports a distributed architecture, and the GitLab interface calls adopt a read-write separation principle. Write operations are consumed by the master and slave database services via a message queue service, while read operations are routed to different master database nodes through Nginx (the aforementioned proxy server). The slave database only takes over when the master database fails. The failover can be completed by modifying only two configurations: 1. Modify the Nginx configuration to replace the IP address of a master database service with the IP address of the slave database service; 2. Developers can replace the slave database by modifying the GitLab repository address configured in the GitLab client and IDE development tools. Once the GitLab master database service recovers, the configuration is changed back, thus completing the master-slave failover.

[0126] For example, to view the read operations of a certain user on GitLab service 2, the DevOps platform directly calls the query API interface of the GitLab service, and then Nginx routes the query to GitLab service 2 for execution.

[0127] Compared to existing technologies that involve system downtime for importing and exporting code repositories, the method provided in this application not only ensures the security of code assets through backup but also guarantees system real-time performance. Hot backups do not require shutting down the code repository. Furthermore, the cost of master-slave switching is lower, improving the user experience. The use of Topic message queues for metadata synchronization and API calls for read-write separation improve the performance of the GitLab code repository. Simultaneously, this invention supports the deployment of a distributed architecture for GitLab code repositories, providing excellent support for production system security.

[0128] The following describes an apparatus embodiment of this application, which can be used to execute the data backup method described above in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the data backup method described above in this application.

[0129] like Figure 13 As shown, Figure 13This is a schematic diagram illustrating the structure of a data backup device according to an exemplary embodiment of this application. The device is deployed in a data backup system and includes:

[0130] The connection module 1310 is used to connect the primary database and the standby database so that the primary database and the standby database can access each other;

[0131] The metadata synchronization module 1320 is used to transmit the generated metadata data to a message queue, so that the metadata can be sent to the primary database and the standby database for synchronization via the message queue;

[0132] The code synchronization module 1330 is used to obtain code files and push them to the master database, so that the master database can synchronize the code files to the backup database via a synchronization script.

[0133] In some embodiments of this application, based on the aforementioned scheme, the number of master databases is at least two; the metadata synchronization module 1320 further creates an exchange in the message queue, as well as a sub-master queue corresponding to each master database and a backup queue corresponding to the backup database; the exchange is bound to each queue through a routing key; the generated metadata is transmitted to the exchange, so that the exchange sends the metadata to the target sub-master queue and the backup queue corresponding to the backup database according to the routing key contained in the metadata, and the metadata is sent to the master database and the backup database through the target sub-master queue and the backup queue for synchronization.

[0134] In some embodiments of this application, based on the foregoing scheme, the metadata synchronization module 1320 is further configured to transmit the metadata data to a message queue if the data type of the generated metadata is write data, so as to send the metadata to the primary database and the backup database for synchronization through the message queue.

[0135] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes an access module, which is used to route the metadata to the main database through an access proxy server if the data type of the generated metadata is read data.

[0136] In some embodiments of this application, based on the aforementioned scheme, the connection module 1310 is further configured to generate key pairs for the primary database and the backup database respectively; establish a connection channel between the primary database and the backup database; send the public key of the primary database to the backup database, and send the public key of the backup database to the primary database, so that the primary database and the backup database can access each other without password.

[0137] In some embodiments of this application, based on the foregoing scheme, the code synchronization module 1330 is further configured to create a code repository in the master repository, and create a synchronization script for triggering code synchronization and a hook function for associating with the backup repository in the directory of the code repository; push code files from the client to the master repository so that the master repository triggers the synchronization script to call the hook function to synchronize the code files to the backup repository.

[0138] In some embodiments of this application, based on the aforementioned scheme, the device further includes a switching module, used to modify the proxy server configuration when the primary database fails, so as to replace the IP address of the primary database service with the IP address of the backup database; and to modify the code repository address of the primary database on the client, thereby completing the replacement of the primary database by the backup database.

[0139] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0140] In the apparatus of this application embodiment, the metadata synchronization module sends messages about user, permission, and code repository management to a Topic-type message queue. These messages are consumed simultaneously by a primary and a backup database in the consumption service, and then the GitLab interface is called to perform the operation.

[0141] Code synchronization module: After the GitLab server completes the creation of the code repository, the invention will automatically create a synchronization script in the corresponding code repository directory. When the user commits code to the master repository in the Git client or IDE development tool, the script can be automatically triggered to synchronize the code to the backup repository service.

[0142] GitLab supports a distributed architecture, and the GitLab API calls follow a read-write separation principle. Write operations are consumed by the primary and backup services via a message service, while read operations are routed to different primary database nodes via Nginx. The backup database only takes over when the primary database fails.

[0143] In one exemplary embodiment, an electronic device includes one or more processors and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method described above. It is understood that the electronic device may be deployed within a server.

[0144] Figure 14 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment.

[0145] It should be noted that this electronic device is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Furthermore, this electronic device should not be interpreted as requiring or depending on any specific feature. Figure 14 One or more components of the exemplary electronic device shown.

[0146] like Figure 14As shown, in an exemplary embodiment, the electronic device includes a processing component 1401, a memory 1402, a power supply component 1403, a multimedia component 1404, an audio component 1405, a processor 1406, a sensor component 1407, and a communication component 1408. Not all of these components are mandatory; the electronic device may add or remove other components according to its functional requirements, and this embodiment does not impose any limitations.

[0147] Processing component 1401 typically controls the overall operation of an electronic device, such as operations associated with display, data communication, and log data synchronization. Processing component 1401 may include one or more processors 1406 to execute instructions to complete all or part of the aforementioned operations. Furthermore, processing component 1401 may include one or more modules to facilitate interaction between processing component 1401 and other components. For example, processing component 1401 may include a multimedia module to facilitate interaction between multimedia component 1404 and processing component 1401.

[0148] The memory 1402 is configured to store various types of data to support operation of the electronic device, examples of which include instructions for any application or method operating on the electronic device. The memory 1402 stores one or more modules configured to be executed by the one or more processors 1406 to perform all or part of the steps in the methods described in the above embodiments.

[0149] Power supply component 1403 provides power to various components of an electronic device. Power supply component 1403 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0150] Multimedia component 1404 includes a screen that provides an output interface between an electronic device and a user. In some embodiments, the screen may include a TP (Touch Panel) and an LCD (Liquid Crystal Display). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0151] Audio component 1405 is configured to output and / or input audio signals. For example, audio component 1405 includes a microphone configured to receive external audio signals when the electronic device is in an operating mode, such as a call mode, recording mode, or voice recognition mode. In some embodiments, audio component 1405 also includes a speaker for outputting audio signals.

[0152] Sensor assembly 1407 includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, sensor assembly 1407 can detect the on / off state of the electronic device and can also detect temperature changes in the electronic device.

[0153] Communication component 1408 is configured to facilitate wired or wireless communication between electronic devices and other devices. Electronic devices can access wireless networks based on communication standards, such as Wi-Fi (Wireless-Fidelity).

[0154] Understandable. Figure 14 The structure shown is for illustrative purposes only; electronic devices may include structures that are more advanced than those shown. Figure 14 The more or fewer components shown, or having the same Figure 14 The different components are shown. Figure 14 Each component shown can be implemented using hardware, software, or a combination thereof.

[0155] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0156] It should be noted that the computer-readable storage medium shown in the embodiments of this application can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0157] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0158] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A data backup method, characterized in that, Applied to a data backup system, the data backup system including a primary database, the primary database having a backup database, the number of primary databases being at least two, the method includes: Connect the primary database to the backup database so that the primary database and the backup database can access each other; The generated metadata is transmitted to a message queue so that the metadata can be sent to the primary database and the backup database for synchronization. The generated metadata is generated for the management of the provided services. Obtain the code file, create a code repository in the main repository, and create a synchronization script for triggering code synchronization and a hook function for associating with the backup repository in the directory of the code repository; push the code file from the client to the main repository, so that the main repository triggers the synchronization script to call the hook function to synchronize the code file to the backup repository; The step of transmitting the generated metadata to a message queue, so as to send the metadata to the primary database and the backup database for synchronization via the message queue, includes: An exchange is created in the message queue, along with sub-master queues corresponding to each master database and backup queues corresponding to each backup database. The exchange is a topic exchange. The switch is bound to each queue using a routing key; The generated metadata is transmitted to the switch, so that the switch sends the metadata to the target sub-master queue and the backup queue corresponding to the backup database according to the routing key contained in the metadata. The metadata is then sent to the master database and the backup database through the target sub-master queue and the backup queue for synchronization.

2. The method according to claim 1, characterized in that, The step of transmitting the generated metadata to a message queue, so as to send the metadata to the primary database and the backup database for synchronization via the message queue, further includes: If the generated metadata is of write data type, the metadata is transmitted to the message queue so that the metadata can be sent to the primary database and the backup database for synchronization.

3. The method according to claim 2, characterized in that, The method further includes: If the generated metadata is of read data type, then the metadata is routed to the master database through the access proxy server.

4. The method according to claim 1, characterized in that, The connection between the primary database and the backup database, enabling mutual access between them, includes: Generate key pairs for the primary database and the backup database respectively; Establish a connection channel between the primary database and the backup database; The public key of the primary database is sent to the backup database, and the public key of the backup database is sent to the primary database, so that the primary database and the backup database can access each other without password.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: When the primary database fails, modify the proxy server configuration to replace the IP address of the primary database service with the IP address of the backup database. Furthermore, the code repository address of the primary database on the client is modified to switch the backup database to the primary database.

6. A data backup device, characterized in that, Deployed in a data backup system, the data backup system including a primary database, the primary database having a backup database, the number of primary databases being at least two, the device comprising: A connection module is used to connect the primary database and the backup database so that the primary database and the backup database can access each other; The metadata synchronization module is used to transmit the generated metadata to a message queue, so that the metadata can be sent to the primary database and the backup database for synchronization through the message queue. The generated metadata is generated for the management of the provided services. The code synchronization module is used to obtain code files, create a code repository in the master repository, and create a synchronization script for triggering code synchronization and a hook function for associating with the backup repository in the directory of the code repository; push the code files from the client to the master repository, so that the master repository triggers the synchronization script to call the hook function to synchronize the code files to the backup repository; The metadata synchronization module is also used to create an exchange in the message queue, as well as sub-master queues corresponding to each master database and a backup queue corresponding to the backup database, wherein the exchange is a topic exchange; the exchange is bound to each queue through a routing key; the generated metadata is transmitted to the exchange, so that the exchange sends the metadata to the target sub-master queue and the backup queue corresponding to the backup database according to the routing key contained in the metadata, and the metadata is sent to the master database and the backup database through the target sub-master queue and the backup queue for synchronization.

7. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a computer's processor, cause the computer to perform the method described in any one of claims 1 to 5.

Images