Method and device for realizing dual-cluster fast flash back of distributed database

Abstract

The invention relates to a method for realizing dual-cluster quick flash back of a distributed database, which is characterized by comprising the following steps: constructing a flash back snapshot library which comprises a first database and a second database, the first database supports multi-place and multi-center deployment, and the second database carries out resource arrangement according to actual requirements; creating a second database snapshot, selecting the second database snapshot and executing mounting, and returning the associated fragmented data to a consistency state through fragmented data recovery operation; and the second database snapshot is managed, so that the efficient database instance supply capability, the database copy flash-back capability at any moment and the data synchronization re-access capability are realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for implementing dual-cluster fast flashback of a distributed database. Background Art

[0002] In the construction of distributed systems, the rapid supply of historical data and the real-time synchronization of incremental data are two core requirements. Existing technologies mainly achieve these capabilities through the following methods:

[0003] Redo log: Generate reverse SQL by parsing the incremental redo log generated during the service, and execute it in reverse order to achieve data flashback. This method has efficiency problems, especially when the flashback point is far away from the current time and the data changes frequently, it is inefficient and cannot handle the flashback of database DDL changes.

[0004] Shared storage device: Deploy the "playback verification library" (the second database in this application) on a centralized storage with snapshot capabilities, and use the snapshot capabilities of the storage to quickly suspend and perform necessary recovery operations, and clone a production library (the first database in this application). At this time, services are provided to the outside world, and the storage continues to be replicated. This solution requires the introduction of third-party equipment, which increases costs. At the same time, it has strict requirements on IO stability and bandwidth when multiple nodes in a distributed system are written simultaneously, which may affect the efficiency of system operation.

[0005] Backup files: With the help of backup files, the "playback verification library" can be fully restored to the historical backup time. However, this method has low timeliness when processing massive data and lacks the necessary flexibility. If the interval between re-entry into the production library is very long, it will face the problem of not being able to catch up quickly, affecting the next use.

[0006] The defects of the existing technology are mainly reflected in the following aspects: First, through the log flashback method, if the flashback point is too far away from the current time and the data changes frequently, the corresponding incremental log may be as large as tens of GB or even hundreds of GB, and the efficiency of parsing and retrieving the log and generating reverse SQL for rollback is low. Second, when flashing back through the log, the DDL changes of the database cannot be flashed back. For example, in the open source MySQL database, the log does not record relevant data information, only the DDL statement itself, so it cannot be flashed back. Third, the use of storage device snapshot capabilities requires the introduction of additional third-party devices, which increases the cost of system construction, and the storage device itself may face IO bandwidth problems when processing the write of multiple data nodes in the distributed database, resulting in an increase in recovery delay and affecting the efficiency of the normal operation of the system. Fourth, for distributed databases, the ability to flash back with storage device snapshots cannot solve the problem of distributed data recovery consistency. At this time, it is still necessary to use the database redo log to perform consistency recovery. Summary of the invention

[0007] To address the deficiencies of the prior art, the present invention achieves fast flashback and data synchronization by building a dual cluster architecture including a master-slave database. The master database is deployed in multiple locations and multiple centers, and the slave database configures resources on demand. Data consistency is ensured by creating and managing slave database snapshots and performing shard data recovery operations. The present invention improves the efficiency of database instance provisioning, achieves data flashback at any time, and maintains data synchronization capabilities.

[0008] To achieve the above objectives, the present invention provides a method for implementing dual-cluster fast flashback of a distributed database in a first aspect, such as Figure 1 As shown, including:

[0009] Building a flashback snapshot library, including a first database and a second database, wherein the first database supports multi-location and multi-center deployment, and the second database arranges resources according to actual needs;

[0010] Create a second database snapshot, select the second database snapshot and mount it, and roll back the associated shard data to a consistent state through a shard data recovery operation;

[0011] The second database snapshot is managed.

[0012] Exemplarily, a quasi-real-time synchronization relationship is established between the first database and the second database via a replication channel.

[0013] Exemplarily, the creation of the second database snapshot includes multiple methods, including: creating a repeated snapshot, an automatic snapshot, a manual command snapshot, and each snapshot uses a globally unique snapshot naming method to facilitate snapshot management;

[0014] The automatic snapshot can automatically create snapshots according to a set policy, including creation at a fixed time or at specific intervals.

[0015] Exemplarily, after the associated shard data is rolled back to a consistent state through the shard data recovery operation, verification operations may be performed, including but not limited to version update, transaction playback, data spot check, and accident reproduction.

[0016] Exemplarily, the managing the second database snapshot includes: supporting snapshot query according to time range and name fuzzy matching;

[0017] Support querying snapshot creation time, snapshot status and snapshot information;

[0018] The snapshot status includes in use, invalid, and valid.

[0019] Exemplarily, the managing the second database snapshot also includes periodically cleaning up the created historical second database snapshots, specifically setting a default retention period or manual cleaning up.

[0020] In a second aspect, the present invention further provides a device for implementing dual-cluster fast flashback of a distributed database, which is used to execute any method described in claims 1 to 6, comprising:

[0021] A snapshot library construction module is used to construct a flashback snapshot library, including a first database and a second database, wherein the first database supports multi-site and multi-center deployment, and the second database arranges resources according to actual needs;

[0022] A snapshot library execution module is used to create a second database snapshot, select the second database snapshot and execute the mount, and roll back the associated shard data to a consistent state through a shard data recovery operation;

[0023] The snapshot management module is used to manage the second database snapshot.

[0024] A third aspect of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

[0025] A fourth aspect of the present invention further provides an electronic device, characterized in that it includes a processor and a memory;

[0026] The memory is used to store computer programs;

[0027] The processor is used to execute the method according to any one of claims 1 to 6 by calling a stored computer program.

[0028] The fifth aspect of the present invention also provides a computer program product, including a computer program and / or instructions, characterized in that when the computer program and / or instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

[0029] The beneficial effects of the present invention are: (1) It realizes efficient database instance provisioning. By maintaining a series of continuous snapshots locally, data can be flashed back at any time point within the snapshot retention time, greatly improving the supply efficiency of database instances; (2) It facilitates the rehearsal of operations in important cyclical periods in the financial industry, such as quarterly interest settlement scenario verification, year-end batch scenario verification, and online transaction deformation verification, providing a reliable testing environment to ensure the correctness of business processes before actual operations and the real fault warning capability for production systems; (3) The early rehearsal of key changes such as application version upgrade verification provides accurate reference data and technical verification for the formal execution of the above important actions, ensuring a smooth transition of changes and the stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a flowchart of the method for implementing the present invention.

[0031] Figure 2 The figure is a schematic diagram of the structure of the device for realizing the present invention.

[0032] Figure 3 A schematic diagram of executing snapshot mounting and consistency recovery according to the present invention.

[0033] Figure 4 FIG. 4 is a schematic diagram of the architecture of the first database and the second database of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0036] The present invention provides a method for implementing dual-cluster fast flashback of a distributed database, such as Figure 1As shown, including:

[0037] Building a flashback snapshot library, including a first database and a second database, wherein the first database supports multi-location and multi-center deployment, and the second database arranges resources according to actual needs;

[0038] Create a second database snapshot, select the second database snapshot and mount it, and roll back the associated shard data to a consistent state through a shard data recovery operation;

[0039] The second database snapshot is managed.

[0040] Specifically, a typical network is Figure 4 As shown, there are a first database (production database) and a second database (replay verification database). The production database is deployed in multiple locations and multiple centers, and the replay verification database can be deployed in a single center or multiple centers. Compared with the resource configuration of the production database, the replay verification database can be reduced according to the usage scenario to reduce the investment cost. Create a synchronous replication relationship between the first database and the second database. The replay verification database establishes a quasi-real-time synchronization relationship with the production database through the replication channel, which is called the synchronization mode. For specific implementation plans, refer to the logical replication technology based on log files. Log file-based logical replication technology is a database replication method that allows data to be synchronized between different database instances or between different types of database systems. In a distributed database, each data node (DN) returns the logical log of the local transaction in the order of transaction submission. The coordination node (CN) coordinates and aggregates the transaction logical logs from each DN through heap sorting, and outputs the logical logs in the order of heap sorting. Some database systems use triggers and stored procedures to implement logical replication. Triggers allow custom application code to be automatically executed when data changes occur in the database system, and there is an opportunity to record the changes in a separate table, which can be read by external programs and replicate the data changes to another system. Therefore, log file-based logical replication technology provides a flexible and efficient method to synchronize and replicate data in a database, and is particularly suitable for scenarios where data needs to be synchronized across database systems or instances.

[0041] Create a snapshot, which can quickly build a playback verification library snapshot based on the file system snapshot capability, covering all components of the distributed database. The snapshot level supports the instance level, that is, all related components in the instance must complete the snapshot action at the same time. For specific implementations, please refer to the ZFS file system snapshot technology. The ZFS snapshot is based on the COW mechanism of ZFS. When data is modified, ZFS will not directly overwrite the original data, but write new data in different locations. Only after the new data is successfully written, ZFS will update the pointer to the data. This mechanism ensures the atomicity of ZFS operations, prevents partial writing of data, and thus ensures data consistency. The snapshot creation methods provided can be repeated snapshots and automatic snapshots, and support manual command line methods to create snapshots of instances. Each snapshot uses a globally unique naming method (default naming format: instance name + unix timestamp + internal identifier) ​​to ensure uniqueness for snapshot management. The method also supports the system to automatically create snapshots according to established policies, such as timing or intervals of specific periods.

[0042] Execute the snapshot, enter the drill mode, select the historical snapshot and mount it. After the consistency data recovery operation, the data will be aligned to a consistent state. At this time, the restored second database, namely the "playback verification library", can perform verification operations, such as version changes, transaction playback, production history data extraction, major fault reproduction and other scenarios.

[0043] This method also includes destroying the snapshot. You can choose to destroy the snapshot after the drill is over. At this time, the playback verification library continues to be in a synchronous replication state and is not affected by the snapshot operation;

[0044] Snapshot management can centrally manage all instance-level snapshots managed by the current system, including providing a unified management and control platform. The following operations can be performed:

[0045] 1) Snapshot query: Through snapshot query, you can obtain information such as the creation time of the snapshot, the status of the snapshot, and the usage of the snapshot. It supports querying by time range, name fuzzy matching, and status. The snapshot status includes: in use, invalid, and valid.

[0046] 2) Snapshot usage: select the required snapshot to mount. For distributed databases, snapshots of other components at the same time will be mounted by default, and the status of the snapshot will be recorded as "in use".

[0047] 3) Snapshot cleanup: As the number of snapshots increases, long-term retention will occupy storage resources, and historical snapshots will not be retained for a long time considering the cycle of the playback verification library. Snapshot cleanup can be configured with default retention periodic cleanup and manual cleanup.

[0048] Data consistency recovery requires ensuring that the data in each shard of the distributed database is in a strictly consistent state before the snapshot is used. This includes data consistency judgment and data rollback, such as Figure 3 As shown in the figure, the distributed storage nodes are distributed in different servers or file systems, and the consistency of the snapshots of each node cannot be guaranteed. Therefore, it is necessary to perform necessary data rollback according to the location (GTID or SCN) of the snapshot time of the playback verification library when mounting the snapshot. The data rollback process is based on the global transaction ID, i.e. GTID, and can be persisted to the data row storage. After executing the snapshot mounting and pulling up, the status of the tenant's shard data can be queried, as shown in the following figure: Figure 3 As shown, the synchronization of shards is uneven. By comparing the persisted GTIDs of each shard, the smallest GTID (recorded as GTID_MIN) is selected as the consistency moment. At this time, the archive logs of the shards with GTID greater than GID_MIN are parsed to generate SQL statements, which are then converted into reverse SQL statements and executed in reverse order until they are restored to the GTID_MIN moment. At this time, all shards are at the data consistency moment.

[0049] The present invention also provides a dual-cluster fast flashback implementation device for a distributed database, which is used to execute the above method, such as Figure 2 As shown, including:

[0050] A snapshot library construction module is used to construct a flashback snapshot library, including a first database and a second database, wherein the first database supports multi-site and multi-center deployment, and the second database arranges resources according to actual needs;

[0051] A snapshot library execution module is used to create a second database snapshot, select the second database snapshot and execute the mount, and roll back the associated shard data to a consistent state through a shard data recovery operation;

[0052] The snapshot management module is used to manage the second database snapshot.

[0053] By using this device, the above-mentioned calculation processing method can be executed and the corresponding technical effects can be achieved.

[0054] An embodiment of the present invention also provides an electronic device for executing the above method. As an implementation device of the method, the electronic device has at least a processor and a memory, and in particular, the memory stores data and related computer programs required for executing the method, and the processor calls the data and programs in the memory to execute all steps of the implementation method and obtains the corresponding technical effect.

[0055] Preferably, the electronic device may include a bus architecture, the bus may include any number of interconnected buses and bridges, and the bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are all well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same component, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor when performing operations.

[0056] In addition, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operation control) used may include a microprocessor or other processor device and / or a logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices, which may store the above-mentioned related data information, and may also store a program for executing related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it may be a key or a touch input device; the power supply is used to provide power to the electronic device; the display is used to display display objects such as images and text, for example, it may be an LCD display. The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to a central processor, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.

[0057] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, electronic devices or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0059] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0060] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0061] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for implementing dual-cluster fast flashback of a distributed database, characterized in that: include: Building a flashback snapshot library, including a first database and a second database, wherein the first database supports multi-location and multi-center deployment, and the second database arranges resources according to actual needs; Create a second database snapshot, select the second database snapshot and mount it, and roll back the associated shard data to a consistent state through a shard data recovery operation; The second database snapshot is managed.

2. The method according to claim 1, characterized in that A quasi-real-time synchronization relationship is established between the first database and the second database via a replication channel.

3. The method according to claim 2, characterized in that The second database snapshot may be created in a variety of ways, including: creating a repeated snapshot, an automatic snapshot, a manual command snapshot, and each snapshot is named in a globally unique manner to facilitate snapshot management; The automatic snapshot can automatically create snapshots according to a set policy, including creation at a fixed time or at specific intervals.

4. The method according to claim 3, characterized in that After the associated shard data is rolled back to a consistent state through the shard data recovery operation, verification operations may be performed, including but not limited to version update, transaction playback, data spot check, and accident reproduction.

5. The method according to claim 4, characterized in that The management of the second database snapshot includes: supporting snapshot query according to time range and name fuzzy matching; Support querying snapshot creation time, snapshot status and snapshot information; The snapshot status includes in use, invalid, and valid.

6. The method according to claim 5, characterized in that The managing the second database snapshot also includes regularly cleaning up the created historical second database snapshots, specifically setting a default retention period or manually cleaning up.

7. A device for implementing dual-cluster fast flashback of a distributed database, characterized in that: Used to perform any of the methods described in claims 1 to 6, comprising: A snapshot library construction module is used to construct a flashback snapshot library, including a first database and a second database, wherein the first database supports multi-site and multi-center deployment, and the second database arranges resources according to actual needs; A snapshot library execution module is used to create a second database snapshot, select the second database snapshot and execute the mount, and roll back the associated shard data to a consistent state through a shard data recovery operation; The snapshot management module is used to manage the second database snapshot.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 6 by calling a stored computer program.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images