Database system and database system management method

Abstract

The invention provides a database system and a database system management method, and relates to the field of databases. The master database, the slave databases, the read-only databases and the cloud file system form a database system with separated operation and storage, the master database and the slave databases are subjected to data synchronization in a copying mode, and each read-only database and the slave database are subjected to data synchronization based on log release / subscription taking the cloud file system as a medium. The master database does not need to synchronize the read-only database, so that the pressure of the master database is effectively relieved, the network uplink bandwidth of the master database is saved, and the overall service performance of the database system is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of databases, and in particular to a database system and a database system management method. Background Art

[0002] A database is a collection of various interrelated data stored in a computer memory and organized in a certain format for users to process data quickly and effectively. A database can store a large amount of structured and unstructured data, such as text, numbers, images, audio and other types of data.

[0003] In some database technologies, in order to cope with business pressure, a database system is deployed with one master database and multiple slave databases. The master database and each slave database use replication to synchronize data, so that the data location when the slave database is backed up is synchronized to the latest data location of the master database. The synchronization process takes seconds to minutes, and the master database needs to synchronize log data to all slave databases, which puts pressure on the master database and occupies more network uplink bandwidth of the master database. Summary of the invention

[0004] In the disclosed embodiment, a master database, a slave database, a read-only database and a cloud file system constitute a database system with separated computing and storage, wherein the master database and the slave databases are synchronized with each other by replication, and each read-only database is synchronized with the slave database based on log publishing / subscription using the cloud file system as a medium. The master database does not need to synchronize with the read-only database, which effectively relieves the pressure on the master database, saves the network uplink bandwidth of the master database, and improves the overall service performance of the database system.

[0005] Some embodiments of the present disclosure propose a database system management method, including: creating and initializing a database directory, and creating a writable snapshot of a slave database in a cloud file system based on the initialized database directory; synchronizing data between a master database and a slave database through replication; synchronizing locally cached log data from the slave database to the writable snapshot of the slave database in the cloud file system; and subscribing a read-only database to the log data in the writable snapshot of the slave database in the cloud file system, and synchronizing data with the slave database.

[0006] In some embodiments, the database system management method also includes: before the read-only database is subscribed, based on the latest snapshot of the master database, creating a writable snapshot of the read-only database in the cloud file system; based on the writable snapshot of the read-only database, starting the database instance corresponding to the read-only database, and configuring the data files to be preferentially stored in the local cache of the read-only database.

[0007] In some embodiments, the database system management method further includes: in response to the read-only database satisfying a preset release condition, stopping a database instance corresponding to the read-only database, and deleting a writable snapshot of the read-only database.

[0008] In some embodiments, synchronizing the locally cached log data from the slave database to a writable snapshot of the slave database in the cloud file system includes: the slave database stores the log data obtained by replaying the received logs of the master database in the local cache, and asynchronously synchronizing the locally cached log data to the writable snapshot of the slave database.

[0009] In some embodiments, the database system management method also includes: in response to the slave database satisfying a preset reconstruction condition, creating a writable snapshot of the first database in the cloud file system based on the latest writable snapshot of the master database; based on the writable snapshot of the first database, starting the corresponding database instance of the first database, and configuring metadata and log data to be preferentially stored in the local cache of the first database; controlling the first database to subscribe to the log data in the writable snapshot of the slave database until the subscription is stopped when the subscription reaches the latest log data; synchronizing data between the master database and the first database through replication; and using the first database as the reconstructed slave database.

[0010] In some embodiments, the database system management method further includes: stopping the database instance corresponding to the slave database before reconstruction, and deleting the writable snapshot of the slave database before reconstruction.

[0011] In some embodiments, the reconstruction condition includes at least one of the following: the slave database before reconstruction cannot be started or the storage space of the cloud file system occupied by the writable snapshot of the slave database before reconstruction exceeds a preset value.

[0012] In some embodiments, the database system management method also includes: the slave database replays the received log of the master database and stores it in the local cache, and persistently stores the locally cached data in the cloud file system; in the cloud file system, the latest writable snapshot is created based on the writable snapshot of the slave database; the slave database is set to be writable and switched to the master database; in the cloud file system, the writable snapshot of the master database before the master-slave switch is deleted, and the latest writable snapshot of the slave database before the master-slave switch is switched to the master branch snapshot.

[0013] In some embodiments, the database system management method also includes: in the cloud file system, based on the latest writable snapshot of the primary database after the master-slave switch, creating a writable snapshot of the second database; based on the writable snapshot of the second database, starting the corresponding database instance of the second database, and configuring metadata and log data to be preferentially stored in the local cache of the second database; the second database subscribes to the log data in the writable snapshot of the primary database after the master-slave switch until the subscription is stopped when the subscription reaches the latest log data; through replication, the primary database after the master-slave switch synchronizes data with the second database; and the second database is used as a rebuilt slave database.

[0014] In some embodiments, the database system management method further includes: the master database and / or the slave database preferentially stores metadata and log data in a local cache, and then stores them in a cloud file system when overflowing or synchronizing, and stores table space data in a cloud file system.

[0015] In some embodiments, the database system management method also includes: the master database stores the log corresponding to the transaction in the local cache and synchronizes it to the local disk in response to the transaction submitted by the client, and sends the log corresponding to the transaction to the slave database, receives a confirmation message returned by the slave database after storing the log in the form of a relay log, confirms that the transaction was successfully submitted, and returns the successful transaction submission to the client.

[0016] Some embodiments of the present disclosure propose a database system, including: a cloud file system, configured to store writable snapshots of a master database, writable snapshots of a slave database, and writable snapshots of a read-only database, wherein the writable snapshots of the slave database and the writable snapshots of the read-only database are created based on the writable snapshot of the master database; the master database and the slave database are configured to synchronize data by replication; the slave database is configured to synchronize locally cached log data to the writable snapshot of the slave database in the cloud file system; the read-only database is configured to subscribe to the log data in the writable snapshot of the slave database in the cloud file system, synchronize data with the slave database, and process read transactions.

[0017] In some embodiments, the master database and / or the slave database are configured to store metadata and log data in a local cache first, and then store them in a cloud file system when overflowing or synchronizing, and store table space data in a cloud file system.

[0018] In some embodiments, the read-only database is configured to store data files in a local cache first, and then store them in a cloud file system when overflow occurs.

[0019] In some embodiments, the master database is configured to store the log corresponding to the transaction in the local cache and synchronize it to the local disk in response to the transaction submitted by the client, and send the log corresponding to the transaction to the slave database, receive the confirmation message returned by the slave database after storing the log in the form of a relay log, confirm that the transaction was successfully submitted, and return the successful transaction submission to the client.

[0020] Some embodiments of the present disclosure provide a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, steps of a database system management method are implemented.

[0021] Some embodiments of the present disclosure provide a computer program product, including computer instructions, which implement steps of a database system management method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0023] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0024] Figure 1 A schematic diagram showing a cloud-native file system according to some embodiments of the present disclosure.

[0025] Figure 2 A schematic diagram showing a hook-based POSIX interface implementation scheme according to some embodiments of the present disclosure.

[0026] Figure 3 A schematic diagram showing a FUSE-based POSIX interface implementation solution of some embodiments of the present disclosure.

[0027] Figure 4 A schematic diagram showing a writable snapshot based on a copy-on-write mechanism according to some embodiments of the present disclosure.

[0028] Figure 5 A schematic diagram showing a local cache according to some embodiments of the present disclosure.

[0029] Figure 6 A schematic diagram showing striped storage and pre-reading according to some embodiments of the present disclosure.

[0030] Figure 7 A schematic diagram showing a database system with separated storage and computation according to some embodiments of the present disclosure.

[0031] Figure 8A schematic diagram showing a database system management method according to some embodiments of the present disclosure, in particular, a method for synchronizing data between databases.

[0032] Fig. 9 A schematic diagram illustrating a database system management method according to some embodiments of the present disclosure, particularly, creation / release of a read-only database.

[0033] Fig.10 A schematic diagram illustrating a database system management method according to some embodiments of the present disclosure, particularly reconstruction / release from a database.

[0034] Fig.11 A schematic diagram showing a database system management method according to some embodiments of the present disclosure, in particular, a master / slave database switching.

[0035] Fig.12 A schematic diagram showing a database system management method according to some embodiments of the present disclosure, in particular, transaction processing of a database system. DETAILED DESCRIPTION

[0036] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0037] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0038] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0039] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0040] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0041] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0042] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0043] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0044] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0045] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0046] In addition, in order to avoid obscuring the present disclosure due to unnecessary details, only the processing steps and / or device structures closely related to at least the scheme according to the present disclosure are shown in the drawings, and other details that are not closely related to the present disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it does not need to be discussed again for subsequent drawings.

[0047] The database system of this embodiment includes a master database, a slave database, a read-only database, and a cloud file system, which is a storage and computing separation architecture. In the storage and computing separation architecture, the storage of the database is separated from the database instance, and most of the data in the database is stored in an independent storage system. This embodiment uses a cloud file system as a storage system. Cloud file systems are usually distributed, so they are also called cloud distributed file systems. Based on the cloud file system, the database system of the storage and computing separation of this embodiment can be implemented.

[0048] The cloud file system has the following capabilities: support for cloud-native deployment; support for POSIX interface access; support for writable snapshots; support for local cache; support for striped storage and pre-reading. For example, the cloud file system can be implemented based on the open source JuiceFS. The following introduces each capability of the cloud file system one by one.

[0049] Cloud file systems can be deployed based on the following two storage systems: Figure 1 shown.

[0050] (1) A database with transaction processing capabilities, used to store metadata and log data. Taking JuiceFS as an example, JuiceFS supports Redis, MySQL, TiKV and other database systems. Redis can provide better file writing performance, TiKV can provide larger storage capacity, and MySQL's capabilities are between the above two. JuiceFS can access Redis, MySQL and TiKV provided by different cloud vendors.

[0051] (2) Distributed object storage, used to store data objects. Taking JuiceFS as an example, JuiceFS supports cloud services of different object storage products of many cloud vendors.

[0052] Cloud file systems, such as JuiceFS, can be directly built on general cloud services and support cloud-native deployment without the need to develop customized cloud services. Therefore, cloud file systems are also called cloud-native file systems.

[0053] In terms of access interface, cloud file systems (such as JuiceFS) support POSIX (Portable Operating System Interface) interface, so that cloud file systems can be accessed like ordinary file systems. POSIX is a set of standards designed to solve the compatibility between different operating systems. POSIX defines a series of APIs (application programming interfaces) and command line tools. These APIs and tools specify what functions the operating system should provide, and specify the calling methods and behaviors of these functions, mainly covering file management, process control, user permissions, system calls, etc. Compliance with POSIX standards can ensure the compatibility of programs when porting from one platform to another, thereby improving the portability and development efficiency of software.

[0054] The POSIX interface can be implemented in the following two ways, for example.

[0055] Hook-based POSIX interface implementation, such as Figure 2As shown in the figure. The operating system provides a series of application programming interfaces (APIs) for file operation functions through the runtime library. File operation functions include open, close, read, write, lseek (current file displacement), fsync (synchronize data from the user space buffer to the disk), etc. The APIs of these file operation functions are compatible with the POSIX protocol. By injecting hook functions at runtime, the file operation functions in the runtime library can be replaced by hook functions through rewriting, thereby simulating access to the cloud file system as local file operations.

[0056] POSIX interface implementation based on FUSE (Filesystem in Userspace), such as Figure 3 As shown. Reversely penetrate from kernel state back to user state and call the callback function specified by the user state file system interface. The POSIX interface of JuiceFS is implemented in this way. When the user process performs file operations on JuiceFS (such as reading, writing, attribute query, etc.), these requests first pass through the kernel's VFS (Virtual Filesystem Switch) layer; the VFS layer forwards the request to the kernel's FUSE kernel module; the FUSE kernel module communicates with the JuiceFS client process through the / dev / fuse (character device file) device; the JuiceFS client process calls the corresponding processing function to perform the operation according to the type and parameters of the request; after the operation is completed, the JuiceFS client process returns the result to the FUSE kernel module, which then returns it to the kernel's VFS layer and finally returns it to the user process.

[0057] The cloud file system supports the POSIX interface, which allows transparent access to the cloud file system without intruding or reconstructing the database source code, so that the cloud file system can be accessed just like a normal file system.

[0058] A writable snapshot is a technology that can retain a copy of data at a certain point in time and allow users to write to this copy. When creating a snapshot, the storage system records the physical address and status of the current data and generates a pointer to this data, which is the snapshot. Copy-On-Write (COW): When a user writes to a snapshot, the storage system first checks the original location of the data being written. If the data has not been modified (that is, it still points to the data when the snapshot was created), the storage system will first copy the data to a new location, and then update the pointer in the snapshot to point to the new data location. In this way, the original data will not be overwritten, and the snapshot will contain the latest write operation results.

[0059] Cloud file systems, such as JuiceFS, support writable snapshots. Figure 4 As shown in the figure, multiple branch snapshots can be split from the same baseline snapshot. The baseline snapshot is also called the main branch snapshot. When there is no write operation, all read operations can be read directly from the baseline snapshot. When a write operation is performed, the corresponding data page (or data block) in the baseline snapshot is copied and written to this copy (branch snapshot). Based on COW technology, when creating a new snapshot based on a snapshot version, you only need to copy the metadata of the old snapshot (that is, the data that marks the location of all the old snapshot data pages) to split a new snapshot based on this metadata copy. Since the metadata is very small, snapshots can be achieved in seconds.

[0060] Based on the above writable snapshot technology, a database instance can be created quickly (in seconds) without copying data files (such as table space data).

[0061] This embodiment also proposes a local cache technology to improve file writing performance. Figure 5 As shown in the figure, when writing a file, the data is not directly synchronized to the disk, but is first written to the operating system cache to speed up the write operation, and then the data file is asynchronously stored in the local file system of the local disk, and then asynchronously persisted in the cloud file system (cloud storage). Therefore, between the application and the cloud storage, there are two levels of local cache: the local cache and the local disk.

[0062] The cloud file system also supports a striped storage mechanism, which helps improve data read / write performance. In the cloud file system, data files are composed of several large chunks, such as 64MB. When the file size is not an integer multiple of 64MB, the last chunk may be less than 64MB. A write operation on a file is composed of several slices, and a slice cannot span across chunks. In other words, a write operation spans as many slices as the number of chunks. Figure 6 As shown in the figure, an empty file is appended with 160MB of data, which takes up about 2.5 slices. In actual storage, a slice is divided into several small blocks, such as 4MB, and these blocks are stored in the object storage. The read / write operations of multiple block objects can be completed in parallel, which can fully utilize the network up / downlink bandwidth and improve the read / write efficiency.

[0063] Based on striped storage, cloud file system also supports "pre-reading" technology. Figure 6 As shown in the figure, when a block is read, other adjacent blocks are also read and asynchronously cached locally. This can minimize the latency of continuous read operations and improve performance.

[0064] For databases such as MySQL, as long as the logs (Write Ahead Log (WAL) and physical logs) and metadata are persisted, the transaction can be confirmed to be successful, because data files (such as tablespace files) can be repaired through log replay. In addition, because log writing is sequential writing (that is, write ahead) and can achieve streaming (pipelining) replication, the performance of log persistence is much higher than that of data files.

[0065] Metadata is data about data, mainly information describing data properties, used to support functions such as indicating storage location, historical data, resource search, and file records.

[0066] Physical logs, such as redo logs, record the physical modification information of data pages, that is, the actual data changes. It is mainly used to ensure the persistence and atomicity of transactions. When a system failure occurs, the physical log can be used to recover uncommitted transaction changes to ensure data integrity.

[0067] Logical logs (pre-written logs), such as binary logs (binlogs), record the logical changes of SQL (Structured Query Language) statements, that is, the specific SQL operations performed. They are mainly used for data replication, recovery, and auditing.

[0068] Based on the aforementioned cloud file system, a database system can be built. The database system includes a master database, a slave database, a read-only database, and a cloud file system. The master database, the slave database, and the read-only database are responsible for computing and processing various transactions, and the cloud file system is responsible for storage, thereby realizing a database system with separated storage and computing. Figure 7 .

[0069] For each database: the master database is used to handle various client transactions, including but not limited to write transactions, read transactions, etc. Of course, if the master database is under a heavy load, you can also configure the master database to handle write transactions, and read transactions to other databases; the slave database is a slave database of the master database, which is used to back up the data of the master database, and when the master database fails or has other abnormalities, it switches to the master database and performs the functions of the master database; the read-only database is a slave database of the slave database, which synchronizes data with the slave database and can be used to handle client read transactions. If there are many read transactions, you can add a read-only database, and use multiple read-only databases to deal with a large number of read transactions.

[0070] The main database is introduced below.

[0071] The data directory of the master database can be stored in a cloud file system (such as JuiceFS). Based on the data directory of the master database, a writable snapshot of the master database is created. Based on the latest writable snapshot of the master database, a writable snapshot of the slave database can be created, or a writable snapshot of the read-only database can be created. The branch where the writable snapshot of the master database is located is called the "master branch" (i.e., Original branch).

[0072] Under normal circumstances, the metadata and log data of the master database are only stored in the local cache (also called local file cache), and do not need to be synchronized to the cloud file system (i.e. cloud storage). Asynchronous persistence to cloud storage is required only when the local cache of the master database overflows, or the local cache has not been synchronized to the cloud for a long time and synchronization is required.

[0073] The master database needs to be backed up periodically. During the backup, all local caches of the master database are persisted to cloud storage, and the latest writable snapshot of the master database (that is, the latest snapshot of the master branch) is created based on the current data status. Snapshot creation can be completed in seconds and is only performed at the distributed file system layer (cloud file system), which does not increase the burden on the master database.

[0074] The data files of the primary database (such as tablespace data (.ibd)) can be directly uploaded to the cloud storage without being cached in the local file system. Alternatively, if the local storage specification of the primary database is large, part of the data files can be cached locally. When the local cache space overflows, the old cache data can be eliminated and synchronized to the cloud storage, thereby accelerating reading and writing.

[0075] The master database can still be synchronized with the slave database through the master-slave replication mechanism, and there is no need to establish a master-slave replication connection between the master database and the read-only database, thereby saving the network bandwidth of the master database.

[0076] The following describes the slave database.

[0077] The metadata and log data of the slave database are stored in the local cache and do not need to be synchronized to the cloud file system (i.e., cloud storage) under normal circumstances.

[0078] For database data files (such as tablespace data (.ibd)), one way is to not cache them in the local file system at all and upload them directly to the cloud file system's own writable snapshot. Another way is that if the local storage specification of the database is large, part of the data file can be cached locally and asynchronously persisted to the cloud file system's own writable snapshot.

[0079] The log data of the slave database (such as binary logs and other logical logs) is first written to the local cache and asynchronously synchronized to the cloud storage periodically. The binlog is uploaded to the cloud storage by the slave database, which can save the bandwidth of the master database and reduce the burden on the master database.

[0080] When the difference between the slave database and the master database is too large (that is, the slave database writable snapshot occupies too much space on the cloud storage), you can free up this part of the storage space on the cloud storage by rebuilding the slave database. You can trigger the backup of the master database in a suitable time window (that is, create the latest snapshot of the master database), and rebuild the slave database based on the latest snapshot. For example, based on the latest snapshot of the master database, create a new writable snapshot, and start a new slave database on this new writable snapshot. The new slave database starts master-slave replication with the master database for data synchronization. At this time, the old slave database and its writable snapshot can be recycled, thereby freeing up storage space.

[0081] The following describes a read-only database.

[0082] Based on the latest snapshot of the master database, N writable snapshots can be created. Based on the N writable snapshots, N read-only databases can be started, and each read-only database corresponds to a writable snapshot.

[0083] When deactivating a read-only database, simply release the writable snapshot corresponding to the read-only database.

[0084] Under normal circumstances, all files of the read-only database are stored in the local cache, including metadata, log data, data files (such as tablespace data files), etc. When there is too much local cache data and the local cache overflows, the data files will be uploaded to the cloud file system's own writable snapshot. Since the read-only database has a relatively short duration, it will not be written to the cloud storage most of the time.

[0085] The read-only database does not directly connect to the master database, but synchronizes data by subscribing to the log files (such as binlog) in the writable snapshot where the slave database is located.

[0086] The following describes the database system management method, wherein the database system management content involves, for example, data synchronization between databases, creation / release of read-only databases, reconstruction / release of slave databases, master / slave database switching, write transaction processing, read transaction processing, etc. These are introduced below.

[0087] Figure 8 The following is a schematic diagram showing a database system management method according to some embodiments of the present disclosure, in particular, a method for synchronizing data between databases. Figure 8 As shown, the method of this embodiment includes the following steps.

[0088] In step 810, a database directory (ie, a master branch) is created and initialized, and based on the initialized database directory, a writable snapshot of the slave database is created in the cloud file system.

[0089] In step 820, the slave database is started based on the writable snapshot of the slave database, and the master database and the slave database are synchronized with each other through replication.

[0090] The master database synchronizes the data that needs to be synchronized, such as log data, to the slave database through replication protocols, such as the binlogdump command. After the slave database stores the received log data locally in the form of relay logs, it can return a confirmation message to the master database without having to complete the log replay operation. This speeds up the data synchronization process. The slave database can then complete the log replay operation at the appropriate time, usually periodically, without having to complete it at the same time as the relay log.

[0091] At step 830 , the slave database synchronizes the locally cached log data to a writable snapshot of the slave database in the cloud file system.

[0092] The slave database stores the log data obtained by replaying the logs of the master database in the local cache, and asynchronously synchronizes the locally cached log data to the writable snapshot of the slave database in the cloud file system. The slave database uploads the log data to the cloud storage, which can reduce the burden on the master database.

[0093] In step 840 , the read-only database subscribes to the log data in the writable snapshot of the slave database in the cloud file system, and performs data synchronization with the slave database.

[0094] In this embodiment, data synchronization is performed between the master database and the slave database by replication. Each read-only database and the slave database perform data synchronization based on log publishing / subscription using the cloud file system as the medium. The master database does not need to synchronize the read-only database, which effectively alleviates the pressure on the master database, saves the network uplink bandwidth of the master database, and improves the overall service performance of the database system.

[0095] Fig. 9 The following is a schematic diagram showing a database system management method according to some embodiments of the present disclosure, particularly a schematic diagram of creating / releasing a read-only database. Fig. 9 As shown, the method of this embodiment includes the following steps, wherein steps 910-930 are a read-only database creation process, and step 940 is a read-only database release process.

[0096] At step 910 , a writable snapshot of the read-only database is created in the cloud file system based on the latest snapshot of the primary database.

[0097] In step 920, based on the writable snapshot of the read-only database, the database instance corresponding to the read-only database is started, and the data files are configured to be preferentially stored in the local cache of the read-only database, so that the read-only database can quickly provide data reading services to the client based on the local cache.

[0098] In step 930 , the read-only database subscribes to the log data in the writable snapshot of the slave database in the cloud file system, and performs data synchronization with the slave database.

[0099] The subscriber of the read-only database obtains new log data from the writable snapshot of the slave database in the cloud file system, parses the log data into SQL, and after executing these SQLs, the corresponding SQL operations are replayed to the read-only database.

[0100] Based on second-level snapshots, a new read-only database can be created quickly (in seconds) to quickly meet the needs of business growth.

[0101] In step 940, in response to the read-only database meeting the preset release condition, the database instance corresponding to the read-only database is stopped, and the writable snapshot of the read-only database is deleted, thereby quickly (in seconds) releasing a read-only database.

[0102] This embodiment enables the read-only database to have the ability to scale in seconds, and can quickly meet the needs of changes in business volume.

[0103] Fig.10 The following is a schematic diagram showing a database system management method according to some embodiments of the present disclosure, particularly a method for rebuilding / releasing a database. Fig.10 As shown, the method of this embodiment includes the following steps, wherein steps 1010-1050 are a process of rebuilding from a database, and step 1060 is a process of releasing from a database.

[0104] In step 1010, in response to the slave database satisfying a preset reconstruction condition, a writable snapshot of the first database is created in the cloud file system based on the latest writable snapshot of the master database, so as to rebuild the first database into a new slave database.

[0105] The reconstruction conditions include, for example, that the slave database cannot be started before the reconstruction, or that the storage space of the cloud file system occupied by the writable snapshot of the slave database before the reconstruction exceeds a preset value, etc. The larger the storage space of the cloud file system occupied by the writable snapshot of the slave database, the greater the difference between the slave database and the master database. Therefore, the preset value of the storage space of the cloud file system occupied by the writable snapshot of the slave database often indicates that the difference between the slave database and the master database is too large, and this problem can be solved by rebuilding the slave database.

[0106] In step 1020, based on the writable snapshot of the first database, the database instance corresponding to the first database is started, and the metadata and log data are configured to be preferentially stored in the local cache of the first database, and then stored in the cloud file system when overflow or synchronization occurs, and data files such as table space data are stored in the cloud file system.

[0107] In step 1030, the first database subscribes to the log data in the writable snapshot of the slave database until the latest log data is subscribed and then stops subscribing, so that the log data of the first database is consistent with that of the original slave database.

[0108] In step 1040, the primary database and the first database are synchronized with each other through replication.

[0109] The master database synchronizes the data that needs to be synchronized, such as log data, to the first database through replication protocols, such as the binlogdump command. After the first database stores the received log data locally in the form of relay logs, it can return a confirmation message to the master database without having to complete the log replay operation. This speeds up the data synchronization process. The first database can then complete the log replay operation at an appropriate time, usually periodically, without having to complete it at the same time as the relay log.

[0110] In step 1050, the first database may be set to read-only, and the first database may be used as a rebuilt slave database.

[0111] This completes the reconstruction of the slave database. The new slave database is consistent with the latest snapshot content of the master database and occupies a relatively small storage space of the cloud file system.

[0112] In step 1060, the database instance corresponding to the slave database before reconstruction is stopped, and the writable snapshot of the slave database before reconstruction is deleted, so as to quickly release the original slave database.

[0113] Fig.11 The following is a schematic diagram showing a database system management method according to some embodiments of the present disclosure, particularly a master / slave database switching method. Fig.11 As shown, the method of this embodiment includes the following steps 1110-1140.

[0114] In step 1110, the slave database replays the received log of the master database and stores it in the local cache, and persistently stores the locally cached data in the cloud file system, specifically, persistently stores it in the corresponding writable snapshot of the slave database in the cloud file system.

[0115] Log replay refers to parsing log data into SQL statements. After executing these SQL statements, the corresponding SQL operations are replayed.

[0116] In step 1120, in the cloud file system, a latest writable snapshot is created based on the writable snapshot of the slave database, so that a master branch snapshot can be subsequently formed based on the latest directory.

[0117] In step 1130, the slave database is set to be writable and switched to the master database, so that the slave database takes over the work of the master database and can process write transactions.

[0118] In step 1140, in the cloud file system, the writable snapshot of the master database (original master database) before the master-slave switch is deleted, and the latest writable snapshot of the slave database (new master database) before the master-slave switch is switched to the master branch snapshot.

[0119] Thus, the switch from the slave database to the master database is completed.

[0120] After the slave database is switched to the master database, the slave database can be rebuilt based on the writable snapshot of the current master database (the new master branch snapshot). The specific process is as follows: (1) In the cloud file system, based on the latest writable snapshot of the primary database after the master-slave switch, a writable snapshot of the second database is created, so as to rebuild the second database into a slave database; (2) Based on the writable snapshot of the second database, start the database instance corresponding to the second database, and configure the metadata and log data to be stored in the local cache of the second database first, and then stored in the cloud file system when overflow or synchronization occurs; (3) The second database subscribes to the log data in the writable snapshot of the primary database after the master-slave switch until it stops subscribing to the latest log data, so that the log data of the second database is consistent with that of the new primary database; (4) Through replication, the primary database after the master-slave switch is synchronized with the second database. The synchronization method can refer to the aforementioned data synchronization between the primary database and the first database, which will not be repeated here; (5) The second database can be set to read-only and used as a rebuilt slave database.

[0121] Therefore, after the slave database is switched to the master database, the slave database can be rebuilt based on the writable snapshot of the current master database (new master branch snapshot) to complete the reconstruction of the slave database. The new slave database is consistent with the latest snapshot content of the master database and occupies a relatively small storage space of the cloud file system.

[0122] Fig.12 The following is a schematic diagram showing a database system management method according to some embodiments of the present disclosure, particularly a transaction processing diagram of a database system. Fig.12 As shown, the method of this embodiment includes the following steps 1210-1250.

[0123] In step 1210, the client submits a transaction to the primary database. The primary database generally processes write transactions, and may also process read transactions.

[0124] Step 1220 , the transaction processing process of the primary database stores the log corresponding to the transaction in a local cache, such as a page cache, and then synchronizes a set of logs corresponding to the transaction from the local cache to a local disk.

[0125] Step 1230, the master database transfers the log to the slave database through master-slave replication. The slave database can return a confirmation message after storing the log in the form of a relay log, without having to complete the log replay operation and then return the confirmation message, thereby improving transaction processing efficiency.

[0126] Step 1240: The primary database receives a confirmation message returned by the secondary database, confirming that the transaction is successfully committed.

[0127] Step 1250, after receiving the confirmation message returned from the database, the transaction processing process of the primary database returns the successful transaction submission to the client without having to wait for the primary database to synchronize the changes of the data file (such as the table space file ".ibd") to the local disk, thereby improving transaction processing efficiency.

[0128] The transaction processing mechanism of this embodiment has taken a variety of measures to improve transaction processing efficiency, and will not cause a significant performance drop due to data storage in the cloud. In particular, for write transactions, it can achieve write performance close to that under a master database and a slave database architecture. Specifically, as a slow storage, cloud storage only provides data storage capacity and elasticity in this embodiment; the submission and synchronization of write transactions (i.e., binlog file writing and replication) are still based on the master-slave replication mode, that is, the performance of write transactions still depends on the local disk writing performance and the transmission efficiency of the master-slave replication protocol, and the update of data files (such as tablespace files ".ibd") is written to the cloud file system asynchronously, which does not affect the speed of write transaction submission.

[0129] In addition, the database system of this embodiment is very advantageous in terms of storage space occupancy rate. A comparative analysis is performed below.

[0130] Based on the operational experience of cloud databases, the normal capacity of database instances is generally less than 50% of the maximum capacity. For a database instance with a maximum capacity of 1TB and a normal storage of 300GB, under the traditional master / slave database synchronization solution, two containers or cloud hosts with a storage specification of 1TB are required to deploy a master database and a slave database respectively. Regardless of how much data is actually stored in the database, the total storage space occupied is 2TB.

[0131] Under the premise of the same specifications, 300GB of data is stored in a 3-copy object storage, and 20% of the incremental space is reserved for the writable snapshot of the slave database (that is, the difference between the writable snapshot of the slave database and the master database), which actually occupies 300GB×1.2×3 = 1TB of space. Considering that cloud storage needs to reserve a certain amount of space (based on the operational experience of cloud storage, the overall storage space utilization rate of an object storage is generally above 75%), the total space occupied is 1TB / 0.75 = 1.3TB. Under normal circumstances, the two database nodes, one master and one slave, reserve 20% of the local file cache respectively, that is, 300GB×0.2×2= 120GB. Therefore, under normal circumstances, the total storage space occupied by the database instance is between 1.4TB and 1.5TB, which saves more storage capacity than the traditional master / slave database system. In addition, the unit price of object storage is lower than that of local disks, and it can also be elastically expanded according to the actual amount of data stored in the warehouse, so the cost of the first purchase is lower.

[0132] Those skilled in the art should understand that the embodiments of the present disclosure may be provided as methods, systems, (non-transitory) storage media or computer program products. The storage medium stores computer instructions, which, when executed by a processor, implement the steps of the database system management method of each embodiment. The computer program product includes computer instructions, which, when executed by a processor, implement the steps of the database system management method of each embodiment. A computer program product should be understood as a software product that implements its solution mainly through a computer program. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0133] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A database system management method, characterized in that: include: Create a database directory and initialize it. Based on the initialized database directory, create a writable snapshot of the slave database in the cloud file system. Through replication, the master database and the slave database synchronize data; The slave database synchronizes the locally cached log data to a writable snapshot of the slave database in the cloud file system; The read-only database subscribes to the log data in the writable snapshot of the slave database in the cloud file system and synchronizes data with the slave database.

2. The database system management method according to claim 1, characterized in that: Also includes: Before the read-only database is subscribed, a writable snapshot of the read-only database is created in the cloud file system based on the latest snapshot of the master database; Based on the writable snapshot of the read-only database, start the database instance corresponding to the read-only database, and configure the data files to be stored preferentially in the local cache of the read-only database.

3. The database system management method according to claim 2, characterized in that: Also includes: In response to the read-only database satisfying a preset release condition, a database instance corresponding to the read-only database is stopped, and a writable snapshot of the read-only database is deleted.

4. The database system management method according to claim 1, characterized in that: The slave database synchronizes the locally cached log data to the writable snapshot of the slave database in the cloud file system, including: The slave database stores the log data obtained by replaying the logs of the received master database in the local cache, and asynchronously synchronizes the locally cached log data to the writable snapshot of the slave database.

5. The database system management method according to claim 1, characterized in that: Also includes: In response to the slave database satisfying a preset reconstruction condition, creating a writable snapshot of the first database in the cloud file system based on the latest writable snapshot of the master database; Based on the writable snapshot of the first database, start the database instance corresponding to the first database, and configure the metadata and log data to be preferentially stored in the local cache of the first database; Control the first database to subscribe to the log data in the writable snapshot of the slave database until the subscription reaches the latest log data and stops the subscription; The primary database synchronizes data with the first database through replication; Use the first database as the rebuilt slave database.

6. The database system management method according to claim 5, characterized in that: Also includes: Stop the database instance corresponding to the slave database before reconstruction, and delete the writable snapshot of the slave database before reconstruction.

7. The database system management method according to claim 5, characterized in that: The reconstruction condition includes at least one of the following: the slave database before reconstruction cannot be started or the storage space of the cloud file system occupied by the writable snapshot of the slave database before reconstruction exceeds a preset value.

8. The database system management method according to claim 1, characterized in that: Also includes: The slave database replays the received logs of the master database and stores them in the local cache, and persistently stores the locally cached data in the cloud file system; In the cloud file system, creating the latest writable snapshot based on the writable snapshot of the slave database; The slave database is set to be writable and switched to the master database; In the cloud file system, delete the writable snapshot of the master database before the master-slave switch, and switch the latest writable snapshot of the slave database before the master-slave switch to the master branch snapshot.

9. The database system management method according to claim 8, characterized in that: Also includes: In the cloud file system, based on the latest writable snapshot of the primary database after the master-slave switch, a writable snapshot of the secondary database is created; Based on the writable snapshot of the second database, start the database instance corresponding to the second database, and configure the metadata and log data to be preferentially stored in the local cache of the second database; The second database subscribes to the log data in the writable snapshot of the primary database after the master-slave switch, and stops subscribing when it subscribes to the latest log data; Through replication, the primary database after the master-slave switch is synchronized with the secondary database; Use the second database as the rebuilt slave database.

10. The database system management method according to claim 1, characterized in that: Also includes: The master database and / or slave database stores metadata and log data in the local cache first, and then stores them in the cloud file system when overflowing or synchronizing, and stores table space data in the cloud file system.

11. The database system management method according to claim 1, characterized in that: Also includes: In response to the transaction submitted by the client, the master database stores the corresponding log of the transaction in the local cache and synchronizes it to the local disk, sends the corresponding log of the transaction to the slave database, receives the confirmation message returned by the slave database after storing the log in the form of a relay log, confirms that the transaction was successfully submitted, and returns the successful transaction submission to the client.

12. A database system, characterized in that: include: A cloud file system is configured to store a writable snapshot of a master database, a writable snapshot of a slave database, and a writable snapshot of a read-only database, wherein the writable snapshot of the slave database and the writable snapshot of the read-only database are created based on the writable snapshot of the master database; The master database and the slave database are configured to synchronize data through replication; The slave database is configured to synchronize the locally cached log data to a writable snapshot of the slave database in the cloud file system; A read-only database that is configured to subscribe to log data in the writable snapshot of the slave database in the cloud file system, synchronize data with the slave database, and process read transactions.

13. The database system according to claim 12, characterized in that: The master database or / and the slave database are configured to store metadata and log data in the local cache first, and then store them in the cloud file system when overflowing or syncing, and store table space data in the cloud file system; The read-only database is configured to store data files in the local cache first, and then store them in the cloud file system when overflow occurs.

14. The database system according to claim 12, characterized in that: The master database is configured to respond to the transaction submitted by the client, store the corresponding log of the transaction in the local cache and synchronize it to the local disk, send the corresponding log of the transaction to the slave database, receive the confirmation message returned by the slave database after storing the log in the form of relay log, confirm that the transaction is successfully submitted, and return the successful transaction submission to the client.

15. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the database system management method described in any one of claims 1 to 11 are implemented.

16. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the database system management method described in any one of claims 1 to 11 are implemented.

Images