Maintenance Method, Device, Computer Equipment and Storage Medium of TiDB Database

Abstract

The present invention discloses a maintenance method, device, computer equipment and storage medium for a TiDB database. The method comprises: collecting data in a TiDB production library through a TiDB-Binlog cluster component, obtaining a binlog file corresponding to the data in the TiDB production library, synchronizing the binlog file to a TiDB local disaster recovery library, and then using a preset scheduled backup script to synchronize the binlog file in the TiDB local disaster recovery library to a flash NAS device. When an abnormality in the TiDB production library is monitored, data restoration processing is performed on the TiDB production library based on the binlog file stored in the flash NAS device, thereby avoiding the problem of slow backup and restoration speed caused by directly saving the data in the production library in an archive tape in the traditional method, and improving the efficiency of TiDB database maintenance.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a TiDB database maintenance method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of the economy, more and more corporate organizations have expanding their business scope and user groups. Many companies have begun to use distributed databases for data storage in order to improve their business processing capabilities. TiDB is an open source distributed relational database with high security, compatibility with MySQL protocols and ecology, easy migration, and extremely low operation and maintenance costs, making it a relatively popular distributed database.

[0003] When the database is processing transactions, data backup is required so that in the event of a failure, data recovery can be performed based on the backup files. The existing TiDB database uses archive tapes for data backup, which has weak IO transmission capabilities. TiDB databases often have large amounts of data, which makes data backup and recovery time longer, resulting in low database maintenance efficiency. Summary of the invention

[0004] Embodiments of the present invention provide a TiDB database maintenance method, apparatus, computer device, and storage medium to improve the maintenance efficiency of the TiDB database.

[0005] In order to solve the above technical problems, the present application embodiment provides a TiDB database maintenance method, including:

[0006] Collect data from the TiDB production database through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production database, and synchronize the binlog file to the TiDB local disaster recovery database;

[0007] Use the preset scheduled backup script to synchronize the binlog files in the TiDB local disaster recovery database to the flash NAS device;

[0008] If an abnormality is detected in the TiDB production database, data restoration is performed on the TiDB production database based on the binlog file stored in the flash NAS device.

[0009] Optionally, the TiDB-Binlog cluster component includes a Drainer component and a pump component. The collecting data in the TiDB production database through the TiDB-Binlog cluster component, obtaining a binlog file corresponding to the data in the TiDB production database, and synchronizing the binlog file to the TiDB local disaster recovery database includes:

[0010] The pump component collects the binglog file of the TiDB node server corresponding to the pump component as a basic file, and sends the basic file to the Drainer component;

[0011] The basic files are received by the Drainer component and merged to obtain a merged binglog file;

[0012] Synchronize the merged binglog file to the TiDB local disaster recovery database.

[0013] Optionally, the flash NAS device includes a first flash device and a second flash device.

[0014] Optionally, the using a preset scheduled backup script to synchronize the binlog file in the TiDB local disaster recovery database to the flash NAS device includes:

[0015] Move and save the binlog file in the TiDB local disaster recovery database to the first flash memory device;

[0016] Regularly performing incremental and full difference verification on the first flash memory device and the second flash memory device to obtain a verification result;

[0017] If the verification result shows that there is a difference, the data in the second flash memory device is updated using the data in the first flash memory device according to the difference in the verification result.

[0018] Optionally, moving and saving the binlog file in the TiDB local disaster recovery repository to the first flash memory device includes:

[0019] Use the binlog file in the TiDB local disaster recovery database as the data to be imported, and obtain the storage path of the first flash memory device corresponding to the data to be imported;

[0020] Determining the number of target threads based on the data to be imported and the first flash memory device;

[0021] Based on the target number of threads, multiple threads are constructed, and the multiple threads are used to synchronize the data to be imported to the storage path of the first flash memory device.

[0022] Optionally, it is characterized in that if the TiDB production database is monitored to be abnormal, then based on the binlog file stored in the flash NAS device, performing data restoration processing on the TiDB production database includes:

[0023] Stop the application service from writing tables to the TiDB production database, and clear the data in the data table of the TiDB production database;

[0024] Obtaining the full backup file and the additional backup file from the flash NAS device as files to be restored;

[0025] Load the full backup file into the data table in the TiDB production database, and then load the additional backup file into the data table in the TiDB production database;

[0026] Verify the data in the data table in the TiDB production library. After the verification passes, confirm that the data restoration is successful and start the application service.

[0027] In order to solve the above technical problems, the embodiment of the present application also provides a maintenance device for a TiDB database, including:

[0028] The data collection module is used to collect data from the TiDB production database through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production database, and synchronize the binlog file to the TiDB local disaster recovery database;

[0029] A file synchronization module, used to synchronize the binlog files in the TiDB local disaster recovery database to the flash NAS device using a preset scheduled backup script;

[0030] The data recovery module is used to restore the data of the TiDB production database based on the binlog file stored in the flash NAS device if an abnormality is detected in the TiDB production database.

[0031] Optionally, the data acquisition module includes:

[0032] A collection unit, configured to collect the binglog file of the TiDB node server corresponding to the pump component through the pump component as a basic file, and send the basic file to the Drainer component;

[0033] A merging unit, configured to receive the basic files through the Drainer component and merge the basic files to obtain a merged binglog file;

[0034] A synchronization unit is used to synchronize the merged binglog file to the TiDB local disaster recovery database.

[0035] Optionally, the flash NAS device includes a first flash memory device and a second flash memory device, and the file synchronization module includes:

[0036] A synchronization unit, configured to move and save the binlog files in the TiDB metropolitan disaster recovery database to the first flash device;

[0037] A verification unit, configured to perform incremental and full-scale differential verification on the first flash device and the second flash device at regular intervals to obtain a verification result;

[0038] An update unit, configured to, if there are differences in the verification result, update the data in the second flash device with the data in the first flash device according to the differences in the verification result.

[0039] Optionally, the synchronization unit includes:

[0040] A path acquisition subunit, configured to use the binlog files in the TiDB metropolitan disaster recovery database as data to be imported, and acquire the storage path of the first flash device corresponding to the data to be imported;

[0041] A thread number determination subunit, configured to determine the number of target threads based on the data to be imported and the first flash device;

[0042] A multi-thread synchronization subunit, configured to build multi-threads based on the number of target threads, and use the multi-threads to synchronize the data to be imported to the storage path of the first flash device.

[0043] Optionally, the data recovery module includes:

[0044] A data cleaning unit, configured to stop the write table process of the application service for the TiDB production database, and empty the data in the data tables of the TiDB production database;

[0045] A file to be restored determination unit, configured to obtain a full backup file and an incremental backup file from the flash NAS device as files to be restored;

[0046] A file restoration unit, configured to load the full backup file into the data tables in the TiDB production database, and then load the incremental backup file into the data tables in the TiDB production database;

[0047] A data verification unit, configured to verify the data in the data tables in the TiDB production database. After the verification is passed, confirm that the data restoration is successful, and start the application service.

[0048] To solve the above technical problems, an embodiment of the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above TiDB database maintenance method are implemented.

[0049] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned TiDB database maintenance method.

[0050] The TiDB database maintenance method, device, computer equipment and storage medium provided in the embodiments of the present invention collect data in the TiDB production library through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production library, and synchronize the binlog file to the TiDB local disaster recovery library, and then use the preset scheduled backup script to synchronize the binlog file in the TiDB local disaster recovery library to the flash NAS device. When the TiDB production library is monitored to be abnormal, the TiDB production library is restored based on the binlog file stored in the flash NAS device, avoiding the problem of slow backup and restore speed caused by the traditional method of directly saving the data in the production library in the archive tape, thereby improving the efficiency of database maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0052] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0053] Figure 2 is a flowchart of an embodiment of the TiDB database maintenance method of the present application;

[0054] Figure 3 is a schematic diagram of the structure of an embodiment of a maintenance device for a TiDB database according to the present application;

[0055] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0057] References herein to "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase does not necessarily refer to the same embodiment each time it appears in the specification, nor is it an independent or alternative embodiment mutually exclusive of other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0058] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] Please refer to Figure 1 As shown in Figure 1 , the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0060] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc.

[0061] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture E interface display perts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture E interface display perts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc.

[0062] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0063] It should be noted that the TiDB database maintenance method provided in the embodiment of the present application is executed by a server, and accordingly, the TiDB database maintenance device is set in the server.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 101, 102, 103 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0065] See also Figure 2 , Figure 2 A TiDB database maintenance method provided by an embodiment of the present invention is shown. The method is applied in Figure 1 The server in is used as an example to illustrate the details as follows:

[0066] S201: Collect data from the TiDB production database through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production database, and synchronize the binlog file to the TiDB local disaster recovery database.

[0067] Among them, the TiDB production database is used in the production environment and is based on the TiDB open source distributed relational database. TiDB is a hybrid database product positioned for online transaction processing / online analytical processing (HTAP), which realizes one-click horizontal scaling, strong consistency of multi-copy data security, distributed transactions, real-time OLAP and other important features. At the same time, it is compatible with the MySQL protocol and ecology, easy to migrate, and has extremely low operation and maintenance costs.

[0068] Specifically, this embodiment includes several TiDB node servers. During operation, the TiDB node servers will update data (including but not limited to addition, deletion, modification, query and backup, etc.). Through the TiDB-Binlog cluster component, the updated data in each TiDB node server is obtained, and the corresponding binlog files are generated according to the updated data, and the binlog files are synchronized to the TiDB local disaster recovery library.

[0069] Among them, TiDB local disaster recovery database refers to the backup database in the same region based on TiDB.

[0070] The binlog file is a binary log file used to record the SQL statements used by users to update the database. For example, SQL statements for changing database tables and content will be recorded in the binlog, but queries on database tables and other content will not be recorded. By default, the binlog file is in binary format and cannot be viewed using commands for viewing text tools (such as cat, vi, etc.). Instead, it can be viewed using mysqlbinlog parsing.

[0071] It should be noted that when data is written to the database, the updated SQL statements will also be written to the corresponding binlog file. When using dump backup, only the data for a period of time is fully backed up.

[0072] S202: Use the preset scheduled backup script to synchronize the binlog files in the TiDB local disaster recovery database to the flash NAS device.

[0073] Existing external storage is divided into direct-attached storage (DAS) and fabric-attached storage (FAS) according to the connection method; network-attached storage is further divided into network-attached storage (NAS) and storage area network (SAN) according to the transmission protocol.

[0074] Among them, the flash NAS device is a device connected to the network with data storage function, so it is also called "network storage". It is a special dedicated data storage server, including storage devices (such as disk arrays, CD / DVD drives, tape drives or removable storage media) and embedded system software, which can provide cross-platform file sharing function. NAS usually occupies its own node on a LAN, without the intervention of an application server, allowing users to access data on the network. In this configuration, NAS centrally manages and processes all data on the network, unloads the load from the application or enterprise server, effectively reduces the total cost of ownership, and protects the user's investment. NAS itself can support multiple protocols (such as NFS, CIFS, FTP, HTTP, etc.), and can support various operating systems. Through any workstation, the NAS device can be intuitively and conveniently managed using an IE or Netscape browser. Its cost is much lower than using server storage, while its efficiency is much higher than the latter.

[0075] The NAS device is a truly plug-and-play product, so it is also called a flash NAS device. It generally supports multiple computer platforms. Users can access the same documents through network support protocols. Therefore, the NAS device can be used in a mixed Unix / Windows NT local area network without modification. At the same time, the application of the NAS device is very flexible. Compared with other external storage methods, a key problem with using a flash NAS device is the bandwidth consumption during the backup process. Different from the storage area network (SAN) that transfers the backup data stream out of the LAN, NAS still uses the network for backup and recovery. Therefore, data backup and recovery can be performed in a multi-threaded manner to improve the efficiency of data recovery and backup.

[0076] Among them, the preset timed backup script is used to synchronize the binlog file at regular intervals. For example, through the timed backup script, at 2 o'clock every day, the binlog files created within the previous day in the drainer are transferred to the flash NAS device. The types of the preset timed backup script include but are not limited to: shell script, JavaScript script, Lua script, python script, etc.

[0077] In the existing TiDB database backup method, when a production database fails, it is necessary to restore the full backup and incremental backup files to the local server through an archival tape and then import them into the database. Since the full backup files of the database that need to be restored from the archival tape are generally relatively large (the database scale varies, generally from dozens of gigabytes to several terabytes), the tape I / O capacity is weak, and it takes a long time to obtain the restored files, resulting in an extended application stop time and seriously affecting the continuous operation of the business system. In this embodiment, a flash NAS device is used for data backup, which can greatly improve the data backup and recovery speed and the maintenance efficiency of the TiDB database.

[0078] It should be noted that the flash NAS device is not suitable for permanently storing data. In this embodiment, after moving and saving the binlog file to the flash NAS device, the data stored in the flash NAS device is then synchronized to the archival tape so that the data can be saved and read offline.

[0079] Among them, the archival tape, also known as magnetic tape storage, is a storage device with a magnetic tape as the storage medium, consisting of a tape drive and its controller, and is an auxiliary memory of a computer. The tape drive consists of a tape drive mechanism and a magnetic head, etc., which can drive the tape to move relative to the magnetic head, perform electromagnetic conversion with the magnetic head, and sequentially record or read data on the magnetic tape. The magnetic tape storage is one of the computer peripheral devices. The tape controller is a control circuit device used by the central processing unit to access data on the tape drive. The magnetic tape storage accesses data in a sequential manner. The magnetic tape storing data can be saved offline and read interchangeably.

[0080] S203: If it is monitored that the TiDB production database is abnormal, then based on the binlog file stored in the flash NAS device, data restoration processing is performed on the TiDB production database.

[0081] Specifically, the server is configured with application log monitoring and database performance metric monitoring, which monitor various data in the TiDB production database in real time. When it is monitored that some data in the TiDB production database exceeds the normal preset range, it is confirmed that the TiDB production database is abnormal. At this time, an alarm prompt will be sent via email or phone, and data restoration processing will be performed through the binlog file stored in the flash NAS device.

[0082] Among them, for the specific process of performing data restoration processing through the binlog file stored in the flash NAS device, reference can be made to the description of subsequent embodiments. To avoid repetition, it will not be elaborated here.

[0083] In this embodiment, the data in the TiDB production library is collected through the TiDB-Binlog cluster component to obtain the binlog file corresponding to the data in the TiDB production library, and the binlog file is synchronized to the TiDB local disaster recovery library. Then, the preset scheduled backup script is used to synchronize the binlog file in the TiDB local disaster recovery library to the flash NAS device. When the TiDB production library is monitored to be abnormal, the TiDB production library is restored based on the binlog file stored in the flash NAS device, avoiding the problem of slow backup and restore speed caused by the traditional method of directly saving the data in the production library in the archive tape, thereby improving the efficiency of database maintenance.

[0084] In some optional implementations of this embodiment, in step S201, the TiDB-Binlog cluster component includes a Drainer component and a pump component. Collecting data in the TiDB production database through the TiDB-Binlog cluster component, obtaining a binlog file corresponding to the data in the TiDB production database, and synchronizing the binlog file to the TiDB local disaster recovery database includes:

[0085] Use the pump component to collect the binglog files of the TiDB node server corresponding to the pump component as the basic files, and send the basic files to the Drainer component;

[0086] Receive the basic files through the Drainer component and merge the basic files to obtain the merged binglog file;

[0087] Synchronize the merged binglog file to the TiDB local disaster recovery database.

[0088] Specifically, the server includes multiple TiDB node servers, each TiDB node server corresponds to a pump component. After the TiDB node server generates a binglog file, the pump component collects the binglog file as the basic file and sends the basic component to the Drainer component. The Drainer component merges all the basic files received, and the merged binglog file is synchronized to the TiDB local disaster recovery library.

[0089] It is easy to understand that the data format in the binglog file is binary. Using binglog files for synchronous backup can greatly improve the backup efficiency.

[0090] Among them, the pump component generates a daemon process, which runs in the background of each TiDB node server. Its main function is to record the Binlog files generated by the TiDB node server when performing data processing in real time and write them sequentially to the corresponding storage space.

[0091] Among them, the Drainer component interacts with the pump component to collect Binlog files.

[0092] In this embodiment, the binglog files of the TiDB node server corresponding to the pump component are collected through the pump component as basic files, and the basic files are sent to the Drainer component. The basic files are received and merged through the Drainer component to obtain a merged binglog file, and then the merged binglog file is synchronized to the TiDB local disaster recovery database, thereby realizing the rapid synchronization of binglog files to the TiDB local disaster recovery database and improving the efficiency of rapid synchronization of binglog files to the TiDB local disaster recovery database.

[0093] In some optional implementations of this embodiment, in step S202, the flash NAS device includes a first flash device and a second flash device, and using a preset scheduled backup script to synchronize the binlog file in the TiDB local disaster recovery database to the flash NAS device includes:

[0094] Move and save the binlog files in the TiDB local disaster recovery database to the first flash memory device;

[0095] Regularly perform incremental and full difference verification on the first flash memory device and the second flash memory device to obtain a verification result;

[0096] If the verification result shows that there is a difference, the data in the second flash memory device is updated using the data in the first flash memory device according to the difference in the verification result.

[0097] Specifically, the flash NAS device in this embodiment includes a first flash device and a second flash device, the first flash device is a primary storage device, the second flash device is a backup storage device, and the second flash device is set up in the same region as the TiDB production library. In the event of an abnormal network interruption, a backup storage device in the same regional network environment as the TiDB production library is used to ensure that even if cross-regional network access is unavailable, the database backup file can be obtained from the backup storage device in the same city to quickly restore the database.

[0098] Furthermore, the binlog files in the TiDB local disaster recovery library are moved and saved to the first flash memory device, and incremental and full difference checks are performed on the first flash memory device and the second flash memory device at regular intervals to obtain the check results. When the check results show that there are differences, the data in the second flash memory device is updated with the data in the first flash memory device based on the differences in the check results, thereby quickly realizing the synchronous update of the first flash memory device and the second flash memory device.

[0099] Among them, incremental refers to the data of the most recent update, and full refers to all data.

[0100] The first flash memory device and the second flash memory device are periodically verified for differences in increments and full amounts, and the verification result is obtained by comparing the full and incremental file differences between the first flash memory device and the second flash memory device through a timing script, and the main comparison points include but are not limited to: file size, file modification time, and file creation time, etc. The specific comparison method can be a string matching method.

[0101] In this embodiment, the binlog file in the TiDB local disaster recovery library is moved and saved to the first flash memory device, and incremental and full difference checks are performed on the first flash memory device and the second flash memory device at regular intervals to obtain the check results. When the check results show that there are differences, the data in the second flash memory device is updated with the data in the first flash memory device based on the differences in the check results, thereby achieving rapid synchronous update of the first flash memory device and the second flash memory device.

[0102] In some optional implementations of this embodiment, moving the binlog file in the TiDB local disaster recovery repository and saving it to the first flash memory device includes:

[0103] Use the binlog file in the TiDB local disaster recovery database as the data to be imported, and obtain the storage path of the first flash memory device corresponding to the data to be imported;

[0104] Determine the number of target threads based on the data to be imported and the first flash memory device;

[0105] Based on the target number of threads, multiple threads are constructed, and the data to be imported is synchronized to the storage path of the first flash memory device by using the multiple threads.

[0106] Specifically, each TiDB node server corresponds to a preset storage path in the first flash memory device, and uses the binlog file in the TiDB local disaster recovery library as the data to be imported. The storage path of the first flash memory device corresponding to the data to be imported is determined based on the TiDB node server corresponding to the data to be imported. The number of target threads is determined based on the imported data and the first flash memory device, and multiple threads of this number are constructed for data synchronization.

[0107] Among them, based on the data to be imported and the first flash memory device, the number of target threads is determined. The specific process is: according to the number of host CPU cores, the minimum number of threads is set at the beginning, and then it is gradually multiplied. At the same time, the host IO and network load are observed, and the number of import threads is dynamically determined according to the host load.

[0108] In this embodiment, the number of threads is dynamically determined, and in a multi-threaded manner, the binlog file is synchronized to the first flash device, which helps save system resources and improve data backup efficiency.

[0109] In some optional implementation manners of this embodiment, in step S203, if it is monitored that the TiDB production library is abnormal, then based on the binlog file stored in the flash NAS device, the data restoration process for the TiDB production library includes:

[0110] Stop the write table process of the application service for the TiDB production library, and empty the data in the data tables of the TiDB production library;

[0111] Obtain the full backup file and the incremental backup file from the flash NAS device as the files to be restored;

[0112] Load the full backup file into the data tables in the TiDB production library, and then load the incremental backup file into the data tables in the TiDB production library;

[0113] Verify the data in the data tables of the TiDB production library. After the verification passes, confirm that the data restoration is successful, and start the application service.

[0114] Specifically, take the files corresponding to all the data stored in the flash NAS device as the full backup files, and take the files corresponding to the data of the last backup as the incremental backup files. When it is monitored that the TiDB production library is abnormal, stop the write table process of the application service for the TiDB production library, and empty the data in the data tables of the TiDB production library. Then, take the full backup file and the incremental backup file as the files to be restored, and use the files to be restored to perform data restoration on the data tables in the TiDB production library. After the data restoration is completed, verify the data in the data tables of the TiDB production library. After the verification passes, confirm that the data restoration is successful, and start the application service.

[0115] In this embodiment, when it is monitored that the TiDB production library is abnormal, stop the read and write process of the data tables in the TiDB production library, obtain the files to be restored from the flash NAS device and restore them to the data tables in the TiDB production library, and verify the data in the data tables in the restored TiDB production library. After the verification passes, enable the application service, so as to realize automatic and rapid restoration when the TiDB production library is abnormal and improve the database restoration efficiency.

[0116] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0117] Figure 3The schematic diagram of the maintenance device of the TiDB database is shown in FIG. Figure 3 As shown, the maintenance device of the TiDB database includes a data acquisition module 31, a file synchronization module 32 and a data recovery module 33. The detailed description of each functional module is as follows:

[0118] The data collection module 31 is used to collect data in the TiDB production database through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production database, and synchronize the binlog file to the TiDB local disaster recovery database;

[0119] The file synchronization module 32 is used to synchronize the binlog files in the TiDB local disaster recovery database to the flash NAS device using a preset scheduled backup script;

[0120] The data recovery module 33 is used to restore the data of the TiDB production database based on the binlog file stored in the flash NAS device if an abnormality is detected in the TiDB production database.

[0121] Optionally, the data acquisition module 31 includes:

[0122] The collection unit is used to collect the binglog files of the TiDB node server corresponding to the pump component through the pump component as the basic files, and send the basic files to the Drainer component;

[0123] The merging unit is used to receive the basic files through the Drainer component and merge the basic files to obtain the merged binglog file;

[0124] The synchronization unit is used to synchronize the merged binglog files to the TiDB local disaster recovery database.

[0125] Optionally, the flash NAS device includes a first flash device and a second flash device, and the file synchronization module 32 includes:

[0126] A synchronization unit, used to move and save the binlog files in the TiDB local disaster recovery database to the first flash memory device;

[0127] A verification unit, used for periodically performing incremental and full difference verification on the first flash memory device and the second flash memory device to obtain a verification result;

[0128] The updating unit is used to update the data in the second flash memory device with the data in the first flash memory device according to the difference in the verification result if the verification result shows that there is a difference.

[0129] Optionally, the synchronization unit includes:

[0130] A path acquisition subunit, configured to use the binlog file in the TiDB local disaster recovery library as the data to be imported, and acquire the storage path of the first flash device corresponding to the data to be imported;

[0131] A thread number determination subunit, configured to determine the number of target threads based on the data to be imported and the first flash device;

[0132] A multi-thread synchronization subunit, configured to build multiple threads based on the number of target threads, and use the multiple threads to synchronize the data to be imported to the storage path of the first flash device.

[0133] Optionally, the data recovery module 33 includes:

[0134] A data cleaning unit, configured to stop the write table process of the application service to the TiDB production library, and empty the data in the data table of the TiDB production library;

[0135] A file to be restored determination unit, configured to obtain the full backup file and the incremental backup file from the flash NAS device as the files to be restored;

[0136] A file restoration unit, configured to load the full backup file into the data table in the TiDB production library, and then load the incremental backup file into the data table in the TiDB production library;

[0137] A data verification unit, configured to verify the data in the data table in the TiDB production library. After the verification passes, confirm that the data restoration is successful, and start the application service.

[0138] For the specific limitations of the maintenance device for the TiDB database, reference can be made to the limitations of the maintenance method for the TiDB database in the above text, which will not be elaborated here. Each module in the above-mentioned maintenance device for the TiDB database can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor in the computer device in hardware form or be independent of the processor, or be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0139] To solve the above technical problems, an embodiment of the present application also provides a computer device. For details, please refer to Figure 4 , Figure 4 which is the basic structure block diagram of the computer device in this embodiment.

[0140] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are communicatively connected to each other via a system bus. It should be noted that only the computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43 is shown in the figure. However, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of the present technology can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0141] The computer device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, etc.

[0142] The memory 41 includes at least one type of readable storage medium. The readable storage medium includes flash memory, hard disks, multimedia cards, card-type memories (such as SD or D interface display memories, etc.), random access memories (RAMs), static random access memories (SRAMs), read-only memories (ROMs), electrically erasable programmable read-only memories (EEPROMs), programmable read-only memories (PROMs), magnetic memories, magnetic disks, optical disks, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 41 can also include both the internal storage unit and the external storage device of the computer device 4. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as program codes for controlling electronic files, etc. In addition, the memory 41 can also be used to temporarily store various data that have been output or will be output.

[0143] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code for controlling electronic files.

[0144] The network interface 43 may include a wireless network interface or a wired network interface, and the network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0145] The present application also provides another implementation manner, that is, to provide a computer-readable storage medium storing an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to execute the steps of the maintenance method of the TiDB database as described above.

[0146] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0147] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements on some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields is equally within the scope of the patent protection of the present application.

Claims

1. A maintenance method for a TiDB database, characterized in that, The TiDB database maintenance method includes: Collect data from the TiDB production database through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production database, and synchronize the binlog file to the TiDB local disaster recovery database; Using a preset scheduled backup script, synchronize the binlog files in the TiDB local disaster recovery database to the flash NAS device, wherein the flash NAS device includes a first flash device and a second flash device, the first flash device is a primary storage device, the second flash device is a backup storage device, and the second flash device is set up in the same region as the TiDB production database; If an abnormality is detected in the TiDB production database, data restoration is performed on the TiDB production database based on the binlog file stored in the flash NAS device; The method of using a preset scheduled backup script to synchronize the binlog file in the TiDB local disaster recovery database to the flash NAS device includes: Move and save the binlog file in the TiDB local disaster recovery database to the first flash memory device; Regularly performing incremental and full difference verification on the first flash memory device and the second flash memory device to obtain a verification result, wherein the incremental refers to the most recently updated data and the full refers to all data; If the verification result shows that there is a difference, updating the data in the second flash memory device with the data in the first flash memory device according to the difference in the verification result; The method periodically performs incremental and full difference verification on the first flash memory device and the second flash memory device, and obtains the verification result including: comparing the full and incremental file differences between the first flash memory device and the second flash memory device through a timing script, the main comparison points include file size, file modification time and file creation time, and the specific comparison method adopts a string matching method.

2. The maintenance method of the TiDB database according to claim 1, characterized in that The TiDB-Binlog cluster component includes a Drainer component and a pump component. The TiDB-Binlog cluster component is used to collect data in the TiDB production database, obtain the binlog file corresponding to the data in the TiDB production database, and synchronize the binlog file to the TiDB local disaster recovery database, including: The pump component collects the binglog file of the TiDB node server corresponding to the pump component as a basic file, and sends the basic file to the Drainer component; The basic files are received by the Drainer component and merged to obtain a merged binglog file; Synchronize the merged binglog file to the TiDB local disaster recovery database.

3. The maintenance method of the TiDB database according to claim 1, characterized in that, The moving and saving the binlog file in the TiDB local disaster recovery database to the first flash memory device includes: Use the binlog file in the TiDB local disaster recovery database as the data to be imported, and obtain the storage path of the first flash memory device corresponding to the data to be imported; Determining the number of target threads based on the data to be imported and the first flash memory device; Based on the target number of threads, multiple threads are constructed, and the multiple threads are used to synchronize the data to be imported to the storage path of the first flash memory device.

4. The maintenance method of the TiDB database according to any one of claims 1 to 3, characterized in that If the TiDB production database is monitored to be abnormal, then based on the binlog file stored in the flash NAS device, data restoration processing of the TiDB production database includes: Stop the application service from writing tables to the TiDB production database, and clear the data in the data table of the TiDB production database; Obtaining the full backup file and the additional backup file from the flash NAS device as files to be restored; Load the full backup file into the data table in the TiDB production database, and then load the additional backup file into the data table in the TiDB production database; Verify the data in the data table in the TiDB production library. After the verification passes, confirm that the data restoration is successful and start the application service.

5. A maintenance device for a TiDB database, characterized in that, The maintenance device of the TiDB database includes: The data collection module is used to collect data from the TiDB production database through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production database, and synchronize the binlog file to the TiDB local disaster recovery database; A file synchronization module, used to synchronize the binlog files in the TiDB local disaster recovery database to the flash NAS device using a preset scheduled backup script, wherein the flash NAS device includes a first flash device and a second flash device, the first flash device is a primary storage device, the second flash device is a backup storage device, and the second flash device is set up in the same region as the TiDB production database; A data recovery module, configured to restore the TiDB production database based on the binlog file stored in the flash NAS device if an abnormality is detected in the TiDB production database; The file synchronization module includes: A synchronization unit, used to move and save the binlog file in the TiDB local disaster recovery database to the first flash memory device; a verification unit, configured to periodically perform incremental and full difference verification on the first flash memory device and the second flash memory device to obtain a verification result, wherein the incremental refers to the data updated most recently and the full refers to all data; an updating unit, configured to update the data in the second flash memory device with the data in the first flash memory device according to the difference in the verification result if the verification result shows that there is a difference; The verification unit is specifically used to compare the full and incremental file differences between the first flash memory device and the second flash memory device through a timing script. The main comparison points include file size, file modification time and file creation time. The specific comparison method adopts a string matching method.

6. The maintenance device of the TiDB database according to claim 5, characterized in that, The TiDB-Binlog cluster components include the Drainer component and the pump component. The data collection module includes: A collection unit, configured to collect the binglog file of the TiDB node server corresponding to the pump component through the pump component as a basic file, and send the basic file to the Drainer component; A merging unit, configured to receive the basic files through the Drainer component and merge the basic files to obtain a merged binglog file; A synchronization unit is used to synchronize the merged binglog file to the TiDB local disaster recovery database.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the TiDB database maintenance method according to any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the TiDB database maintenance method according to any one of claims 1 to 4 is implemented.

Images