Block-level incremental backup method and device for database, equipment and medium

Abstract

The invention provides a block-level incremental backup method and device for a database, equipment and a medium. The method comprises the steps that data files in the database are traversed; for any traversed data file, whether the data file is a small file or not is determined according to a preset size, and the small file is a data file with the file size smaller than the preset size; under the condition that at least two data files which are small files are determined, merging the at least two data files which are small files to obtain at least one merged file; and partitioning the at least one combined file to obtain at least one data block, and performing block-level incremental backup according to the at least one data block. On the basis, the number of the data blocks can be reduced, the backup time consumption is reduced, and the database incremental backup efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a block-level incremental backup method, device, equipment and medium for a database. Background Art

[0002] Block-level incremental backup can be understood as backing up only the data blocks that have changed in the file since the last backup. When using the block-level incremental backup method to back up a database, each data file in the database is divided into blocks with a preset block size to obtain multiple data blocks. Then, according to each data block, the data blocks that need to be backed up are backed up to achieve block-level incremental backup of the database.

[0003] In the existing block-level incremental backup method, the blocks are divided based on the file as the basic unit. For example, if the size of a data file is smaller than the block size, no matter how small the size of the data file is, it will be divided into a data block for processing. Therefore, after the block is divided based on the existing block-level incremental backup method, the total amount of data blocks obtained is large, which will reduce the efficiency of incremental backup. Summary of the invention

[0004] The present application provides a block-level incremental backup method, device, equipment and medium for a database, so as to improve the backup efficiency of the database during block-level incremental backup.

[0005] A first aspect of the present application provides a block-level incremental backup method for a database, the method comprising:

[0006] Traversing the data files in the database;

[0007] For any of the traversed data files, determine whether the data file is a small file according to a preset size, wherein the small file is a data file whose file size is smaller than the preset size;

[0008] When at least two data files that are small files are determined, merging the at least two data files that are small files to obtain at least one merged file;

[0009] The at least one merged file is divided into blocks to obtain at least one data block, and a block-level incremental backup is performed according to the at least one data block.

[0010] Furthermore, in the method as described above, performing block-level incremental backup according to the at least one data block includes:

[0011] According to a preset change condition, determining a first target data block that satisfies the preset change condition in the at least one data block, wherein the preset change condition includes a preset condition for determining a change range of a data block since a last backup;

[0012] When at least one first target data block is determined, a block-level incremental backup is performed on the at least one first target data block in a sequential input and output manner.

[0013] Furthermore, the method as described above further comprises:

[0014] According to a preset change condition, determining a second target data block that does not meet the preset change condition in the at least one data block;

[0015] When at least one second target data block is determined, data files in the at least one second target data block that have been changed since the last backup are referenced to a new data block to obtain at least one referenced data block;

[0016] The at least one reference data block is backed up at a block level incrementally in a sequential input and output manner.

[0017] Furthermore, in the above method, determining whether the data file is a small file according to a preset size includes:

[0018] Determining whether the data file has been changed since the last backup according to the modification time of the data file;

[0019] In the case where it is determined that the data file has changed since the last backup, determining whether the data file is a small file according to a preset size;

[0020] and / or,

[0021] Determining, according to the creation time of the data file, whether the data file is a data file created after the last backup;

[0022] In the case where it is determined that the data file is a data file created after the last backup, it is determined whether the data file is a small file according to a preset size.

[0023] Furthermore, in the above method, traversing the data files in the database includes:

[0024] For any of the traversed data files, according to the modification time of the data files, a data file that has not been modified for more than a preset time is determined as an old data file;

[0025] When at least one old age data file is determined, the at least one old age data file is stored in a designated storage area.

[0026] Furthermore, in the method as described above, merging the at least two small data files to obtain at least one merged file includes:

[0027] Sequentially write at least part of the at least two small data files into the same designated file to obtain a merged file among the at least one merged file;

[0028] According to the writing order and the size of at least part of the data files, the position information of at least part of the data files in the merged file is recorded in the configuration file corresponding to the merged file.

[0029] A second aspect of the present application provides a block-level incremental backup device for a database, the device comprising:

[0030] A processing module, used for traversing the data files in the database;

[0031] The processing module is further used to determine whether the data file is a small file according to a preset size, and the small file is a data file whose file size is smaller than the preset size;

[0032] A merging module, configured to merge the at least two data files that are small files to obtain at least one merged file when at least two data files that are small files are determined;

[0033] The block backup module is used to divide the at least one merged file into blocks to obtain at least one data block, and perform block-level incremental backup according to the at least one data block.

[0034] Furthermore, in the above device, the block backup module is specifically used for:

[0035] According to a preset change condition, determining a first target data block that satisfies the preset change condition in the at least one data block, wherein the preset change condition includes a preset condition for determining a change range of a data block since a last backup;

[0036] When at least one first target data block is determined, a block-level incremental backup is performed on the at least one first target data block in a sequential input and output manner.

[0037] Furthermore, in the above device, the block backup module is also used for:

[0038] According to a preset change condition, determining a second target data block that does not meet the preset change condition in the at least one data block;

[0039] When at least one second target data block is determined, data files in the at least one second target data block that have been changed since the last backup are referenced to a new data block to obtain at least one referenced data block;

[0040] The at least one reference data block is backed up at a block level incrementally in a sequential input and output manner.

[0041] Further, in the above device, the processing module is specifically used for:

[0042] Determining whether the data file has been changed since the last backup according to the modification time of the data file;

[0043] In the case where it is determined that the data file has changed since the last backup, determining whether the data file is a small file according to a preset size;

[0044] and / or,

[0045] Determining, according to the creation time of the data file, whether the data file is a data file created after the last backup;

[0046] In the case where it is determined that the data file is a data file created after the last backup, it is determined whether the data file is a small file according to a preset size.

[0047] Further, in the above device, the processing module is specifically used for:

[0048] For any of the traversed data files, according to the modification time of the data files, a data file that has not been modified for more than a preset time is determined as an old data file;

[0049] When at least one old age data file is determined, the at least one old age data file is stored in a designated storage area.

[0050] Further, in the above device, the merging module is specifically used for:

[0051] Sequentially write at least part of the at least two small data files into the same designated file to obtain a merged file among the at least one merged file;

[0052] According to the writing order and the size of at least part of the data files, the position information of at least part of the data files in the merged file is recorded in the configuration file corresponding to the merged file.

[0053] A third aspect of the present application provides an electronic device, including: a memory and a processor;

[0054] The memory stores computer-executable instructions;

[0055] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0056] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any of the methods described in the first aspect.

[0057] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any method described in the first aspect.

[0058] The present application provides a method, device, equipment and medium for block-level incremental backup of a database, the method comprising: traversing data files in the database; for any traversed data file, determining whether the data file is a small file according to a preset size, a small file is a data file whose file size is less than the preset size; in the case of determining at least two data files that are small files, merging the at least two data files that are small files to obtain at least one merged file; dividing the at least one merged file into blocks to obtain at least one data block, and performing block-level incremental backup according to the at least one data block. Based on this, after determining at least two data files that are small files, merging them to obtain a merged file can reduce the total amount of data files, and when dividing the blocks with each data file as the basic unit of the blocks, the total amount of the divided data blocks can be reduced. And when dividing the merged file into blocks, the amount of data in each data block can be made more saturated, thereby promoting the reduction of the total amount of data blocks. In the case of a large reduction in the total amount of data blocks, performing block-level incremental backup according to the divided data blocks can reduce the processing operation of the data block backup, reduce the time consumption of the backup, and thus improve the backup efficiency during the incremental backup. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0060] Figure 1 A schematic diagram of a block-level incremental backup of a database in the related art;

[0061] Figure 2 One of the flowcharts of the block-level incremental backup method of a database provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of data file merging provided in an embodiment of the present application;

[0063] Figure 4 A second flowchart of a block-level incremental backup method for a database provided in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of the structure of a block-level incremental backup device for a database provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0066] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0068] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first target data and the second target data are only used to distinguish different target data, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0069] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0070] In the embodiments of the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0071] In the technical solutions of the embodiments of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0073] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0074] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced.

[0075] In the related art, in the existing method of performing block-level incremental backup on a database, files are used as the basic unit of blocks. Figure 1 Schematic diagram of block-level incremental backup of a database in the related art, such as Figure 1 As shown in the figure, the sizes of the data files in the database are different. Some of the larger sizes can be understood as large files, and some of the smaller sizes can be understood as small files. When performing block-level incremental backup, each large file or small file will be divided into blocks one by one according to the pre-set block size to obtain multiple data blocks after the block division. If the pre-set block size cannot match the size of each data file in the database, the total amount of data blocks obtained after the block division will be large.

[0076] For example, the preset block size is 8 kilobytes (KB). If there are multiple small files in the database, for example, the size is 3KB, and each small file is segmented into blocks, one data block will be obtained, so the database will obtain more data blocks after segmentation. If the block size is preset to 3KB or smaller in order to adapt to these small files, it is not suitable for the large files in the database. For example, for a large file with a size of 800KB, compared with the block size of 8KB, after the large file is segmented into blocks with a block size of 3KB or smaller, many more data blocks will be obtained. Therefore, after reducing the block size, segmenting each large file into blocks with a smaller block size will also increase the total amount of data blocks after segmentation.

[0077] When faced with the above problem, the general solution is to determine a compromise block size to try to adapt to the size of each data file, but this method is difficult to implement in block-level incremental backup in database scenarios.

[0078] The inventors have found in practice that as the database is used, the size of each data file in the database changes dynamically, and the size of each data file varies greatly. Moreover, during a backup, the number of data files of the same size does not have a strong distribution pattern. It can be understood that no size has an absolute advantage in number, so it is difficult to determine a suitable block size. In addition, when the database is backed up incrementally at the block level, subsequent backups will use the block size preset during the first backup. However, as the data files in the database change dynamically, the block size preset for the first time is difficult to adapt to the sizes of the data files in subsequent backups.

[0079] Furthermore, even if a compromised block size is determined to reduce the total amount of data blocks after some block divisions, there are still cases where small files smaller than the block size are divided into a single data block. There are also some small files slightly larger than the block size, although they can be divided into two or more data blocks, the amount of data in some of the data blocks is not saturated, indicating that the result of the block division is still not in a good state.

[0080] When performing incremental backup, each data block will be logically compared to determine whether there is any data change, and then determine the data block that needs to be backed up. After that, the input and output (I / O) operation of writing the backup set will be performed on each data block that needs to be backed up. Therefore, if the block division of each data file in the database is unreasonable, resulting in a large total amount of data blocks, the number of logical comparisons required and the number of write I / O operations will increase, which will increase the time consumption of the backup and reduce the backup efficiency when performing incremental backup of the database.

[0081] In view of this, an embodiment of the present application provides a block-level incremental backup method for a database. The method traverses the data files in the database, determines the small files whose data file size is smaller than the preset size, and merges these small files to obtain the merged file. In this way, the total amount of data files can be reduced. When each data file is used as the basic unit for block division, the total amount of data blocks after block division can be reduced. Further, the merged file is divided into blocks to obtain data blocks and block-level incremental backup is performed according to the data blocks. In this way, since the merged file contains continuous data content, when the merged file is divided into blocks, the continuous data content can be divided into blocks, so that the amount of data in each obtained data block can be saturated as much as possible, and the total amount of data blocks can also be reduced. Therefore, when the total amount of data blocks is reduced as much as possible, the number of logical comparisons and write I / O operations performed on each data block can be reduced, thereby reducing the time consumption of backup and improving the backup efficiency during incremental backup.

[0082] The embodiments of the present application are introduced below in conjunction with the drawings in the specification.

[0083] Figure 2 This is one of the flowcharts of the block-level incremental backup method of the database provided in the embodiment of the present application. The execution subject of the embodiment of the present application may be an electronic device with data processing capabilities, for example, an electronic device such as a computer, a server or a server cluster. Figure 2 As shown, the method includes:

[0084] S201, traverse the data files in the database.

[0085] Exemplarily, the database may be one of multiple types of databases, for example, the database may be a relational database, a NoSQL database, a columnar database, or a graph database, etc. The database may be a database in any application scenario, for example, a database of an enterprise resource planning system, a database of an e-commerce platform, or a database of a social media platform, etc.

[0086] Data files can be understood as files used to store data in a database, such as data tables, index files, log files, or control files. Data files can be objects that are traversed each time a backup is performed. During an incremental backup, data blocks corresponding to the changed data in data files that have changed since the last backup will be backed up.

[0087] When traversing the data files in the database, it can be a sequential traversal or a random traversal, etc., and it can be a one-by-one traversal or a concurrent multi-threaded traversal. The following steps can be performed for each currently traversed data file. When traversing any data file, the file information of the data file can be read. The file information can be called the file metadata of the file, including various attributes and information of the data file, such as the file name, file path, file size, creation time and modification time, etc., where the file size is the file size.

[0088] S202: for any traversed data file, determine whether the data file is a small file according to a preset size. A small file is a data file whose file size is smaller than a preset size.

[0089] Exemplarily, the preset size may be a preset file size value of any size, which is used to judge and classify the size of the data file and determine whether the data file is a small file. For example, the preset size may be 10KB, 100KB, or 1 megabyte (MB).

[0090] Taking the preset size of 10KB as an example, when traversing any data file, the file information of the data file is read, and the file size therein is compared with the preset size of 10KB. If the file size of the data file is greater than or equal to 10KB, the data file is not a small file, and it can be understood that the data file is a large file. If the file size of the data file is less than 10KB, it can be determined that the data file is a small file. For the convenience of the following description, a data file that is a small file can be referred to as a small file, and a data file that is not a small file can be referred to as a large file.

[0091] S203: When at least two data files are determined to be small files, merge the at least two data files to obtain at least one merged file.

[0092] Exemplarily, when merging at least two small data files, any data merging operation applicable to incremental backup can be used. For example, the cat command in the command line tool can be used to merge data, or the data in at least two small files can be written into an empty file through a write operation such as moving or copying, and at least one merged file can be obtained after the writing is completed.

[0093] For example, after traversing multiple data files concurrently and determining 10 small files, the data in the 10 small files can be copied to one file, and a merged file is obtained after the copying is completed. Alternatively, the data of 5 of the 10 small files can be copied to one file to obtain a merged file, and the data of the other 5 small files can be copied to another file to obtain another merged file, thus obtaining two merged files in total.

[0094] In a possible implementation, when merging at least two small data files to obtain at least one merged file, specifically: writing at least part of the at least two small data files into the same designated file in sequence to obtain a merged file in at least one merged file; and recording position information of at least part of the data files in a merged file in a configuration file corresponding to the merged file according to the writing order and the size of at least part of the data files.

[0095] Exemplarily, at least part of at least two small data files may be sequentially written into the same designated file according to the order of traversing data files or the order of file positions, to obtain a merged file. The writing operation may be data copying, data moving, or data referencing.

[0096] Figure 3 A schematic diagram of a data file merging provided in an embodiment of the present application is as follows: Figure 3 As shown, when merging multiple small files, each small file is taken as a binary data stream, multiple small files are written into a large file in sequence, and the location information of the large file to which the small file belongs is recorded in the configuration file. Exemplarily, when merging, the location information can be recorded through one configuration file, or through multiple configuration files, and the location information can be determined according to the amount of data recorded in the configuration file, or the number of configuration files can be determined according to other requirements.

[0097] For a small file, the offset and file size of the small file in the large file to which it belongs can be recorded in the corresponding configuration file. The offset can be understood as the starting position of the small file in the large file to which it belongs. The ending position of the small file in the large file to which it belongs can be determined by the starting position and the file size of the small file. Therefore, according to the offset and file size of each small file in the configuration file, the position of each small file in the merged file can be determined. When the merged file is subsequently divided into blocks, whether the block data needs to be backed up, block data backup and / or block data recovery and other operations are performed, the small file that needs to be operated can be queried and determined based on the position information recorded in the configuration file. For example, the corresponding small file can be uniquely obtained from a merged file by calling a search function (such as a seek function) so that the data of the small file will not be lost.

[0098] In this embodiment, based on the writing order and the size of at least part of the data files, the location information of at least part of the data files in a merged file is recorded in a configuration file corresponding to the merged file. The corresponding small files can be accurately queried and determined in subsequent backup-related operations, thereby improving the accuracy of the backup.

[0099] S204, dividing the at least one merged file into blocks to obtain at least one data block, and performing block-level incremental backup according to the at least one data block.

[0100] After the small files are merged, the merged file may be divided into blocks to obtain at least one data block. The blocks may be divided into blocks with a preset block size.

[0101] Exemplarily, the block size can be a size measure used when dividing a data file into blocks, and can also be understood as an upper limit of the size of a data block, and the size of each divided data block will not be greater than the block size. The block size can be any preset file size value, such as 1KB, 2KB, or 1MB.

[0102] For example, if the block size is preset to 1KB, when a data file of 9KB is divided into blocks, 9 data blocks will be divided, each of which is 1KB in size. When a data file of 9.5KB is divided into blocks, 10 data blocks will be divided, including 9 1KB data blocks and 1 0.5KB data block.

[0103] The preset size and the block size may be the same or different. For example, if the preset size is 10 KB and the block size is also 10 KB, then a data file smaller than the block size is a small file.

[0104] In one possible implementation, in order to make the amount of data in each data block as saturated as possible when the merged file is divided into blocks, that is, the amount of data in the data block is as close as possible to the size of the block size, and the number of unsaturated data blocks is reduced, all small files can be merged to obtain a merged file.

[0105] For example, after traversing all data files in the database, all small files are determined, and all of these small files are merged into a merged file to obtain a large file. The merged file is divided into blocks according to the block size.

[0106] Based on this, all small files can form a longer continuous data, which can greatly reduce the total amount of data files. When dividing the blocks, the amount of data in each data block can be made as close to the block size as possible, reducing the total amount of data blocks, reducing the number of logical comparisons of data blocks, and also reducing the number of write I / O operations for incremental backup of data blocks, thereby improving backup efficiency.

[0107] When performing block-level incremental backup based on at least one data block, it can be determined whether the data in each data block has changed since the last backup, and a block-level incremental backup can be performed for the changed data blocks, thereby realizing incremental backup of the database.

[0108] The block-level incremental backup method of a database provided by an embodiment of the present application can reduce the total amount of data files by merging at least two data files that are small files to obtain a merged file. When each data file is used as the basic unit for block division, the total amount of data blocks after block division can be reduced. When the merged file is divided into blocks, the amount of data in each data block can be made more saturated, thereby reducing the total amount of data blocks. When the total amount of data blocks is greatly reduced, block-level incremental backup is performed according to the divided data blocks, which can reduce the processing operations of data block backup, reduce the time consumption of backup, and thus improve the backup efficiency during incremental backup.

[0109] In one possible implementation, performing block-level incremental backup based on at least one data block includes: determining a first target data block that meets the preset change condition in at least one data block based on a preset change condition, the preset change condition including a preset condition for determining the magnitude of change of the data block since the last backup; when at least one first target data block is determined, performing block-level incremental backup on the at least one first target data block using a sequential input and output method.

[0110] Exemplarily, the preset change condition may be a preset condition for determining the magnitude of the change in the data in the data block since the last backup, wherein the change may be understood as modification, addition or deletion of data. The magnitude of the change may be, for example, the ratio of the amount of data that has changed to the total amount of data in the data block.

[0111] The preset change condition may include a first change value of the number of data files that have changed in the data block and the total number of data files in the data block. A first change threshold for determining the size of the first change value is preset, and the first change threshold may be any preset threshold, such as 50%. Whether the preset change condition is satisfied, for example, it may be preset that the preset change condition is satisfied when the first change value is greater than or equal to the first change threshold, and the preset change condition is not satisfied when the first change value is less than the first change threshold.

[0112] Taking the first change threshold of 50% as an example, for a data block, if the data in the data block comes from 10 data files, and it is determined that the data in 6 of the data files have changed since the last backup, then the first change value corresponding to the data block is 6 / 10, that is, 60%, which is greater than the first change threshold of 50%. It is determined that the data block meets the preset change condition, and the data block is the first target data block.

[0113] Alternatively, the preset change condition may include a second change value of the amount of data that has changed in the data block and the total amount of data in the data block. A second change threshold for determining the size of the second change value is preset, and the second change threshold may be any preset threshold, such as 50%. Whether the preset change condition is satisfied, for example, it may be preset that the preset change condition is satisfied when the second change value is greater than or equal to the second change threshold, and the preset change condition is not satisfied when the second change value is less than the second change threshold.

[0114] Taking the second change threshold of 50% as an example, for a 10KB data block, if it is determined that 6KB of data in the data block has changed since the last backup, then the second change value corresponding to the data block is 6KB / 10KB, that is, 60%, which is greater than the second change threshold of 50%. Then it is determined that the data block meets the preset change condition, and the data block is the first target data block.

[0115] When at least one first target data block is determined, a block-level incremental backup is performed on the at least one first target data block using a sequential input / output method. For example, the data in each first target data block can be updated using a sequential I / O read / write method for each first target data block as a whole according to the order of the locations of the first target data blocks, thereby realizing an incremental backup of the data blocks.

[0116] In this embodiment, the data blocks with large changes can be quickly determined according to the preset change conditions, and the sequential I / O method for each first target data block can better utilize the parallelism of reading and writing of the electronic device to improve the reading and writing speed. For example, compared with the random I / O method, the sequential I / O can perform fewer addressing operations, reduce the addressing time consumption, improve the reading and writing speed, and thus improve the backup efficiency.

[0117] In one possible implementation, the method further includes: determining, based on a preset change condition, a second target data block in at least one data block that does not meet the preset change condition; when at least one second target data block is determined, referencing a data file in at least one second target data block that has changed since the last backup to a new data block to obtain at least one referenced data block; and performing a block-level incremental backup of at least one referenced data block in a sequential input and output manner.

[0118] Exemplarily, the second target data block may be understood as a data block determined according to the preset change condition and not satisfying the preset change condition.

[0119] For example, the preset change condition includes a first change value, and a first change threshold is preset for judging the size of the first change value. When the first change value is less than the first change threshold, the preset change condition is not met, and the corresponding data block is the second target data block.

[0120] Taking the first change threshold of 50% as an example, for a data block, if the data in the data block comes from 10 data files, and it is determined that the data of 4 data files have changed since the last backup, and the others have not changed, then the first change value corresponding to the data block is 4 / 10, that is, 40%, which is less than the first change threshold of 50%. It is determined that the data block does not meet the preset change condition, and the data block is the second target data block.

[0121] After determining at least one second target data block, for example, all data blocks are traversed, and after determining all second target data blocks, the data files in each second target data block that have changed since the last backup are referenced to the new data block to obtain one or more reference data blocks. The reference data block can be understood as a reassembled data block obtained by referencing the data in the original data block. Block-level incremental backup is performed on these reference data blocks in a sequential input and output manner.

[0122] Based on this, the second target data block with a smaller change range can be temporarily not backed up. When the second target data block reaches a certain level, it can be referenced and then backed up. This can reduce the operation of backing up the data that has not changed in the second target data block, reduce the amount of data backup, and perform backup by sequential input and output, which can improve backup efficiency.

[0123] In order to further improve the efficiency of backup, when determining whether a data file is a small file, you can first determine whether the data file has changed since the last backup. If the data file has not changed, you do not need to back it up, and then there is no need to determine and merge small files, so as to reduce the amount of data calculations and improve backup efficiency.

[0124] In one possible implementation, determining whether a data file is a small file based on a preset size includes: determining whether the data file has changed since the last backup based on the modification time of the data file; if it is determined that the data file has changed since the last backup, determining whether the data file is a small file based on the preset size; and / or, determining whether the data file is a data file created since the last backup based on the creation time of the data file; if it is determined that the data file is a data file created since the last backup, determining whether the data file is a small file based on the preset size.

[0125] Exemplarily, the modification time may be a specific time when the data file was last modified, and the creation time may be a specific time when the data file was created. The modification time and the creation time may be obtained through file information.

[0126] By comparing the last backup time and the modification time of the data file, if the modification time of the data file is earlier than the last backup time, it can be determined that the data file has not changed since the last backup. If the modification time of the data file is later than the last backup time, it can be determined that the data file has changed since the last backup.

[0127] By comparing the last backup time with the data file creation time, if the data file creation time is later than the last backup time, it can be determined that the data file is a newly created data file after the last backup. In this case, the data file needs to be backed up, which is equivalent to changes having occurred since the last backup.

[0128] In this embodiment, by first determining whether the data file has changed since the last backup before determining whether the data file is a small file, the steps of small file determination and / or file merging for data files that have not changed can be reduced, thereby reducing the amount of data calculations and further improving the efficiency of backup.

[0129] Exemplarily, according to the modification time and / or creation time of the data file, the data file can be determined as a New Year data file, a middle-aged data file or an old age data file. A New Year data file can be understood as a data file that is backed up for the first time during this backup process. A New Year data file can be determined by having a creation time later than the last backup time. A New Year data file can also be understood as a data file that appears in a backup set for the first time. A backup set can be understood as a directory file corresponding to the file information of each data file. The first appearance of the file information of a data file in a directory file indicates that the data file is a newly created data file.

[0130] Middle-aged data files can be understood as data files that appeared in the last backup. In block-level incremental backups, middle-aged data files account for a large proportion and can be understood as data files with relatively active data changes. Old-aged data files can be understood as data files that have not undergone data changes for a long time, such as some configuration files or full data files.

[0131] In one possible implementation, traversing data files in a database includes: for any traversed data file, determining a data file that has not been modified for more than a preset period of time as an old-age data file based on the modification time of the data file; and when at least one old-age data file is determined, storing the at least one old-age data file in a designated storage area.

[0132] Exemplarily, the preset duration may be any preset time length, such as one week, two weeks or one month, etc. A data file that has not been modified for more than the preset duration may be a data file that will not send data changes for a long time, such as a full data file.

[0133] During the traversal process, if the data file is determined to be an old data file, it can be stored in a designated storage area, so that in subsequent backups, data changes in the data files in the storage area can be determined in a way with less computational effort. For example, only by comparing the modification time of the data file with the time of the last backup, without the need to divide the data into blocks and compare each data block, it can be determined whether it needs to be backed up.

[0134] Based on this, it is convenient to centrally manage elderly data files and improve the efficiency of subsequent data backup and data management.

[0135] Exemplarily, block-level incremental backup may also be referred to as block incremental backup or block backup, etc. In a backup set of a block backup, data corresponding to each data file may be divided into three logical modules, namely, super block, block map and storage file.

[0136] The superblock can be used to mark the status and location of each data file. For example, the superblock can record multiple file-level configuration files, through which the file information of the data file can be obtained, such as file size, creation time, modification time, and file location, etc., so as to achieve the purpose of marking the status and location of the data file. The information marked by the superblock helps to reduce the complexity of backup.

[0137] For example, the super block can also be used for file recovery after backup. When restoring data files after backup, the corresponding relationship between data blocks and data files can be determined through the configuration file in the super block, so as to query and use the data blocks corresponding to the data files for file recovery. It should be understood that the configuration file is used to record the corresponding relationship between data files and corresponding data blocks, and the configuration file when merging small files in the above embodiment is to record the corresponding relationship between small files and corresponding large files.

[0138] Block mapping can be used to store verification data of data blocks. By comparing the verification data, it can be determined whether the data of the data block has changed since the last backup. The verification data includes, for example, a checksum and / or a hash value. Block mapping can also be used to record the mapping relationship between a data file and its corresponding data block. The mapping relationship can be used to characterize the corresponding relationship between a data file and its corresponding data block and the verification data of the data block.

[0139] Block mapping stores information at the block level rather than at the file level. For example, it can store the hash value corresponding to the last backup or all previous backups of each data block. For example, for a data block, if the hash value of the data block calculated this time is different from the hash value of the data block stored in the block mapping when it was last backed up, it means that the data in the data block has changed since the last backup, and the data block needs to be backed up during this backup.

[0140] The storage file can actually store the changed data blocks. It can be understood that after the data blocks to be backed up are determined, the data blocks to be backed up are stored in the storage file for block backup.

[0141] In the solution of the present application, small files may be merged when implementing block backup. Therefore, the data file when divided into blocks may be a large file that has not been merged, or it may be a merged file, so a data block may contain data from multiple small files. Based on this, the file information of the file block and the data files it contains, such as the identity or file name of the data file, can be recorded in the super block. For example, the configuration file when merging files can be recorded in the super block.

[0142] Below, the block-level incremental backup method of the database provided in the embodiment of the present application is further described through a possible implementation method.

[0143] Figure 4 The second flowchart of the block-level incremental backup method of the database provided in the embodiment of the present application is as follows: Figure 4 As shown, when performing block backup, the data files in the database are traversed. For any data file, the file information of the data file is obtained, and the state of the data file is determined by the file information, wherein the state can be understood as the period of the data file being in the new year, middle age or old age, that is, the state indicates that the data file is a new year data file, a middle age data file or an old age data file.

[0144] like Figure 4 As shown, if it is determined that the state of the data file is a new year, the data file can be added to the super block. Further, the data file can be divided into blocks or stored as a data block in the block mapping and storage file.

[0145] like Figure 4 As shown, if it is determined that the status of the data file is middle-aged, then determine whether the data of the data file has changed since the last backup. If not, that is, it is determined that there is no change, then the data file does not need to be backed up, and other data files can be traversed. If so, that is, it is determined that a change has occurred, then it is necessary to determine whether the data file is a small file. If it is not a small file, then the data file is treated as a large file for block division and block backup, for example, the divided data blocks are stored in a block mapping and storage file. If it is a small file, then the data file is marked as a small file in the super block, and after determining other small files, the small files are merged to obtain a merged large file. After obtaining the merged large file, the large file is divided into blocks and backed up, for example, the divided data blocks are stored in a block mapping and storage file.

[0146] like Figure 4 As shown, if it is determined that the state of the data file is in the old age stage, it is determined whether the data of the data file has changed since the last backup. If not, that is, it is determined that there is no change, it can be recorded separately in a super block, for example, registered in the super block. Generally, the possibility of modifying a data file that has not been modified for a long time is also small, so only the modification time of the data file can be compared. If it is determined that the data has not changed, the previous data is kept unchanged. If it is determined that the data has changed, the information can be updated in the super block, and the state of the old age data file is returned to the middle age stage, and processed according to the processing flow corresponding to the middle age state.

[0147] In a possible implementation, after determining the state of a data file, if the data file is not determined to have changed data content, for a large file or a merged large file, when more than half of the data files in the data blocks corresponding to the large file have changed, the entire data block in the large file will be re-registered in the super block and recorded in the block mapping, and the data of the large file will be stored in the storage file. If the data file is not determined to have changed data content, for a small file, the parameters of the configuration file can be directly modified to new parameters in the super block and / or block mapping, and the file information can be inserted into the storage file. The parameters include the file name, the location and size of the corresponding data block, the modification time of the data block, the state of the data block, etc. In the block mapping, the verification data of the data block is updated to determine whether the data block has changed.

[0148] Exemplarily, when generating storage files, at least two storage files may be generated. One storage file is used to store each old-age data file, and the other storage file is used to store each new-age data file and each middle-age data file. Since the probability of data change in the old-age data file is relatively small, it is convenient to manage it.

[0149] There can be at least one storage file for storing the new year data file and the mid-year data file. If the backup is performed in a multi-process and / or multi-threaded manner, the same number of storage files as the process can be generated. This helps to accelerate reading and writing by using multi-process and / or multi-threading, thereby improving the backup speed.

[0150] Since each small file can be written into the large file in the form of a binary data stream, and the offset and file size corresponding to the small file are recorded in the super block, the data block corresponding to the small file can be uniquely determined during data recovery to achieve data recovery with higher accuracy.

[0151] In the embodiment of the present application, the above method can be used to merge small files, effectively reduce the write I / O during block incremental backup, and speed up the backup. The block incremental backup method can reduce the total amount of data blocks, reduce the number of logical judgment comparisons of data blocks, and reduce unsaturated data blocks, thereby achieving the effect of speeding up the block incremental backup speed and efficiency.

[0152] Figure 5 A schematic diagram of the structure of a block-level incremental backup device for a database provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, an embodiment of the present application provides a block-level incremental backup device for a database, the device comprising:

[0153] Processing module 501, used for traversing data files in the database;

[0154] The processing module 501 is further used to determine whether the data file is a small file according to a preset size, and a small file is a data file whose file size is smaller than a preset size;

[0155] A merging module 502 is used to merge the at least two data files that are small files to obtain at least one merged file when at least two data files that are small files are determined;

[0156] The block backup module 503 is used to divide at least one merged file into blocks to obtain at least one data block, and perform block-level incremental backup according to the at least one data block.

[0157] Optionally, the block backup module 503 is specifically used to: determine a first target data block that meets the preset change condition in at least one data block according to a preset change condition, the preset change condition including a preset condition for determining the magnitude of change of the data block since the last backup; when at least one first target data block is determined, perform block-level incremental backup of the at least one first target data block in a sequential input and output manner.

[0158] Optionally, the block backup module 503 is also used to: determine, according to a preset change condition, a second target data block in at least one data block that does not meet the preset change condition; when at least one second target data block is determined, reference the data files in at least one second target data block that have changed since the last backup to a new data block to obtain at least one reference data block; and perform block-level incremental backup of at least one reference data block in a sequential input and output manner.

[0159] Optionally, processing module 501 is specifically used to: determine whether the data file has changed since the last backup based on the modification time of the data file; if it is determined that the data file has changed since the last backup, determine whether the data file is a small file based on a preset size; and / or, determine whether the data file is a data file created since the last backup based on the creation time of the data file; if it is determined that the data file is a data file created since the last backup, determine whether the data file is a small file based on the preset size.

[0160] Optionally, the processing module 501 is specifically used to: for any traversed data file, determine a data file that has not been modified for more than a preset time as an old-age data file based on the modification time of the data file; when at least one old-age data file is determined, store the at least one old-age data file in a designated storage area.

[0161] Optionally, the merging module 502 is specifically used to: sequentially write at least part of the data files of at least two small files into the same designated file to obtain a merged file in at least one merged file; and record the position information of at least part of the data files in a merged file in a configuration file corresponding to the merged file according to the writing order and the size of at least part of the data files.

[0162] The block-level incremental backup device of a database provided in an embodiment of the present application can be used to execute the technical solution of the block-level incremental backup method of a database in any of the above embodiments of the present application. The implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0163] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the electronic device of this embodiment may include: at least one processor 601; and a memory 602 communicatively connected to the at least one processor; wherein the memory 602 stores instructions executable by the at least one processor 601, and the instructions are executed by the at least one processor 601 so that the electronic device executes a method as in any of the above embodiments.

[0164] Optionally, the memory 602 may be independent or integrated with the processor 601 .

[0165] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0166] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any of the above embodiments is implemented.

[0167] An embodiment of the present application further provides a computer program product, including a computer program, which implements the method described in any of the above embodiments when executed by a processor.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0169] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0170] It should be understood that the above-mentioned processor can be a processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include RAM (Random Access Memory), and may also include NVM (Non-Volatile Memory), such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0171] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0172] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in a dedicated integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0173] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0174] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0176] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

[0177] Those skilled in the art will readily come up with other implementations of the embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modifications, uses or adaptations of the embodiments of the present application, which follow the general principles of the embodiments of the present application and include common knowledge or customary technical means in the art that are not disclosed in the embodiments of the present application.

[0178] It should be understood that the embodiments of the present application are not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the embodiments of the present application is limited only by the appended claims.

Claims

1. A block-level incremental backup method for a database, characterized in that: The method comprises: Traversing the data files in the database; For any of the traversed data files, determine whether the data file is a small file according to a preset size, wherein the small file is a data file whose file size is smaller than the preset size; When at least two data files that are small files are determined, merging the at least two data files that are small files to obtain at least one merged file; The at least one merged file is divided into blocks to obtain at least one data block, and a block-level incremental backup is performed according to the at least one data block.

2. The method according to claim 1, characterized in that The performing block-level incremental backup according to the at least one data block comprises: According to a preset change condition, determining a first target data block that satisfies the preset change condition in the at least one data block, wherein the preset change condition includes a preset condition for determining a change range of a data block since a last backup; When at least one first target data block is determined, a block-level incremental backup is performed on the at least one first target data block in a sequential input and output manner.

3. The method according to claim 2, characterized in that The method further comprises: According to a preset change condition, determining a second target data block that does not meet the preset change condition in the at least one data block; When at least one second target data block is determined, data files in the at least one second target data block that have been changed since the last backup are referenced to a new data block to obtain at least one referenced data block; The at least one reference data block is backed up at a block level incrementally in a sequential input and output manner.

4. The method according to any one of claims 1 to 3, characterized in that: The determining whether the data file is a small file according to a preset size includes: Determining whether the data file has been changed since the last backup according to the modification time of the data file; In the case where it is determined that the data file has changed since the last backup, determining whether the data file is a small file according to a preset size; and / or, Determining, according to the creation time of the data file, whether the data file is a data file created after the last backup; In the case where it is determined that the data file is a data file created after the last backup, it is determined whether the data file is a small file according to a preset size.

5. The method according to any one of claims 1 to 3, characterized in that: The traversing of the data files in the database includes: For any of the traversed data files, according to the modification time of the data files, a data file that has not been modified for more than a preset time is determined as an old data file; When at least one old age data file is determined, the at least one old age data file is stored in a designated storage area.

6. The method according to any one of claims 1 to 3, characterized in that: The step of merging the at least two small data files to obtain at least one merged file includes: Sequentially write at least part of the at least two small data files into the same designated file to obtain a merged file among the at least one merged file; According to the writing order and the size of at least part of the data files, the position information of at least part of the data files in the merged file is recorded in the configuration file corresponding to the merged file.

7. A block-level incremental backup device for a database, characterized in that: The device comprises: A processing module, used for traversing the data files in the database; The processing module is further used to determine whether the data file is a small file according to a preset size, and the small file is a data file whose file size is smaller than the preset size; A merging module, configured to merge the at least two data files that are small files to obtain at least one merged file when at least two data files that are small files are determined; The block backup module is used to divide the at least one merged file into blocks to obtain at least one data block, and perform block-level incremental backup according to the at least one data block.

8. An electronic device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images