A data merging method and device for a database

Abstract

The present specification provides a data merging method and device for a database, wherein the database is used to store an incremental file containing incremental data and a baseline file containing baseline data, the baseline data in the baseline file is sequentially divided into a group of baseline data blocks; the method comprises: in response to a merging instruction for the incremental file, reading the incremental data in the incremental file; determining the baseline data block corresponding to each piece of incremental data in the baseline file; wherein the baseline data in each baseline data block and the corresponding incremental data constitute a to-be-written data set; generating a target file and sequentially writing each to-be-written data set into a target data block in the target file; wherein at least one target data block reserves a preset proportion of storage space.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of database, and in particular, to a data merging method and device for a database. BACKGROUND

[0002] Data in a database can be managed based on a hierarchical structure, so that the data in the database is stored in a plurality of layers of files. Figure 1 For example, a database implemented based on a log-structured merge tree structure (LSM-Tree) can merge (merge or Major Compaction) an incremental file containing incremental data in Ck-1 on a disk with a baseline file containing baseline data in Ck on the disk after a MemTable in memory is dumped (dump or Minor Compaction) to an SSTable on the disk, so as to write the merged data into a data block of a new file in the next layer, so as to save storage space and improve data query efficiency.

[0003] In related technologies, the number of data blocks in the new file is easily affected by the data of the data blocks contained in the incremental file and the baseline file, resulting in that the data blocks do not fully store the merged data, and the space waste is serious. Therefore, a data merging scheme with higher space utilization is needed. SUMMARY

[0004] Therefore, the present specification provides a data merging method and device for a database to solve the problems in related technologies.

[0005] Specifically, the present specification is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present specification, a data merging method for a database is provided, the database being configured to store an incremental file containing incremental data and a baseline file containing baseline data, the baseline data in the baseline file being sequentially divided into a group of baseline data blocks; the method comprising:

[0007] reading the incremental data in the incremental file in response to a merging instruction for the incremental file;

[0008] determining the baseline data blocks corresponding to each piece of incremental data in the baseline file; wherein the baseline data in each baseline data block and the corresponding incremental data constitute a to-be-written data set;

[0009] The target file is generated, and each to-be-written data set is written into a target data block in the target file in sequence; wherein at least one target data block reserves a preset proportion of storage space.

[0010] According to a second aspect of the embodiments of the present specification, a data merging device for a database is provided, the database is used to store an incremental file containing incremental data and a baseline file containing baseline data, the baseline data in the baseline file is sequentially divided into a group of baseline data blocks; the device comprises:

[0011] an incremental data reading unit, configured to read the incremental data in the incremental file in response to a merging instruction for the incremental file;

[0012] a data set forming unit, configured to determine the baseline data block corresponding to each piece of incremental data in the baseline file; wherein the baseline data in each baseline data block and the corresponding incremental data form a to-be-written data set;

[0013] a data set writing unit, configured to generate a target file, and write each to-be-written data set into a target data block in the target file in sequence; wherein at least one target data block reserves a preset proportion of storage space.

[0014] According to a third aspect of the embodiments of the present specification, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, the program is executed by a processor to implement the steps of the method according to the first aspect.

[0015] According to a fourth aspect of the embodiments of the present specification, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method according to the first aspect.

[0016] In the technical solutions provided in the specification, the database can sequentially write each to-be-written data set into a target data block in a target file, and make at least one target data block reserve a preset proportion of storage space. In the technical solutions, the target data block reserves the preset proportion of storage space, which can be used to store subsequent incremental data, thereby avoiding the case of adding a new data block due to storing additional incremental data, reducing the probability of generating a new block, and improving the space utilization rate of the data block. In addition, since the merged data can be written into a newly generated target file, the storage space of the data block in the target file after each merge is actually redistributable, for example, two to-be-written data sets are written into the same target data block, or a to-be-written data set with a large data volume is written into two continuous target data blocks. The baseline data and the incremental data maximally avoid interfering with the storage space of each target data block, further improving the storage space utilization rate on the basis of reserving storage space for each target data block.

[0017] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the specification. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the specification or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the specification, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0019] Figure 1 is a schematic diagram of an architecture of a database based on a log-structured merge tree structure according to an example embodiment of the specification;

[0020] Figure 2 is a flowchart of a data merging method for a database according to an example embodiment of the specification;

[0021] Figure 3a is a schematic diagram of a data merging method in different scenarios according to an example embodiment of the specification;

[0022] Figure 4 is a schematic structural diagram of an electronic device according to an example embodiment of the specification;

[0023] Figure 5 is a structural schematic diagram of a data merging apparatus for a database according to an example embodiment of the specification. DETAILED DESCRIPTION

[0024] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description herein relates to the drawings, in which the same numbers represent the same or similar elements throughout several views. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the present description. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present description.

[0025] It is noted that the steps of the methods in other embodiments need not necessarily be performed in the order shown and described in the present description. In some other embodiments, the steps of the methods can be more or less than those described in the present description. Furthermore, a single step described in the present description can be split into multiple steps in other embodiments; and multiple steps described in the present description can be combined into a single step in other embodiments. It is to be understood that although the terms first, second, third, etc. can be employed in this description to describe various information, these information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information of the same type. For example, a first information can also be termed a second information, and similarly, a second information can also be termed a first information, without departing from the scope of the present description. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "upon" or "in response to determining."

[0026] In the related art, a database can store data in the form of files, and take a data block in a file as a basic unit of data read and write. Specifically, data can be stored in data blocks of a corresponding file in sequence according to storage time, however, the order of the data itself is often irrelevant to the storage time, causing the database to be difficult to establish a corresponding index, the query efficiency of the data to be low, and different versions of the same data to be respectively stored in different data blocks, causing data management to be chaotic. In summary, in order to ensure the order and effectiveness of the data, the database needs to constantly move and merge the data, thereby wasting a large amount of time and space.

[0027] Based on this, the traditional database usually adopts an in-place update manner to solve the above problems. The in-place update refers to replacing old data in the database with new data without moving or copying the data. For example, new data is directly inserted into the data block where the original data is located, or the data block to be deleted is deleted from the data block where the original data is located. This manner can save time and space, especially when the database is large. However, the space occupied by the data after in-place update is often greater than or much smaller than the space occupied by the original data. For example, in the case where the data block where the original data is located has been completely written, any new data cannot continue to be written. If there is no remaining space in the current data block at this time, row migration occurs, that is, the database cannot complete the migration of the data stored in the above current data block to a new data block, and only sets a pointer in the original space, the pointer points to the address of the above new data block, and the remaining space of the original space is not used by other data blocks. It can be understood that the above remaining space will cause waste of storage space and additional I / O (Input / Output) overhead in the data access process. For another example, in the case where the data block where the original data is located has been written with incremental data for deleting data, resulting in a large amount of remaining storage space, other new data cannot be written into the above storage space because it can only be written into the corresponding original data.

[0028] Of course, in order to solve the above problems caused by in-place update, the parameters of pctfree and pctused can be introduced to control the use of data block space to reduce the lack of available space of data block when in-place update occurs, for example, when the space utilization rate of data block reaches 1-pctfree, the database can stop inserting data into the data block, and the remaining space is only used for data update operation, or when the space utilization rate of the stopped data block falls below pctused, the data block is re-enabled to insert data. For example, the pctfree of the data block in the database is set to 10%, and the pctused is set to 40%, then when the space utilization rate of the data block reaches 90% (i.e. 1-10%), the database will stop inserting data into the data block (but the inserted data can still be updated), and then some deletion operations occur on the data block, so that the utilization rate of the data block decreases, and when the space utilization rate falls below 40%, the database can use the data block for data insertion again. However, the above method of controlling the remaining space of data block by parameters still has potential risks, for example, in the case that the space utilization rate of the original data block reaches 90%, the subsequent inserted data can only be stored in a new data block (even if the remaining space of the original data block is sufficient to store the data to be inserted), resulting in waste of storage space; or, in the case that the space utilization rate of the original data block reaches 90%, the updated data is exactly 10%, so that the space utilization rate of the original data block is 100%, then in the subsequent data update process, the data block will not have extra space to store new data, in other words, the in-place update method causes the data block to fail to continue to play the role of the remaining space; or, in the case that the space utilization rates of the first data block and the second data block reach 90%, 80% of the space is deleted respectively, so that the space utilization rates of the two data blocks are only 10% (90%-80%), and a total of 180% of storage space is wasted. In summary, even if the technical means of remaining space is used, the data block based on in-place update still has a common problem of space waste.

[0029] Figure 2 is a flowchart of a data merging method for a database according to an exemplary embodiment of the present specification, as shown in Figure 2 The above database is used to store an incremental file containing incremental data and a baseline file containing baseline data, and the baseline data in the baseline file is sequentially divided into a group of baseline data blocks; the method can include the following steps:

[0030] S201, in response to a merging instruction for the incremental file, reading the incremental data in the incremental file.

[0031] When the incremental file needs to be merged, the corresponding incremental data can be read from the incremental file; the specific content of the incremental data can be determined according to the corresponding data operation, for example, when the incremental data is insertion / update data, the incremental data can include the key of the corresponding data row primary key and the inserted / updated content; or when the incremental data is deletion data, the incremental data can include the key of the corresponding data row primary key and the tombstone mark. The above-mentioned merging instruction can be triggered by the user actively controlling the database software, or triggered automatically by the database according to the preset merging condition (for details, see the above-mentioned merging part of SSTable), and the present specification does not limit this.

[0032] It should be noted that the number of incremental files to which the above-mentioned merging instruction is directed can be plural. Specifically, in the LSM-Tree structure greater than the zeroth layer, the key in each SSTable file can be stored in order from small to large, and the key range (i.e. the range formed by the minimum key and the maximum key of the data contained in the file) between different SSTable files will not overlap. However, considering that the zeroth layer SSTable file is directly generated by dumping, so any two zeroth layer SSTable files may have overlapping key ranges. Therefore, when merging, for layers greater than the zeroth layer, one of the files can be selected as an incremental file, but for the zeroth layer, after a certain incremental file is specified, it is likely that there are other incremental files in this layer whose key range overlaps with this file, at this time all overlapping SSTable files can be found as incremental data and the baseline file of the next layer to be merged, that is, the database when selecting the incremental file for the zeroth layer, can involve one or more incremental files. Similarly, the above-mentioned merging instruction can also correspond to multiple baseline files, specifically, all files in the next layer that overlap with the incremental file of the current layer in the key range can be selected as the baseline file to be merged with the incremental file.

[0033] S202, determine the baseline data block corresponding to each incremental data in the baseline file; wherein the baseline data in each baseline data block and the corresponding incremental data constitute a to-be-written data set.

[0034] As mentioned above, the SSTable as the baseline file can record the index of each baseline data block contained therein, and further determine the key range of each baseline data block, so that each piece of incremental data can determine the corresponding baseline data block in the baseline file according to its own key, wherein the baseline data in each baseline data block and the corresponding incremental data can constitute a set of data to be written. For example, there is a baseline file containing baseline data blocks A, B and C, wherein the key range of the baseline data block A is [0, 99], the key range of the baseline data block B is [100, 199], and the key range of the baseline data block C is [200, 299], the incremental data of an incremental file is incremental data A: {50, new content} and incremental data B: {250, updated content}, obviously, since the key of the incremental data A is located in the baseline data block A and the key of the incremental data B is located in the baseline data block C, the incremental data A and the baseline data block A, and the incremental data B and the baseline data block B can be respectively regarded as a set of data to be written, in addition, the baseline data block B which does not correspond to any incremental data can be reused in this merging process, so as to reduce unnecessary I / O overhead.

[0035] S203, generating a target file, and sequentially writing each set of data to be written into a target data block in the target file; wherein at least one target data block reserves a preset proportion of storage space.

[0036] In order to store the data after merging the incremental data and the baseline data, a new target file can be generated, and each set of data to be written can be sequentially written into a target data block in the target file, and by reserving a preset proportion of storage space for at least one target data block, it can be ensured that the target data block of the target file after the merging operation has the ability to accommodate subsequent incremental data.

[0037] It should be noted that the technical solution is targeted for the storage space design of the merging operation when the baseline file does not exist (i.e. the first merging operation).

[0038] In an embodiment, when the above baseline file does not exist, a baseline file is generated, and the incremental data is sequentially written into a baseline data block in the baseline file; wherein at least one baseline data block reserves a preset proportion of storage space. In this embodiment, since the baseline file does not exist, the incremental data itself can be regarded as baseline data. For example, as mentioned above, there is a baseline file containing baseline data blocks A, B and C, wherein the key range of the baseline data block A is [0, 99], the key range of the baseline data block B is [100, 199], and the key range of the baseline data block C is [200, 299], the incremental data of an incremental file is incremental data A: {50, new content} and incremental data B: {250, updated content}, obviously, since the key of the incremental data A is located in the baseline data block A and the key of the incremental data B is located in the baseline data block C, the incremental data A and the baseline data block A, and the incremental data B and the baseline data block B can be respectively regarded as a set of data to be written, in addition, the baseline data block B which does not correspond to any incremental data can be reused in this merging process, so as to reduce unnecessary I / O overhead. Figure 3aAs shown, assuming that the data amount in an incremental file first reaches the preset merging threshold, the database can merge it into the target file of the next layer, wherein the target data block in the target file can reserve a preset proportion of storage space compared with the incremental data that completely fills the data block. The target data block with reserved space and the preset proportion can be changed according to the actual use scenario. For example, in the case where the average unit data amount of the incremental data is small, space can be reserved for all target data blocks, and each target data block is guaranteed to reserve 10% of the storage space to facilitate the storage of subsequent incremental data (for example Figure 3a Of course, the last data block of the target file can be floating with the total size of the incremental data, so that the remaining space x of the data block m is greater than 10%.

[0039] Of course, the present application also designs a series of technical solutions for the merging operation when the baseline file exists. First, the method for the database to determine whether each target data block is set to reserve a preset proportion of storage space is explained.

[0040] In an embodiment, the database can determine the current target data block corresponding to any to-be-written data set in the target file during the process of writing any to-be-written data set into the target file, and calculate the sum of the written data of the current target data block and the data amount of the baseline data contained in the any to-be-written data set. In the case where the calculated sum of data amounts is less than the maximum storage data amount of the current target data block, it can be set that no storage space is reserved for the current target data block. In the case where the calculated sum of data amounts is greater than or equal to the maximum storage data amount, it can be set that the current target data block reserves the preset proportion of storage space. Wherein, the database can control whether the current target data block reserves the preset proportion of storage space through the setting operation of reserving storage space, and the setting operation can be performed before each to-be-written data set is written or at the same time as the writing of the preset proportion of storage space, which is not limited in the present specification.

[0041] The technical solutions of the above embodiment are introduced below in combination with Figures 3b-4 d. multiple data merging scenarios.

[0042] For example, as shown in FIG. 1, the baseline file has 10 data blocks, and the data amount of each data block is 1000 bytes. The incremental file has 5 data blocks, and the data amount of each data block is 1000 bytes. The preset merging threshold is 1000 bytes, and the preset proportion of storage space is 10%. Figure 3bAs shown, it is assumed that only two baseline data blocks 1, 2 in the baseline data are modified in the incremental data, specifically, there is insertion data for the baseline data block 1, data volume accounting for 2% of the data block 2, and insertion data for the baseline data block 2, data volume accounting for 3% of the data block 3. In other words, the baseline data block 1 and the corresponding incremental data constitute the to-be-written data set 1 accounting for 92% of the data block, and the baseline data block 2 and the corresponding incremental data constitute the to-be-written data set 2 accounting for 93% of the data block 3. Then, in the process of sequentially writing the to-be-written data sets 1, 2, a target data block 1 can be newly created in the generated target file, and it is assumed that the target data block 1 to be written in the to-be-written data set 1 is taken as the current target data block, the remaining space is 100%, and the data volume of the written data is 0%. Obviously, the sum of the current target data block and the baseline data of the to-be-written data set 1 is still 90% (0% + 90%), and the calculated data volume sum is less than the maximum storage data volume of the current target data block (100%). Therefore, the database can be set not to reserve storage space for the current target data block, and the baseline data and the incremental data in the to-be-written data set 1 are written into the target data block 1. When the to-be-written data set 1 is written, a target data block 2 can be newly created as the current target data block, the data volume of the written data is 0%, the sum of the current target data block and the baseline data of the to-be-written data set 2 is still 90% (0% + 90%), and the calculated data volume sum is less than the maximum storage data volume of the current target data block (100%). Therefore, the database can be set not to reserve storage space for the current target data block, and the baseline data and the incremental data in the to-be-written data set 2 are written into the target data block 2. When the to-be-written data set 2 is written, the comparison between the target file and the baseline file shows that the databases of the two are not increased due to the new data, thereby reducing the space waste caused by the data merging in the related art. In addition, for any baseline data block in the incremental file, since there is no corresponding incremental data, any baseline data block can be reused in the target file to save I / O overhead.

[0043] It should be emphasized that, since the writing process of the to-be-written data set actually belongs to a mode similar to stream processing, the database cannot predict the data volume of the incremental data in each to-be-written data set in advance, and determine whether to set the reserved storage space for the current target data block according to the data volume. Similarly, due to the unpredictability of the incremental data in the writing process, there can be a single target data block that can store the baseline data of the to-be-written data set (i.e., the sum of the data volume is less than the maximum storage data volume of the current target data block), but cannot store the incremental data of the to-be-written data set. Based on this, the present specification proposes a storage scheme for larger incremental data.

[0044] In an embodiment, in the case that any to-be-written data set cannot be completely written into the current target data block and continues to occupy the subsequent one or more target data blocks, a preset proportion of storage space is reserved for the subsequent one or more target data blocks. As shown in Figure 3c Assuming that only baseline data block 1 in the baseline data is modified in the incremental data, specifically, there is 150% of the data block of the insertion data for baseline data block 1 in the incremental data, in other words, baseline data block 1 and the corresponding incremental data constitute 240% of the to-be-written data set 1, then in the process of writing the to-be-written data set 1, target data block 1 can be newly created in the generated target file, assuming that the target data block 1 to be written into the to-be-written data set 1 at this time is the current target data block, the remaining space is 100%, and the data amount of the written data is 0%. Obviously, the sum of the current target data block and the baseline data of the to-be-written data set 1 is still 90% (0% + 90%), and the calculated sum of the data amount is less than the maximum storage data amount (100%) of the current target data block, so the database can set not to reserve storage space for the current target data block, and write the baseline data and the incremental data in the to-be-written data set 1 into the target data block 1. When the target data block 1 is completely written (i.e., the remaining space is 0%), the database determines that the incremental data of the to-be-written data set 1 has not been completely written, so target data block 2 can be newly created after target data block 1 to continue writing the to-be-written data set 1 (at this time, the to-be-written data set 1 still has 140% of the data amount), and a preset proportion (assuming 10%) of storage space can be reserved for the target data block 2, so that the target data block 2 stops writing after being written into 90% of the to-be-written data set 1, and target data block 3 is newly created after the target data block 2 to continue writing the to-be-written data set 3 (at this time, the to-be-written data set 1 still has 50% of the data amount), and a preset proportion of storage space can be reserved for the target data block 3, so that the target data block 3 stops writing after being written into 90% of the to-be-written data set 1. Of course, at this time, since the to-be-written data set 1 only has 50% of the data amount and the target data block after the target data block 3 is a multiplexed data block, the target data block 3 can stop writing after being written into 50% of the to-be-written data set 1. In this embodiment, when the incremental data of any to-be-written data set is extremely large, the above method can maximize the guarantee that the target data block can reserve a preset proportion of storage space, thereby saving the storage space waste caused by the subsequent merging operation while maintaining the storage space utilization rate.

[0045] Next, combined with Figure 3d The case that the sum of the data amount calculated above is greater than or equal to the maximum storage data amount is exemplified.

[0046] In an embodiment, it is assumed that only three consecutive baseline data blocks 1, 2, 3 in the baseline data are modified in the incremental data, specifically, there is 10% of the data block for baseline data block 1, 8% of the data block for baseline data block 2, and 8% of the data block for baseline data block 3 in the incremental data. In other words, baseline data block 1 and the corresponding incremental data constitute 80% of the data block of the to-be-written data set 1, baseline data block 2 and the corresponding incremental data constitute 98% of the data block of the to-be-written data set 2, and baseline data block 3 and the corresponding incremental data constitute 98% of the data block of the to-be-written data set 3. Then in the process of sequentially writing the to-be-written data sets 1, 2, 3, a target data block 1 can be newly created in the generated target file, and it is assumed that the target data block 1 to be written in the to-be-written data set 1 is taken as the current target data block, the remaining space of which is 100%, and the data amount of the written data is 0%. Obviously, the sum of the current target data block and the baseline data of the to-be-written data set 1 is 90% (0%+90%), and the calculated sum of the data amount is less than the maximum storage data amount (100%) of the current target data block, so the database can be set not to reserve storage space for the current target data block, and the baseline data and the incremental data in the to-be-written data set 1 are written into the target data block 1. When the to-be-written data set 1 is written, the remaining space of the target data block 1 is 20%, and at this time, the to-be-written data set 2 is attempted to be written into the target data block 1. It is found that the sum of the written data of the target data block 1 and the baseline data of the to-be-written data set 2 (80%+98%) is greater than the maximum storage data amount (100%) of the target data block 1, so the to-be-written data set 2 continues to be written into the target data block 1 until the target data block 1 has a reserved space of 10% (at this time, the data amount of the to-be-written data set 2 is 98%-10%), and then the target data block 1 is written into a non-volatile storage such as a disk, and a target data block 2 is newly created to be prepared as the current target data block. It can be understood that in the case that the maximum storage data amount of the baseline data block in the baseline file and the target data block in the target file is the same, since the target data block 2 is a newly created data block, the data amount of the written data must be 0%, and the baseline data in the to-be-written data set 2 must be completely written into the target data block 2, so the next target data block (i.e., the target data block 2) of the current data block (i.e., the target data block 1) can be set not to reserve the preset proportion of the storage space without considering the incremental data. Since the target data block 1 has written 10% of the data amount in the to-be-written data set 2, the target data block 2 still has a storage space of 12% under the condition of completely writing the to-be-written data set 2.At this time, the data set 3 to be written is attempted to be written into the target data block 2, and it is found that the sum of the written data of the target data block 2 and the baseline data of the data set 3 to be written (88%+98%) is greater than the maximum storage data amount (100%) of the target data block 2, so the data set 3 to be written can continue to be written into the target data block 2 until the target data block 2 has 10% of the reserved space left (at this time, the data amount of the data set 2 to be written is 98%-2%), then the target data block 1 is written into the disk, and a new target data block 3 is created to prepare as the current target data block, and it is set that no storage space of the preset proportion is reserved for the next target data block (i.e. the target data block 3), and when the data set 3 to be written is completely written into the target data block 3, it is found that there is 4% of the storage space left, and it is assumed that the target data block 3 is followed by a multiplexed data block, so the target data block 3 can be written into the disk to complete the entire merging process.

[0047] In the above example, the setting of the storage space of any target data block can be changed, and is set according to the last change, for example Figure 3d The target data block 2 in the above example is set not to reserve storage space during the writing of the data set 1 to be written, but is re-set to reserve storage space during the subsequent writing of the data sets 2 and 3 to be written.

[0048] In addition, the present specification can be combined with Figure 3d and 3e to explain the writing of the data blocks before and after the multiplexed data block into the data set to be written.

[0049] In an embodiment, during the writing of any data set to be written into a target file, if the last data block in the target file is a multiplexed baseline data block, the next target data block can be created after the multiplexed baseline data block for writing any data set to be written; if the last data block in the target file is not a multiplexed baseline data block, the last data block is used to write the data set to be written. Wherein, the multiplexed baseline data block is a data block that is not reserved for writing the data set to be written. Figure 3eFor example, it is assumed that only two baseline data blocks 1 and 2 in the baseline data are modified in the incremental data, specifically, there is 1% of the data block 2 of the deletion data for the baseline data block 1 and 1% of the data block 2 of the insertion data for the baseline data block 2. In other words, the baseline data block 1 and the corresponding incremental data constitute 88% of the data block of the to-be-written data set 1, and the baseline data block 2 and the corresponding incremental data constitute 92% of the data block of the to-be-written data set 2. During the writing process of the to-be-written data set 1, since the last data block in the target file (i.e., the data block before the target data block 1) is a reused baseline data block, the next target data block (i.e., the target data block 1) can be created after the reused baseline data block for writing the to-be-written data set 1; during the writing process of the to-be-written data set 2, since the last data block in the target file (i.e., the data block before the target data block 2) is a reused baseline data block, the next target data block (i.e., the target data block 2) can be created after the reused baseline data block for writing the to-be-written data set 2. Meanwhile, it is assumed that the last data block in the target file (i.e., the data block before the target data block 1) is a reused baseline data block for the to-be-written data set 2, and the last data block in the target file (i.e., the data block before the target data block 2) is a reused baseline data block for the to-be-written data set 3. Figure 3d For example, it is assumed that only two baseline data blocks 1 and 2 in the baseline data are modified in the incremental data, specifically, there is 1% of the data block 2 of the deletion data for the baseline data block 1 and 1% of the data block 2 of the insertion data for the baseline data block 2. In other words, the baseline data block 1 and the corresponding incremental data constitute 88% of the data block of the to-be-written data set 1, and the baseline data block 2 and the corresponding incremental data constitute 92% of the data block of the to-be-written data set 2. During the writing process of the to-be-written data set 1, since the last data block in the target file (i.e., the data block before the target data block 1) is a reused baseline data block, the next target data block (i.e., the target data block 1) can be created after the reused baseline data block for writing the to-be-written data set 1; during the writing process of the to-be-written data set 2, since the last data block in the target file (i.e., the data block before the target data block 2) is a reused baseline data block, the next target data block (i.e., the target data block 2) can be created after the reused baseline data block for writing the to-be-written data set 2. Meanwhile, it is assumed that the last data block in the target file (i.e., the data block before the target data block 1) is a reused baseline data block for the to-be-written data set 2, and the last data block in the target file (i.e., the data block before the target data block 2) is a reused baseline data block for the to-be-written data set 3.

[0050] It can be known from the above embodiments that in the technical solution of the present specification, the database can sequentially write each to-be-written data set into a target data block in the target file, and at least one target data block is reserved with a preset proportion of storage space. In the technical solution, the target data block is reserved with a preset proportion of storage space, which can be used to store subsequent incremental data, avoiding the case of adding a new data block due to storing additional incremental data, reducing the probability of generating a new block, and improving the space utilization rate of the data block. In addition, since the merged data can be written into a newly generated target file, the storage space of the data block in the target file after each merging is actually redistributable, for example, two to-be-written data sets are written into the same target data block, or a to-be-written data set with a larger data volume is written into two consecutive target data blocks. The baseline data and the incremental data interfere with the storage space of each target data block to the greatest extent, and the storage space utilization rate is improved on the basis of reserving storage space for each target data block.

[0051] Figure 4Fig. 1 is a schematic structural diagram of an electronic device according to an example embodiment. Please refer to Figure 4 At the hardware level, the electronic device comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory, and can further comprise other required hardware. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, and at the logical level, forms a data merging apparatus for a database. Of course, the present specification does not exclude other implementation manners, such as a logic device or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to the logical units, but can also be hardware or a logic device.

[0052] Corresponding to the above-mentioned embodiment of the data merging method for a database, the present specification further provides an embodiment of a data merging apparatus for a database.

[0053] Fig. 2 is a schematic structural diagram of a data merging apparatus for a database according to an example embodiment. Please refer to Figure 5 Figure 5 Fig. 3 is a schematic structural diagram of a data merging apparatus for a database according to an example embodiment. As shown in the figure, in the software implementation manner, the database is used to store an incremental file containing incremental data and a baseline file containing baseline data, the baseline data in the baseline file is orderly divided into a group of baseline data blocks; the apparatus can comprise: Figure 5

[0054] An incremental data reading unit 501 is configured to read the incremental data in the incremental file in response to a merging instruction for the incremental file.

[0055] A data set constituting unit 502 is configured to determine the baseline data block corresponding to each piece of incremental data in the baseline file; wherein the baseline data in each baseline data block and the corresponding incremental data constitute a to-be-written data set.

[0056] A data set writing unit 502 is configured to generate a target file and write each to-be-written data set into a target data block in the target file in sequence; wherein at least one target data block reserves a preset proportion of storage space.

[0057] Optionally, the apparatus further comprises:

[0058] A first data amount comparison unit 504 is configured to determine the current target data block corresponding to any to-be-written data set in the target file in the process of writing the to-be-written data set into the target file.

[0059] Calculate the sum of the written data of the current target data block and the data amount of the baseline data contained in the any to-be-written data set.

[0060] ​​In a case where the sum of the calculated data amounts is less than the maximum storage data amount of the current target data block, it is determined not to reserve storage space for the current target data block.

[0061] In a case where the sum of the calculated data amounts is greater than or equal to the maximum storage data amount, it is determined to reserve the preset proportion of storage space for the current target data block.

[0062] Optionally, the maximum storage data amount of the baseline data block in the baseline file is the same as that of the target data block in the target file, and the apparatus further comprises:

[0063] The second data amount comparison unit 505 is configured to, in a case where the sum of the calculated data amounts is greater than or equal to the maximum storage data amount, determine not to reserve the preset proportion of storage space for the next target data block.

[0064] Optionally, the apparatus further comprises:

[0065] The data set complete writing unit 506 is configured to, in a case where the any to-be-written data set cannot be completely written into the current target data block and continues to occupy one or more subsequent target data blocks, determine to reserve the preset proportion of storage space for the one or more subsequent target data blocks.

[0066] Optionally, the apparatus further comprises:

[0067] The baseline file generation unit 507 is configured to, in a case where the baseline file does not exist, generate the baseline file and sequentially write the incremental data into baseline data blocks in the baseline file, wherein at least one baseline data block reserves a preset proportion of storage space.

[0068] Optionally, the apparatus further comprises:

[0069] The baseline data block multiplexing unit 508 is configured to, in a case where any baseline data block in the baseline file does not have corresponding incremental data, multiplex the any baseline data block into the target file.

[0070] Optionally, the apparatus further comprises:

[0071] The target data block creating unit 509 is configured to, in a process of writing any to-be-written data set into the target file, if the last data block in the target file is a multiplexed baseline data block, create a next target data block after the multiplexed baseline data block for writing the any to-be-written data set; if the last data block in the target file is not a multiplexed baseline data block, use the last data block for writing the any to-be-written data set.

[0072] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0073] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the scheme of the present specification according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0074] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier to be executed by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0075] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit), and the apparatus can also be implemented as special purpose logic circuitry.

[0076] Computers suitable for the execution of a computer program include, by way of example, general and / or special purpose microprocessors or any other kind of processing unit. Generally, a processing unit will receive instructions and data from a read-only memory and / or a random access memory. The essential elements of a computer are a processing unit for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0077] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0078] While the specification contains many specifics, these should not be construed as limiting the scope of any invention or of any claim, but as merely providing illustrations of some of the embodiments of the inventions. Certain features that are, for clarity, described above in the context of separate embodiments, can also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described above in the context of a single embodiment, can also be provided separately or in any suitable subcombination. In addition, while features can be described above as being implemented in digital electronic circuitry, forms of hardware implementation can include, but are not limited to, analog electronic circuitry, or forms of hardware implementation can be implemented together with software implementations. In fact, the disclosure covers hardware implementations whether or not any software is permitted. Moreover, while embodiments have been described above in the context of fully functioning apparatuses, the disclosure covers both apparatuses and partial apparatuses, and both methods of using apparatuses and methods of manufacturing apparatuses. Accordingly, the disclosure covers both the following.

[0079] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring or implying that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0080] Thus, specific embodiments of the subject matter have been described. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0081] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A data merging method for a database, wherein the database is used to store an incremental file containing incremental data and a baseline file containing baseline data, wherein the baseline data in the baseline file is orderly divided into a set of baseline data blocks; The method comprises: In response to a merge instruction for the incremental file, reading incremental data in the incremental file; Determine the baseline data blocks corresponding to each incremental data in the baseline file; wherein the baseline data in each baseline data block and the corresponding incremental data constitute the data set to be written; A target file is generated, and each data set to be written is sequentially written into a target data block in the target file; wherein a preset proportion of storage space is reserved for at least one target data block.

2. The method according to claim 1, further comprising: In the process of writing any data set to be written into the target file, determining a current target data block corresponding to the data set to be written into the target file; Calculating the sum of the amount of data already written to the current target data block and the amount of baseline data contained in any data set to be written; When the calculated sum of the data amounts is less than the maximum storage data amount of the current target data block, setting not to reserve storage space for the current target data block; When the sum of the calculated data amounts is greater than or equal to the maximum storage data amount, the preset proportion of storage space is reserved for the current target data block.

3. The method according to claim 2, wherein the baseline data block in the baseline file and the target data block in the target file have the same maximum storage data volume, and the method further comprises: When the sum of the calculated data amounts is greater than or equal to the maximum storage data amount, it is set not to reserve the preset proportion of storage space for the next target data block.

4. The method according to claim 2, further comprising: In the case that any of the to-be-written data sets cannot be completely written into the current target data block and continues to occupy one or more subsequent target data blocks, the preset proportion of storage space is reserved for the one or more subsequent target data blocks.

5. The method according to claim 1, further comprising: If the baseline file does not exist, the baseline file is generated, and the incremental data is written into the baseline data blocks in the baseline file in sequence; wherein a preset proportion of storage space is reserved for at least one baseline data block.

6. The method according to claim 1, further comprising: In the case that any baseline data block in the baseline file does not have corresponding incremental data, the baseline data block is reused in the target file.

7. The method according to claim 6, further comprising: During the process of writing any data set to be written into the target file, if the last data block in the target file is a reused baseline data block, then creating a next target data block after the reused baseline data block for writing any data set to be written; If the last data block in the target file is not a reused baseline data block, the last data block is used to write into any one of the data sets to be written.

8. A data merging device for a database, wherein the database is used to store incremental files containing incremental data and baseline files containing baseline data, wherein the baseline data in the baseline files is orderly divided into a set of baseline data blocks; The device comprises: an incremental data reading unit, configured to read the incremental data in the incremental file in response to a merge instruction for the incremental file; The data set forming unit is used to determine the baseline data blocks corresponding to each incremental data in the baseline file; wherein the baseline data in each baseline data block and the corresponding incremental data constitute the data set to be written; The data set writing unit is used to generate a target file and write each data set to be written into a target data block in the target file in sequence; wherein at least one target data block has a preset proportion of storage space reserved.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.

Images