Data processing method and device for target database

Abstract

Embodiments of the present specification provide a data processing method and device for a target database, the storage architecture of the target database comprises a first storage layer, a second storage layer and a third storage layer, the method comprises: determining the smallest primary key value in the primary key values of the first storage layer, the second storage layer and the third storage layer; in response to the source storage layer of the smallest primary key value only comprising the third storage layer, taking the smallest primary key value as a first boundary primary key value, and determining the smallest primary key value as a second boundary primary key value in the primary key values of the first storage layer and the second storage layer; obtaining, from the third storage layer, a data row whose primary key value is within the range of the first boundary primary key value and the second boundary primary key value, and outputting the obtained data row to a calculation operator of an upper layer; wherein the range does not contain the second boundary primary key value.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of database, and in particular, to a data processing method and device for a target database. BACKGROUND

[0002] At present, the storage architecture adopted by some databases includes multiple storage layers, and data with the same primary key value may be stored in different storage layers due to different writing and updating times. Based on such a situation, when computing in the database, such as executing push-down computation in the data query execution process, the multi-layer data needs to be fused first, which not only becomes a bottleneck of the computation, but also blocks the batch computation.

[0003] Therefore, there is an urgent need for a reasonable and reliable solution to improve the computing performance of the database adopting the above storage architecture. SUMMARY

[0004] Embodiments of the present specification provide a data processing method and device for a target database, which can improve the computing performance of the target database.

[0005] In a first aspect, the embodiments of the present specification provide a data processing method for a target database, the storage architecture of the target database includes a first storage layer, a second storage layer and a third storage layer, and the method includes: determining the smallest primary key value in the primary key values of the first storage layer, the second storage layer and the third storage layer; in response to the source storage layer of the smallest primary key value including only the third storage layer, taking the smallest primary key value as a first boundary primary key value, and determining the smallest primary key value as a second boundary primary key value in the primary key values of the first storage layer and the second storage layer; obtaining data rows with primary key values in the range of the first boundary primary key value and the second boundary primary key value from the third storage layer, and outputting the obtained data rows to an upper layer computing operator; wherein the range does not include the second boundary primary key value.

[0006] In some embodiments, the determining the smallest primary key value in the primary key values of the first storage layer, the second storage layer and the third storage layer includes: respectively reading the row of data with the smallest primary key value in each storage layer, thereby obtaining a first data row from the first storage layer, a second data row from the second storage layer, and a third data row from the third storage layer; determining the smallest primary key value in the primary key values of the first data row, the second data row and the third data row; and the determining the smallest primary key value in the primary key values of the first storage layer and the second storage layer includes: determining the smallest primary key value in the primary key values of the first data row and the second data row.

[0007] In some embodiments, each storage layer has a respective scan iterator for scanning data in its corresponding storage layer in ascending order of primary key values, and each scan iterator of each storage layer has been put into a loser tree; and the reading of a row of data with the smallest primary key value in each storage layer respectively comprises: reading a row of data in each scan iterator of each storage layer respectively by using the loser tree; the determining of the smallest primary key value from among the primary key values of the first data row, the second data row and the third data row comprises: determining the smallest primary key value from among the primary key values of the first data row, the second data row and the third data row by using the loser tree; and the determining of the smallest primary key value from among the primary key values of the first data row and the second data row comprises: determining the smallest primary key value from among the primary key values of the first data row and the second data row by using the loser tree.

[0008] In some embodiments, the determining of the smallest primary key value from among the primary key values of the first data row and the second data row by using the loser tree comprises: removing the scan iterator of the third storage layer from the loser tree; and reading the data row with the smallest primary key value from among the first data row and the second data row from the remaining scan iterators by using the loser tree, so as to determine the smallest primary key value.

[0009] In some embodiments, after the determining of the smallest primary key value, the method further comprises: putting the scan iterator of the third storage layer back into the loser tree.

[0010] In some embodiments, the method further comprises: creating the respective scan iterator of each storage layer; and putting the created scan iterators into the loser tree.

[0011] In some embodiments, after the determining of the smallest primary key value from among the primary key values of the first storage layer, the second storage layer and the third storage layer, the method further comprises: recording the source storage layer of the smallest primary key value.

[0012] In some embodiments, the third storage layer comprises a plurality of first sub-blocks, any first sub-block comprises a plurality of second sub-blocks, and any second sub-block comprises a plurality of data rows; and the obtaining of the data row with the primary key value in the range of the first boundary primary key value and the second boundary primary key value from the third storage layer comprises: determining each second sub-block satisfying the range in the second sub-blocks of the third storage layer; and obtaining the data row from the each second sub-block.

[0013] In some embodiments, any second sub-block records a maximum primary key value of each data row contained therein; and the determining each second sub-block satisfying the range comprises: determining each second sub-block whose recorded maximum primary key value is within the range.

[0014] In some embodiments, after the determining each second sub-block satisfying the range, the method further comprises: adding a target mark to each second sub-block respectively, which is used to indicate that the corresponding second sub-block satisfies the range; and the obtaining data rows from the each second sub-block comprises: sequentially traversing each second sub-block added with the target mark, so as to obtain data rows in the each second sub-block.

[0015] In some embodiments, the storage architecture is a Log-Structured Merge Tree (LSM-Tree).

[0016] In some embodiments, the first storage layer is located in memory, and the second storage layer and the third storage layer are located in disk, and data in each storage layer is arranged in ascending order based on primary keys.

[0017] In a second aspect, the embodiments of the present specification provide a data processing apparatus for a target database, a storage architecture of the target database comprising a first storage layer, a second storage layer and a third storage layer, the apparatus comprising: a first determining unit configured to determine a minimum primary key value from primary key values of the first storage layer, the second storage layer and the third storage layer; a second determining unit configured to, in response to a source storage layer of the minimum primary key value comprising only the third storage layer, take the minimum primary key value as a first boundary primary key value, and determine a minimum primary key value from primary key values of the first storage layer and the second storage layer as a second boundary primary key value; and a processing unit configured to obtain data rows with primary key values within a range of the first boundary primary key value and the second boundary primary key value from the third storage layer, and output the obtained data rows to a computing operator of an upper layer; wherein the range does not contain the second boundary primary key value.

[0018] In a third aspect, the embodiments of the present specification provide a computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer executes the method as described in any implementation manner of the first aspect.

[0019] In a fourth aspect, the embodiments of the present specification provide a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method as described in any implementation manner of the first aspect.

[0020] In a fifth aspect, an embodiment of the present specification provides a computer program, which, when executed in a computer, causes the computer to perform the method described in any implementation manner of the first aspect.

[0021] The target database in the scheme provided by the above embodiments of the present specification includes a first storage layer, a second storage layer and a third storage layer. In the scheme, the minimum primary key value can be determined among the primary key values of the first storage layer, the second storage layer and the third storage layer. Then, when the source storage layer of the minimum primary key value only includes the third storage layer, the minimum primary key value is taken as the first boundary primary key value, and the minimum primary key value is determined among the primary key values of the first storage layer and the second storage layer as the second boundary primary key value. Since the second boundary primary key value is the minimum primary key value among the primary key values of the first storage layer and the second storage layer, the data row whose primary key value is greater than or equal to the first boundary primary key value and less than the second boundary primary key value should only exist in the third storage layer, and should not exist in the first storage layer and the second storage layer.

[0022] Therefore, the data row whose primary key value is in the range of the first boundary primary key value and the second boundary primary key value can be obtained from the third storage layer in sequence, and the obtained data row is output to the upper computing operator. The range does not include the second boundary primary key value. By using such a data processing process, it is not necessary to judge whether the primary key value of the data row still exists in the first storage layer and the second storage layer, and the data fusion process can be avoided. In this way, the invalid data fusion process can be avoided, so that the overhead of fusion can be avoided as much as possible to become the bottleneck of calculation and block the batch calculation, thereby the computing performance of the target database can be greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in the present specification, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only a part of the embodiments disclosed in the present specification, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0024] Figure 1 is a schematic diagram of a storage architecture adopted by the target database;

[0025] Figure 2 is a structural schematic diagram of the target database;

[0026] Figure 3 is a data distribution schematic diagram of each storage layer;

[0027] Figure 4 is a flowchart of an embodiment of a data processing method for the target database;

[0028] Figure 5 is a schematic diagram of data of the first storage layer, the second storage layer and the third storage layer;

[0029] Figure 6 is a structural schematic diagram of a data processing apparatus for a target database. DETAILED DESCRIPTION

[0030] The present specification will be further described in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. The described embodiments are only part of the embodiments of the present specification, not all embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of the present application.

[0031] It should be noted that, for the convenience of description, only the part related to the application is shown in the drawings. The embodiments in the present specification and the features in the embodiments can be combined with each other without conflict.

[0032] As mentioned before, some databases employ a storage architecture including multiple storage layers, and data with the same primary key value may be stored in different storage layers due to different writing and updating times. Based on such a situation, when computing in the database, such as performing pushdown computation in the data query execution process, the multi-layer data needs to be fused first, which not only becomes a bottleneck of the computation, but also blocks the batch computation.

[0033] Based on this, some embodiments of the present specification provide a data processing method for a target database, which can improve the computing performance of the target database.

[0034] Referring to Figure 1 , which is a schematic diagram of a storage architecture 100 employed by a target database. The storage architecture 100 can include multiple storage layers, such as Figure 1 storage layer a, storage layer b, storage layer c shown in

[0035] Generally, the data update frequency of the storage layer a is the highest, the data update frequency of the storage layer c is the lowest, and the data update frequency of the storage layer b is between the storage layer a and the storage layer c. Correspondingly, the data in the storage layer a is generally the latest data, the data in the storage layer c is generally the oldest data, and the data in the storage layer b is between the storage layer a and the storage layer c.

[0036] The storage layer a can be located in the memory, and is used to store the incremental data of the target database. The storage layer b and the storage layer c can be located in the disk, and are both used to store the inventory data (which can also be referred to as baseline data) of the target database. It should be understood that the data of the target database other than the incremental data can be referred to as the inventory data.

[0037] The data in the storage layer a can be dumped to the storage layer b, and the data in the storage layer b can be dumped to the storage layer c. Specifically, the storage layer a and the storage layer b can have respective dump conditions. The data in the storage layer a can be dumped to the storage layer b when the dump condition associated with the storage layer a is met. The data in the storage layer b can be dumped to the storage layer c when the dump condition associated with the storage layer b is met.

[0038] In practice, the data in each storage layer is generally row data (which can also be referred to as data row). Any row of data can include a primary key value and one or more field values corresponding to the primary key value. The data with the same primary key value can be stored in different storage layers due to different times of writing and updating. Since the data in the storage layer c is relatively old, compared with the storage layer a and the storage layer b, the storage layer c generally includes more data rows, and there are more complete data rows in the storage layer c, that is, the data rows that have not been updated and deleted.

[0039] In addition, the data in each storage layer is generally arranged in ascending order of the primary key. In an example, the storage architecture 100 can be a layered, ordered and disk-oriented data structure, such as an LSM-Tree (Log Structured Merge Tree), and the core idea can be to use the characteristic that the sequential writing of the disk batch is higher in performance than the random writing to achieve better writing performance. In the case where the storage architecture 100 is an LSM-Tree, the storage layer a can be a MemTable, the storage layer b can be a Minor SSTable, and the storage layer c can be a Major SSTable.

[0040] The MemTable can be a data memory organization structure located in the memory, and provides read and write services. The data newly inserted / deleted / updated by the user can be first written into the MemTable, and the transaction is guaranteed through the Redo Log. When the size of the data in the MemTable exceeds a threshold value, the internal data is generally frozen (Freeze) and persisted on the disk. The file written by the frozen data can be a Minor SSTable. The number of Minor SSTables is limited, and when certain conditions are met, multiple Minor SSTables can be merged, and the complete data rows are written into the file in the increasing order of the primary key value. At this time, the Major SSTable is generated.

[0041] Since the LSM-Tree is a well-known storage architecture, the present specification does not further explain each storage layer itself.

[0042] Referring to Figure 2 , which is a structural diagram of the target database. In addition to the storage architecture 100 described above, the target database can also include data processing operators 101 and computing operators 102 as shown in Figure 2 . The data processing operators 101 can be used to read data from each storage layer of the storage architecture 100 and perform analysis and other processing on the read data, and can also output data to the computing operators 102. The computing operators 102 can be used to perform data computation. Both the data processing operators 101 and the computing operators 102 can be implemented as software units or devices, etc., and are not specifically limited here.

[0043] According to the description above, the storage layer c generally includes more data rows than the storage layer a and the storage layer b, and there are more complete data rows in the storage layer c, i.e., data rows that have not been updated and deleted. For complete data rows, there is no need to perform a data fusion process. Starting from the point of view of how to reduce data fusion, for the storage layer c, a data boundary that has not been updated and deleted can be found, such as the boundary primary key values key1 and key2 shown in Figure 3 . For data rows in the storage layer c that are within [key1, key2), expression calculation can be directly performed without the need to perform a data fusion process.

[0044] wherein, Figure 3 is a data distribution diagram of each storage layer. In Figure 3 , the rectangular region in the rectangular frame corresponding to the storage layer a / storage layer b, where both the upper and lower edges are solid lines, represents the data of the storage layer a / storage layer b, and the rectangular region represented by the rectangular frame where both the upper and lower edges are dashed lines represents data that is not stored in the storage layer a / storage layer b. The rectangular frame corresponding to the storage layer c represents the data of the storage layer c. Figure 3 The data of each storage layer shown in the diagram is arranged in ascending order of the primary key. According to the diagram, Figure 3 , the data rows with primary key values within [key1, key2) only appear in the storage layer c.

[0045] Based on the above thinking, when the target database performs computation, for example, when performing pushdown computation in a data query execution process, the data processing operators 101 can first determine the data boundary that has not been updated and deleted for the storage layer c, and output the data rows in the storage layer c within the boundary range to the computing operators 102 for computation.

[0046] Specifically, as shown in Figure 2As shown, the data processing operator 101 can determine the minimum primary key value among the primary key values of the storage layers, and determine whether the minimum primary key value is only from the storage layer c. If the result of the determination is yes, the data processing operator 101 can take the minimum primary key value as the boundary primary key value key1, and then determine the minimum primary key value among the primary key values of the storage layer a and the storage layer b as the boundary primary key value key2.

[0047] It should be understood that the data rows with the primary key values greater than or equal to key1 and less than key2 should only exist in the storage layer c, and are data rows that have not undergone data update or deletion. Based on this, the data processing operator 101 can then obtain the data rows with the primary key values within [key1, key2) from the storage layer c, and directly output the obtained data rows to the computing operator 102 of the upper layer, without the need to determine whether the primary key values of the data rows still exist in the storage layer a and the storage layer b, and the data fusion process can be avoided.

[0048] In the prior art, each time a data row is scanned from the storage architecture, whether the data row is a complete data row or not, a data fusion process is performed according to the primary key value of the data row, for example, whether the primary key value exists in each storage layer is queried, and after the primary key value is queried, the data row containing the primary key value is taken out for fusion, and the like. In the case of small data volume, the demand can still be met, but when the data reaches a certain scale, the fusion overhead will become a computing bottleneck and block batch computation. For example, in the case of sequential insertion or data with less update, most of the data has no deletion and update operation, and only one version exists in the storage, and it is not necessary to determine whether the primary key value exists in each storage layer, and the data fusion process is also not necessary.

[0049] Compared with the prior art, the above data processing process provided by the embodiments of the present specification can avoid performing invalid data fusion process, so that the fusion overhead can be avoided as much as possible to become a computing bottleneck and block batch computation, thereby greatly improving the computing performance of the target database.

[0050] In the following, the specific implementation steps of the above method are described in combination with specific embodiments.

[0051] Referring to Figure 4FIG. 4 shows a flow 400 of one embodiment of a data processing method for a target database. The storage architecture of the target database (e.g., the storage architecture 100 in the foregoing) includes a first storage layer (e.g., the storage layer a in the foregoing), a second storage layer (e.g., the storage layer b in the foregoing), and a third storage layer (e.g., the storage layer c in the foregoing). The first storage layer can be located in the memory, for example, and the second storage layer and the third storage layer can be located in the disk, for example, and the data in each storage layer can be arranged in ascending order based on the primary key. In one example, the storage architecture of the target database can be an LSM-Tree, for example. For the explanation of each storage layer, refer to the related description in the foregoing, which will not be repeated here.

[0052] The execution subject of the method can be a data processing operator (e.g., the data processing operator 101 in the foregoing) in the target database. The method includes the following steps:

[0053] In step 402, the minimum primary key value is determined among the primary key values in the first storage layer, the second storage layer, and the third storage layer.

[0054] In step 404, in response to the source storage layer of the minimum primary key value including only the third storage layer, the minimum primary key value is taken as the first boundary primary key value, and the minimum primary key value is determined among the primary key values in the first storage layer and the second storage layer as the second boundary primary key value.

[0055] In step 406, the data rows with the primary key values in the range from the first boundary primary key value to the second boundary primary key value are obtained from the third storage layer, and the obtained data rows are output to the upper computing operator; wherein the range does not include the second boundary primary key value.

[0056] The above steps are further described below.

[0057] In step 402, the minimum primary key value can be determined among the primary key values in the first storage layer, the second storage layer, and the third storage layer. Then, the source storage layer of the minimum primary key value can be recorded, for example.

[0058] It should be noted that each primary key value mentioned in the embodiments of the present specification is the primary key value of the same primary key. In addition, the primary key values of the first storage layer, the second storage layer, and the third storage layer in step 402 can be all or part of the primary key values of each storage layer. The part of the primary key values can be the primary key values that meet the specified primary key requirements.

[0059] The specified primary key requirement can include, for example, a primary key name, and a primary key value range corresponding to the primary key name. For example, the above process 400 can be performed based on a computing task, and the specified primary key requirement can be included in the computing task. Alternatively, the above process 400 can be performed during execution of a data query, and the specified primary key requirement can be included in a data query request corresponding to the data query.

[0060] After determining the minimum primary key value by performing the step 402, it can be determined whether the source storage layer of the minimum primary key value only includes the third storage layer. If it is determined that the source storage layer of the minimum primary key value only includes the third storage layer, the step 404 can be performed subsequently.

[0061] In the step 404, the minimum primary key value can be determined as the first boundary primary key value, and the minimum primary key value among the primary key values of the first storage layer and the second storage layer can be determined as the second boundary primary key value, in response to the source storage layer of the minimum primary key value only including the third storage layer. It should be noted that the "first" and "second" here are only used to distinguish different boundary primary key values, and do not have any limiting effect.

[0062] Since the second boundary primary key value is the minimum primary key value among the primary key values of the first storage layer and the second storage layer, the data rows whose primary key values are greater than or equal to the first boundary primary key value and less than the second boundary primary key value should only exist in the third storage layer, and are data rows that have not been updated or deleted.

[0063] Based on this, after the step 404, the step 406 can be performed subsequently. In the step 406, the data rows whose primary key values are within the range of the first boundary primary key value and the second boundary primary key value can be obtained from the third storage layer, and the obtained data rows can be directly output to the upper computing operator (such as the computing operator 102 in the foregoing), without the need to determine whether the primary key values of the data rows still exist in the first storage layer and the second storage layer, thereby avoiding the data fusion process. The range can be represented as [the first boundary primary key value, the second boundary primary key value). Through this representation, it can be seen that the range includes the first boundary primary key value but does not include the second boundary primary key value.

[0064] In an example, the third storage layer can include a plurality of first sub-blocks, any first sub-block can include a plurality of second sub-blocks, and any second sub-block can include a plurality of data rows.

[0065] It should be noted that in the case of the storage architecture of the target database being an LSM-Tree, the first block can be referred to as a macro block, and the second block can be referred to as a micro block. In practice, in order to reduce the disk load caused by frequent read and write of the storage layer located on the disk, the data located in any storage layer on the disk can be divided by unit size to form a multi-layer structure of storage layer -> macro block -> micro block -> data row. Among them, the macro block is generally fixed-length, and is the basic unit of data file write IO (Input Output). The micro block is generally variable-length, and is the minimum unit of data file read IO.

[0066] In order to reduce the granularity of data reading and the number of primary key value comparisons, the second block in the third storage layer can be taken as a conflict detection unit. Specifically, in the second block of the third storage layer, each second block that satisfies the range of the first boundary primary key value and the second boundary primary key value can be determined, and the data row is obtained from the each second block. It should be noted that the second block of the third storage layer described here can be all or part of the second block of the third storage layer. The part of the second block can be the second block that satisfies the specified primary key requirement.

[0067] Further, after determining the each second block, a target mark can be added to the each second block respectively, which is used to indicate that the corresponding second block satisfies the current primary key range. Then, the each second block with the target mark can be traversed in turn, so as to obtain the data row in the each second block.

[0068] In order to further reduce the number of primary key value comparisons, the second block can record the maximum primary key value of each data row contained in the second block. In this way, in the second block of the third storage layer, each second block whose recorded maximum primary key value is within the range of the first boundary primary key value and the second boundary primary key value can be determined, and the data row is obtained from the each second block.

[0069] Figure 4 Compared with the prior art described in the foregoing, the data processing process shown in the corresponding embodiment can avoid executing invalid data fusion process, so as to avoid the overhead of fusion as much as possible to become the bottleneck of calculation and block batch calculation, thereby greatly improving the calculation performance of the target database. In addition, when the data processing process is executed in the data query execution process, the data query efficiency can be effectively improved.

[0070] Next, the data processing process shown in the corresponding embodiment will be described in combination with the following Figure 5 The data processing process shown in the corresponding embodiment will be described in combination with the following Figure 4 The data processing process shown in the corresponding embodiment will be described in combination with the following Figure 5 is a schematic diagram of the data of the first storage layer, the second storage layer and the third storage layer. InFigure 5 In the example, A is the primary key, and the data in each storage layer is arranged in ascending order of the primary key A. It should be understood that the data is only exemplary data and does not have any limiting effect.

[0071] In Figure 5 In the primary key values of each storage layer shown, the minimum primary key value 2 can be determined, and it can be known that the primary key value 2 only comes from the third storage layer. Therefore, the primary key value 2 can be taken as the first boundary primary key value. Then, the minimum primary key value 4 can be determined in the primary key values of the first storage layer and the second storage layer, and the primary key value 4 can be taken as the second boundary primary key value. Then, the data rows with the primary key values in [2, 4) can be obtained in the third storage layer, for example, the first row and the second row of data of the third storage layer are obtained, and the obtained data rows are output to the computing operator of the upper layer.

[0072] In an embodiment, for the minimum primary key value determined in step 402, if it is determined that the source storage layer of the minimum primary key value includes the first storage layer or the second storage layer, one of the following can be performed: when the source storage layer is a single storage layer, the data row in which the minimum primary key value is located is output to the computing operator of the upper layer; when the source storage layer is multiple storage layers, the data rows in which the minimum primary key value is located are fused, and the fused data rows are output to the computing operator of the upper layer. By using this processing manner, repeated scanning of data outside the boundary range (such as the first boundary primary key value and the second boundary primary key value range) in the data fusion process can be avoided, the overall data fusion efficiency can be improved, and the data query efficiency and the computing performance can be improved.

[0073] In an embodiment, step 402 in the foregoing can be specifically implemented by reading the row with the minimum primary key value in each storage layer, thereby obtaining the first data row from the first storage layer, the second data row from the second storage layer, and the third data row from the third storage layer, and determining the minimum primary key value in the primary key values of the first data row, the second data row, and the third data row. Correspondingly, in step 404, the minimum primary key value can be determined in the primary key values of the first data row and the second data row.

[0074] Further, in steps 402 and 404, the determination of the minimum primary key value can be implemented by using a Loser Tree. In practice, the Loser Tree can be a complete binary tree for data sorting, and each leaf node in the Loser Tree can store the record currently participating in comparison in the merging process of each merge segment, and each non-leaf node can remember the node with a small record sorting code (i.e., the loser) in its two child nodes.

[0075] Specifically, before step 402, a scan iterator of each storage layer can be created, and the created scan iterators can be put into the loser tree. The scan iterator can be used to scan data of the corresponding storage layer in ascending order of the primary key value.

[0076] In the case of applying the loser tree, in step 402, the loser tree can be used to read a row of data in each scan iterator of each storage layer, and the loser tree can be used to determine the smallest primary key value from the first data row, the second data row and the third data row. In practice, the cursor of the scan iterator can point to the row of data with the smallest primary key value, and the loser tree can be used to read the row of data pointed to by the cursor in the scan iterator. In step 404, the loser tree can be used to determine the smallest second primary key value from the primary key values of the first data row and the second data row.

[0077] Generally, the loser tree reads rows of data from each scan iterator therein and compares the sizes. In order to avoid the third data row participating in the comparison of the sizes of the primary key values, the scan iterator of the third storage layer can be removed from the loser tree before the loser tree is used to read the row of data with the smallest primary key value from the first data row and the second data row from the remaining scan iterators, so as to determine the smallest primary key value.

[0078] Since the data of the third storage layer can be required in the subsequent rounds, after the loser tree is used to read the row of data with the smallest primary key value from the first data row and the second data row from the remaining scan iterators, so as to determine the smallest primary key value, the scan iterator of the third storage layer can be put back into the loser tree.

[0079] In summary of the foregoing description, it can be learned that the scheme provided by the embodiments of the present specification considers from the starting point of how to reduce data fusion, finds the primary key range in which no update and deletion occur, further explores the conflict boundary algorithm for determining the data of the storage architecture hierarchy, and directly pushes the calculation to the storage layer based on the boundaries for single-side calculation, which can effectively improve the calculation performance. Moreover, for the data in the primary key range, the scan result can be directly calculated like accessing a normal table, which can avoid the process of multi-way merging and fusion, thereby greatly improving the query efficiency.

[0080] Further reference Figure 6 , the present specification provides an embodiment of a data processing apparatus for a target database, which corresponds to the method embodiment shown in Figure 4 . The storage architecture of the target database includes a first storage layer, a second storage layer and a third storage layer. The apparatus can be applied to a data processing operator (such as the data processing operator 101 in the foregoing description) in the target database.

[0081] As shown in Figure 6 The data processing apparatus 600 for the target database in this embodiment includes a first determining unit 601, a second determining unit 602, and a processing unit 603. The first determining unit 601 is configured to determine the minimum primary key value among the primary key values of the first storage layer, the second storage layer, and the third storage layer. The second determining unit 602 is configured to, in response to the source storage layer of the minimum primary key value including only the third storage layer, take the minimum primary key value as the first boundary primary key value, and determine the minimum primary key value among the primary key values of the first storage layer and the second storage layer as the second boundary primary key value. The processing unit 603 is configured to obtain, from the third storage layer, the data rows whose primary key values are within the range of the first boundary primary key value and the second boundary primary key value, and output the obtained data rows to the computing operator of the upper layer, wherein the range does not include the second boundary primary key value.

[0082] In some embodiments, the first determining unit 601 can be further configured to read the row of data with the minimum primary key value in each storage layer respectively, thereby obtaining the first data row from the first storage layer, the second data row from the second storage layer, and the third data row from the third storage layer. The second determining unit 602 can be further configured to determine the minimum primary key value among the primary key values of the first data row and the second data row.

[0083] In some embodiments, each storage layer has a respective scan iterator for performing data scanning on the corresponding storage layer in the order of the primary key values from small to large, and the respective scan iterator of each storage layer has been put into a loser tree. The first determining unit 601 can be further configured to read a row of data in the respective scan iterator of each storage layer respectively by using the loser tree. The second determining unit 602 can be further configured to determine the minimum primary key value among the primary key values of the first data row, the second data row, and the third data row by using the loser tree.

[0084] In some embodiments, the second determining unit 602 can be further configured to remove the scan iterator of the third storage layer from the loser tree, and read the row of data with the minimum primary key value in the first data row and the second data row from the remaining scan iterators by using the loser tree, thereby determining the minimum primary key value.

[0085] In some embodiments, the second determining unit 602 can be further configured to put the scan iterator of the third storage layer back into the loser tree.

[0086] In some embodiments, the apparatus 600 further comprises a creating unit (not shown in the figure) configured to create a scan iterator for each storage layer; and the created scan iterators are put into the loser tree.

[0087] In some embodiments, the first determining unit 601 is further configured to record the source storage layer.

[0088] In some embodiments, the third storage layer comprises a plurality of first sub-blocks, any first sub-block comprises a plurality of second sub-blocks, and any second sub-block comprises a plurality of data rows; and the processing unit 603 is further configured to determine, in the second sub-blocks of the third storage layer, a second sub-block that satisfies the range; and obtain the data rows from the second sub-block.

[0089] In some embodiments, any second sub-block records a maximum primary key value of each data row contained in the second sub-block; and the processing unit 603 is further configured to determine a second sub-block whose recorded maximum primary key value is within the range.

[0090] In some embodiments, the processing unit 603 is further configured to add a target mark to each second sub-block respectively, the target mark being used to indicate that the corresponding second sub-block satisfies the range; and sequentially traverse each second sub-block added with the target mark to obtain the data rows in the second sub-block.

[0091] In some embodiments, the storage architecture is a Log-Structured Merge Tree (LSM-Tree).

[0092] In some embodiments, the first storage layer is located in the memory, and the second storage layer and the third storage layer are located in the disk, and the data in each storage layer is arranged in ascending order based on the primary key.

[0093] In Figure 6 In the corresponding apparatus embodiments, the specific processing of each unit and the resulting technical effects can refer to the related descriptions of the method embodiments in the foregoing, and will not be described here again.

[0094] The embodiments of the present specification further provide a computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer program causes the computer to execute the data processing method for the target database described in each of the method embodiments.

[0095] The embodiments of the present specification further provide a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the data processing method for the target database described in each of the method embodiments.

[0096] The embodiments of the present specification further provide a computer program, wherein when the computer program is executed in a computer, the computer is caused to perform the data processing method for the target database described in the above method embodiments respectively.

[0097] Those skilled in the art should be aware that in one or more examples described above, the functions described in the embodiments of the present specification can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0098] In some cases, the acts or steps recited in the claims can be performed in a different order and still accomplish desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order in order to achieve desirable results. In some implementations, multitasking and parallel processing can be advantageous.

[0099] The above detailed description merely describes the specific implementation of the embodiments of the present specification, and is not intended for limiting the protection scope of the embodiments of the present specification. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the present specification shall be included in the protection scope of the embodiments of the present specification.

Claims

1. A data processing method for a target database, a storage architecture of the target database comprising a first storage layer, a second storage layer and a third storage layer, the method comprising: determining a minimum primary key value among primary key values of the first storage layer, the second storage layer and the third storage layer, data in each storage layer being arranged in ascending order of primary key; in response to the source storage layer of the minimum primary key value only comprising the third storage layer, taking the minimum primary key value as a first boundary primary key value, and determining a minimum primary key value among primary key values of the first storage layer and the second storage layer as a second boundary primary key value; obtaining, from the third storage layer, a data row whose primary key value is within a range of the first boundary primary key value and the second boundary primary key value, and outputting the obtained data row to an upper layer computing operator without judging whether the primary key value of the data row still exists in the first storage layer and the second storage layer to avoid data fusion process, wherein the range does not include the second boundary primary key value.

2. The method of claim 1, wherein, The determining a minimum primary key value among primary key values of the first storage layer, the second storage layer and the third storage layer comprises: respectively reading a row of data with the minimum primary key value in each storage layer, thereby obtaining a first data row from the first storage layer, a second data row from the second storage layer, and a third data row from the third storage layer; determining a minimum primary key value among primary key values of the first data row, the second data row and the third data row; The determining a minimum primary key value among primary key values of the first storage layer and the second storage layer comprises: determining a minimum primary key value among primary key values of the first data row and the second data row.

3. The method of claim 2, wherein, Each storage layer has a respective scan iterator for scanning the corresponding storage layer in order of primary key value from small to large, and the respective scan iterator of each storage layer has been put into a loser tree. And The respectively reading a row of data with the minimum primary key value in each storage layer comprises: respectively reading a row of data in the respective scan iterator of each storage layer by using the loser tree; The determining a minimum primary key value among primary key values of the first data row, the second data row and the third data row comprises: determining a minimum primary key value among primary key values of the first data row, the second data row and the third data row by using the loser tree; The determining a minimum primary key value among primary key values of the first data row and the second data row comprises: determining a minimum primary key value among primary key values of the first data row and the second data row by using the loser tree.

4. The method of claim 3, wherein, The determining a minimum primary key value among primary key values of the first data row and the second data row by using the loser tree comprises: removing the scan iterator of the third storage layer from the loser tree; reading a row of data with the minimum primary key value in the first data row and the second data row from the remaining scan iterators by using the loser tree, thereby determining the minimum primary key value.

5. The method of claim 4, wherein, after the minimum primary key value is determined from among the primary key values of the first storage layer, the second storage layer, and the third storage layer, further comprising: putting back the scan iterator of the third storage layer into the loser tree.

6. The method of claim 3, further comprising: creating a scan iterator for each of the storage layers; putting the created scan iterators into the loser tree.

7. The method of claim 1, wherein, after the minimum primary key value is determined from among the primary key values of the first storage layer, the second storage layer, and the third storage layer, further comprising: recording the source storage layer of the minimum primary key value.

8. The method of claim 1, wherein, the third storage layer includes a plurality of first chunks, any first chunk includes a plurality of second chunks, and any second chunk includes a plurality of data rows; and the obtaining, from the third storage layer, the data rows whose primary key values are within a range between the first boundary primary key value and the second boundary primary key value, comprises: determining, in the second chunks of the third storage layer, the second chunks that satisfy the range; obtaining the data rows from the second chunks that satisfy the range.

9. The method of claim 8, wherein, any second chunk records a maximum primary key value of each data row contained in the second chunk; and the determining the second chunks that satisfy the range, comprises: determining the second chunks whose recorded maximum primary key values are within the range. after the second chunks that satisfy the range are determined, further comprising:

10. The method of claim 8, wherein, adding a target mark to each of the second chunks, respectively, to indicate that the corresponding second chunk satisfies the range; the obtaining the data rows from the second chunks that satisfy the range, comprises: sequentially traversing the second chunks that have the target mark added thereto, to obtain the data rows in the second chunks. the storage architecture is a Log-Structured Merge Tree (LSM-Tree).

11. The method of claim 1, wherein, the first storage layer is located in a memory, and the second storage layer and the third storage layer are located in a disk.

12. The method of claim 1, wherein, 13. A data processing apparatus for a target database, a storage architecture of the target database including a first storage layer, a second storage layer, and a third storage layer, the apparatus comprising: a first determining unit configured to determine a minimum primary key value from among primary key values of the first storage layer, the second storage layer, and the third storage layer, data in each storage layer being arranged in ascending order based on a primary key; a second determining unit configured to, in response to a source storage layer of the minimum primary key value including only the third storage layer, take the minimum primary key value as a first boundary primary key value, and determine a minimum primary key value from among primary key values of the first storage layer and the second storage layer as a second boundary primary key value; a processing unit configured to obtain, from the third storage layer, data rows whose primary key values are within a range between the first boundary primary key value and the second boundary primary key value, and output the obtained data rows to a computing operator at a higher level, without judging whether a primary key value of the data row still exists in the first storage layer and the second storage layer, to avoid a data fusion process; wherein the range does not include the second boundary primary key value. the computer program, when executed in a computer, causes the computer to perform the method of any one of claims 1-12.

14. A computer readable storage medium having stored thereon a computer program, wherein, the computer program, when executed in a computer, causes the computer to perform the method of any one of claims 1-12.

15. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-12. The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-12. The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-12. The memory stores

Images