Table file merging method, database instance, device, medium, and program product

Abstract

Embodiments of the present application provide a table file merging method, a database instance, a device, a medium, and a program product. In the method, a cache comprises first data pages and second data pages which belong to different table files to be merged; a database instance can first sort records in a fixed number of first data pages, and merge the table files according to a sorting result; when any data page among the first data pages is completely merged into a merged table file, a page fault occurs in the database instance; and in this case, the database instance can continue to sort and merge the second data pages belonging to the same table file as any data page in the cache and the remaining data pages among the first data pages.

Description

Table File Merging Method, Database Instance, Device, Medium, and Program Product - Technical Field

[0001] This application relates to the field of database technology, and particularly to a table file merging method, a database instance, a device, a medium, and a program product. Background Art

[0002] In a database, data is stored in the form of table files. In practice, table files can be sorted (compacted), and multiple table files can be merged according to the sorting process. This kind of merging can be considered as merging multiple small table files to obtain a large table file. By merging, the number of table files in the database can be reduced, thereby improving the read / write speed and performance of the table files in the database. Based on the above description, how to improve the merging speed has become an urgent problem to be solved. Summary of the Invention [03 In view of this, embodiments of this application provide a table file merging method, a database instance, a device, a medium, and a program product to ensure the hot migration effect of virtual machines.

[0004] In a first aspect, embodiments of this application provide a table file merging method, including: sorting the records included in a preset number of first data pages in a cache to obtain a first sorting result, where the to-be-merged table files to which the first data pages belong are different; according to the first sorting result, if any one of the first data pages is written into the merged table file, then sorting the remaining data pages in the first data pages and the records included in the second data pages in the cache respectively to obtain a second sorting result; according to the second sorting result, writing the remaining data pages and the second data pages that belong to the same to-be-merged table file as the any one data page into the merged table file.

[0005] In a second aspect, embodiments of this application provide a database instance, including: an input / output 10 thread and a compression thread; the 10 thread is used to read the first data page and the second data page from the to-be-merged table file into the cache; the compression thread is used to sort the records included in the first data page to obtain a first sorting result, where the to-be-merged table files to which the first data pages belong are different; according to the first sorting result, if any one of the first data pages is written into the merged table file, then sorting the remaining data pages in the first data pages and the records included in the second data pages in the cache respectively to obtain a second sorting result; according to the second sorting result, writing the remaining data pages and the second data pages that belong to the same to-be-merged table file as the any one data page into the merged table file.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, including: the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, the table file merging method described in the first aspect above is implemented. The electronic device may further include a communication interface for communicating with other devices or communication systems.

[0007] In a fourth aspect, an embodiment of the present application provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a computing system of an electronic device, the computing system can at least implement the table file merging method described in the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product. The computer program product includes a computer program or instruction, and when the computer program or instruction is executed by a processor, the processor can be caused to implement the table file merging method described in the first aspect.

[0009] For the first data page and the second data page that belong to the table files to be merged and have been stored in the cache in the table file merging method provided by the embodiment of the present application, the database instance can first sort the records in a preset number of the first data pages to obtain a first sorting result. Among them, the merging of table files can be considered as a process of writing the records of the data pages included in different table files into the same new table file. Then, during the file merging process, the database instance can write the records in the first data page into the merged table file in order according to the above first sorting result.

[0010] And during the process of file merging according to the first sorting result, after any one of the data pages in the first data page is completely written into the merged table file, the database instance can continue to sort the second data page and the remaining data pages in the first data page in the cache to obtain a second sorting result, and continue to write the records in the data page into the merged table file according to the second sorting result. Among them, the table files to which the first data pages belong are different, and the second data page and any one of the above data pages belong to the same table file to be merged. The total amount of the second data page and the remaining data pages is also a preset number. And after all the data pages of the table files to be merged are written into the merged table file, that is, the merging of the table files is completed.

[0011] In the above merging method, the database instance can first sort a preset number of first data pages. When any data page is written into the file to be merged, the number of sorted data pages decreases, that is, the database instance has a page fault. At this time, the database instance can use the second data page that has been stored in the cache as a supplement to continue sorting. Since the second data page is already in the cache, there will be no 10 process in the database instance during a page fault, thus improving the efficiency of table file merging. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0013] FIG. 1 is a flowchart of a table file merging method provided by an embodiment of the present application;

[0014] FIG. 2 is a schematic diagram of a record sorting process provided by an embodiment of the present application;

[0015] FIG. 3a is a flowchart of another table file merging method provided by an embodiment of the present application;

[0016] FIG. 3b is a flowchart of yet another table file merging method provided by an embodiment of the present application;

[0017] FIG. 3c is a flowchart of yet another table file merging method provided by an embodiment of the present application;

[0018] FIG. 3d is a flowchart of yet another table file merging method provided by an embodiment of the present application;

[0019] FIG. 4 is a schematic diagram of a table file merging process provided by an embodiment of the present application;

[0020] FIG. 5 is a structural diagram of a database instance provided by an embodiment of the present application;

[0021] FIG. 6 is a structural diagram of another database instance provided by an embodiment of the present application;

[0022] FIG. 7 is a structural schematic diagram of a table file merging device provided by an embodiment of the present application;

[0023] FIG. 8 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0025] The terms used in the embodiments of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms "a", "the", and "said" used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. "Plural" generally includes at least two, but does not exclude the case of including at least one.

[0027] Depending on the context, the words "if", "when" as used herein can be interpreted as "when...", "at the time when...", or "in response to determining", or "in response to recognizing". Similarly, depending on the context, the phrase "if determined" or "if recognized (stated condition or event)" can be interpreted as "when determined", "in response to determining", "when recognizing (stated condition or event)", or "in response to recognizing (stated condition or event)". It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0029] It should also be noted that the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, so that a commodity or system including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such commodity or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the commodity or system including the said element. In the case of no further limitations, an element defined by the statement "including one..." does not exclude the existence of another identical element in the commodity or system including the said element.

[0030] Before describing the methods and examples provided in the following embodiments of the present application, the following concepts can also be explained:

[0031] A distributed database, that is, a database of servers deployed at different physical locations. Among them, multiple servers located at one physical location can be called a regional cluster, and any server in a regional cluster provides a database instance. The storage level of data in the database can be described as follows: The data in the database can be stored in the form of table files. A table file can include at least one data page, and a data page can include at least one record.

[0032] Input / Output (I / O for short): In the database scenario, I / O is the process of reading data stored on the disk into the cache or writing the data in the cache to the disk. The above I / O process can occur in the same server, that is, the data is transferred between the cache and the disk of one server. For a distributed database, the above I / O process can also occur in multiple servers, that is, the data is transferred between the memory of one server and the disk of another server. And the above I / O process usually occurs when the database instance responds to an access request generated by the user.

[0033] Table file merging: Continuing from the background technology, table file merging is the process of sorting by records in the file to merge multiple small table files to be merged into a large merged table file. Among them, the records contained in the data pages of each small table file to be merged are also stored in an orderly manner. Then, through sorting, the records in the table files before and after merging are stored in an orderly manner. And after the table file merging is completed, the table files to be merged can also be deleted from the database.

[0034] Based on the above introduction, the following will describe some embodiments of the present application in detail with reference to the accompanying drawings. Without conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step timings in the following method embodiments are only examples, not strictly limited.

[0035] FIG. 1 is a flowchart of a table file merging method provided by an embodiment of the present application. The method provided by the embodiment of the present application can be executed by a database instance in a server. More precisely, the steps in this embodiment can be executed by a compression thread in the server instance. As shown in FIG. 1, the method can include the following steps:

[0036] S101. Sort the records included in a preset number of first data pages in the cache to obtain a first sorting result. The to-be-merged table files to which the first data pages belong are different from each other.

[0037] S102. According to the first sorting result, if any data page in the first data pages is written into the merged table file, then sort the records included in the remaining data pages in the first data pages and the second data pages in the cache respectively to obtain a second sorting result.

[0038] S103. According to the second sorting result, write the remaining data pages and the second data pages that belong to the same to-be-merged table file as any data page into the merged table file.

[0039] The database instance can periodically or in response to a manually triggered file merging operation to start executing the table file merging method provided in this embodiment and the following embodiments.

[0040] The to-be-merged files can be stored on the disk of the server where the database instance is located. Each to-be-merged table file can contain multiple data pages. The data pages included in the to-be-merged table files can be collectively referred to as to-be-merged data pages. Optionally, the to-be-merged table file can be a file in the database with a file size less than a preset value. Optionally, the to-be-merged table file can be a file in columnar storage format, such as (Optimized Row Columnar, abbreviated as ORC) file, parquet file, etc.

[0041] The merging process of the to-be-merged table files can be generally described as follows: The I / O thread in the database instance can first read the data pages in the to-be-merged table files from the disk into the cache. Then, the compression thread in the database instance can sort the records included in each data page in the cache and write the records included in the data pages into the merged table file according to the sorting result. After writing all the records in the data pages of each to-be-merged table file into the merged table file, that is, the merging of the table files is completed. Optionally, the merged table file can also be stored on the disk of the server where the database instance is located.

[0042] For the above merging process, specifically, the I / O thread in the database instance can first read the first data page in the to-be-merged table file from the disk into the cache. Optionally, the to-be-merged table files to which the first data pages belong are different from each other. Then, the compression thread in the database instance can sort the records included in each first data page in the cache to obtain a first sorting result. Optionally, the compression thread can use the primary key value of each record in the data page as a basis to sort the records in the first data pages.

[0043] Then, according to the first sorting result, the compression thread can gradually write the records contained in each of the first data pages into the merged table file, that is, the process of gradually merging the records in the first data pages into the merged table file. Optionally, during the above table file merging process, the compression thread can always sort a preset number of data pages, and the data pages participating in the sorting belong to different files to be merged. Based on this sorting requirement, the number of the above first data pages is a preset number, and the first data pages can respectively belong to a preset number of files to be merged.

[0044] During the process of writing records into the merged table file according to the first sorting result, when any one of the first data pages is written into the merged table file, that is, when the records in any one of these data pages are all written into the merged table file, it can be considered that the compression thread has a page fault, that is, the number of data pages participating in the sorting in the cache is insufficient. In order to keep the sorting process of the compression thread uninterrupted, the compression thread can use the second data page that has been read into the cache as a supplementary data page, and continue to sort this second data page and the remaining data pages in the first data pages, so as to obtain a second sorting result. Among them, the second data page and any one data page belong to the same file to be merged table. Further, according to this second sorting result, the compression thread can continue to gradually write the remaining data pages and the records in the second data page into the merged table file, that is, continue the merging of the table file.

[0045] When the compression thread has the above page fault situation, since the second data page as the supplementary data page and any one data page that has been merged come from the same file to be merged table, the same as before the data page is supplemented, the compression thread still sorts a preset number of data pages belonging to a preset number of files to be merged, that is, meets the sorting requirement of the compression thread. Therefore, it can ensure that the sorting process is uninterrupted.

[0046] Regarding the timing of reading the second data page from the disk into the cache, optionally, the second data page can be read into the cache together with the first data page; optionally, it can also read the second data page into the cache in response to the start of the compression thread sorting the records in the first data page; optionally, it can also read the second data page into the cache during the process of the compression thread sorting the records in the first data page.

[0047] It should be noted that this embodiment does not limit the reading timing of the second data page. The above are just several examples. As long as the second data page is read into the cache before any one data page is completely written into the merged table file.

[0048] In this embodiment, for the first data page and the second data page that have been stored in the cache and belong to different to-be-merged table files, the compression thread in the database instance can first sort the records in a preset number of the first data pages to obtain a first sorting result, and write the records in the first data pages into the merged table file in sequence according to the first sorting result. And during the process of file merging according to the first sorting result, when a page fault occurs to the compression thread when any data page is completely written into the merged table file, the compression thread can continue to sort the second data page and the remaining data pages in the cache to obtain a second sorting result, and continue to write the records in the data pages into the merged table file according to the second sorting result. After all the data pages of the to-be-merged table files are written into the merged table file, that is, the merging of the table files is completed.

[0049] In the above method, the compression thread in the database instance can first sort a preset number of the first data pages. When any data page is written into the to-be-merged file, the number of data pages being sorted in the cache will decrease, and the compression thread is in a page fault state. At this time, the compression thread can directly use the second data page that has been stored in the cache as a supplement to achieve uninterrupted sorting. And since the second data page is already in the cache, when a page fault occurs to the compression thread, the database instance will not experience the process of data pages, and thus the sorting interruption caused by it can be improved, thereby improving the efficiency of table file merging.

[0050] In addition, the technical effects achievable by the table file merging method provided in each embodiment of this application can be understood in combination with the following content.

[0051] When a page fault occurs to the compression thread as described in the embodiment shown in FIG. 1, a common way can be that the io thread in the database instance can, in response to the occurrence of the page fault, read the supplementary data page from the disk into the memory, and then the compression thread can perform sorting. However, in the above method, the sorting and merging of data pages and the reading of supplementary data pages have a sequential execution relationship, that is, when a page fault occurs, the 10 thread needs to first read the supplementary data page, and the compression thread needs to wait until the supplementary data page is read into the cache before it can continue to sort. This waiting of the compression thread for the 10 thread will obviously reduce the merging speed of the table file.

[0052] In the method provided in each embodiment of this application, the sorting and merging process of the data pages in the cache and the process of reading the supplementary data pages into the cache can be carried out independently. For example, the sorting and merging of data pages and the reading of supplementary data pages can be carried out synchronously, and the compression thread does not need to wait for the 10 thread. Therefore, the above problem can be improved.

[0053] In addition, the relationship between the table file to be merged and the server is not mentioned in the above embodiments. Optionally, the table file to be merged is stored in one server. Optionally, for a distributed database, the table file to be merged can also be stored in different servers. In this case, when a page fault occurs in the compression thread, there is also a data page 10 process across devices, that is, the 10 thread needs to read the data page in another server to the local. Then the waiting of the compression thread for the 10 thread mentioned in the first point above can specifically occur during the 10 process of the data page on the same device or across devices. In practice, the 10 of the data page is often time-consuming, and the 10 across devices is even more time-consuming, which will further reduce the merging speed of the table file.

[0054] However, when using the methods provided in the embodiments of the present application, since there is no 10 process when a page fault occurs in the compression thread, the compression thread can continuously perform sorting and will not cause the compression thread to hang or pause due to waiting for the 10 thread, thereby improving the merging speed of the table file.

[0055] According to the description in the above embodiments, during the merging of the table file, the sorting and merging of the records in the data page by the compression thread is an important step. Then, in order for the compression thread to sort and merge the records in the first data page, optionally, the attribute of the file to be merged can be set as an iterable object, and each file to be merged also has a corresponding iterator, and the iterator is used to sequentially read the records in the data page. The records included in the data page of the table file to be merged can be stored in an orderly manner according to the size of the primary key value. For example, the first record in a data page has the smallest primary key value in the data page, and the last record has the largest primary key value in the data page.

[0056] Based on this, an optional way for the compression thread to record sorting and merging can be: the primary key value of the record in the first data page that the iterator can read. The compression thread can sort the records in the first data page according to the primary key value read by the iterator to obtain the first sorting result. The compression thread can then write the records in the first data page into the merged table file according to the first sorting result. Among them, the table files to be merged to which the first data page belongs are different.

[0057] More specifically, for the above sorting and merging method, the iterators corresponding to the table files to be merged to which the first data page belongs can respectively read a record pointed to by the iterator currently. The records read by the iterator are put into the heap as alternative records. Then, the compression thread can sort the alternative records in the heap according to the primary key values of the alternative records so that the heap forms a minimum heap. In this minimum heap, the target record corresponding to the minimum primary key value in the alternative records is located at the top of the heap, and the remaining records in the alternative records are located non-at the top of the heap. The compression thread can call the second function in the database instance to read the target record located at the top of the heap and write the target record into the merged table file. Optionally, the second function can be the peek function.

[0058] In practice, the above target record can be included in any one of the data pages in the first data page. One case can be that the target record is included in any one of the data pages mentioned in the embodiment shown in FIG. 1. Among them, for the subsequent description of the request, the table file to be merged to which any one of the data pages belongs can be called the target table file to be merged. And while the compression thread writes the target record in any one of the data pages into the table file to be merged, the database instance can also call the first function to make the iterator corresponding to the target table file to be merged point to the next record of the target record. Optionally, the first function can be.hasNext„

[0059] After the target record is written into the merged table file, the next record of the target record is read into the heap by the iterator. At this time, the heap can include the remaining records that were previously located non-at the top of the heap and the next record of the target record. The remaining records are respectively included in the remaining data pages in the first data page except any one of the data pages. That is, the iterators corresponding to the table files to be merged to which the remaining data pages belong still point to the current record, and only the pointer of the iterator corresponding to the target table file to be merged slides.

[0060] The above can be considered as a round of sorting and merging process of records by the compression thread. After multiple rounds of sorting, the situation mentioned in the embodiment shown in FIG. 1 will occur: all the records included in any one of the data pages are written into the merged table file, that is, the page fault situation that occurs to the compression thread. situation, that is, the page fault situation that occurs to the compression thread.

[0061] The above process of the compression thread using the iterator and the heap structure to perform a round of sorting and merging of records can also be understood in combination with the following example.

[0062] For example, data pages 1 to 3 can belong to the to-be-merged table files 1 to 3 respectively as the first data pages. Data page 4 can belong to the to-be-merged table file 1 as the second data page. And each to-be-merged table file is set as an iterable object, and the 3 to-be-merged table files respectively correspond to iterators 1 to 3.

[0063] Based on the above assumptions, when data pages 1 to 4 are read from the disk to the cache by 10 threads, iterator 1 can read the currently pointed record 1 from data page 1. Similarly, iterator 2 and iterator 3 can also read the currently pointed record 2 and record 3 from data page 2 and data page 3 respectively. Records 1 to 3 can be put into the heap, and the primary key values of records 1 to 3 can be 100, 200, and 300 respectively. Then the compression thread can sort the above 3 records according to the primary key values of the records so that the heap forms a minimum heap. Among them, record 1 with the smallest primary key value is at the top of the heap. After that, the compression thread can further write the record 1 at the top of the heap into the merged table file. At the same time, iterator 1 can point to the next record of record 1 in data page 1. Since records 2 and 3 are not written into the merged table file, iterator 2 and iterator 3 still point to records 2 and 3 respectively.

[0064] The above example of a round of sorting and merging process can also be understood in combination with the content of Figure 2.

[0065] In the above manner, the compression thread mostly sorts the same number of records in each round and writes the record at the top of the heap into the merged table file. And as the number of sorting rounds increases, the records contained in the to-be-merged data pages can be gradually written into the merged table file, thus finally completing the merging of the table files. And after merging in the above manner, the records in the merged table file are also stored in ascending order according to the primary key values.

[0066] According to the description in the above embodiments, in the process of merging table files, it is another important step for 10 threads to read data pages into the cache. And in order to prevent the 10 threads from not having data pages when the compression thread has a page fault, the 10 threads can read the supplementary data page (i.e., the second data page) into the cache in advance before the compression thread has a page fault. And in order to achieve the pre-reading of the supplementary data page, the 10 threads need the compression thread to know the reading order of the to-be-merged data pages before sorting and merging, so as to read the data pages according to this reading order.

[0067] For the determination of the reading order, in an optional manner, the compression thread can determine the reading order of the data pages to be merged according to the primary key values of the records included in the data page packets to be merged in the disk. More specifically, the compression thread can perform an ascending sort on the maximum primary key values in each data page to be merged, and the sorting result is the reading order. Among them, the records in the data pages to be merged are stored in an orderly manner according to the primary key values, and each data page included in each table file to be merged can also store the extreme values of the primary keys corresponding to each data page it contains.

[0068] The above sorting of the data pages to be merged is also the preprocessing process of the compression thread before the table file merging. And this reading order can also be obtained by the 10 threads to read the data pages according to this reading order.

[0069] In this embodiment, the compression thread can first sort the data pages to be merged according to the primary key values of the records to obtain the reading order of the data pages. The 10 threads can pre-read the supplementary data pages into the cache according to this reading order. When the compression thread has a page fault, the compression thread can directly continue the sorting based on the supplementary data pages that have been stored in the cache. The process of the supplementary data pages by the 10 threads is not carried out only after a page fault occurs. Therefore, the speed of table file merging can be improved.

[0070] In addition, for the cache used to store data pages, when the table files to be merged have different table structures, the 10 threads can also create corresponding cache areas for the table files to be merged with different table structures respectively. That is, cache areas can be set in the server where the database instance is located, and the data pages in the table files to be merged with one table structure are stored in each area. Among them, the table structure can be considered as the fields corresponding to the records included in the data pages in the table file. If the fields are different, the table structures are different.

[0071] The above process of creating the cache can be the preprocessing process performed by the 10 threads before the table file merging.

[0072] Optionally, the database embodiment can provide multiple 10 threads, and the data pages with the same table structure can be read into the corresponding cache by the same 10 thread.

[0073] It should be noted here that the caches storing data pages mentioned in the embodiments of the present application can all be considered as the cache areas corresponding to the table structures of the data pages, rather than all the cache areas provided by the server.

[0074] ​Since 10 threads can create different read tasks according to the table structure of the data page to ensure that the 10 threads can correctly read the data page into the cache by executing the read tasks. Obviously, the creation of read tasks requires a certain amount of memory and computational overhead. In this embodiment, by setting corresponding cache areas for data pages with different table structures, the classification management of data pages can be achieved, which is convenient for reusing the read tasks of 10 threads, thereby reducing the overhead of 10 threads creating read tasks.

[0075] Based on the above cache allocation process and the determination process of the read order, when 10 threads want to read the first data page and the second data page according to the read order, 10 threads can read the first data page into the cache corresponding to the table structure of the to-be-merged table file to which it belongs, and 10 threads can also read the second data page into the corresponding cache. After that, 10 threads can also add cache location information to the first data page and the second data page respectively. The addition of this cache location information can be considered as setting the attribute of the first data page that has been read into the cache to a recyclable object (Recyclable object). This cache location information is used to reflect which cache area the data page comes from.

[0076] Continuing the situation in the embodiment shown in FIG. 1, when the records in any data page are written into the merged table file, it indicates that the compression thread has finished consuming the any data page. Then the compression thread can also delete the any data page from the cache corresponding to its cache location information, thereby releasing the cache space occupied by this data page. And it is easy to understand that considering the limited size of the cache, therefore, the number of data pages stored in the cache is limited. Before any data page is released from the cache, no new data page can be written into this cache.

[0077] In this embodiment, by setting the data page as a recyclable object and adding cache location information to the data page, the cache space occupied by the data page can be recycled after the data page is written into the merged table file, so as to improve the utilization rate of the cache.

[0078] In each of the above embodiments, a situation is mentioned: when the compression thread has a page fault, 10 threads have read the supplementary data page into the cache according to the read order, and the compression thread can directly use this supplementary data page to continue sorting. However, in practice, it is also possible that the supplementary data page is not read into the cache for various reasons. Then when the compression thread has a page fault, the compression thread can determine its next processing logic according to the reason why the data page is not read into the cache, that is, determine whether it can continue sorting.

[0079] Optionally, whether the supplementary data page is read into the cache and the reason for not being read into the cache can be reflected by the read status of the table file to be merged to which the supplementary data page belongs. Optionally, the read status of the table file to be merged to which the supplementary data page belongs can be monitored in real time by the status management module in the database instance. The status management module (Async Prefetch Manager) can monitor the aforementioned read status during the process of the 10 threads reading the data page from the disk into the cache.

[0080] Among them, the read status of the table file to be merged can include: ready status (READY), blocked status (BLOCK), completed status (FINISHED), failed status (FAILED), and closed status (CLOSED).

[0081] Among them, the ready status indicates that the supplementary data page has been read into the cache.

[0082] The blocked status indicates that the supplementary data page has not been read into the cache. Optionally, the reasons for not being read into the cache can include any one of the supplementary data page being read from the disk into the cache, or the cache corresponding to the table file to be merged to which the supplementary data page belongs being full, etc.

[0083] The failed status indicates that an exception or interruption occurred during the 10 process of the supplementary data page.

[0084] The completed status indicates that all data pages in a table file to be merged have been written into the merged table file.

[0085] The closed status indicates that all data pages in a table file to be merged have been written into the merged table file, and the resources required to read this table file to be merged (such as 10 resources) are recycled. And it can enter the closed status after maintaining the completed status for a certain period of time.

[0086] Based on the above description, when a page fault occurs in the compression thread, the compression thread can obtain the read status of the target table file to be merged to which any data page belongs from the status management module. The relationship between this read status and the processing logic of the compression thread can be as follows.

[0087] One case: If the read status is the ready status, that is, the read status reflects that the second data page as the supplementary data page already exists in the cache, then the compression thread can continue to execute steps S102 - S103 in the embodiment shown in FIG. 1.

[0088] In this case, the merging process of the table file can also be understood in combination with FIG. 3a.

[0089] In another case, if the read status is the blocked state, indicating that the second data page has not been read into the cache, the compression thread can wait, and at this time, the merging process of the table file pauses. And during the waiting process, the compression thread can continuously obtain the read status of the target table file to be merged from the status management module. When the read status becomes ready, the compression thread can continue to execute steps S102 to S103 in the embodiment shown in FIG. 1. In this case, the merging process of the table file can also be understood in combination with FIG. 3b.

[0090] And because the reasons for the blocked state of the table file to be merged are different, therefore, the content reflected by the change of the read status to the ready state is also different: when the read status becomes the ready state, it can indicate that the target cache corresponding to the target table file to be merged to which the second data page belongs is not full; it can also indicate that the previous abnormality of the second data page is eliminated, and then the 10 thread can continue to read the second data page into the target cache; it can also indicate that the 10 thread successfully reads the second data page into the target cache.

[0091] In yet another case, if the read status is the completed state, reflecting that all data pages in the target table file to be merged have been read into the cache, then after the 10 thread can read other data pages belonging to other table files to be merged into the cache, the compression thread can sort the remaining data pages and other data pages in the cache to continue the merging process of the table file. In this case, the merging process of the table file can also be understood in combination with FIG. 3c.

[0092] In this case, the merging process of the table file can also be understood in combination with FIG. 3c.

[0093] In yet another case, if the read status is the fault state or the closed state, reflecting that the data page reading in the target table file to be merged fails, then after reading other data pages belonging to other table files to be merged into the cache, the compression thread can sort the remaining data pages and other data pages in the cache to continue the merging process of the table file.

[0094] In this case, the merging process of the table file can also be understood in combination with FIG. 3d.

[0095] In summary of the above embodiments, on the one hand, before merging the table files, the compression thread can first sort the data pages to be merged using the recorded primary key values. This sorting result is also the reading order of the data pages by the 10 threads, that is, the sorting order of the compression thread. Then, during the process of merging the table files, the 10 threads can supplement the data pages into the cache according to this reading order before the compression thread encounters a page fault. Thus, when the compression thread encounters a page fault, it can directly obtain the supplementary data pages from the cache instead of through the 10 process, so as to continuously merge the table files, improving the interruption of table file merging caused by the 10 process, and thus increasing the speed of table file merging.

[0096] In addition, when the compression thread encounters a page fault, the compression thread can also determine its own processing logic according to the reading status provided by the status management module. For specific content, reference can be made to the embodiments shown in FIGS. 32 to 3d above.

[0097] On the other hand, before merging the table files, for the table files to be merged with different table structures, corresponding caches can also be set for them, so that the data pages to be merged with the same table structure are stored in the same cache. During the process of the 10 threads reading the data pages, cache location information can also be set for the read data pages. Then, during the process of merging the table files, when all the records in a data page in a cache are written into the merged table file, the storage space occupied by the data page can be determined using the cache location information of the data page, and further the storage space occupied by the data page can be released, thereby realizing the reuse of the cache and improving the utilization rate of the cache.

[0098] For the table file merging method provided in the above embodiments, it can be specifically understood in combination with the example shown in FIG. 4.

[0099] Continuing with the example in FIG. 2, assume that there are table files to be merged 1 to table files to be merged 3 in the database. Among them, table files to be merged 1 and table files to be merged 2 have the first table structure, and table files to be merged 3 have the second table structure.

[0100] Before merging the table files, the 10 threads can set cache 1 for table files to be merged 1 and table files to be merged 2, and set cache 2 for table files to be merged. At the same time, the compression thread can also sort the data pages in the table files according to the maximum primary key values corresponding to each data page in table files to be merged 1 to table files to be merged 3 to obtain the reading order of the data pages.

[0101] Among them, each table file to be merged is set as an iterable object, and the three table files to be merged respectively correspond to iterators 1 to 3.

[0102] After the above preprocessing process is completed, the merging of the table files can begin.

[0103] Thread 10 can read data page 1 in the table file to be merged 1 and data page 2 in the table file to be merged 2 into cache 1 in the reading order, and thread 10 can read data page 3 in the table file to be merged 3 into cache 2. Among them, data pages 1 to 3 are the first data pages in the above embodiments. The compression thread can sort records 1 to 3 in data pages 1 to 3 read by iterators 1 to 3 respectively. The specific sorting process can refer to the description in the embodiment shown in Figure 2.

[0104] And during the process of thread 10 reading the data pages, thread 10 can also modify the reading status of the table file to which the data page belongs, and the reading status can be specifically managed by the status management module in the database instance. The specific content and meaning of the reading status can refer to the description in the above related embodiments and will not be elaborated here.

[0105] As the number of sorting rounds increases, when all the records in data page 1 have been written into the merged table file and the compression thread encounters a page fault, the compression thread can further obtain the reading status of the table file to be merged 1 to which data page 1 belongs from the status management module.

[0106] If the reading status of the table file to be merged 1 is READY, it indicates that data page 4 (which is the second data page in the above embodiments) in the table file to be merged 1 has been read into the cache, and this data page 4 is also the one that thread 10 should read in the reading order. Then the compression thread can continue to perform the next round of sorting on data pages 2 to 4.

[0107] If the reading status of the table file to be merged 1 is BLOCK, the compression thread can wait and continue with the next round of sorting after data page 4 is read into the cache.

[0108] If the reading status of the table file 1 to be merged is any one of FINISHED, FAILED, or CLOSED, then the compression thread can, after the data pages 5 belonging to the table file 4 to be merged (i.e., the other data pages in the above embodiments) are read by the 10 threads in the reading order, continue to perform the next round of sorting on the data pages 2, 3, and 5. Among them, the data page 5 can be stored in the cache 2.

[0109] And during the above sorting process, assuming that the cache 1 can store at most 3 data pages, then the data pages 1, 2, and 4 involved in the above sorting process have filled the cache 1. If no processing is performed, subsequent sorting cannot be carried out.

[0110] To ensure the continuous progress of sorting, after the data page 1 is written into the merged table file, the data page 1 can also be deleted from the cache 1, that is, the storage space in the cache 1 is released. The released storage space is used to store the data page 6 with the table structure 1 read by the 10 threads 1 in the reading order. The data page 6 can belong to the table file 1 to be merged.

[0111] In Figure 4, the data page 1 is represented by ①, and similarly, the data page 2 is represented by ②, and so on. The data page 1 deleted from the cache 1 in the figure is marked with a diagonal line.

[0112] And this embodiment schematically gives the overall process of table file synthesis. For the content not described in detail, reference can also be made to the relevant descriptions in the above embodiments, which will not be elaborated here.

[0113] The above has described from the method perspective the process of the 10 threads and the compression thread achieving table file merging through collaborative work. And the above threads can specifically be provided by the database instance. Then the following can also describe the table file merging process from the perspective of the database instance. Figure 5 is a schematic structural diagram of a database instance provided by an embodiment of the present application. As shown in Figure 5, the database instance can include: 10 threads and a compression thread.

[0114] The 10 threads are used to read the first data page and the second data page from the table file to be merged into the cache.

[0115] The compression thread is used to sort the records contained in the first data page to obtain a first sorting result. The first data pages belong to different table files to be merged. Then, the compression thread continuously writes the records in the first data page into the merged table file according to the first sorting result. During this process, if any data page in the first data page is written into the merged table file, the records contained in the remaining data pages in the first data page and the second data pages in the cache are sorted respectively to obtain a second sorting result; the compression result can also write the remaining data pages and the second data pages belonging to the same table file to be merged as any data page into the merged table file according to this second sorting result.

[0116] When all the records contained in the data pages of the table file to be merged are written into the merged table file, the merging of the table file is completed.

[0117] Optionally, the sorting process of the compression thread for the first data page can refer to the relevant description in the embodiment shown in FIG. 2, which will not be elaborated here.

[0118] In this embodiment, for the first data page and the second data page that have been stored in the cache and belong to different table files to be merged, the compression thread in the database instance can first sort the records in a preset number of first data pages to obtain a first sorting result, and write the records in the first data page into the merged table file in order according to the first sorting result. And during the process of file merging according to the first sorting result, when a page fault occurs to the compression thread when any data page is completely written into the merged table file, the compression thread can continue to sort the second data pages and the remaining data pages in the cache to obtain a second sorting result, and continue to write the records in the data pages into the merged table file according to the second sorting result. After all the data pages of the table file to be merged are written into the merged table file, that is, the merging of the table file is completed.

[0119] In the above method, the compression thread in the database instance can first sort a preset number of first data pages. When any data page is written into the file to be merged, the number of data pages being sorted in the cache will decrease, and the compression thread is in a page fault state. At this time, the compression thread can directly use the second data page that has been stored in the cache as a supplement to continue sorting. Since the second data page is already in the cache, when a page fault occurs to the compression thread, no data page process will occur in the database instance, and the problem of low table file merging efficiency caused by the process can be shortened, that is, the efficiency of table file merging can be improved.

[0120] In addition, for the content not described in detail in this embodiment and the achievable technical effects, reference may be made to the relevant descriptions in the above embodiments, which will not be elaborated herein.

[0121] Before merging the table files, optionally, the compression thread can pre-set corresponding caches for the data pages in the table files to be merged with different table structures, and can also set cache location information for the data pages, so as to realize the reuse of the cache by using this cache location information and improve the utilization rate of the cache.

[0122] Before merging the table files, optionally, the compression thread can also sort the data pages to be merged to obtain the reading order of the data pages. The compression thread can use this reading order to read the supplementary data pages into the cache before a page fault occurs in the compression thread, so that the compression thread can continuously perform sorting. Since there is no I / O process of data pages when a page fault occurs in the compression thread, the speed of table file merging can also be improved.

[0123] For the content not described in detail in this embodiment and the achievable technical effects, reference may be made to the relevant descriptions in the above embodiments, which will not be elaborated herein.

[0124] In most cases, the compression thread can read the supplementary data pages into the cache before a page fault occurs in the compression thread according to the reading order. However, there may be exceptions. Optionally, when a page fault occurs in the compression thread, it can determine its own processing logic according to the reading state of the table files to be merged. FIG. 6 is a schematic structural diagram of another database instance provided by an embodiment of the present application. On the basis of the embodiment shown in FIG. 5, the database instance may further include a status management module.

[0125] Continuing with the situation in the embodiment shown in FIG. 5, when the compression thread writes all the records included in any data page in the first data page into the merged table file, that is, when a page fault occurs in the compression thread, the compression thread can first obtain the reading state of the target table file to be merged to which any data page belongs from the status management module, and determine its next processing logic according to this reading state. The relationship between the processing logic and the reading state can be referred to the relevant descriptions in the embodiments shown in FIGS. 33 to 3d above.

[0126] In addition, for the content not described in detail in this embodiment and the achievable technical effects, reference may be made to the relevant descriptions in the above embodiments, which will not be elaborated herein.

[0127] The table file merging device according to one or more embodiments of the present application will be described in detail below. Those skilled in the art can understand that these table file merging devices can all be configured by using commercially available hardware components through the steps taught by this solution.

[0128] FIG. 7 is a schematic structural diagram of a table file merging device provided by an embodiment of the present application. As shown in FIG. 7, the device may include the following modules.

[0129] The first sorting module 11 is configured to sort the records included in a preset number of first data pages in the cache to obtain a first sorting result, and the to-be-merged table files to which the first data pages belong are different from each other.

[0130] The second sorting module 12 is configured to, according to the first sorting result, if any one of the first data pages is written into the merged table file, sort the remaining data pages in the first data pages and the records included in the second data pages in the cache respectively to obtain a second sorting result.

[0131] The writing module 13 is configured to write the remaining data pages and the second data pages that belong to the same to-be-merged table file as any one of the data pages into the merged table file according to the second sorting result.

[0132] Optionally, the device further includes: a determination module 14 and a reading module 15.

[0133] The determination module is configured to determine the reading order of the to-be-merged data pages according to the primary key values of the records included in the to-be-merged data pages on the disk, and the to-be-merged data pages belong to multiple to-be-merged table files.

[0134] The reading module 15 is configured to read the first data pages and the second data pages in the to-be-merged data pages into the cache according to the reading order.

[0135] Optionally, the device further includes: a creation module 16, configured to create corresponding caches for the to-be-merged table files with different table structures.

[0136] The reading module 15 is configured to read the first data pages and the second data pages into the corresponding caches according to the table structures of the to-be-merged table files to which the first data pages and the second data pages belong respectively.

[0137] Optionally, the device further includes: an adding module 17 and a deleting module 18.

[0138] The adding module is configured to add cache location information to the first data page and the second data page respectively in response to the reading of the first data page and the second data page.

[0139] The deleting module 18 is configured to delete any one of the data pages from the cache corresponding to the cache location information after the any one of the data pages is written into the merged table file.

[0140] Optionally, the apparatus further includes: a modifying module 19, configured to modify the read status of the to-be-merged table files to which the first data page and the second data page respectively belong in response to the reading of the first data page and the second data page.

[0141] Optionally, the second sorting module 12 is configured to, if any one of the data pages is written into the merged table file, obtain the read status of the target to-be-merged table file to which the any one of the data pages belongs; if the read status indicates that there is an un-sorted second data page in the target to-be-merged table file in the cache, sort the records included in the remaining data pages and the second data page respectively.

[0142] Optionally, the second sorting module 12 is further configured to, if the read status indicates that the second data page does not exist in the cache, sort the records included in the remaining data pages and the second data page respectively after the second data page is read into the cache.

[0143] Optionally, there is a corresponding cache for the to-be-merged table files with different table structures.

[0144] The reading module 15 is further configured to read the second data page into the target cache when the target cache corresponding to the target to-be-merged table file is not full; or, in response to the recovery of an abnormal data page reading, read the second data page into the target cache.

[0145] Optionally, the second sorting module 12 is configured to, if the read status indicates that all data pages in the target to-be-merged table file have been read into the cache or the data page reading in the target to-be-merged table file fails, sort the records included in the remaining data pages and the other data pages respectively after reading the other data pages into the cache in accordance with the reading order.

[0146] Wherein, the other data pages and the to-be-merged table files to which the first data page belongs are different from each other.

[0147] Optionally, the records in the data page are stored in an ordered manner according to the primary key values. Any of the data pages belongs to the target table file to be merged. The attribute of the table file to be merged is an iterable object, and the table file to be merged corresponds to an iterator one by one.

[0148] The apparatus further includes: a calling module 20.

[0149] The first sorting module 11 is further configured to use the iterators corresponding to the table files to be merged to which the first data page belongs to read the records in the first data page; write the target record corresponding to the smallest primary key value read by the iterator into the merged table file, and the target record is included in any of the data pages.

[0150] The calling module 20 is configured to call a first function to make the iterator corresponding to the target table file to be merged point to the next record of the target record.

[0151] Optionally, the writing module 13 is further configured to put the alternative records read by the iterator from the first data page into a heap; sort the alternative records according to their respective primary key values to form a minimum heap; call a second function to write the target record at the top of the heap into the merged table file.

[0152] The apparatus shown in FIG. 7 can execute the method of the embodiments shown in FIGS. 1 to 4. For parts not described in detail in this embodiment, reference may be made to the relevant descriptions of the embodiments shown in FIGS. 1 to 4. The execution process and technical effects of this technical solution are described in the embodiments shown in FIGS. 1 to 4 and will not be elaborated here.

[0153] In a possible design, the table file merging method provided in the above embodiments can be applied to an electronic device. As shown in FIG. 8, the electronic device may include: a processor 31 and a memory 32. Among them, the memory 32 is used to store a program for supporting the electronic device to execute the table file merging method provided in the above embodiments shown in FIGS. 1 to 4, and the processor 31 is configured to execute the program stored in the memory 32.

[0154] The program includes one or more computer instructions. When the one or more computer instructions are executed by the first processor 31, the following steps can be implemented: sorting the records included in a preset number of first data pages in the cache to obtain a first sorting result, where the to-be-merged table files to which the first data pages belong are different; according to the first sorting result, if any one of the first data pages is written into the merged table file, then sorting the records included in the remaining data pages in the first data pages and the second data pages in the cache respectively to obtain a second sorting result; according to the second sorting result, writing the remaining data pages and the second data pages that belong to the same to-be-merged table file as the any one data page into the merged table file.

[0155] Optionally, the processor 31 is further configured to execute all or part of the steps in the foregoing embodiments shown in FIGS. 1 to 4.

[0156] Wherein, the structure of the electronic device may further include a communication interface 33 for the electronic device to communicate with other devices or communication systems.

[0157] In addition, an embodiment of the present application provides a computer storage medium for storing the computer software instructions used by the foregoing electronic device, which includes a program involved in executing the table file merging method shown in FIGS. 1 to 4 above.

[0158] In addition, an embodiment of the present application provides a computer program product. The computer program product includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is caused to implement the steps or functions of the table file merging method shown in FIGS. 1 to 4 above.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

Claims 1. A method for merging table files, wherein, Including: Sorting the records included in a preset number of first data pages in the cache to obtain a first sorting result, where the to-be-merged table files to which the first data pages belong are different; according to the first sorting result, if any one of the first data pages is written into the merged table file, then sorting the records included in the remaining data pages in the first data pages and the second data pages in the cache respectively to obtain a second sorting result; According to the second sorting result, writing the remaining data pages and the second data pages that belong to the same to-be-merged table file as the any one data page into the merged table file.

2. The method according to claim 1, wherein The method further includes: determining the reading order of the to-be-merged data pages according to the primary key values of the records included in the to-be-merged data pages on the disk, where the to-be-merged data pages belong to multiple to-be-merged table files; reading the first data pages and the second data pages in the to-be-merged data pages into the cache according to the reading order.

3. The method according to claim 2, wherein The method further includes: creating corresponding caches for the to-be-merged table files with different table structures; the step of reading the first data pages and the second data pages in the to-be-merged data pages into the cache includes: reading the first data pages and the second data pages into the corresponding caches according to the table structures of the to-be-merged table files to which the first data pages and the second data pages belong respectively.

4. The method according to claim 2, wherein The method further includes: in response to the reading of the first data pages and the second data pages, adding cache location information to the first data pages and the second data pages respectively; after any one of the data pages is written into the merged table file, deleting the any one data page from the cache corresponding to the cache location information.

5. The method according to claim 2, wherein, The method further includes: in response to the reading of the first data pages and the second data pages, modifying the reading status of the to-be-merged table files to which the first data pages and the second data pages belong respectively.

6. The method according to claim 5, wherein, If any one of the first data pages is written into the merged table file, then sorting the records included in the remaining data pages in the first data pages and the second data pages in the cache respectively, includes: if any one of the data pages is written into the merged table file, then obtaining the reading status of the target to-be-merged table file to which the any one data page belongs; if the reading status reflects that there are second data pages in the cache that have not been sorted in the target to-be-merged table file, then sorting the records included in the remaining data pages and the second data pages respectively.

7. The method according to claim 6, wherein The method further includes: if the reading status reflects that the second data pages do not exist in the cache, then after the second data pages are read into the cache, sorting the records included in the remaining data pages and the second data pages respectively.

8. The method according to claim 7, wherein There are corresponding caches for the table files to be merged with different table structures; the method further includes: when the target cache corresponding to the target table file to be merged is not full, reading the second data page into the target cache; or, in response to the recovery of an abnormal data page reading, reading the second data page into the target cache.

9. The method according to claim 6, wherein The method further includes: if the reading status reflects that all data pages in the target table file to be merged have been read into the cache or the data page reading in the target table file to be merged fails, then after reading other data pages into the cache in accordance with the reading order, sorting the remaining data pages and the records included in each of the other data pages; wherein, the other data pages and the table file to be merged to which the first data page belongs are different from each other.

10. The method according to claim 1, wherein The records in the data page are stored in an orderly manner according to the primary key value, any data page belongs to the target table file to be merged, the attribute of the table file to be merged is an iterable object, and the table file to be merged corresponds to an iterator one by one; The sorting of the records included in the first data page in the cache includes: using the iterators corresponding to the table files to be merged to which the first data page belongs, reading the records in the first data page; writing the target record corresponding to the smallest primary key value read by the iterator into the merged table file, and the target record is included in any data page; the method further includes: calling a first function to make the iterator corresponding to the target table file to be merged point to the next record of the target record.

11. The method according to claim 10, wherein, The writing the target record corresponding to the smallest primary key value read by the iterator into the merged table file includes: putting the alternative records read by the iterator from the first data page into a heap; sorting the alternative records according to their respective primary key values to form a minimum heap; calling a second function to write the target record at the top of the heap into the merged table file.

12. A database instance, wherein, Including: Input / output 10 threads and a compression thread; The 10 threads are used to read the first data page and the second data page from the table file to be merged into the cache; The compression thread is used to sort the records included in the first data page to obtain a first sorting result, and the table files to be merged to which the first data page belongs are different from each other; according to the first sorting result, if any data page in the first data page is written into the merged table file, then sorting the remaining data pages in the first data page and the records included in the second data page in the cache respectively to obtain a second sorting result; according to the second sorting result, writing the remaining data pages and the second data page belonging to the same table file to be merged as any data page into the merged table file.

13. The example according to claim 12, wherein The instance further includes a status management module; the 10 threads are used to modify the first The read status of the table file to which each of the first data page and the second data page belongs; The status management module is configured to obtain the read status; The compression thread is configured to obtain the read status from the status management module; If the read status indicates that there is an unsorted second data page in the target table file to be merged in the cache, then sort the records included in each of the remaining data pages and the second data page.

14. An electronic device, wherein, Comprising: A memory and a computing system; wherein, executable code is stored on the memory, and when the executable code is executed by the computing system, the computing system is caused to execute the table file merging method according to any one of claims 1 to 11.

15. A non-transitory machine-readable storage medium, wherein, Executable code is stored on the non-transitory machine-readable storage medium, and when the executable code is executed by the computing system of the electronic device, the computing system is caused to execute the table file merging method according to any one of claims 1 to 11.

16. A computer program product, wherein, Comprising a computer program or instruction, and when the computer program or instruction is executed by a processor, the processor is caused to be able to implement the steps in the table file merging method according to any one of claims 1 to 11. 19

Images