Shentong database three-layer cache segment search management structure and page search method

Abstract

The present invention relates to a three-layer cache segment search management structure and page search method of a Shentong database, and its technical characteristics are: including thread-level cache, segment search global cache and physical space; thread-level cache is a local cache of a thread; segment search global cache includes global cache node and a batch of free page items managed by global cache node, and the page items managed by each cache node are organized into a multi-level linked list according to the free space gear; physical space is managed by PFS page, all PFS pages in the segment are multi-branch tree structure, one PFS page manages several data pages, and each PFS page records a batch of data pages and their space free values. The present invention is reasonably designed, improves the performance of multi-threaded concurrent page search, effectively improves concurrency and throughput, and solves the problem of a large number of I / Os caused by accessing the PFS tree of the segment every time a page is searched, and the problem of resource competition caused by multiple threads concurrently searching the PFS tree.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and in particular to a three-layer cache segment search management structure and a page search method for a Shentong database. Background Art

[0002] In the database, a page is the smallest unit of data files exchanged between internal and external memory, and the common size is 8KB. A segment is organized by several pages, corresponding to an entity (table or index) in the database. The segment module is one of the core modules of the Shentong database storage subsystem. The performance and function of the segment module directly affect the performance and function of the database, so its good design can ensure the high scalability and high availability of the database.

[0003] PFS (page free space) page is a special page used by the segment to manage data pages. One PFS page can manage several data pages. All PFS pages of a segment are in a multi-branch tree structure. Each PFS page records a batch of data pages and their free space values. The free space value maintained on the PFS page cannot be the exact free space value of the data page. Otherwise, every time the free space of the data page changes, the value on the PFS page needs to be modified, which will cause a large concurrency conflict. A free space classification (PFS gear) strategy is adopted to reduce concurrency conflicts. The size of the free space of the page can be roughly determined based on the PFS gear of the page.

[0004] An important operation in the segment module is the segment search operation, which requires finding a page that can be inserted in all the pages of the segment. Searching for a suitable available page in the segment will have the following problems: (1) When the space managed by the segment is very large, accessing the segment's PFS tree every time a page with free space is searched will cause a lot of I / O and be very costly; (2) During concurrent access, multiple threads searching the PFS tree at the same time will cause a lot of resource competition. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a three-layer cache segment search management structure and a page search method for a Shentong database, which solves the problem of a large amount of I / O caused by accessing the PFS tree of the segment every time a page is searched, and the problem of resource competition caused by multiple threads concurrently searching the PFS tree.

[0006] The present invention solves the existing technical problems by adopting the following technical solutions:

[0007] A page search method for a three-layer cache segment search management structure of a Shentong database, including a thread-level cache page search method, a segment search global cache page search method, and a physical space page search method;

[0008] The segment search global cache page search method comprises the following steps:

[0009] ⑴Get the cache node of the corresponding segment and add a mutex lock;

[0010] ⑵ Polling allocation page items: Use the best fit algorithm in the global cache node to find the page item that satisfies the free space gear less than or equal to the search value, and then remove it from the linked list. If it is not found, load the required page from the physical space through the PFS tree search;

[0011] ⑶ Update the statistical information of the cache node and release the node mutex lock;

[0012] ⑷Return the found page item;

[0013] The three-layer cache segment search management structure of the Shentong database includes a thread-level cache, a segment search global cache and a physical space; the thread-level cache is a local cache of a thread, each thread has an array structure to maintain a fixed number of page items, each page item of the thread-level cache is thread-private, a thread caches at most one page item for a certain segment, and saves the global cache node pointer and a sequence number of this segment, the sequence number is used to determine whether the saved global cache node is invalid; the segment search global cache includes a global cache node and a batch of free page items managed by the global cache node, the page items managed by each cache node are organized into a multi-level linked list according to the free space gear, and statistical information is recorded in the global cache node;

[0014] The physical space is managed by PFS pages. All PFS pages in a segment are multi-branch tree structures. One PFS page manages several data pages. Each PFS page records a batch of data pages and their free space values.

[0015] Furthermore, the thread-level cache page search method is implemented by using a binary search method.

[0016] Furthermore, the physical space page search method includes the following steps:

[0017] ⑴ Continue searching from the PFS position after the last search, and determine whether the last PFS page is invalid. If yes, proceed to the next step, otherwise proceed to step ⑹;

[0018] ⑵Search the page in the PFS tree. If the search is successful, proceed to the next step, otherwise proceed to step ⑸;

[0019] ⑶ Determine whether there is a conflict with the existing pages in the PFS search array, if yes, proceed to the next step, otherwise proceed to step ⑹;

[0020] (4) Determine whether the segment has been extended. If yes, proceed to step (2). Otherwise, perform segment extension.

[0021] ⑸ Is it the first search or has it been reset? If so, perform segment expansion, otherwise reset the tree traversal and go to step ⑴;

[0022] (6) Search the corresponding PFS page for available data pages. If the search is successful, proceed to the next step. Otherwise, the current element value of the search array is set to invalid and return to step (1).

[0023] Furthermore, the page items of the thread-level cache are stored in order according to the segment numbers to which they belong. When a transaction searches for a page item of a certain segment in the cache array, a binary search is used to search.

[0024] Further, the page entries of the thread-level cache are allocated from the segment search global cache.

[0025] Furthermore, the multi-level linked list includes eight levels, from level one to level eight are respectively:

[0026] PFS_UNINIT: uninitialized page;

[0027] PFS_FREE: Pages with 100% free space;

[0028] PFS_LEVEL1: Pages with free space between 50% and 100%;

[0029] PFS_LEVEL2: Pages with free space between 17% and 50%;

[0030] PFS_LEVEL3: Pages with free space between 5% and 17%;

[0031] PFS_LEVEL4: Pages with free space between 0% and 5%;

[0032] PFS_OVFL: overflow page;

[0033] PFS_SPECIAL: Special page.

[0034] Furthermore, the statistical information recorded in the global cache node includes: the number of page items currently stored in the global cache node, the number of page items that have been allocated, and the sequence number.

[0035] The advantages and positive effects of the present invention are:

[0036] The present invention adopts a three-layer cache segment search management method of thread-level cache, segment search global cache and physical space search, which improves the performance of multi-threaded concurrent page search, effectively improves concurrency and throughput, and solves the problem of a large amount of I / O caused by accessing the PFS tree of the segment every time a page is searched, and solves the problem of resource competition caused by multiple threads concurrently searching the PFS tree. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a diagram of the segment search global cache structure of the present invention;

[0038] Figure 2 It is a flow chart of the PFS tree search algorithm of the present invention;

[0039] Figure 3 This is the overall search strategy schematic diagram of the present invention:

[0040] Figure 4 It is a flow chart of searching for the next page of the present invention. DETAILED DESCRIPTION

[0041] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0042] The present invention proposes a three-layer cache segment search structure of a Shentong database, including a thread-level cache, a segment search global cache and a physical space, wherein:

[0043] 1. Thread-level cache.

[0044] The thread-level cache is a thread-local cache. Each thread has an array structure that maintains a fixed number of page entries. These page entries are thread-private. A thread can cache at most one page entry for a segment and save the global cache node pointer and a sequence number (seq) of the segment. The sequence number is used to determine whether the saved global cache node is invalid.

[0045] The advantages of using thread-level caching are:

[0046] (1) A page item belongs to only one thread, so there will be no read and write competition for the same data page;

[0047] (2) Reduce the number of accesses to the global cache and reduce critical section conflicts in the global cache;

[0048] (3) Since the cached page items are thread-private, when a transaction performs continuous insert operations, the records in the page are clustered by transaction, which improves the locality of the data and can effectively reduce the IO overhead of the query.

[0049] The page items of the thread-level cache are stored in order according to the segment number to which they belong. Therefore, when a transaction searches for a page item of a segment in the cache array, a binary search can be used to improve the search efficiency.

[0050] The page items of the thread-level cache are allocated from the segment search global cache. When allocating, the page items will be removed from the global cache. Therefore, it can be guaranteed that at any time, a page item will not be allocated to multiple threads at the same time. This can eliminate the hidden danger of page contention caused by threads concurrently inserting data into the same physical page.

[0051] When a transaction is committed, the thread-level cache page will not be cleared. Only when the thread ends will the thread-level cache be cleared and the page item will be returned to the global cache of the segment to which it belongs, if the page item has not expired (the segment to which the page item belongs has not been deleted).

[0052] There are two situations for replacing page entries in the thread-level cache:

[0053] (1) The capacity of the thread-level cache is limited. Therefore, when the cache reaches its upper limit and a page entry of a new segment needs to be added, an old page entry in the cache needs to be replaced.

[0054] (2) When a transaction searches for a page in a certain free space level, if the page item in the thread cache does not meet the level requirements, it will search for a page item that meets the requirements from the global cache and replace the old page item. If the PFS value recorded in the old page item is PFS_OVFL, it means that the corresponding physical page can no longer insert data. In this case, the page item is discarded directly. Otherwise, it is returned to the segment search global cache.

[0055] 2. Segment Search Global Cache

[0056] Each segment has an independent segment search global cache and a batch of free page entries for the segment managed by the segment search cache. All threads that perform insertion operations on the same segment share a segment search global cache. When a thread has no private page entry or the free space of the private page entry does not meet the requirements, the thread will access the segment search global cache to find a new available page entry.

[0057] like Figure 1 As shown in the figure, the segment search global cache includes the global cache node (CacheNode) and a batch of free page items (PageItem) managed by the global cache node. In order to improve the utilization of physical space and speed up the search, we organize the page items managed by each cache node into a multi-level linked list according to the free space level. There are eight levels in the linked list, corresponding to:

[0058] 1. PFS_UNINIT (uninitialized page)

[0059] 2. PFS_FREE (pages with 100% free space)

[0060] 3. PFS_LEVEL1 (pages with free space between 50% and 100%)

[0061] 4. PFS_LEVEL2 (pages with free space between 17% and 50%)

[0062] 5. PFS_LEVEL3 (pages with free space between 5% and 17%)

[0063] 6. PFS_LEVEL4 (pages with free space between 0% and 5%)

[0064] 7. PFS_OVFL (overflow page)

[0065] 8. PFS_SPECIAL (Special Page)

[0066] At the same time, the global cache node also records some statistical information, such as the number of page items currently stored in the node, the number of page items that have been allocated, the sequence number, etc. When a segment is deleted, the sequence number (seq) ++.

[0067] In the process of segment search global cache management, the search algorithm in the global cache node is:

[0068] (1) Get the cache node of the corresponding segment and add a mutex lock.

[0069] (2) Polling allocation page items: Search for page items in the node that meet the free space gear less than or equal to the search value - BestFit (best fit algorithm) strategy, and then remove them from the linked list. If not found, load the required page from the physical space through PFS tree search.

[0070] (3) Update the statistical information of the cache node and release the node mutex lock.

[0071] (4) Return the found page item.

[0072] In the above search algorithm, when the loading operation obtains a page that meets the requirements, in order to ensure the uniqueness of the page in the global cache node, a unique hash structure is maintained in the global cache node, and hash insertion is performed according to the page number. The number of page items maintained in the global cache node has a certain threshold. Before inserting the loaded page, it is necessary to calculate whether the number of existing page items in the current global cache node and the number of pages to be inserted exceeds a certain threshold. If it exceeds, it is necessary to remove the page item along the PFS linked list (starting from PFS_OVFL) and hash the loaded page into the unique hash structure according to the page number. When the page item does not exist in the global cache, it is inserted into the PFS linked list and the statistical information is updated.

[0073] 3. Physical Space

[0074] Physical space is managed using PFS pages. All PFS pages in a segment are in a multi-branch tree structure (PFS tree). A PFS page manages several data pages, and each PFS page records a batch of data pages and their free space values.

[0075] A segment in the physical space consists of multiple extents, and each extent consists of a certain number of consecutive blocks.

[0076] The search of the PFS tree adopts a round-robin method, polling available pages (stored in an array) in a fixed m PFS pages, and a PFS search handle is saved in the global cache node, which stores the PFS tree traversal handle and the PFS page location information at the end of the last search. Each search starts from the end position of the last search, and each searching thread is tried to get different pages, and the pages are not concentrated in one PFS as much as possible to reduce PFS conflicts.

[0077] PFS tree search algorithm is as follows Figure 2 As shown, the specific steps include:

[0078] (1) Continue searching from the PFS position after the last search and determine whether the last PFS is invalid. If so, proceed to the next step; otherwise, proceed to step (6).

[0079] (2) Search the page in the PFS tree. If the search is successful, proceed to the next step; otherwise, proceed to step (5).

[0080] (3) Determine whether there is a conflict with an existing page in the PFS search array. If so, proceed to the next step; otherwise, proceed to step (6).

[0081] (4) Determine whether segment expansion has been performed. If so, proceed to step (2); otherwise, perform segment expansion.

[0082] (5) Is this the first search or has it been reset? If so, perform segment expansion; otherwise, reset the tree traversal and go to step (1).

[0083] (6) Search the corresponding PFS page for an available data page. If the search is successful, proceed to the next step. Otherwise, the current element value of the search array is set to invalid and return to step (1).

[0084] (7) Set the PFS page as the current page, record the index (position) of the search page, and return the ID of the search page.

[0085] Based on the above description, the present invention proposes a page search method for searching three-layer cache segments of a Shentong database, such as Figure 3 As shown, it includes a page search method for a thread-level cache, a page search method for a segment search global cache, and a page search method for a physical space (PFS tree).

[0086] Thread-level cache page search method: Since the page entries managed by the on-site cache are thread-private, they are used to reduce page conflicts and access to the global cache. Therefore, when a transaction searches for a page entry for a segment in the cache array, a binary search method can be used.

[0087] Segment search global cache page search method: allocate page items in a bestfit manner to find the optimal page. Cache a batch of page items, delete the page items from the global cache when allocating, and insert them into the thread cache to eliminate competition for the same page. Organize into a multi-level linked list according to PFS to improve physical space utilization and speed up search. The specific method includes the following steps:

[0088] (1) Get the cache node of the corresponding segment and add a mutex lock.

[0089] (2) Polling allocation page items: Search for page items in the node that meet the free space gear less than or equal to the search value - BestFit (best fit algorithm) strategy, and then remove them from the linked list. If not found, load the required page from the physical space through PFS tree search.

[0090] (3) Update the statistical information of the cache node and release the node mutex lock.

[0091] (4) Return the found page item.

[0092] Physical space page search method: load physical page information into the global cache by round-robin searching the PFS tree. The specific method includes the following steps:

[0093] (1) Continue searching from the PFS position after the last search and determine whether the last PFS is invalid. If so, proceed to the next step; otherwise, proceed to step (6).

[0094] (2) Search the page in the PFS tree. If the search is successful, proceed to the next step; otherwise, proceed to step (5).

[0095] (3) Determine whether there is a conflict with an existing page in the PFS search array. If so, proceed to the next step; otherwise, proceed to step (6).

[0096] (4) Determine whether segment expansion has been performed. If so, proceed to step (2); otherwise, perform segment expansion.

[0097] (5) Is this the first search or has it been reset? If so, perform segment expansion; otherwise, reset the tree traversal and go to step (1).

[0098] (6) Search the corresponding PFS page for an available data page. If the search is successful, proceed to the next step. Otherwise, the current element value of the search array is set to invalid and return to step (1).

[0099] (7) Set the PFS page as the current page, record the index (position) of the search page, and return the ID of the search page.

[0100] In the segment search process of the present invention, a mode of start search (BeginSearch) - search next page (SearchNext) - end search (EndSearch) is adopted. Each search gets a search handle at the beginning, and calls the search next page interface through this handle, and after successfully obtaining a physical page with enough free space, calls the end scan to destroy the handle.

[0101] Below Figure 4 The process of searching for the next page is given to illustrate the segment search algorithm, which is also an iterative process of a handle. The specific steps are as follows:

[0102] Step 1: If it is the first search, search the thread-level cache and try to get the page item from the thread-level cache. If found, jump to step 4, otherwise go to step 2; if it is not the first search, go to step 2.

[0103] Step 2: The search times reach the maximum search times for this gear, and the downshift is successful.

[0104] Step 3: Search the segment search global cache for a page entry that satisfies the free space level. If not found, return an invalid page (NULL); otherwise, jump to step 4.

[0105] Step 4: Entering this step indicates that a page item that meets the gear has been found. According to the page number recorded in the page item, the data page is read from the data cache and an exclusive lock is added. The actual free space gear (PFS) of the page is checked, and the page is updated.

[0106] Step 5: If the number of searches is greater than the search value, it means that the page does not meet the requirements. The corresponding page item is returned to the global cache, the page lock is released, and the search is repeated in step 1. Otherwise, the page item is added to the thread-level cache, the number of searches is increased, and the page is returned.

[0107] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. A page search method for a three-layer cache segment search management structure of a Shentong database, characterized in that: It includes a thread-level cache page search method, a segment search global cache page search method, and a physical space page search method; The segment search global cache page search method comprises the following steps: ⑴Get the cache node of the corresponding segment and add a mutex lock; ⑵ Polling allocation page items: Use the best fit algorithm in the global cache node to find the page item that satisfies the free space gear less than or equal to the search value, and then remove it from the linked list. If it is not found, load the required page from the physical space through the PFS tree search; ⑶ Update the statistical information of the cache node and release the node mutex lock; ⑷Return the found page item; The three-layer cache segment search management structure of the Shentong database includes a thread-level cache, a segment search global cache and a physical space; the thread-level cache is a local cache of a thread, each thread has an array structure to maintain a fixed number of page items, each page item of the thread-level cache is thread-private, a thread caches at most one page item for a certain segment, and saves the global cache node pointer and a sequence number of this segment, the sequence number is used to determine whether the saved global cache node is invalid; the segment search global cache includes a global cache node and a batch of free page items managed by the global cache node, the page items managed by each cache node are organized into a multi-level linked list according to the free space gear, and statistical information is recorded in the global cache node; The physical space is managed by PFS pages. All PFS pages in a segment are multi-branch tree structures. One PFS page manages several data pages. Each PFS page records a batch of data pages and their free space values.

2. The page search method of the three-layer cache segment search management structure of the Shentong database according to claim 1 is characterized in that: The thread-level cache page search method is implemented by using a binary search method.

3. The page search method of the three-layer cache segment search management structure of the Shentong database according to claim 1 is characterized in that: The physical space page search method comprises the following steps: ⑴ Continue searching from the PFS position after the last search, and determine whether the last PFS page is invalid. If yes, proceed to the next step, otherwise proceed to step ⑹; ⑵Search the page in the PFS tree. If the search is successful, proceed to the next step, otherwise proceed to step ⑸; ⑶ Determine whether there is a conflict with the existing pages in the PFS search array, if yes, proceed to the next step, otherwise proceed to step ⑹; (4) Determine whether the segment has been extended. If yes, proceed to step (2). Otherwise, perform segment extension. ⑸ Is it the first search or has it been reset? If so, perform segment expansion, otherwise reset the tree traversal and go to step ⑴; (6) Search the corresponding PFS page for available data pages. If the search is successful, proceed to the next step. Otherwise, the current element value of the search array is set to invalid and return to step (1).

4. The page search method of the three-layer cache segment search management structure of the Shentong database according to claim 1 is characterized in that: The page items of the thread-level cache are stored in order according to the segment numbers to which they belong. When a transaction searches for a page item of a certain segment in the cache array, a binary search is used to search.

5. The page search method of the three-layer cache segment search management structure of the Shentong database according to claim 1 is characterized in that: The page entries of the thread-level cache are allocated from the segment search global cache.

6. The page search method of the three-layer cache segment search management structure of the Shentong database according to claim 1 is characterized in that: The multi-level linked list includes eight levels, from level one to level eight respectively: PFS_UNINIT: uninitialized page; PFS_FREE: Pages with 100% free space; PFS_LEVEL1: Pages with free space between 50% and 100%; PFS_LEVEL2: Pages with free space between 17% and 50%; PFS_LEVEL3: Pages with free space between 5% and 17%; PFS_LEVEL4: Pages with free space between 0% and 5%; PFS_OVFL: overflow page; PFS_SPECIAL: Special page.

7. The page search method of the three-layer cache segment search management structure of the Shentong database according to claim 1 is characterized in that: The statistical information recorded in the global cache node includes: the number of page items currently stored in the global cache node, the number of page items that have been allocated, and the sequence number.

Images