A method, system, and storage medium for managing data blocks.

Abstract

This application discloses a data block management method, system, and storage medium. The data block management method effectively improves the data block lookup efficiency by using a data block ID query queue, an LRU management queue, and management nodes in combination. Furthermore, by traversing or using the LRU management queue and / or the data block ID query queue in a skip list manner, the performance problems of data movement caused by using arrays are avoided, thereby improving data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of computer application technology, and more specifically, to a method, system, and storage medium for managing data blocks. Background Technology

[0002] On a server, all disks need to use a large shared buffer. This shared buffer is used to cache data, reducing the number of times users copy data when reading data, and improving the server's data processing efficiency.

[0003] Each disk's corresponding service is managed by a process. Therefore, when we need to manage the cache, we need to use shared memory. Furthermore, the memory allocation in shared memory needs to be dynamically allocated based on the disk's activity level; the more active the disk, the more shared memory is allocated for data caching.

[0004] When using shared memory, it is necessary to divide the shared memory into a management area and a data area. The data area is divided into multiple data blocks, while the management area uses an LRU (Least Recently Used) queue for each disk. The LRU management queue records the priority of the data blocks in the data area. When a data block needs to be evicted, the data block with the lowest priority in the LRU management queue is selected for eviction.

[0005] However, in actual use, it was found that the existing methods for managing shared memory data blocks have problems such as low efficiency in looking up data nodes and low efficiency in data processing. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides a data block management method, system, and storage medium to improve the search efficiency and data processing efficiency of the shared memory data block management method.

[0007] To achieve the above technical objectives, the embodiments of this application provide the following technical solutions:

[0008] A data block management method applied to shared memory, the shared memory including a management area and a data area, the data block management method comprising:

[0009] The data area is divided into multiple data blocks, and the size of each data block is a preset size.

[0010] After obtaining the shared memory mapped to the current process, the starting address of the shared memory in the current process is obtained, and the difference between the starting address of the data block and the starting address of the shared memory is used as the offset address of the data block;

[0011] For each physical disk, a corresponding LRU management queue and a data block ID query queue are established. The data block ID query queue includes the IDs of data blocks arranged in an ordered manner, and the LRU management queue uses a priority eviction algorithm.

[0012] A management node is established for the data block. The management node includes multiple first pointers. The first pointers point to adjacent nodes in the LRU management queue, or to adjacent nodes in the data block ID query queue, or are address pointers of data blocks in the data area, or represent the ID of data blocks in the physical disk, or represent the priority in the LRU management queue. The address of the first pointer is an offset address based on the starting address of the shared memory. The address pointer of the data block is used to represent the offset address of the data block.

[0013] A manager is established for each physical disk. The manager stores multiple second pointers, which point to the first or last node of the LRU management queue, the first or last node of the block ID query queue, or represent the number of nodes in the LRU management queue, the number of nodes in the block ID query queue, or the ID of the physical disk.

[0014] Based on the management node and the manager, the queue is traversed in a skip list manner or queried using the data block ID.

[0015] Optionally, the first pointer may be an lru_prev pointer pointing to the previous node of the LRU management queue, or an lru_next pointer pointing to the next node of the LRU management queue, or a bid_prev pointer pointing to the previous node of the data block ID query queue, or a bid_next pointer pointing to the next node of the data block ID query queue, or an address pointer representing the offset address of a data block in the shared memory data area, or a blockid field representing the ID of a data block in the physical disk, or a priv field representing the priority in the LRU management queue;

[0016] The address pointer points to an address other than the starting address of the shared memory.

[0017] Optionally, the second pointer may be an lru_head pointer pointing to the first node of the LRU management queue, or an lru_tail pointer pointing to the last node of the LRU management queue, or a bid_head pointer pointing to the first node of the data block ID query queue, or a bid_tail pointer pointing to the last node of the data block ID query queue, or an lru_size field representing the number of nodes in the LRU management queue, or a bid_size field representing the number of nodes in the data block ID management queue, or a did field representing the ID of the physical disk.

[0018] Optionally, the step of traversing the data block ID query queue in a skip list manner according to the management node and the manager includes:

[0019] The manager obtains the bid_head pointer and uses the bid_head pointer to obtain the offset address of the first node in the data block ID query queue.

[0020] Calculate the sum of the starting address of the shared memory and the offset address of the first node of the data block ID query queue to obtain the address of the first node of the data block ID query queue based on the address space of the current process;

[0021] Based on the management node, the bid_next pointer and bid_prev pointer of the node in the data block ID query queue are obtained sequentially;

[0022] The offset addresses of adjacent nodes of the obtained node are obtained based on the bid_next pointer and bid_prev pointer of the obtained node. The sum of the obtained offset address and the starting address of the shared memory is calculated to obtain the address of the adjacent node of the first node of the data block ID query queue based on the address space of the current process.

[0023] Optionally, the LRU management queue uses a priority eviction algorithm including:

[0024] The manager obtains the lru_head pointer and uses the lru_head pointer to obtain the offset address of the first node in the LRU management queue.

[0025] Calculate the sum of the starting address of the shared memory and the offset address of the first node of the LRU management queue to obtain the address of the first node of the LRU management queue based on the address space of the current process;

[0026] Based on the management node, obtain the lru_next pointer and lru_prev pointer of the node in the LRU management queue in sequence;

[0027] The offset addresses of the adjacent nodes of the obtained node are obtained based on the lru_next pointer and lru_prev pointer. The sum of the obtained offset address and the starting address of the shared memory is calculated to obtain the address of the adjacent node of the first node of the LRU management queue based on the address space of the current process.

[0028] Optionally, when it is necessary to adjust the order of nodes in the LRU management queue, the step of traversing the data block ID query queue in a skip list manner according to the management node and the manager includes:

[0029] Using the data block ID, query the data block ID query queue containing the data block ID in a skip list manner;

[0030] The target node is determined by querying the queue using the data block ID;

[0031] Obtain the lru_next and lru_prev pointers of the target node;

[0032] After connecting the node pointed to by the lru_next pointer and the node pointed to by the lru_prev pointer, the target node is placed at the position of the first node in the LRU management queue, so as to adjust the order of the nodes in the LRU management queue.

[0033] Optionally, when it is necessary to access the data block corresponding to the target node, the step of querying the queue using the data block ID in a skip list manner according to the management node and the manager includes:

[0034] Obtain the data address pointer corresponding to the target node from the data block ID query queue;

[0035] Calculate the sum of the offset address represented by the data address pointer and the starting address of the shared memory, and access the data block corresponding to the target node based on the calculation result.

[0036] Optionally, when it is necessary to store data from the physical disk in the shared memory, the step of traversing or using the LRU management queue and the data block ID query queue in a skip list manner according to the management node and the manager includes:

[0037] At the end of the LRU management queue, select a node with a priv field value of 0 as a storage node. If there is no node with a priv field value of 0 at the end of the LRU management queue, determine whether the value of the priv field of the node at the end of the LRU management queue minus 1 is 0. If not, move the node at the end of the LRU management queue to the position of the first node in the LRU management queue and start a new round of elimination process until a node with a priv field value of 0 is found.

[0038] After the storage node is determined, it is removed from the LRU management queue, and the value of the lru_size field is decremented by 1. If the storage node also exists in the data block ID query queue, it is removed from the data block ID query queue, and the value of the bid_size field of the data block ID query queue is decremented by 1.

[0039] The starting address of the data block pointed to by the storage node is determined by the address pointer of the storage node;

[0040] The data in the physical disk is stored into the determined data block according to the starting address of the determined data block;

[0041] After the data is stored, the ID of the data block is placed in the blockid field of the storage node;

[0042] According to the ID of the data block, the storage node is placed in the data block ID query queue, and the value of the bid_size field is incremented by 1;

[0043] The storage node is placed into the first node position of the LRU management queue, and the value of the bid_size field is incremented by 1.

[0044] Optionally, when it is necessary to read target data in shared memory, the step of traversing or using the LRU management queue and the data block ID query queue in a skip list manner according to the management node and the manager includes:

[0045] Based on the target data, determine the data block ID in the physical disk corresponding to the target data, and read the offset of the data within the data block and the size of the target data;

[0046] Using the determined data block ID, retrieve the target node corresponding to the determined data block ID from the data block ID query queue;

[0047] The data block corresponding to the target node is obtained through the address pointer of the target node. The starting address of the data block is obtained by the sum of the starting address of the shared memory and the offset address represented by the address pointer.

[0048] The starting address of the target data in the data block is obtained by summing the starting address of the data block with the offset within the data block.

[0049] Read the target data based on the starting address of the target data.

[0050] A data block management system is applied to shared memory, the shared memory including a management area and a data area, the data block management system comprising:

[0051] A data block partitioning module is used to divide the data area into multiple data blocks, each data block having a preset size.

[0052] The address determination module is used to obtain the starting address of the shared memory in the current process after the shared memory is mapped to the current process, and to use the difference between the starting address of the data block and the starting address of the shared memory as the offset address of the data block.

[0053] The queue creation module is used to create a corresponding LRU management queue and a data block ID query queue for each physical disk. The data block ID query queue includes the IDs of data blocks arranged in an ordered manner, and the LRU management queue uses a priority eviction algorithm.

[0054] A management node module is used to establish a management node for the data block. The management node includes multiple first pointers, which point to adjacent nodes in the LRU management queue, adjacent nodes in the data block ID query queue, address pointers of data blocks in the data area, IDs of data blocks in the physical disk, or priority in the LRU management queue. The address of the first pointer is an offset address based on the starting address of the shared memory, and the address pointer of the data block is used to represent the offset address of the data block.

[0055] The manager module is used to establish a manager for each physical disk. The manager stores a plurality of second pointers, which point to the first or last node of the LRU management queue, or to the first or last node of the data block ID query queue, or represent the number of nodes in the LRU management queue, or represent the number of nodes in the data block ID query queue, or represent the ID of the physical disk.

[0056] The queue usage module is used to traverse the queue in a skip list manner or query the queue using the data block ID based on the management node and the manager.

[0057] A storage medium storing a program that, when triggered, executes the data block management method described in any of the preceding claims.

[0058] As can be seen from the above technical solutions, the embodiments of this application provide a data block management method, system, and storage medium. The data block management method effectively improves the data block lookup efficiency by using a data block ID query queue, an LRU management queue, and management nodes in combination. Furthermore, by traversing or using the LRU management queue and / or the data block ID query queue in a skip list manner, the performance problems of data movement caused by using arrays are avoided, thereby improving data processing efficiency. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating a data block management method provided in one embodiment of this application. Detailed Implementation

[0061] As described in the background section, existing methods for managing shared memory data blocks suffer from low efficiency in both data node lookup and data processing.

[0062] Specifically, existing technologies divide shared memory into a management area and a data area. The management area uses an LRU (Least Recently Used) queue for each disk, which is implemented using an array. This array approach inserts data in the middle of the array. During insertion, data after the currently needed data is first shifted to the right, and then the new data is inserted. This is inefficient because the data in the array is constantly moving. Furthermore, if there are many nodes, the amount of data being moved in the array increases. Excessive data movement negatively impacts performance. For example, if the ID being searched is at the end of the queue, the entire array is moved, and the node is moved to the front. If the array is large and operations are frequent, this reduces data processing efficiency.

[0063] In view of this, embodiments of this application provide a data block management method applied to shared memory, wherein the shared memory includes a management area and a data area, and the data block management method includes:

[0064] The data area is divided into multiple data blocks, and the size of each data block is a preset size.

[0065] After obtaining the shared memory mapped to the current process, the starting address of the shared memory in the current process is obtained, and the difference between the starting address of the data block and the starting address of the shared memory is used as the offset address of the data block;

[0066] For each physical disk, a corresponding LRU management queue and a data block ID query queue are established. The data block ID query queue includes the IDs of data blocks arranged in an ordered manner, and the LRU management queue uses a priority eviction algorithm.

[0067] A management node is established for the data block. The management node includes multiple first pointers. The first pointers point to adjacent nodes in the LRU management queue, or to adjacent nodes in the data block ID query queue, or are address pointers of data blocks in the data area, or represent the ID of data blocks in the physical disk, or represent the priority in the LRU management queue. The address of the first pointer is an offset address based on the starting address of the shared memory. The address pointer of the data block is used to represent the offset address of the data block.

[0068] A manager is established for each physical disk. The manager stores multiple second pointers, which point to the first or last node of the LRU management queue, the first or last node of the block ID query queue, or represent the number of nodes in the LRU management queue, the number of nodes in the block ID query queue, or the ID of the physical disk.

[0069] Based on the management node and the manager, the queue is traversed in a skip list manner or queried using the data block ID.

[0070] The data block management method effectively improves the data block lookup efficiency by using a data block ID query queue, an LRU management queue, and management nodes in combination. Furthermore, by traversing or using the LRU management queue and / or the data block ID query queue in a skip list manner, the performance issues of data movement caused by using arrays are avoided, thus improving data processing efficiency.

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] This application provides a method for managing data blocks, such as... Figure 1As shown, this method is applied to shared memory, which includes a management area and a data area. The data block management method includes:

[0073] S101: Divide the data area into multiple data blocks, each data block having a preset size;

[0074] For example, assuming the preset size is 4M (Mbyte), the data area will be divided into multiple 4M data blocks.

[0075] S102: After obtaining the shared memory mapped to the current process, the starting address of the shared memory in the current process is obtained, and the difference between the starting address of the data block and the starting address of the shared memory is used as the offset address of the data block.

[0076] In this embodiment, the starting address of each data block minus the starting address of the shared memory equals the offset address of that data block in the shared memory. We use this offset address as the address at which the data block is accessed in the shared memory. If a data block is offset by 10000 in the shared memory, then for all mapped shared memory addresses, the starting address is always 10000, and the space for that data block is within the address space [10000, 10000 + 4 × 1024 × 1024). Based on this, we can establish related data structures in the shared memory using this method. The address space of the next data block is calculated immediately after the previous address space; that is, if the previous address space is 10000, then the starting position of the next address space is 10000 + 4 × 1024 × 1024.

[0077] S103: Establish a corresponding LRU management queue and a data block ID query queue for each physical disk. The data block ID query queue includes the IDs of data blocks arranged in an ordered manner. The LRU management queue uses a priority eviction algorithm.

[0078] S104: Establish a management node for the data block. The management node includes multiple first pointers. The first pointers point to adjacent nodes in the LRU management queue, or to adjacent nodes in the data block ID query queue, or are address pointers of data blocks in the data area, or represent the ID of data blocks in the physical disk, or represent the priority in the LRU management queue. The address of the first pointer is an offset address based on the starting address of the shared memory. The address pointer of the data block is used to represent the offset address of the data block.

[0079] S105: Establish a manager for each physical disk, wherein the manager stores a plurality of second pointers, the second pointers pointing to the first or last node of the LRU management queue, or pointing to the first or last node of the data block ID query queue, or representing the number of nodes in the LRU management queue, or representing the number of nodes in the data block ID query queue, or representing the ID of the physical disk;

[0080] S106: Based on the management node and the manager, traverse the queue using a skip list or query the data block ID.

[0081] A skip list is a linked list with added forward pointers. It's a type of linked list, but faster than a regular linked list for searching data in ordered sets. Skip lists combine the functionality of arrays and linked lists, making them highly efficient. The full name of a skip list is a randomized data structure; essentially, it's an ordered linked list that allows for binary search. Skip lists add multi-level indexes to the original ordered linked list, enabling fast lookups. Skip lists not only improve search performance but also the performance of insertion and deletion operations.

[0082] We know that each node in a linked list includes pointers to `next` and `prev`. However, these `next` and `prev` pointers are not pointers to the actual virtual addresses of the current process's environment, but rather pointers to shared memory locations with an offset (offset address) based on the starting address of the current process (i.e., the first address of the shared memory).

[0083] In actual linked list operations, the starting address plus this offset is the address pointer of the actual node in the current process. The data of the node can be accessed through this actual address pointer.

[0084] Based on this, the data block management method provided in this embodiment effectively improves the data block lookup efficiency by using the data block ID query queue, LRU management queue and management node in combination; and by traversing or using the LRU management queue and / or the data block ID query queue in a skip list manner, the performance problem of data movement caused by using arrays is avoided, thus improving data processing efficiency.

[0085] Optionally, the first pointer may be an lru_prev pointer pointing to the previous node of the LRU management queue, or an lru_next pointer pointing to the next node of the LRU management queue, or a bid_prev pointer pointing to the previous node of the data block ID query queue, or a bid_next pointer pointing to the next node of the data block ID query queue, or an address pointer representing the offset address of a data block in the shared memory data area, or a blockid field representing the ID of a data block in the physical disk, or a priv field representing the priority in the LRU management queue;

[0086] The address pointer points to an address other than the starting address of the shared memory.

[0087] Optionally, the second pointer may be an lru_head pointer pointing to the first node of the LRU management queue, or an lru_tail pointer pointing to the last node of the LRU management queue, or a bid_head pointer pointing to the first node of the data block ID query queue, or a bid_tail pointer pointing to the last node of the data block ID query queue, or an lru_size field representing the number of nodes in the LRU management queue, or a bid_size field representing the number of nodes in the data block ID management queue, or a did field representing the ID of the physical disk.

[0088] Refer to Tables 1 and 2 below. Table 1 shows the data structure fields, the names of the first pointers, and their corresponding descriptions; Table 2 shows the manager field information and their descriptions.

[0089] Table 1

[0090] Data structure fields illustrate lru_prev Points to the previous node in the LRU management queue lru_next Points to the next node in the LRU management queue. bid_prev The previous node in the disk block ID lookup queue bid_next The next node in the disk's data block ID lookup queue block_ptr Address pointers of data blocks in shared memory data regions blockid The corresponding data block ID on the hard drive priv LRU management queue priority

[0091] The lru_prev pointer points to the previous node in the LRU management queue. The value of the first node is 0. The address of the lru_prev pointer is based on the offset address of the starting address of the shared memory.

[0092] The lru_next pointer points to the next node in the LRU management queue. The value of the last node is 0. The address of the lru_next pointer is based on the offset address of the starting address of the shared memory.

[0093] The bid_prev pointer points to the previous node in the disk's data block ID lookup queue. The first node has a value of 0. The address of the bid_prev pointer is based on the offset address of the shared memory's starting address.

[0094] The bid_next pointer points to the next node in the disk's data block ID lookup queue. The last node has a value of 0. The address of the bid_next pointer is based on the offset address of the shared memory's starting address.

[0095] For the address pointer (block_ptr), it represents the address pointer of the data block in the data region of the shared memory. The address of the address pointer is based on the offset address of the starting address of the shared memory.

[0096] For the blockid field, it corresponds to the data block ID on the hard drive. This value is initialized to -1. When the value is greater than or equal to 0, it indicates that it points to the data block ID corresponding to the disk. In the data block ID query queue, it is also based on this value as an ordered set.

[0097] The `priv` field represents the priority of the LRU management queue. This priority serves as the eviction criterion. When `priv` is 0, it means the node can be evicted from the queue and is always at the tail of the LRU management queue. Each time a user uses the node, this value increments by 1, and the node is moved to the head of the LRU management queue. If the value is greater than 0 at the tail, it is first decremented by 1. If the value is still greater than 0, the node is moved to the head of the queue for a new round of eviction. If the value is 0, it means the node has already been evicted and can be removed from the LRU management queue, and is also removed from the data block ID query queue.

[0098] Table 2

[0099]

[0100] The following describes the specific possible scenarios of "traversing the queue in a skip list manner or using the data block ID to query the queue based on the management node and the manager".

[0101] Optionally, the step of traversing the data block ID query queue in a skip list manner according to the management node and the manager includes:

[0102] The manager obtains the bid_head pointer and uses the bid_head pointer to obtain the offset address of the first node in the data block ID query queue.

[0103] Calculate the sum of the starting address of the shared memory and the offset address of the first node of the data block ID query queue to obtain the address of the first node of the data block ID query queue based on the address space of the current process;

[0104] Based on the management node, the bid_next pointer and bid_prev pointer of the node in the data block ID query queue are obtained sequentially;

[0105] The offset addresses of adjacent nodes of the obtained node are obtained based on the bid_next pointer and bid_prev pointer of the obtained node. The sum of the obtained offset address and the starting address of the shared memory is calculated to obtain the address of the adjacent node of the first node of the data block ID query queue based on the address space of the current process.

[0106] Optionally, the LRU management queue uses a priority eviction algorithm including:

[0107] The manager obtains the lru_head pointer and uses the lru_head pointer to obtain the offset address of the first node in the LRU management queue.

[0108] Calculate the sum of the starting address of the shared memory and the offset address of the first node of the LRU management queue to obtain the address of the first node of the LRU management queue based on the address space of the current process, and access the first node of the data block ID query queue based on the address of the first node of the data block ID query queue based on the address space of the current process.

[0109] Based on the management node, obtain the lru_next pointer and lru_prev pointer of the node in the LRU management queue in sequence;

[0110] The offset addresses of the adjacent nodes of the obtained node are obtained based on the lru_next pointer and lru_prev pointer. The sum of the obtained offset address and the starting address of the shared memory is calculated to obtain the address of the adjacent node of the first node of the LRU management queue based on the address space of the current process. Based on the address of the adjacent node of the first node of the data block ID query queue based on the address space of the current process, the adjacent node of the first node of the data block ID query queue is accessed.

[0111] Optionally, when it is necessary to adjust the order of nodes in the LRU management queue, the step of traversing the data block ID query queue in a skip list manner according to the management node and the manager includes:

[0112] Using the data block ID, query the data block ID query queue containing the data block ID in a skip list manner;

[0113] The target node is determined by querying the queue using the data block ID;

[0114] Obtain the lru_next and lru_prev pointers of the target node;

[0115] After connecting the node pointed to by the lru_next pointer and the node pointed to by the lru_prev pointer, the target node is placed at the position of the first node in the LRU management queue, so as to adjust the order of the nodes in the LRU management queue.

[0116] Optionally, when it is necessary to access the data block corresponding to the target node, the step of querying the queue using the data block ID in a skip list manner according to the management node and the manager includes:

[0117] Obtain the data address pointer corresponding to the target node from the data block ID query queue;

[0118] Calculate the sum of the offset address represented by the data address pointer and the starting address of the shared memory, and access the data block corresponding to the target node based on the calculation result.

[0119] Optionally, when it is necessary to store data from the physical disk in the shared memory, the step of traversing or using the LRU management queue and the data block ID query queue in a skip list manner according to the management node and the manager includes:

[0120] At the end of the LRU management queue, select a node with a priv field value of 0 as a storage node. If there is no node with a priv field value of 0 at the end of the LRU management queue, determine whether the value of the priv field of the node at the end of the LRU management queue minus 1 is 0. If not, move the node at the end of the LRU management queue to the position of the first node in the LRU management queue and start a new round of elimination process until a node with a priv field value of 0 is found.

[0121] After the storage node is determined, it is removed from the LRU management queue, and the value of the lru_size field is decremented by 1. If the storage node also exists in the data block ID query queue, it is removed from the data block ID query queue, and the value of the bid_size field of the data block ID query queue is decremented by 1.

[0122] The starting address of the data block pointed to by the storage node is determined by the address pointer of the storage node;

[0123] The data in the physical disk is stored into the determined data block according to the starting address of the determined data block;

[0124] After the data is stored, the ID of the data block is placed in the blockid field of the storage node;

[0125] According to the ID of the data block, the storage node is placed in the data block ID query queue, and the value of the bid_size field is incremented by 1;

[0126] The storage node is placed into the first node position of the LRU management queue, and the value of the bid_size field is incremented by 1.

[0127] Optionally, when it is necessary to read target data in shared memory, the step of traversing or using the LRU management queue and the data block ID query queue in a skip list manner according to the management node and the manager includes:

[0128] Based on the target data, determine the data block ID in the physical disk corresponding to the target data, and read the offset of the data within the data block and the size of the target data;

[0129] Using the determined data block ID, retrieve the target node corresponding to the determined data block ID from the data block ID query queue;

[0130] The data block corresponding to the target node is obtained through the address pointer of the target node. The starting address of the data block is obtained by the sum of the starting address of the shared memory and the offset address represented by the address pointer.

[0131] The starting address of the target data in the data block is obtained by summing the starting address of the data block with the offset within the data block.

[0132] Read the target data based on the starting address of the target data.

[0133] The data block management system provided in the embodiments of this application will be described below. The data block management system described below can be referred to in correspondence with the data block management method described above.

[0134] Accordingly, embodiments of this application provide a data block management system applied to shared memory, wherein the shared memory includes a management area and a data area, and the data block management system includes:

[0135] A data block partitioning module is used to divide the data area into multiple data blocks, each data block having a preset size.

[0136] The address determination module is used to obtain the starting address of the shared memory in the current process after the shared memory is mapped to the current process, and to use the difference between the starting address of the data block and the starting address of the shared memory as the offset address of the data block.

[0137] The queue creation module is used to create a corresponding LRU management queue and a data block ID query queue for each physical disk. The data block ID query queue includes the IDs of data blocks arranged in an ordered manner, and the LRU management queue uses a priority eviction algorithm.

[0138] A management node module is used to establish a management node for the data block. The management node includes multiple first pointers, which point to adjacent nodes in the LRU management queue, adjacent nodes in the data block ID query queue, address pointers of data blocks in the data area, IDs of data blocks in the physical disk, or priority in the LRU management queue. The address of the first pointer is an offset address based on the starting address of the shared memory, and the address pointer of the data block is used to represent the offset address of the data block.

[0139] The manager module is used to establish a manager for each physical disk. The manager stores a plurality of second pointers, which point to the first or last node of the LRU management queue, or to the first or last node of the data block ID query queue, or represent the number of nodes in the LRU management queue, or represent the number of nodes in the data block ID query queue, or represent the ID of the physical disk.

[0140] The queue usage module is used to traverse the queue in a skip list manner or query the queue using the data block ID based on the management node and the manager.

[0141] Accordingly, this application also provides a storage medium storing a program, which, when triggered, executes the data block management method described in any of the above embodiments.

[0142] In summary, the embodiments of this application provide a data block management method, system, and storage medium. The data block management method effectively improves the data block lookup efficiency by using a data block ID query queue, an LRU management queue, and management nodes in combination. Furthermore, by traversing or using the LRU management queue and / or the data block ID query queue in a skip list manner, the performance problems of data movement caused by using arrays are avoided, thus improving data processing efficiency.

[0143] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0144] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for managing data blocks, characterized in that, Applied to shared memory, the shared memory includes a management area and a data area, and the data block management method includes: The data area is divided into multiple data blocks, and the size of each data block is a preset size. After obtaining the shared memory mapped to the current process, the starting address of the shared memory in the current process is obtained, and the difference between the starting address of the data block and the starting address of the shared memory is used as the offset address of the data block; For each physical disk, a corresponding LRU management queue and a data block ID query queue are established. The data block ID query queue includes the IDs of data blocks arranged in an ordered manner, and the LRU management queue uses a priority eviction algorithm. A management node is established for the data block. The management node includes multiple first pointers. The first pointers point to adjacent nodes in the LRU management queue, or to adjacent nodes in the data block ID query queue, or are address pointers of data blocks in the data area, or represent the ID of data blocks in the physical disk, or represent the priority in the LRU management queue. The address of the first pointer is an offset address based on the starting address of the shared memory. The address pointer of the data block is used to represent the offset address of the data block. A manager is established for each physical disk. The manager stores multiple second pointers, which point to the first or last node of the LRU management queue, the first or last node of the block ID query queue, or represent the number of nodes in the LRU management queue, the number of nodes in the block ID query queue, or the ID of the physical disk. Based on the management node and the manager, the queue is traversed in a skip list manner or queried using the data block ID.

2. The method according to claim 1, characterized in that, The first pointer is either an lru_prev pointer pointing to the previous node of the LRU management queue, or an lru_next pointer pointing to the next node of the LRU management queue, or a bid_prev pointer pointing to the previous node of the data block ID query queue, or a bid_next pointer pointing to the next node of the data block ID query queue, or an address pointer representing the offset address of a data block in the shared memory data area, or a blockid field representing the ID of a data block in the physical disk, or a priv field representing the priority in the LRU management queue; The address pointer points to an address other than the starting address of the shared memory.

3. The method according to claim 2, characterized in that, The second pointer is either an lru_head pointer pointing to the first node of the LRU management queue, or an lru_tail pointer pointing to the last node of the LRU management queue, or a bid_head pointer pointing to the first node of the data block ID query queue, or a bid_tail pointer pointing to the last node of the data block ID query queue, or an lru_size field representing the number of nodes in the LRU management queue, or a bid_size field representing the number of nodes in the data block ID query queue, or a did field representing the ID of the physical disk.

4. The method according to claim 3, characterized in that, The step of traversing the data block ID query queue in a skip list manner based on the management node and the manager includes: The manager obtains the bid_head pointer and uses the bid_head pointer to obtain the offset address of the first node in the data block ID query queue. Calculate the sum of the starting address of the shared memory and the offset address of the first node of the data block ID query queue to obtain the address of the first node of the data block ID query queue based on the address space of the current process; Based on the management node, the bid_next pointer and bid_prev pointer of the node in the data block ID query queue are obtained sequentially; The offset addresses of adjacent nodes of the obtained node are obtained based on the bid_next pointer and bid_prev pointer of the obtained node. The sum of the obtained offset address and the starting address of the shared memory is calculated to obtain the address of the adjacent node of the first node of the data block ID query queue based on the address space of the current process.

5. The method according to claim 3, characterized in that, The LRU management queue uses a priority eviction algorithm including: The manager obtains the lru_head pointer and uses the lru_head pointer to obtain the offset address of the first node in the LRU management queue. Calculate the sum of the starting address of the shared memory and the offset address of the first node of the LRU management queue to obtain the address of the first node of the LRU management queue based on the address space of the current process; Based on the management node, obtain the lru_next pointer and lru_prev pointer of the node in the LRU management queue in sequence; The offset addresses of the adjacent nodes of the obtained node are obtained based on the lru_next pointer and lru_prev pointer. The sum of the obtained offset address and the starting address of the shared memory is calculated to obtain the address of the adjacent node of the first node of the LRU management queue based on the address space of the current process.

6. The method according to claim 3, characterized in that, When it is necessary to adjust the order of nodes in the LRU management queue, the step of traversing the data block ID query queue in a skip list manner based on the management node and the manager includes: Using the data block ID, query the data block ID query queue containing the data block ID in a skip list manner; The target node is determined by querying the queue using the data block ID; Obtain the lru_next and lru_prev pointers of the target node; After connecting the node pointed to by the lru_next pointer and the node pointed to by the lru_prev pointer, the target node is placed at the position of the first node in the LRU management queue, so as to adjust the order of the nodes in the LRU management queue.

7. The method according to claim 3, characterized in that, When it is necessary to access the data block corresponding to the target node, the step of querying the queue using the data block ID in a skip list manner according to the management node and the manager includes: Obtain the data address pointer corresponding to the target node from the data block ID query queue; Calculate the sum of the offset address represented by the data address pointer and the starting address of the shared memory, and access the data block corresponding to the target node based on the calculation result.

8. The method according to claim 3, characterized in that, When data from the physical disk needs to be stored in the shared memory, the step of traversing or using the LRU management queue and the data block ID query queue in a skip list manner according to the management node and the manager includes: At the end of the LRU management queue, select a node with a priv field value of 0 as a storage node. If there is no node with a priv field value of 0 at the end of the LRU management queue, determine whether the value of the priv field of the node at the end of the LRU management queue minus 1 is 0. If not, move the node at the end of the LRU management queue to the position of the first node in the LRU management queue and start a new round of elimination process until a node with a priv field value of 0 is found. After the storage node is determined, it is removed from the LRU management queue, and the value of the lru_size field is decremented by 1. If the storage node also exists in the data block ID query queue, it is removed from the data block ID query queue, and the value of the bid_size field of the data block ID query queue is decremented by 1. The starting address of the data block pointed to by the storage node is determined by the address pointer of the storage node; The data in the physical disk is stored into the determined data block according to the starting address of the determined data block; After the data is stored, the ID of the data block is placed in the blockid field of the storage node; According to the ID of the data block, the storage node is placed in the data block ID query queue, and the value of the bid_size field is incremented by 1; The storage node is placed into the first node position of the LRU management queue, and the value of the bid_size field is incremented by 1.

9. The method according to claim 3, characterized in that, When it is necessary to read target data in shared memory, the step of traversing or using the LRU management queue and the data block ID query queue in a skip list manner according to the management node and the manager includes: Based on the target data, determine the data block ID in the physical disk corresponding to the target data, and read the offset of the data within the data block and the size of the target data; Using the determined data block ID, retrieve the target node corresponding to the determined data block ID from the data block ID query queue; The data block corresponding to the target node is obtained through the address pointer of the target node. The starting address of the data block is obtained by the sum of the starting address of the shared memory and the offset address represented by the address pointer. The starting address of the target data in the data block is obtained by summing the starting address of the data block with the offset within the data block. Read the target data based on the starting address of the target data.

10. A data block management system, characterized in that, Applied to shared memory, the shared memory includes a management area and a data area, and the management system for the data blocks includes: A data block partitioning module is used to divide the data area into multiple data blocks, each data block having a preset size. The address determination module is used to obtain the starting address of the shared memory in the current process after the shared memory is mapped to the current process, and to use the difference between the starting address of the data block and the starting address of the shared memory as the offset address of the data block. The queue creation module is used to create a corresponding LRU management queue and a data block ID query queue for each physical disk. The data block ID query queue includes the IDs of data blocks arranged in an ordered manner, and the LRU management queue uses a priority eviction algorithm. A management node module is used to establish a management node for the data block. The management node includes multiple first pointers, which point to adjacent nodes in the LRU management queue, adjacent nodes in the data block ID query queue, address pointers of data blocks in the data area, IDs of data blocks in the physical disk, or priority in the LRU management queue. The address of the first pointer is an offset address based on the starting address of the shared memory, and the address pointer of the data block is used to represent the offset address of the data block. The manager module is used to establish a manager for each physical disk. The manager stores a plurality of second pointers, which point to the first or last node of the LRU management queue, or to the first or last node of the data block ID query queue, or represent the number of nodes in the LRU management queue, or represent the number of nodes in the data block ID query queue, or represent the ID of the physical disk. The queue usage module is used to traverse the queue in a skip list manner or query the queue using the data block ID based on the management node and the manager.

11. A storage medium, characterized in that, The storage medium stores a program that, when triggered, executes the data block management method according to any one of claims 1-9.

Images