A persistence method for graph data index based on parallelism

Abstract

The present invention discloses a method for persisting graph data index based on parallelism, comprising the following steps: (1) detecting whether there is an index file at a specified position of a hard disk, and recreating an empty index file if there is no index file; if there is an index file, reading the index file and reconstructing the index; (2) calculating the size of the continuous available memory space and the size of the space occupied by the index structure, and initializing the index structure; (3) using a memory mapping method to open up a continuous memory, initialize the index header, and map the existing index file into the memory; (4) applying for multiple threads to a computer system, and completing the persistence of the index structure through multi-threaded processing. The present invention implements a parallel persistence method for graph data query indexes, improves the efficiency of index persistence, and is applicable to a variety of multi-way query trees, with good scalability.

Description

Technical Field

[0001] The present invention relates to a persistence method for graph data indexes, and in particular to a persistence method for graph data indexes based on parallelism. Background Art

[0002] With the development and maturity of database technology, as well as the development of Internet communication applications and their derivative software, the amount of data storage is increasing. Emerging applications such as social network analysis, Web semantic analysis, bioinformatics network analysis, and traffic navigation are also growing rapidly. Large graph data with large scale, complex internal structure, and diverse query requirements have appeared in different fields. When querying these graph data, direct global search will result in a lot of time overhead. By building a suitable index structure to store relevant information of the graph data, the efficiency of querying the graph data can be improved through the relevant auxiliary information in the index.

[0003] At present, many technologies for graph data indexing have been proposed, and there are also efficient index structures in various scenarios. This also leads to different main functions of index structures. Some indexes need to calculate strongly connected components in the preprocessing stage, while others record the relationship between vertices, which leads to some problems. For example, when the hardware devices of the host are too different, the host with general memory capacity will not be able to accommodate the index structure that occupies a large space. Due to the problem of memory space, the index technology has the characteristic of limited space. When encountering unavoidable natural disasters such as power outages, the instability of memory will cause the index to be lost, making the index in memory unstable. At the same time, when the computer is restarted and the software is restarted, the memory will be automatically cleaned and recycled, and the index structure in the memory will be lost together, making the index in the memory non-reusable.

[0004] The capacity of memory is often far behind that of external storage devices such as hard disks. Therefore, when the index structure occupies a large amount of space, placing the index structure in external storage is an effective solution to ensure that the index structure is not limited by space. Even if the data scale increases, storing part of the index in external storage can alleviate the pressure on memory. Due to the volatility of memory, data is easily lost due to power outages in the event of inevitable natural disasters. By making the data structure persistent, property losses will not be caused by the volatility of memory when encountering uncontrollable situations. In the subsequent reconstruction of the index structure, there is no need to re-establish the index. The index structure can be directly read from the existing index file, saving the time of building the index structure while ensuring the integrity of the data structure.

[0005] B-tree is a common index structure that improves data search speed based on the binary search strategy. It can fully utilize the advantages brought by the principle of disk block storage. It is often used as a database query index together with the similar B+ tree. At the same time, R-tree, as a tree data structure, is a multi-way search tree like the above index. There are many similarities in the search process. Therefore, if one index is successfully persisted in parallel, the other indexes can also be persisted in parallel. With the development of computer hardware technology, more and more enterprise-level servers choose multi-CPU computers. The existing main memory indexing technology is a single-core serial technology, which cannot meet the needs of the era of parallel computing. For multi-core high-performance servers, this technology will cause a huge waste of computing resources. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a method for persisting graph data indexes based on parallelism that can achieve main memory index persistence and is applicable to multi-core parallel systems.

[0007] Technical solution: The method for persisting graph data index of the present invention comprises the following steps:

[0008] (1) Detect whether there is an index file at the specified location on the hard disk. If no index file exists, create an empty index file. If an index file exists, read the index file and reconstruct the index.

[0009] (2) Calculate the size of the continuous available memory space and the size of the space occupied by the index structure, and initialize the index structure;

[0010] (3) Use the memory mapping method to open up a continuous memory, initialize the index header, and map the existing index file into the memory;

[0011] (4) Apply for multiple threads from the computer system and complete the persistence of the index structure through multi-threaded processing.

[0012] Furthermore, in step (1), the steps for implementing index reconstruction are as follows:

[0013] (11) Calculate the size of the continuous available memory space and use the memory mapping method to map the index file to the continuous memory area;

[0014] (12) Read the index header of the index file in the mapped memory, obtain information related to the index structure, including: index size, offset of the leaf node linked list head and the leaf node linked list tail, and root node information, and initialize the index structure;

[0015] (13) Reconstruct the index based on the content of the index file.

[0016] Furthermore, in step (13), reconstruction is performed according to the node types stored in the index file:

[0017] (131) Leaf node: read the offsets of the two child pointers of the leaf node and the key value of the leaf node, and add the offset to the index header address to complete the reconstruction of the leaf node;

[0018] (132) Intermediate node: Read the offsets of the two child pointers of the intermediate node and the key value information of the intermediate node, and add the offset to the index header address to complete the reconstruction of the leaf node.

[0019] Furthermore, in step (2), the step of calculating the size of the space occupied by the index structure is as follows:

[0020] (21) Calculate the leaf node space occupancy as S(Y);

[0021] (22) Calculate the space occupied by the intermediate node as S(X);

[0022] (23) Calculate the memory space occupied by the index structure as S = S(Y) + S(X);

[0023] (24) To ensure that the index still has space to accommodate new nodes when it is subsequently updated, the index size F that should be recorded in the index header is k times the memory space occupied by the actual index structure, where k>1, then F=k*S.

[0024] Furthermore, in step (4), the multi-thread processing step includes:

[0025] (41) Record the starting address t of the index structure;

[0026] (42) Apply for n threads and allocate a data buffer m for each thread n and record buffer p n , where data buffer area m n Used to store nodes to be persisted and record cache area p n The record information structure of the persisted node is used to store: the offset of the node's starting address relative to the index structure's starting address t, the offset of the ending address relative to the index structure's starting address t, the node type, the key value, and the offset of the child pointer relative to the index structure's starting address t;

[0027] (43) Load the root node and its child nodes of the index structure into the data cache m of each thread respectively n middle;

[0028] (44) Each thread maintains its own data buffer and record buffer, traverses the data in each thread buffer, records the key value of each node, the start address, the end address and the offset of the child pointer relative to the start address t of the index structure, stores all the information in the record buffer, traverses the child nodes of the node, and loads the child nodes into the specified data buffer according to the load balancing strategy. At the same time, the data in the record buffer is saved and written into the shared memory according to the set K value; wherein K represents a globally set variable and serves as the storage threshold of the buffer;

[0029] (45) The computer system monitors the workload of each thread; each thread traverses a node, and each thread feeds back its own workload to M, which records the number of threads to work, where M is used to record the number of nodes currently traversed by each thread and changes in real time; in step (44), whenever a new child node is found, the thread with the smallest number of threads to work M will be found, and the new task will be loaded into the corresponding data buffer;

[0030] (46) Repeat steps (44) and (45) until all nodes have been visited and all information has been successfully recorded in the shared memory.

[0031] Compared with the prior art, the present invention has the following significant effects: 1. The present invention considers the efficiency of index structure persistence from the hardware aspect, adopts parallel technology, distributes the nodes in the index to different threads according to the load balancing strategy, and multiple threads operate simultaneously, making full use of hardware resources, greatly improving the efficiency of index structure persistence, and at the same time persists the index structure from volatile memory to main memory, avoiding property loss due to uncontrollable factors, and is suitable for various tree indexes such as multi-way search trees; 2. The present invention supports multi-user concurrency. When performing a parallel index persistence method, other users are allowed to query, add, delete, and other operations on the graph. Compared with traditional persistence methods, it ensures that no thread resources are wasted, and when writing to the memory mapping space, it reduces the phenomenon of resource waste caused by simultaneous writing while one party is busy, occupies less resources, and is more user-friendly. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0033] Figure 2 It is a schematic diagram of the index structure of the present invention;

[0034] Figure 3 It is a schematic diagram of the persistence process of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] In order to solve the problem that the persistence of traditional graph data query index structure cannot take advantage of multi-core servers, resulting in waste of resources, the present invention proposes a persistence method applicable to multi-way search trees represented by B-trees and B+ trees, which has good scalability and extensibility and is applicable to tree-like index structures. At the same time, a parallel index persistence method is proposed, and its load balancing method ensures that no thread resources are wasted, and when writing to the memory mapping space, the phenomenon of waste of resources caused by simultaneous writing while one party is busy is reduced.

[0037] The implementation method will be described in conjunction with a transitive closure solution system based on a B+ tree.

[0038] like Figure 1 The figure shows the overall process diagram of the present invention. The implementation steps of the persistence method of the present invention are as follows:

[0039] Step 1: Check whether there is an index file at the specified location of the hard disk. If there is no index file, create an empty index file. If there is an index file, read the index file and reconstruct the index. The steps for reconstructing the index structure are:

[0040] Step 11, calculate the size of the continuous available memory space, and use memory mapping technology to map the index file to a continuous memory area.

[0041] Step 12, read the index header of the index file in the mapped memory, obtain information related to the index structure, such as: index size, offset of the leaf node linked list head and the leaf node linked list tail, root node and other information, and initialize the B+ tree index structure.

[0042] Step 13: Reconstruct the index according to the content of the index file. Reconstruct according to the node type stored in the index file:

[0043] Step 131, leaf node: read the offsets of the two child pointers of the leaf node and the key value of the leaf node, that is, the stored related information and offset, and add the offset to the index header address to complete the reconstruction of the leaf node.

[0044] Step 132, intermediate node: read the offsets of the two child pointers of the intermediate node and the key value information of the intermediate node, and add the offsets to the index header address to complete the reconstruction of the leaf node.

[0045] Step 2, calculate the size of the continuous available memory space and the size of the space occupied by the index structure, and initialize the index structure;

[0046] The steps to calculate the space occupied by the index structure are as follows:

[0047] Step 21, let the key value size be l, the vertex be a, and the node size be c, then the number of keys b that a node can accommodate is Since leaf nodes and key values ​​correspond one to one, the space occupied by leaf nodes is:

[0048]

[0049] Step 22: In step 21, b-2 is the number of key values ​​contained in a leaf node, and the space occupied by the intermediate node is:

[0050]

[0051] Step 23: According to the formulas in steps 21 and 22, the memory space occupied by the index structure is:

[0052]

[0053] Step 24: To ensure that the index still has space to accommodate new nodes when it is subsequently updated, the index size recorded in the index header should be k (k>1) times the memory space occupied by the actual index structure, that is,

[0054] F=k*S (4)

[0055] Step 3: Use memory mapping technology to open up a continuous memory, initialize the index header, and map the existing index file into the memory.

[0056] Step 4: Apply for multiple threads from the computer system and use multi-threaded processing to complete the persistence of the index structure. The steps for recording the index structure offset are:

[0057] Step 41, record the starting address t of the index structure;

[0058] Step 42: Apply for n threads and allocate a data buffer m to each thread. n and record buffer p n , where the data buffer is used to store the nodes to be persisted, and the record buffer is used to store the record information structure of the persisted nodes, and the record information structure is used to store the offset of the node's starting address relative to the index structure's starting address t, the offset of the ending address relative to the index structure's starting address t, the node type, the key value, the offset of the child pointer relative to the index structure's starting address t, and other information;

[0059] Step 43, load the root node and its child nodes of the index structure into the data buffer m of each thread respectively n middle;

[0060] Step 44, each thread maintains its own data buffer and record buffer, traverses the data in each thread buffer, records the key value of each node, the starting address, the ending address, the child pointer and other information relative to the starting address t of the index structure, and after storing all the information in the record buffer, traverses the child nodes of the node, and loads the child nodes into the designated data buffer according to the load balancing strategy. At the same time, the data stored in the record buffer will be written into the shared memory according to the set K value; wherein K represents a globally set variable and serves as the storage threshold of the buffer;

[0061] Step 45, the computer system monitors the workload of each thread. Each thread traverses a node and provides positive feedback to M, which records the number of threads to work. Whenever a new child node is found in step 44, the thread with the smallest number of threads to work M will be found and the new task will be loaded into the corresponding data buffer.

[0062] Step 46, repeat steps 44 and 45 until all nodes have been visited and all information has been successfully recorded in the shared memory.

[0063] Taking the B+ tree as an example, the implementation process of the present invention is as follows:

[0064] like Figure 2 , the steps to implement the pre-calculated index structure space size are as follows:

[0065] Step 1: Let the key value size be l, the vertex be a, and the node size be c. Then the number of keys b that a node can accommodate is Since leaf nodes and key values ​​correspond one to one, the space occupied by leaf nodes is:

[0066]

[0067] Step 2: According to the basic knowledge of B+ tree, the space occupied by the intermediate node is:

[0068]

[0069] Step 3: To ensure that the index still has space to accommodate new nodes when it is subsequently updated, the index size recorded in the index header should be k (k>1) times the memory space occupied by the actual index structure, that is,

[0070]

[0071] like Figure 3 , using multi-threaded processing, the steps to implement the index structure persistence method are as follows:

[0072] Step 1: Record the index structure starting address t of the B+ tree index structure established in the memory, record the overall size of the index structure, the starting address and ending address of the leaf node linked list, the number of vertices and other information, store this information in a structure, and write it into the mapped memory as an index file header.

[0073] Step 2: The computer system applies for n threads according to the pre-set global variables, and allocates a data buffer m for storing vertices and a record buffer p for storing traversed node information to each thread.

[0074] Step 3: Store the root node and child nodes of the index structure in different data caches in order. If the number of nodes is greater than the number of threads, that is, there are surplus nodes in the first task allocation, these nodes are arbitrarily allocated to each thread.

[0075] Step 4: Each thread maintains its own data buffer and record buffer, traverses the data in each thread buffer, records the key value, starting address, ending address, child pointer and other information of each node relative to the starting address t of the index structure, and stores all information in the record buffer. Then, the child nodes of the node are traversed, and the child nodes are loaded into the specified data buffer according to the load balancing strategy. At the same time, the data stored in the record buffer will be written into the shared memory according to the set K value; K represents a globally set variable, which serves as the storage threshold of the cache. Once this threshold is reached, the information stored in the cache will be written into the specified location.

[0076] Step 5: The computer system monitors the workload of each thread. When each thread traverses a node, it will feed back its workload to M, which records the number of threads to work. Whenever it traverses a new child node, it will find the thread with the smallest number of threads to work, and load the new task into the corresponding data buffer. M is used to record the number of nodes currently traversed by each thread, which changes in real time to achieve load balancing.

[0077] Step 6: Repeat steps 4 and 5 until all nodes have been visited and all information has been successfully recorded in the shared memory.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for persisting graph data based on parallelism, characterized in that: The following steps are involved: (1) Check whether there is an index file at the specified location on the hard disk. If no index file exists, create an empty index file. If the index file exists, read the index file and reconstruct the index; (2) Calculate the size of the continuous available memory space and the size of the space occupied by the index structure, and initialize the index structure; (3) Use the memory mapping method to open up a continuous memory, initialize the index header, and map the existing index file into the memory; (4) Apply for multiple threads from the computer system and complete the persistence of the index structure through multi-threaded processing; In step (1), the steps to implement index reconstruction are as follows: (11) Calculate the size of the continuous available memory space and use the memory mapping method to map the index file to the continuous memory area; (12) Read the index header of the index file in the mapped memory, obtain information related to the index structure, including: index size, offset of the leaf node linked list head and the leaf node linked list tail, and root node information, and initialize the index structure; (13) Reconstruct the index according to the content of the index file; wherein, the reconstruction is performed according to the node type stored in the index file: (131) Leaf node: read the offsets of the two child pointers of the leaf node and the key value of the leaf node, and add the offset to the index header address to complete the reconstruction of the leaf node; (132) Intermediate node: read the offsets of the two child pointers of the intermediate node and the key value information of the intermediate node, and add the offset to the index header address to complete the reconstruction of the leaf node; In step (4), the multi-thread processing steps include: (41) Record the starting address t of the index structure; (42) Apply for n threads and allocate a data buffer m for each thread n and record buffer p n , where data buffer area m n Used to store nodes to be persisted and record cache area p n The record information structure of the persisted node is used to store: the offset of the node's starting address relative to the index structure's starting address t, the offset of the ending address relative to the index structure's starting address t, the node type, the key value, and the offset of the child pointer relative to the index structure's starting address t; (43) Load the root node and its child nodes of the index structure into the data cache m of each thread respectively n middle; (44) Each thread maintains its own data buffer and record buffer, traverses the data in each thread buffer, records the key value of each node, the start address, the end address and the offset of the child pointer relative to the start address t of the index structure, stores all the information in the record buffer, traverses the child nodes of the node, and loads the child nodes into the specified data buffer according to the load balancing strategy. At the same time, the data in the record buffer is saved and written into the shared memory according to the set K value; wherein K represents a globally set variable and serves as the storage threshold of the buffer; (45) The computer system monitors the workload of each thread; each thread traverses a node, and each thread feeds back its own workload to M, which records the number of threads to work, where M is used to record the number of nodes currently traversed by each thread and changes in real time; in step (44), whenever a new child node is found, the thread with the smallest number of threads to work M will be found, and the new task will be loaded into the corresponding data buffer; (46) Repeat steps (44) and (45) until all nodes have been visited and all information has been successfully recorded in the shared memory.

2. The method for persisting graph data based on parallel index according to claim 1, characterized in that: In step (2), the steps for calculating the size of the space occupied by the index structure are as follows: (21) Calculate the leaf node space occupancy as S(Y); (22) Calculate the space occupied by the intermediate node as S(X); (23) Calculate the memory space occupied by the index structure as S = S(Y) + S(X); (24) To ensure that the index still has space to accommodate new nodes when it is subsequently updated, the index size F that should be recorded in the index header is k times the memory space occupied by the actual index structure, where k>1, then F=k*S.

Images