A high-speed data query method and system based on continuous memory

Abstract

The present invention relates to a high-speed data query method based on continuous memory. By combining index nodes and index offsets, a globally balanced binary tree implemented by indexes is constructed on a logically continuous memory address structure, breaking the limitations brought by the current method of using pointers for data connection and achieving the purpose of high-speed sharing between processes. The relationships between all nodes in the present invention are represented by indexes and offsets, perfectly solving the problem of memory sharing caused by different memory addresses between processes; a doubly linked list algorithm implemented by indexes is adopted as the free node management structure to directly reuse unused memory blocks, greatly saving the memory overhead for managing free addresses; during query, since the data organization structure adopts a globally balanced binary tree implemented by indexes, high-speed data query is achieved. Since indexes are used to connect each node, the memory overhead of the pointers themselves is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular, to a high-speed data query method and system based on continuous memory. Background Art

[0002] With the advent of the Internet era, earth-shaking changes have taken place in major fields such as media, education, e-commerce, Internet finance, enterprise services, tourism, healthcare, and logistics. Massive amounts of data flow on the network. To comprehensively analyze and detect such huge traffic, it is usually necessary to retrieve key information from it to obtain matching information. However, the current problems are as follows: 1. In such a large traffic scenario, the processing efficiency requirement for each packet is extremely high, and the query step is required to be within 1 millisecond. Neither traditional relational databases such as MySQL and Oracle nor non-relational databases such as MongoDB and HBase can handle this application scenario; 2. If the query dataset is placed in a traditional, query-prior data structure, relatively high query efficiency can be obtained, but it does not support multi-process sharing. If each process holds such a huge amount of data on its own, the memory consumption will be extremely large, and it is also difficult to maintain data consistency.

[0003] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a high-speed data query method based on continuous memory, which solves the problems existing in the existing information query and retrieval.

[0005] The purpose of the present invention is achieved through the following technical solutions: A high-speed data query method based on continuous memory, the query method comprising:

[0006] Memory management step: The memory manager sets an index node, reuses the memory through the result of the index node, realizes the storage of data and the free list, initializes the applied memory by the memory manager, points the root node of the free list in the header structure to the data block, and calculates the type offset through the data node and the index node;

[0007] Data insertion step: Locate the data block where the root node is located according to the root node field of the query tree. Add the type offset to the address of the data block to find the index node of the root node. Compare the inserted data with the data of the current node multiple times. Move according to the comparison result through the index node. Then, retrieve a new data node from the head of the free list, copy the data to be inserted into the retrieved new data node, find the index node through the type offset and set the index node, and then attach it to the tree structure;

[0008] Data query step: Locate the data block where the root node is located according to the root node field of the query tree. Find the index node of the root node according to the type offset, compare the data area, and move through the index node according to the comparison result to implement data query.

[0009] The index node includes a parent node, a left node, a right node, node attributes, index, the index of the previous node, and the index of the next node; the type offset represents the offset of the index node in the data node. Find the index node of each data node through this offset, and assign the index, the index of the previous node, and the index of the next node in the index node to form an index connection for all data blocks.

[0010] The multiple comparisons of the inserted data with the data of the current node and the movement through the index node according to the comparison result include:

[0011] When the inserted data is less than the data of the current node, move to the left node; when the inserted data is greater than the data of the current node, move to the right node; when the inserted data is equal to the data of the current node, it means the data node already exists;

[0012] After moving, find the corresponding index node through the type offset, and continue to compare the inserted data with the data of the current node;

[0013] When the data after left shift is greater than the data of the current node, or the data after right shift is less than the data of the current node, or the current node is empty, data insertion is performed at the current position.

[0014] Attaching the index node to the tree structure includes:

[0015] Set the parent node of the index node to the index of the parent node of the current node. If it is a left shift in the data insertion step, set the left node of the index node to the index of the current node; if it is a right shift, set the right node of the index node to the index of the current node. Then, set the parent node of the current node to the new data node.

[0016] The realization of data query by moving through index nodes according to the comparison result includes:

[0017] If the data to be queried is small, it is moved to the left node. When the left node is empty, it means the data is not found. If the data to be queried is large, it is moved to the right node. When the right node is empty, it means the data is not found. If it is equal, it means the queried data is found. Finally, the user query result is returned.

[0018] A high-speed data query system based on continuous memory, which includes a memory manager, a data insertion module, and a data query module;

[0019] The memory manager: is used to set an index node, reuse the memory through the result of the index node, realize the storage of data and the free linked list, initialize the memory applied by the memory manager, point the root node of the free linked list in the header structure to the data block, and calculate the type offset through the data node and the index node;

[0020] The data insertion module: is used to find the data block where the root node is located according to the root node field of the query tree, find the index node of the root node by adding the type offset to the address of the data block, compare the inserted data with the data of the current node multiple times, move through the index node according to the comparison result, then take out a new data node from the head of the free linked list, copy the data to be inserted into it, find the index node through the type offset and set the index node, and then hang it on the tree structure;

[0021] The data query module: is used to find the data block where the root node is located according to the root node field of the query tree, find the index node of the root node according to the type offset, compare the data area, and realize data query by moving through the index node according to the comparison result.

[0022] The present invention has the following advantages: A high-speed data query method based on continuous memory. The relationships between all nodes are represented by indexes and offsets, perfectly solving the problem of memory sharing caused by different memory addresses between processes; Adopting a doubly linked list algorithm implemented by indexes as the free node management structure, directly reusing unused memory blocks, greatly saving the memory overhead for managing free addresses; During query, since the data organization structure adopts a globally balanced binary tree implemented by indexes, high-speed data query is achieved. In actual projects, the average query time for billions of data is within 3 microseconds (us); Since indexes are used to connect each node, the memory overhead of the pointers themselves is reduced, which is particularly obvious when the amount of data is large. Brief Description of the Drawings

[0023] Figure 1It is the schematic diagram of the inode and data node of the present invention;

[0024] Figure 2 It is the schematic diagram of the memory management of the present invention;

[0025] Figure 3 It is the schematic diagram of insertion and query of the present invention;

[0026] Figure 4 It is the schematic diagram of the actual memory usage of the present invention. Detailed implementation manners

[0027] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the protection scope of the claimed present application, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application. The present invention will be further described below with reference to the accompanying drawings.

[0028] The present invention specifically relates to a high-speed data query method based on continuous memory. Among them, the continuous memory can be a memory block with continuous logical addresses that is private to a process; it can also be a memory block with continuous logical addresses provided by techniques such as shared memory or file memory mapping. It specifically includes the following contents:

[0029] Step 1. Memory manager: As Figure 1 shown, the memory manager first declares a data structure: inode (node), including the following key fields: parent node, left node, right node, node attribute, index, previous node index, and next node index. The data node (the data structure for query and being queried) must contain the inode (regardless of the position). By reusing the memory through the structure of the inode, data can be stored, and the free list can also be stored.

[0030] As Figure 2As shown, the memory manager also initializes the allocated memory, with each data block storing one data node. During initialization, the memory manager first points the root node of the free list in the header structure to the data block. And the value of type_offset is calculated through the data structures of the data node (usrdata) and the index node (node), and the calculation method is as follows: type_offset = ((size_t)&((usrdata*)0)->node). The value of type_offset as Figure 2 shown is the offset of the index node in the data node. Then, through this offset, the index node of each data node is found, and the index, previous node index, and next node index in the index node are assigned, so that all data blocks form a structure connected by indexes, similar to a doubly linked list. Finally, the root node of the query tree is set to null.

[0031] Step 2: Data insertion: First, find the data block where the root node is located according to the root node field of the query tree. Add the type_offset offset to the address of the data block to find the index node of the root node. Compare the inserted data with the data of the current node and move through the index of the index node according to the comparison result: when the inserted data is less than the current node data, move to the left node; when the inserted data is greater than the current node data, move to the right node; if they are equal, it means the data node already exists. After moving, find the corresponding index node through type_offset. Continue to compare the inserted data with the data of the current node: when it is greater than the current node after moving left, or less than the current node after moving right, or the current node is empty, it indicates that insertion is to be performed at this position. At this time, a new data node is taken from the head of the free list, the data to be inserted is copied into it, and the index node is found through type_offset; then the index node is set and attached to the tree structure. The specific operation is: set the parent node of the index node to the index of the parent node of the current node. If it was a left move before, set the left node of the index node to the index of the current node; if it was a right move before, set the right node of the index node to the index of the current node. Then set the parent node of the current node to the new data node.

[0032] Step 3: Data query: First, find the data block where the root node is located according to the root node field of the query tree, and find the index node of the root node according to the type_offset offset; compare the data area and move through the index of the index node according to the comparison result. If the query data is smaller, move to the left node. When the left node is empty, it means the data is not found; if the query data is larger, move to the right node. When the right node is empty, it means the data is not found; if they are equal, it means the query data is found. Finally, return the user query result.

[0033] Further, the data node includes an index node and a data area. The data area is used to store data. When querying, it is necessary to compare the data stored in the data area. Therefore, the data area in the comparison of the data area refers to the data stored in the data area of the data node to which the current index node belongs.

[0034] Further, "equal" indicating that the queried data is found means that the queried data is equal to the data stored in the data area of the data node to which the current index node belongs.

[0035] In actual use of the present invention, after sequentially executing steps one to three, steps two and three are executed again.

[0036] The present invention constructs a globally balanced binary tree implemented by indexes on a logically continuous memory address structure by combining index nodes and index offsets, breaking the limitations brought by the current method of using pointers for data connection, achieving the purpose of high-speed sharing between processes, and having an extremely high key-value pair query speed. When applied to the private memory of a process, it is an efficient key-value pair memory database; when applied to shared memory, it becomes a key-value pair memory database that supports shared data sets and shared data queries.

[0037] The above are only the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be changed within the scope of the concept described herein through the above teachings or the techniques or knowledge in related fields. And any changes and modifications made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the protection scope of the appended claims of the present invention.

Claims

1. A high-speed data query method based on continuous memory, characterized in that: The query method includes: Memory management step: The memory manager sets an index node, reuses memory through the result of the index node to store data and the free linked list. The memory manager initializes the applied memory, points the root node of the free linked list in the header structure to the data block, and calculates the type offset through the data node and the index node; Data insertion step: Locate the data block where the root node is located according to the root node field of the query tree. Add the type offset to the address of the data block to find the index node of the root node. Compare the inserted data with the data of the current node multiple times. Move according to the comparison result through the index node. Then, take out a new data node from the head of the free linked list, copy the data to be inserted into the taken-out new data node, find the index node through the type offset and set the index node, and then hang it on the tree structure; Data query step: Locate the data block where the root node is located according to the root node field of the query tree. Find the index node of the root node according to the type offset, compare the data area, and move through the index node according to the comparison result to implement data query; The multiple comparisons of the inserted data with the data of the current node and the movement through the index node according to the comparison result include: When the inserted data is less than the data of the current node, move to the left node. When the inserted data is greater than the data of the current node, move to the right node. When the inserted data is equal to the data of the current node, it means that the data node already exists; After moving, find the corresponding index node through the type offset and continue to compare the inserted data with the data of the current node; When the data after moving left is greater than the data of the current node, or the data after moving right is less than the data of the current node, or the current node is empty, data insertion is performed at the current position.

2. The high-speed data query method based on continuous memory according to claim 1, wherein: The index node includes a parent node, a left node, a right node, node attributes, index, the index of the previous node, and the index of the next node; The typeoffset offset represents the offset of the index node in the data node. Through this offset, find the index node of each data node and assign values to the index, the index of the previous node, and the index of the next node in the index node, so that all data blocks are connected by indexes.

3. A high-speed data query method based on continuous memory according to claim 1, characterized in that: Hanging the index node on the tree structure includes: Set the parent node of the index node to the index of the parent node of the current node. If it is a left shift in the data insertion step, set the left node of the index node to the index of the current node. If it is a right shift, set the right node of the index node to the index of the current node. Then, set the parent node of the current node to the new data node.

4. A high-speed data query method based on continuous memory according to claim 1, characterized in that: The movement through the index node according to the comparison result to implement data query includes: If the data to be queried is small, move to the left node. When the left node is empty, it means the data is not found. If the data to be queried is large, move to the right node. When the right node is empty, it means the data is not found. Equal means the queried data is found. Finally, return the user query result.

5. A high-speed data query system based on continuous memory, characterized in that: It includes a memory manager, a data insertion module, and a data query module; The memory manager: is used to set an index node, reuse memory through the result of the index node, implement the storage of data and the free linked list, initialize the memory applied by the memory manager, point the root node of the free linked list in the header structure to the data block, and calculate the type offset through the data node and the index node; The data insertion module: is used to find the data block where the root node is located according to the root node field of the query tree, find the index node of the root node by adding the type offset to the address of the data block, compare the inserted data with the data of the current node multiple times, move according to the comparison result through the index node, then take out a new data node from the head of the free linked list, copy the data to be inserted into it, find the index node through the type offset and set the index node, and then hang it on the tree structure; The data query module: is used to find the data block where the root node is located according to the root node field of the query tree, find the index node of the root node according to the type offset, compare the data area, and move through the index node according to the comparison result to implement data query; The multiple comparisons of the inserted data with the data of the current node and the movement through the index node according to the comparison result include: When the inserted data is less than the data of the current node, move to the left node; when the inserted data is greater than the data of the current node, move to the right node; when the inserted data is equal to the data of the current node, it means that the data node already exists; After moving, find the corresponding index node through the type offset and continue to compare the inserted data with the data of the current node; When the data after left shift is greater than the data of the current node, or the data after right shift is less than the data of the current node, or the current node is empty, data insertion is performed at the current position.

Images