A compressed index-based attribute graph data query system

Abstract

This invention discloses an attribute graph data query system based on compressed indexes, applied to attribute graph databases based on compressed indexes. The system includes: a graph computing engine parsing query requests to determine their validity, and converting valid query requests into query plans; determining whether the data involved in the query request is marked in a data update table based on the query plan; the data update table includes the VIDs corresponding to attribute graph data that have been updated by the user but not yet updated to the compressed index module; when the data involved is not marked in the data update table, a positioning index unit obtains the start and end positions of the text string to be extracted; the attribute index unit extracts the text string using the high-order entropy compressed full-text self-indexing (GeCSA) algorithm based on the start and end positions; and a data processor converts the text string into a preset format and outputs the query results. This invention reduces the storage overhead of the attribute graph database, thereby improving query performance.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to an attribute graph data query system based on compressed indexes. Background Technology

[0002] With the rapid development of social networks, e-commerce, and other fields, attribute graph databases are being used more and more widely. They support efficient analysis of complex relationships and are far more efficient than traditional relational databases in processing complex, interconnected network data. However, graph data is usually complex and large, which may cause graph databases to read large amounts of data during query operations. The limitation of disk read speed leads to low query efficiency.

[0003] To improve query performance, mainstream attribute graph database systems employ various strategies and technologies. For example, Neo4j uses an index-free adjacency strategy, ensuring native storage of the graph structure. This means that the pointers between two entity nodes connected by an edge are physical; when accessing a node through an edge, the edge stores the physical address of the target node on disk. This ensures high query efficiency even with a large number of edges. Neo4j uses its own query language, Cypher, a declarative graph database query language that enables precise and efficient querying and updating of graph data. Due to its native graph storage and corresponding graph computation methods, Neo4j's query performance is relatively good compared to similar products, but it is not fully open source, and its community version has limited functionality. TigerGraph is another mainstream attribute graph database that utilizes efficient data compression technology, comprehensively considering storage and computational performance, reducing memory usage, and improving CPU cache hit rate, thereby improving overall query performance. In TigerGraph's official test documentation, its performance surpasses other similar products; however, TigerGraph is not currently available for free commercial use.

[0004] In addition, there are other attribute graph database systems, such as those that use partitioning techniques to distribute data across multiple storage nodes to improve parallel processing capabilities. However, the data consistency, cost, and scalability issues brought about by multiple nodes cannot be ignored. Other methods use caching techniques to reduce disk I / O operations; however, caching also incurs memory overhead. Still others use node indexing techniques, including hash indexes and binary tree indexes. Hash indexes use hash functions to map node attribute values ​​to positions in the index table, enabling fast node location but not supporting range queries. Binary tree indexes use a binary tree structure to organize nodes, supporting range queries. However, in current graph database application scenarios, the data scale and complexity are very high, and the above methods still suffer from low query efficiency and poor scalability when processing graph data.

[0005] Therefore, solving the query performance bottleneck problem in large-scale complex attribute graph scenarios has become an urgent need. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides an attribute graph data query system based on compressed indexes. The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] This invention provides an attribute graph data query system based on compressed index, comprising: a graph computing engine, a backend storage interface, a compressed index module, and an underlying database, wherein the compressed index module comprises: a positioning index unit, an attribute index unit, and a data processor;

[0008] The graph computing engine is used to parse the query request after receiving the user's query request to determine whether it is valid, and if the query request is valid, to convert the query request into a query plan.

[0009] The backend storage interface is used to determine whether the data involved in the query request is marked in the data update table according to the query plan; the data update table includes vertex identifiers (VIDs) corresponding to the attribute graph data that the user has updated but has not yet updated to the compressed index module;

[0010] The positioning index unit is used to obtain the start position startPos and end position endPos of the text string to be extracted when the data involved is not marked in the data update table.

[0011] The attribute indexing unit is used to extract the text string using the high-order entropy compressed full-text self-indexing GeCSA algorithm based on the start position startPos and the end position endPos.

[0012] The data processor is used to convert the extracted text string into a preset format and then output the query results.

[0013] In one embodiment of the present invention, the backend storage interface is further used to define a Key-Column-Value model, and the compressed index module further includes attribute graph data stored in the Key-Column-Value model, wherein "Key" includes the vertex identifier VID of the vertex in the attribute graph, "Column" includes the data type of the vertex in the attribute graph, and "Value" includes the data content of the data type of the vertex in the attribute graph, wherein the vertex identifier of all vertices in the attribute graph increments continuously from 0.

[0014] In one embodiment of the present invention, the data types of vertices in the attribute graph include: metadata, label information, attribute information, and edge information.

[0015] In one embodiment of the present invention, the positioning index unit is further configured to generate attribute graph text based on the attribute graph data stored in the Key-Column-Value model.

[0016] In one embodiment of the present invention, when generating attribute graph text based on the attribute graph data stored in the Key-Column-Value model, the positioning index unit is specifically used to: sort the attribute graph based on the VID of each vertex, store the data type and data content of each vertex in the row where the vertex is located, and obtain the attribute graph text; generate a bitmap based on the attribute graph text; wherein, each row of data in the attribute graph text is the uni-graph data of each vertex, and the bitmap is used to identify the starting position of each uni-graph data.

[0017] In one embodiment of the present invention, the query request includes the user's query requirement, which is: to query the data content of the data type of the vertex to be queried, as determined by the user.

[0018] When obtaining the start position startPos and end position endPos of the text string to be extracted, the positioning index unit is specifically used to: obtain the VID of the vertex to be queried, and calculate the ID of the corresponding monograph data based on the VID of the vertex to be queried; and determine the start position startPos = select1(BitMap,ID) and end position endPos = select1(BitMap,ID+1) of the text string to be extracted based on the bit string BitMap and the ID of the monograph data, where select1(BitMap,ID) and select1(BitMap,ID+1) represent the positions of the ID-th "1" and the ID+1-th "1" in the bit string BitMap, respectively.

[0019] In one embodiment of the present invention, when extracting the text string using the GeCSA (Geographically Compressed Full-Text Self-Indexing) algorithm based on the start position startPos and the end position endPos,

[0020] The attribute indexing unit is specifically used to: determine the length of the text string to be extracted, length = endPos - startPos, based on the start position startPos and the end position endPos; and further extract the text string using the high-order entropy compressed full-text self-indexing (GeCSA) algorithm based on the start position startPos and the length of the text string to be extracted.

[0021] In one embodiment of the present invention, the backend storage interface is further configured to query the underlying database and output query results when the data involved is marked in the data update table.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] This invention provides an attribute graph data query system based on compressed indexing, comprising: a graph computing engine, a backend storage interface, a compressed index module, and an underlying database. The compressed index module includes a positioning index unit and an attribute index unit. During the query process, the positioning index unit can determine the starting position (startPos) and length (length) of the text string to be extracted based on the query plan. Since a mapping relationship between VID and high-order entropy compressed text self-index is pre-established, the attribute index unit efficiently extracts the text string based on the high-order entropy compressed full-text self-indexing GeCSA algorithm. This enables various types of graph query operations on the attribute graph database based on compressed indexing, reduces the storage overhead of the attribute graph database, and thus improves query performance.

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a structure of an attribute graph data query system based on compressed index provided in an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of a query using an attribute graph data query system based on compressed index, provided in an embodiment of the present invention.

[0027] Figure 3 This is another structural diagram of the attribute graph data query system based on compressed index provided in the embodiments of the present invention;

[0028] Figure 4 This is a property diagram provided in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0030] Figure 1 This is a schematic diagram of a structure of an attribute graph data query system based on compressed index provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating a query performed using an attribute graph data query system based on compressed indexes, as provided in an embodiment of the present invention. Please refer to [link / reference]. Figure 1-2This invention provides an attribute graph data query system based on compressed index, comprising: a graph computing engine, a backend storage interface, a compressed index module, and an underlying database, wherein the compressed index module comprises: a positioning index unit, an attribute index unit, and a data processor;

[0031] The graph computing engine is used to parse the query request after receiving it from the user to determine whether it is valid, and if the query request is valid, to transform it into a query plan.

[0032] The backend storage interface is used to determine whether the data involved in the query request is marked in the data update table based on the query plan; the data update table includes the vertex identifiers (VIDs) corresponding to the attribute graph data that the user has updated but has not yet updated to the compressed index module;

[0033] The positioning index unit is used to obtain the start position startPos and end position endPos of the text string to be extracted when the data involved is not marked in the data update table.

[0034] The attribute indexing unit is used to extract the text string based on the start position startPos and the end position endPos using the high-order entropy compressed full-text self-indexing GeCSA algorithm.

[0035] A data processor is used to convert the extracted text string into a preset format and then output the query results.

[0036] Specifically, when performing query operations using the aforementioned attribute graph data query system based on compressed indexes, users can submit query requests using the Gremlin query language. By providing users with the mainstream attribute graph query language Gremlin, interaction between users and the attribute graph database is realized. Users can write graph queries and traversals in a declarative manner and perform complex graph analysis operations. At the same time, a set of commands and functions for performing graph operations is provided. Users can use the Gremlin language to add, modify, and delete graph data, as well as perform data aggregation and filtering.

[0037] The graph computing engine is used to optimize and parse query requests, checking their validity. If a query request is invalid, an exception message is returned to the user. If the query request is valid, it is transformed into a specific query plan. The query plan describes how to specifically execute a query operation, such as the path and operation method of the data to be queried. The query plan includes a set of operators and their execution order. Tinkerpop contains various types of operators such as VertexStep and GraphStep. Each step ultimately interacts with the backend storage interface to extract the corresponding information.

[0038] It should be noted that the backend storage interface can interact with the underlying database to query the corresponding serialized data and perform data parsing.

[0039] Furthermore, the backend storage interface is used to determine whether the data involved in the query request has been marked in the data update table. It should be noted that the compressed index module in this embodiment is not updated in real time. If the data involved has been marked in the data update table, it means that the data updated by the user has not yet been updated to the compressed index module. Therefore, the underlying database can be queried directly to obtain the query results. On the other hand, the backend storage interface is also used to query the underlying database and output the query results when the data involved is marked in the data update table.

[0040] In addition, the backend storage interface is pluggable, allowing users to choose to store graph data in different backend storage systems. Different backend storage interfaces offer different features and performance, allowing users to select different products based on their application scenarios, such as Apache Cassandra, HBase, and BerkleyDB.

[0041] Optionally, the compressed index module is a pluggable module. When the graph data is large and complex, using the compressed index can accelerate the query. The compressed index can not only realize the function of the storage backend, but also provide efficient data extraction function. The function of the positioning index unit is to determine the starting position startPos and length of the text string in the text set according to the query plan. Then, the attribute index unit efficiently extracts the text string, and finally maps the extracted text string to data in a specific format of the Key-Column-Value model.

[0042] For example, the backend storage interface is also used to define a Key-Column-Value model, and the compressed index module also includes attribute graph data stored in the Key-Column-Value model. Here, "Key" includes the vertex identifier (VID) of the vertices in the attribute graph, "Column" includes the data type of the vertices in the attribute graph, and "Value" includes the data content of the data type of the vertices in the attribute graph. The vertex identifiers of all vertices in the attribute graph increment continuously from 0. The data type of the vertices in the attribute graph can include: metadata, label information, attribute information, and edge information.

[0043] Figure 3 This is another structural diagram of the attribute graph data query system based on compressed index provided in this embodiment of the invention. Please refer to... Figure 3The attribute graph data query system based on compressed indexes also includes a transaction engine, which is used to handle transaction operations of the attribute graph database to ensure data consistency and reliability. It works in conjunction with the back-end storage interface to provide ACID transaction support.

[0044] Optionally, the locating index unit is also used to generate attribute graph text based on the attribute graph data stored in a Key-Column-Value model.

[0045] Specifically, when generating attribute graph text based on attribute graph data stored in a Key-Column-Value model, the positioning index unit is specifically used for: sorting based on the VID of each vertex in the attribute graph, storing the data type and data content of each vertex in the row where the vertex is located, and obtaining the attribute graph text; generating a bitmap based on the attribute graph text; wherein, each row of data in the attribute graph text is the uni-graph data of each vertex, and the bitmap is used to identify the starting position of each uni-graph data.

[0046] In other words, during the generation of attribute graph text, the data content of each data type of each vertex in the attribute graph can be stored as a text set, sorted by the VID of each vertex, to obtain the attribute graph text. Each line of data in the attribute graph text is the unigraph data of the corresponding vertex. Furthermore, the bitmap is a string containing only 0s and 1s, with its first and last bits set to 1. The length of the bitmap is the length of the attribute graph text. If a character in the attribute graph text is a newline character, the corresponding bit in the bitmap is set to 1; otherwise, it is set to 0.

[0047] Figure 4 This is a property diagram provided in an embodiment of the present invention, and Table 1 is... Figure 3 The attribute graph text shown below, combined with the attribute graph text, is a combination of the following. Figure 3 Table 1 further explains the generation process of the attribute graph text.

[0048] Specifically, please see Figure 3The attribute graph has 4 vertices, with vertex identifiers VID starting from 0 and incrementing sequentially: VID=0, VID=1, VID=2, and VID=3. The data types of each vertex include: metadata, labels, attribute information, and edge information. Taking vertex VID=0 as an example, its data content is: metadata, demigod, Name:hercules, Age:30, and out-1-mother,out-2-father. Among them, the edge information out-1-mother,out-2-father indicates that one outgoing edge of vertex VID=0 points to vertex VID=1, and the attribute of this edge is "mother". Another outgoing edge of vertex VID=0 points to vertex VID=2, and the attribute of this edge is "father". When generating attribute graph text, vertices VID=0, VID=1, VID=2, and VID=3 are arranged sequentially. The four data types and corresponding data content of vertex VID=0 are stored in lines 1-4, the four data types and corresponding data content of vertex VID=1 are stored in lines 5-8, and so on. This pattern can be followed to generate... Figure 3 The attribute graph text shown.

[0049] Table 1

[0050]

[0051] It should be noted that in Table 1, "in" represents an incoming edge, and "both" represents a bidirectional edge.

[0052] Optionally, the query request includes the user's query requirements, which are: to query the data content of the data type of the vertex to be queried, as determined by the user.

[0053] When obtaining the start position startPos and end position endPos of the text string to be extracted, the positioning index unit is specifically used to: obtain the VID of the vertex to be queried, and calculate the ID of the corresponding monograph data based on the VID of the vertex to be queried; and determine the start position startPos = select1(BitMap,ID) and end position endPos = select1(BitMap,ID+1) of the text string to be extracted based on the bit string BitMap and the ID of the monograph data, where select1(BitMap,ID) and select1(BitMap,ID+1) represent the positions of the ID-th "1" and the ID+1-th "1" in the bit string BitMap, respectively.

[0054] Table 2 is... Figure 3The data content of the vertex data type VID=3 in the attribute diagram shown is stored in the underlying database. BitMap[j]=1 indicates that the value of the j-th column in the bit string BitMap is the index of the bit string BitMap where the bit is "1".

[0055] Table 2

[0056]

[0057] Please refer to Table 2 and... Figure 3 The data type and data content of vertex VID=3 are stored in binary form in the underlying database. The data content is assigned a unique ID in a continuously increasing order according to the four data types: metadata, label information, attribute information, and edge information. In this embodiment, each row can be called the ID of the unidirectional graph data.

[0058] Taking the user's query requirement as: query the attribute information of the vertex with VID=3, we can find that the ID corresponding to this attribute information is VID*4+2=14, the starting position of the text string to be extracted is startPos=select1(BitMap,ID)=208, and the ending position is endPos=select1(BitMap,ID+1)=227.

[0059] Furthermore, when extracting the text string using the GeCSA algorithm for full-text self-indexing based on the start position startPos and the end position endPos, the attribute indexing unit is specifically used for:

[0060] The length of the text string to be extracted is determined by the start position startPos and the end position endPos, where length = endPos - startPos. Then, based on the start position startPos and the length of the text string to be extracted, the text string is extracted using the high-order entropy compression full-text self-indexing algorithm GeCSA.

[0061] As can be seen from the above embodiments, the beneficial effects of the present invention are as follows:

[0062] This invention provides an attribute graph data query system based on compressed indexing, comprising: a graph computing engine, a backend storage interface, a compressed index module, and an underlying database. The compressed index module includes a positioning index unit and an attribute index unit. During the query process, the positioning index unit can determine the starting position (startPos) and length (length) of the text string to be extracted based on the query plan. Since a mapping relationship between VID and high-order entropy compressed text self-index is pre-established, the attribute index unit efficiently extracts the text string based on the high-order entropy compressed full-text self-indexing GeCSA algorithm. This enables various types of graph query operations on the attribute graph database based on compressed indexing, reduces the storage overhead of the attribute graph database, and thus improves query performance.

[0063] In the description of this invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0064] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A property graph data query system based on compressed indexes, characterized in that, include: The system comprises a graph computing engine, a backend storage interface, a compressed index module, and an underlying database. The compressed index module includes a location index unit, an attribute index unit, and a data processor. The graph computing engine is used to parse the query request after receiving the user's query request to determine whether it is valid, and if the query request is valid, to convert the query request into a query plan. The backend storage interface is used to determine whether the data involved in the query request is marked in the data update table according to the query plan; the data update table includes vertex identifiers (VIDs) corresponding to attribute graph data that have been updated by the user but have not yet been updated to the compressed index module; the backend storage interface is also used to define a Key-Column-Value model; wherein, "Key" includes the vertex identifier (VID) of the vertex in the attribute graph, "Column" includes the data type of the vertex in the attribute graph, and "Value" includes the data content of the data type of the vertex in the attribute graph, and the vertex identifiers of all vertices in the attribute graph increment continuously from 0; The positioning index unit is used to obtain the start position startPos and end position endPos of the text string to be extracted when the data involved is not marked in the data update table; it is also used to generate attribute graph text based on the attribute graph data stored in the Key-Column-Value model included in the compression index module. The attribute indexing unit is used to extract the text string using the high-order entropy compressed full-text self-indexing algorithm GeCSA based on the start position startPos and the end position endPos. Furthermore, when the positioning indexing unit generates attribute graph text based on attribute graph data stored in a Key-Column-Value model, the positioning indexing unit is specifically used to: sort the data based on the VID of each vertex in the attribute graph, store the data type and data content of each vertex in the corresponding row, and obtain the attribute graph text; generate a bitmap based on the attribute graph text; wherein, each row of data in the attribute graph text is the univariate graph data of each vertex, and the bitmap is used to identify the starting position of each univariate graph data. The data processor is used to convert the extracted text string into a preset format and then output the query results.

2. The attribute graph data query system based on compressed index according to claim 1, characterized in that, The data types of vertices in the attribute graph include: metadata, label information, attribute information, and edge information.

3. The attribute graph data query system based on compressed index according to claim 2, characterized in that, The query request includes the user's query requirements, which are: to query the data content of the data type of the vertex to be queried, as determined by the user. When obtaining the start position startPos and end position endPos of the text string to be extracted, the positioning index unit is specifically used for: obtaining the VID of the vertex to be queried, and calculating the ID of the corresponding monograph data based on the VID of the vertex to be queried; determining the start position of the text string to be extracted based on the bitmap and the ID of the monograph data. and termination position , , These represent the positions of the ID-th "1" and the (ID+1)-th "1" in the BitMap string, respectively.

4. The attribute graph data query system based on compressed index according to claim 2, characterized in that, When extracting the text string using the High-Order Entropy Compressed Full-Text Self-Indexing (GeCSA) algorithm based on the start position startPos and the end position endPos, the attribute indexing unit is specifically used to: determine the length of the text string to be extracted (length = endPos - startPos) based on the start position startPos and the end position endPos, and further extract the text string using the High-Order Entropy Compressed Full-Text Self-Indexing (GeCSA) algorithm based on the start position startPos and the length of the text string to be extracted.

5. The attribute graph data query system based on compressed index according to claim 1, characterized in that, The backend storage interface is also used to query the underlying database and output the query results when the data involved is marked in the data update table.

Images