A method for storing and encoding graph data based on an attribute graph model

Abstract

The present invention provides a method for storing and encoding graph data based on an attribute graph model, comprising the following steps: S1. Design and construct an attribute graph storage model: In the design of the graph storage model, a graph can be divided into multiple subgraphs according to the relationship type, and the independent life cycle and storage medium of the subgraph can be set; S2. Design the graph data encoding method: The Id of the node data and the relationship data is represented by a fixed-length byte[], the relationship type in the relationship data is directly encoded into the Id of the data. Secondly, the uniqueValue is encoded into the key of the relationship. Finally, in the design of the relationship Id, a direction flag bit is adopted. The present invention adopts a new graph storage model and data encoding, effectively solves the problem of setting the storage period and storage medium for trajectory data according to subgraphs. Secondly, by adopting a custom encoding method, the space occupied by data storage is efficiently compressed and the query efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of graph databases. More specifically, it particularly relates to a method for storing and encoding graph data based on an attribute graph model. Background Art

[0002] With the wide application of big data technology and the rapid development of the Internet of Things field, the rapid growth of massive associated data has made graph databases exhibit their unique storage and computing capabilities. Some people compare relational databases to trains and graph databases to high-speed trains in the big data era. It has become the most active and most obvious development trend in NoSQL databases.

[0003] Compared with traditional relational databases, graph databases are good at processing a large amount of complex, interconnected and changeable network data. Therefore, they have extensive applications in fields such as social networks, recommendation systems, financial credit investigation systems, anti-fraud systems, as well as knowledge graphs and artificial intelligence.

[0004] Graph databases are divided into native graph storage and non-native graph storage according to whether the storage has been specifically optimized. No matter which type of graph storage, it will face problems related to supernodes and trajectory relationship storage caused by unreasonable storage model design, such as:

[0005] 1. It is unable to handle the data entry and hot spot problems brought by supernodes in a friendly manner (Note: A supernode refers to a single node with the number of associated edges exceeding hundreds of thousands or millions). In the existing storage model, after a supernode appears, the linked list for storing relational data will be very long. When a large amount of data is entered into the database in batches, it will form a hot task, resulting in task timeouts and failures;

[0006] 2. For the storage period and calculation of trajectory-type relational data, they cannot be well isolated and set. There is a lack of flexibility and it cannot be customized for storage according to the relationship type. It can only globally uniformly set the storage period and storage medium (as shown in Figure 1 and Figure 2 ). In reality, trajectory-type relational data has a certain timeliness. Therefore, when modeling data, it is necessary to store subgraphs according to static and dynamic relationships to solve the problem of non-uniform storage periods and the problem of flexible selection of storage media, and achieve data isolation at the subgraph level;

[0007] 3. The existing solutions for bidirectional relational data have redundant storage (as shown in Figure 3 ). In the mainstream graph storage model, in a directed graph scenario, bidirectional relationship storage will store according to two different graph data. Except for the direction, other fields and attributes of the data are redundantly stored, wasting a certain amount of space. Based on the existing storage solution, the storage model is optimized so that bidirectional storage only stores the direction mark, reducing the occupied storage space.

[0008] Therefore, in view of this, we study and improve the existing structure and deficiencies, and provide a method for storing and encoding graph data based on the attribute graph model, in order to achieve a more practical purpose. Summary of the Invention

[0009] To solve the above technical problems, the present invention provides a method for storing and encoding graph data based on an attribute graph model, which is achieved by the following specific technical means:

[0010] A method for storing and encoding graph data based on an attribute graph model, comprising the following steps:

[0011] S1. Design and build a property graph storage model: Based on the graph data type, different relational data is divided into different subgraphs for storage to achieve subgraph-level data isolation and data storage lifecycle. In the graph storage model design, a graph can be divided into multiple subgraphs according to relationship type, and the subgraphs can be set with independent lifecycles and storage media.

[0012] S2. Design diagram data encoding method: The IDs of node data and relationship data are represented by fixed-length byte[]. The relationship type in the relationship data is directly encoded into the data ID. Secondly, the uniqueValue is encoded into the key of the relationship. Finally, the direction indicator bit is used in the relationship ID design.

[0013] Furthermore, the hash bit of the data may be set in the schema corresponding to the graph storage model of step S1 to identify the storage starting bit of each subgraph.

[0014] Furthermore, step S1 is specifically as follows: a subgraph corresponds to 5 regions, and the hash bit is 4, then the starting range of the 5 regions corresponding to the subgraph will be marked by a 4-bit byte: [-∞,0200), [0200,0400), [0400,0600), [0600,0800), [0800,∞).

[0015] Furthermore, when graph data is stored, it can be evenly divided into 5 regions based on the ID of the node data through a hash algorithm; when relationship data is stored, it can be evenly divided into 5 regions based on the ID of the starting node through a hash algorithm.

[0016] Furthermore, step S2 includes encoding the node data ID, specifically, hashing the primary key and primary value of the node data to generate a fixed-length node key, the length of which is hash bits + 8. The hash bits are the hash length specified in the graph model of step S1, and 8 is the byte[] length corresponding to the long-type node ID.

[0017] Furthermore, step S2 includes encoding the relationship data ID, specifically generating a fixed-length relationship key from the starting node, relationship type, direction, end node and uniqueValue, with a length of hash bits + 29 bits, and corresponding to the 8-bit byte[] length corresponding to the starting node ID, the 4-bit byte[] length corresponding to the relationship type mapped to an int value, the 1-bit byte corresponding to the direction, the 8-bit byte[] length corresponding to the end node ID, and the 8-bit byte[] length corresponding to the relationship uniqueValue. The uniqueValue is used to uniquely identify a piece of relationship data corresponding to two points, and is used to identify and store trajectory relationship data.

[0018] Furthermore, the hash algorithm is specifically a hash algorithm used to identify the data storage of each subgraph.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] This invention uses a new graph storage model and data encoding to effectively solve the problem of setting the storage period and storage medium for trajectory data by subgraph. Secondly, it uses a custom encoding method to efficiently compress the space occupied by data storage and improve the efficiency of query. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of the mainstream graph storage model in the existing technology.

[0022] Figure 2 This is a schematic diagram of trajectory relationship storage of the main mainstream graph storage model in the prior art.

[0023] Figure 3 This is a schematic diagram of bidirectional relationship storage of the mainstream storage model in the prior art.

[0024] Figure 4 This is a schematic diagram of the relationship ID storage design of the present invention.

[0025] Figure 5 This is a schematic diagram of the property graph storage design based on the present invention.

[0026] Figure 6 This is a schematic diagram of the encoding storage based on the attribute graph of the present invention. DETAILED DESCRIPTION

[0027] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0028] Example 1

[0029] The present invention provides a method for storing and encoding graph data based on an attribute graph model, comprising the following steps:

[0030] S1. Design and build a property graph storage model: Based on the graph data type, different relational data are divided into different subgraphs for storage to achieve subgraph-level data isolation and data storage lifecycle. In the graph storage model design, a graph can be divided into multiple subgraphs according to the relationship type, and the subgraphs can be set with independent lifecycles and storage media. At the same time, data hotspots and skew caused by super nodes can be prevented. The hash bit of the data can be set in the schema corresponding to the graph storage model to identify the storage starting bit of each subgraph.

[0031] For example, if one of the subgraphs corresponds to 5 regions and the hash bit is 4, the starting ranges of the 5 regions corresponding to the subgraph will be marked by 4 bytes: [-∞,0200), [0200,0400), [0400,0600), [0600,0800), [0800,∞);

[0032] When graph data is stored, it is evenly distributed among the five regions mentioned above based on the node data ID and a specified hash algorithm. Similarly, when relationship data is stored, it is evenly distributed among the five regions mentioned above based on the starting node ID and a specified hash algorithm. The hash algorithm is specifically used to identify the storage of each subgraph data, as follows:

[0033]

[0034]

[0035] S2. Design of graph data encoding: This primarily addresses the problem of allowing both node and relationship IDs to be represented using fixed-length byte[] s. The relationship type in the relationship data is directly encoded in the data ID, facilitating data query and data type differentiation. Secondly, by encoding unique values in the relationship key, efficient trajectory data storage is achieved. Finally, by incorporating a direction indicator into the relationship ID design, redundancy in bidirectional data storage is addressed.

[0036] Among them, node ID: the primary key and primary value of the node data are hashed to generate a fixed-length node key, the length of which is hash bits + 8. The hash bits are the hash length specified in the above graph model, and 8 is the byte[] length corresponding to the long type node ID.

[0037] Among them, the relationship ID: starting node, relationship type, direction, end node and uniqueValue generate a fixed-length relationship key, the length of which is the hash bit + 29 bits. 29 corresponds to the 8-bit byte[] length corresponding to the starting node ID, the 4-bit byte[] length corresponding to the relationship type mapped to an int value, the 1-bit byte corresponding to the direction, the 8-bit byte[] length corresponding to the end node ID, and the 8-bit byte[] length corresponding to the relationship uniqueValue. The uniqueValue is used to uniquely identify a piece of relationship data corresponding to two points, and is used to identify and store trajectory relationship data, such as Figure 4 shown.

[0038] Example 2

[0039] like Figure 5 As shown in the figure, the construction of the graph model is mainly based on the graph data type, dividing different relational data into different subgraphs for storage, so as to achieve subgraph-level data isolation and data storage life cycle (TTL).

[0040] The process diagram is as follows:

[0041] 1. Graph Data Partitioning Strategy

[0042] (1) The node data of the graph is stored in the main graph. The storage period of the main graph is permanent storage. This is mainly because the node data is usually associated with the relationship data. In the actual data query process, the corresponding starting node or ending node may be queried through the relationship data.

[0043] (2) The division of relational data can be implemented in a specific way according to the needs. For example, a type of relations with the same storage period can be divided into a subgraph for storage based on the storage period, or a type of related relational data can be divided into a subgraph for storage based on the relationship type.

[0044] 2. Graph Data Encoding Strategy

[0045] The following will explain in detail the node ID encoding, relationship ID encoding and attribute encoding. Attribute data is stored in the form of key-value, where the key and value data correspond to the byte[] data structure, which are called attribute name (key) and attribute value (value).

[0046] (1) Node data ID encoding

[0047] The encoding of the Id of node data is relatively simple. When designing the graph model, the number of hash bits is specified, and the length of the node Id is the hash bits + 8, where 8 is the long value corresponding to the node id. The node id ensures uniform data distribution across regions through hashing, and the scale of node data can reach 2^64.

[0048] (2) Relationship data Id encoding

[0049] The encoding of relationship data Id is relatively complex and mainly consists of six parts: hash bits, start node id, relationship type, relationship direction, end node id, and uniqueValue. Similar to nodes, the hash ensures uniform distribution of relationship data across specified regions. The start node is also a long value, encoded as an 8-byte byte array; the relationship type is a String, which is first mapped to an int value during encoding and encoded as a 4-byte byte array; the relationship direction includes OUTGOING, INCOMING, and BOTH, encoded with a single byte; the end node is encoded as an 8-byte byte array; and uniqueValue is also encoded as an 8-byte byte array. Therefore, when the hash bits are 4, the relationship Id will be encoded as a 33-byte byte array, and the storage scale of relationship data can be designed to be unlimited.

[0050] (3) Node / relationship attribute encoding

[0051] As Figure 6 shown, assume a node A with 3 attributes: ("ID card", "620502*********758"), ("Former name", ["zhangsan", "Zhang Sanfeng"]), ("Age", 26), ("Gender", "male"). The length of the rowValue after encoding this node is 74 bytes. The specific encoding process is as follows:

[0052] 1) This node has 4 attributes, and each attribute key is mapped to an int value for encoding (4 bytes). The composition of this node element is: total number of attributes (4 bytes) + encoding corresponding to the mapping of the first attribute key + encoding corresponding to the mapping of the second attribute key + encoding corresponding to the mapping of the third attribute key + encoding corresponding to the mapping of the fourth attribute key;

[0053] 2) The node Id of ("ID card", "620502*********758") is encoded as 19 bytes: flag indicating whether it is a multi-valued attribute (1 byte) + attribute value ID card number (18-byte byte array);

[0054] 3) The property value of ("Former Name", ["zhangsan", "Zhang Sanfeng"]) will be encoded into 27 bytes: a flag indicating whether it is a multi-valued property (1 byte), the number of property values (4-byte encoding) + the length of the first value (4-byte encoding) + the storage of the first value zhangsan (8-byte encoding) + the length of the first value (4-byte encoding) + the storage of the second value Zhang Sanfeng (6-byte encoding);

[0055] 4) The property value of ("Age", 26) is encoded into 5 bytes: a flag indicating whether it is a multi-valued property (1 byte) + the specific storage of the property value 26 (4-byte);

[0056] 5) The property value of ("Gender", "Male") is encoded into 3 bytes: a flag indicating whether it is a multi-valued property (1 byte), the length of the property value (2-byte encoding);

[0057] In summary, the encoded storage of node A is: 20 bytes after encoding the property name + 54 bytes for the encoded storage of the property value (19 + 27 + 5 + 3). The storage encoding of relationship properties is similar to that of nodes. Among them, the encoded property name can be reused in the storage of other nodes to avoid the problem of duplicate node property names. In addition, for the case of multi-valued columns in property values, it can also well support storage.

[0058] The embodiments of the present invention are given for purposes of illustration and description, and are not exhaustive or limit the present invention to the disclosed form. Many modifications and variations are obvious to those of ordinary skill in the art. The embodiments are chosen and described in order to better illustrate the principles of the present invention and its practical applications, and to enable those of ordinary skill in the art to understand the present invention and design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for storing and encoding graph data based on an attribute graph model, characterized in that: It includes the following steps: S1. Design and construct an attribute graph storage model: According to the graph data type, different relational data are divided into different subgraphs for storage to achieve data isolation and data storage lifecycle at the subgraph level. In the graph storage model design, a graph is divided into multiple subgraphs according to the relationship type, and independent lifecycle and storage media for the subgraphs are set; S2. Design the graph data encoding method: The Id of node data and relational data is represented by a fixed-length byte[]. The relationship type in the relational data is directly encoded into the Id of the relational data. Secondly, the uniqueValue is encoded into the key of the relationship. Finally, in the design of the relationship Id, a direction flag bit is adopted; The step S2 includes the encoding of the node data Id. Specifically, the main key and main value of the node data are hashed to generate a fixed-length node key, with a length of hash bit + 8. The hash bit is the hash length specified in the graph model of step S1, and 8 is the byte[] length corresponding to the long-type node Id; The step S2 also includes the encoding of the relational data Id. Specifically, the start node, relationship type, direction, end node, and uniqueValue are used to generate a fixed-length relationship key, with a length of hash bit + 29 bits, and they respectively correspond to the 8-bit byte[] length corresponding to the start node Id, the 4-bit byte[] length corresponding to the relationship type mapped to an int value, the 1-bit byte[] length corresponding to the direction, the 8-bit byte[] length corresponding to the end node Id, and the 8-bit byte[] length corresponding to the relationship uniqueValue. The uniqueValue is used to uniquely identify a relational data corresponding to two points and is used to identify the storage track relational data.

2. The method for storing and encoding graph data based on an attributed graph model according to claim 1, wherein: In the schema corresponding to the graph storage model of step S1, the hash bit of the data can be set to identify the storage start bit of each subgraph.

3. The method for storing and encoding graph data based on the property graph model according to claim 2, wherein: Specifically, step S1 is as follows: One subgraph corresponds to 5 regions. If the hash bit is 4, the start ranges of the 5 regions corresponding to this subgraph will be identified by 4-bit bytes: [-∞, 0200), [0200, 0400), [0400, 0600), [0600, 0800), [0800, ∞).

4. The method for storing and encoding graph data based on the property graph model according to claim 3, wherein: When storing graph data, according to the Id of the node data, through the hash algorithm, it evenly falls into 5 regions; when storing relational data into the database, according to the Id of the start node, through the hash algorithm, it evenly falls into 5 regions.

5. The method for storing and encoding graph data based on the property graph model according to claim 4, wherein: The hash algorithm is specifically the hashing algorithm used to identify the data storage of each subgraph.

Images