A storage method and device for a distributed native graph database unique index

Abstract

The application relates to the technical field of distributed graph database, and provides a storage method and device for a unique index of a distributed native graph database. According to a unique attribute column of existing data, a unique index is created for the existing data; incremental data is inserted into a current master table, and the current master table is updated according to a unique attribute column of the incremental data, so that the unique index of the incremental data is directly established; the unique index is queried according to an index key of to-be-queried data, the to-be-queried data is queried from the current master table according to the queried unique index, and the query is directly based on the unique index; the current master table can be subjected to an adding, deleting, modifying and inquiring operation without sending a request to data nodes on all partitions related to the current master table, the efficiency of the adding, deleting, modifying and inquiring operation on the current master table is improved by using the unique index, the waste of CPU and I / O resources is avoided, and the overall performance of a database system is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of distributed graph database technology, and in particular to a method and apparatus for storing a unique index of a distributed native graph database. Background Technology

[0002] Graph databases are a type of NoSQL database that primarily uses graph theory to store information about relationships between entities. They are suitable for analyzing complex, multi-layered relationships and are better suited to today's massive data processing scenarios. Graph databases are database systems specifically designed for storing and querying graph data. Their design goal is to handle large-scale graph-structured data—data composed of nodes and edges—providing efficient graph traversal and complex graph query capabilities. Unlike traditional relational databases, graph databases can effectively store and process knowledge graph data and are better suited for handling complex relationships and connections. They have highly flexible data models that can easily represent and process many-to-many relationships and complex graph structures.

[0003] Native graph databases are database systems specifically designed and optimized for storing and querying graph data. They provide specialized query languages ​​and algorithms for efficiently storing and processing graph data, supporting flexible querying and analysis. Unlike traditional relational databases or key-value stores, native graph storage systems are optimized for the characteristics of graph structures to provide higher performance and scalability. Native graph databases use the graph model as the core of data storage and querying, processing and analyzing graph data through specialized graph query languages ​​and algorithms. Non-native graph databases refer to database systems that were not originally designed specifically for graph data but were later extended or had graph data storage and querying capabilities added to support graph data storage and analysis. Non-native graph databases may be less efficient and flexible than native graph databases in processing graph data. Native graph databases typically employ specialized data structures and algorithms internally to optimize the storage and querying of graph data. Therefore, native graph databases often have an advantage when dealing with large-scale and complex graph data.

[0004] Distributed graph databases distribute data across multiple nodes or computers to achieve parallel processing and high scalability. In a distributed graph database, graph data is divided into multiple subgraphs and distributed across different nodes. Each node is responsible for managing and processing a portion of the graph data. Nodes communicate and collaborate through a network, enabling the graph database to handle large-scale graph datasets and improve query and analysis performance through parallel computing.

[0005] In existing technologies, the main table data in distributed native graph databases is distributed across multiple distributed data nodes based on the partition key value. Each data node has an independent data structure, and the data between nodes has no overlap. When inserting data into a main table, an insert request needs to be sent to all data nodes in the relevant partitions of the main table. Uniqueness constraints are applied to all data nodes in the relevant partitions to prevent data leakage. Similarly, when querying data, a query request also needs to be sent to all data nodes in the relevant partitions of the main table. This significantly reduces the efficiency of CRUD operations on the database, resulting in a waste of CPU and I / O resources, leading to low overall database system performance. Furthermore, existing technologies cannot directly create unique indexes for incremental data appended to the database, failing to improve CRUD efficiency through unique indexes, further reducing the overall performance of the database system.

[0006] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a storage method and apparatus for a unique index of a distributed native graph database. The purpose is to improve the efficiency of CRUD operations through unique indexes, thereby improving the overall performance of the database system. This solves the problem in the prior art that when performing CRUD operations on the main table, requests need to be sent to all data nodes on the partitions involved in the main table, resulting in low overall performance and poor usability of the database system.

[0008] The present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for storing a unique index in a distributed native graph database, comprising:

[0010] Create a unique index for the existing data based on the unique attribute columns of the data that already exist in the current main table;

[0011] Insert incremental data into the current main table, and create a unique index for the incremental data based on its unique attribute column to update the current main table;

[0012] The corresponding unique index is retrieved based on the index key of the data to be queried, and the data to be queried is retrieved from the current main table based on the retrieved unique index.

[0013] Furthermore, creating a unique index for the existing data based on the unique attribute columns of the existing data in the current main table includes:

[0014] The data node calculates the partition where the existing data is located based on the unique attribute column, extracts the unique identifier of the existing data from the partition based on the unique attribute column, and uses the mapping relationship between the unique attribute column and the corresponding unique identifier as a unique index.

[0015] The data nodes, based on the data volume and allocation of the unique index, and according to the number of all data nodes involved in the unique index, divide and store the unique index into shards; after storage is completed, they send status information to the management node.

[0016] When a data node discovers that a unique index under creation violates a unique constraint, it sends an error message to the management node; upon receiving the error message, the management node notifies all data nodes involved in the unique index to delete all unique indexes under creation.

[0017] The management node integrates all status information and updates the metadata information of the unique index based on all status information to complete the creation of the unique index.

[0018] Furthermore, the data nodes, based on the data volume and allocation of the unique index, and according to the number of all data nodes involved in the unique index, perform sharding and storage of the unique index, including:

[0019] Based on the data volume and allocation, and according to the number of all data nodes involved in the unique index, determine the index partition information of the multiple data nodes to which the unique index needs to be stored;

[0020] Based on the index partition information, a local index table and a remote index table are established, and the unique index is written to the local index table and the remote index table respectively;

[0021] Based on the index partition information, a synchronization request is sent to the remote data node so that the remote data node can receive the synchronization request and synchronize the remote index table, thereby realizing the sharded storage of the unique index through the local index table and the remote index table.

[0022] Furthermore, each data node synchronously maintains both a local index table and a remote index table.

[0023] Furthermore, during the batch loading of existing data, the storage method for the unique index of the distributed native graph database also includes:

[0024] The data nodes for existing data are determined based on the unique identifiers of the existing data, and the data nodes for storing the corresponding unique indexes are determined based on the unique attribute columns of the existing data.

[0025] Existing data and / or unique indexes that need to be sent to the same data node are grouped and encapsulated to obtain grouped data that need to be sent to each data node;

[0026] Send the grouped data to the corresponding data nodes.

[0027] Furthermore, the step of inserting incremental data into the current main table and creating a unique index for the incremental data based on its unique attribute column to update the current main table includes:

[0028] Insert the incremental data into the current main table and obtain the unique identifier of the incremental data;

[0029] The index partition to be inserted for the incremental data is calculated based on the unique attribute column of the incremental data;

[0030] Based on the unique identifier, determine whether the unique index created in the partition to be inserted violates the uniqueness constraint;

[0031] If the uniqueness constraint is violated, the unique index will not be created;

[0032] If the uniqueness constraint is not violated, the unique attribute column is used as the index key, the mapping relationship between the index key and the corresponding unique identifier is used as the unique index, and the unique index is inserted into the index partition to be inserted.

[0033] Further, inserting the incremental data into the current main table and obtaining the unique identifier of the incremental data includes:

[0034] Perform a bitwise AND operation on the set ID of the graph where the incremental data is located and a preset AND operation parameter to obtain the preset sequence value of the set ID;

[0035] Perform a bitwise OR operation between the node partition ID where the incremental data is located and a preset OR operation parameter to obtain the initial partition result; clear the data in the initial partition result except for the preset partition position to obtain the intermediate partition result; move the data in the intermediate partition result with the preset partition position to the higher bit to obtain the preset partition value;

[0036] Perform a bitwise OR operation on the data type of the incremental data to obtain an initial type result; clear the data in the initial type result except for the preset type bit to obtain an intermediate type result; shift the data of the preset type bit in the intermediate type result to the higher bit to obtain the preset type value;

[0037] The unique identifier is obtained based on the preset sequence value, the preset partition value, and the preset type value.

[0038] Further, obtaining the unique identifier based on the preset sequence value, the preset partition value, and the preset type value includes:

[0039] When the sum of the number of bits of the preset sequence value, the preset partition value, and the preset type value is greater than the number of bits of the unique identifier, the number of bits of the preset sequence value remains unchanged, and the number of bits of the preset partition value and / or the number of bits of the preset type value are reduced according to a preset adjustment strategy, so that the sum of the number of bits of the preset sequence value, the preset partition value, and the preset type value is equal to the number of bits of the unique identifier, so as to obtain the unique identifier.

[0040] Furthermore, the step of querying the corresponding unique index based on the index key of the data to be queried, and then retrieving the data to be queried from the current main table based on the retrieved unique index, includes:

[0041] Based on the index key of the data to be queried, obtain the local index table and / or the remote index table of the data to be queried; obtain the unique index based on the index key of the data to be queried;

[0042] Based on the unique index, the unique identifier of the data to be queried is found in the local index table and / or the remote index table;

[0043] Based on the unique identifier, the data node where the data to be queried is located is obtained, and the data to be queried is retrieved from the data node.

[0044] Secondly, the present invention also provides a storage device for a unique index of a distributed native graph database, used to implement the storage method for a unique index of a distributed native graph database described in the first aspect, wherein the storage device for the unique index of a distributed native graph database includes:

[0045] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the storage method for a unique index of a distributed native graph database as described in the first aspect.

[0046] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the storage method for a unique index of a distributed native graph database as described in the first aspect.

[0047] Unlike existing technologies, the present invention has at least the following beneficial effects:

[0048] This invention creates unique indexes on existing data based on unique attribute columns. By inserting incremental data into the current main table and updating the current main table using the index key of the incremental data's unique index, a unique index is directly established on the incremental data. The invention then queries the corresponding unique index based on the index key of the data to be queried, and retrieves the data from the current main table using the found unique index, enabling direct querying based on the unique index. This invention allows for CRUD operations on the current main table without sending requests to all data nodes across all partitions, optimizing communication and network overhead during computation. It also improves the efficiency of CRUD operations on the current main table using unique indexes, avoids wasting CPU and I / O resources, and significantly enhances the overall performance of the database system. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A schematic diagram of the overall process of a method for storing a unique index in a distributed native graph database, provided by an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of existing data provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of a specific process for step 10 of an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of a specific process of step 102 in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of an indexing process for existing data provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of a process for driving and processing data source data provided in an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram of a GCM process for handling index creation requests provided in an embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram of a process for gStore to process index creation requests, provided by an embodiment of the present invention.

[0058] Figure 9 This is a schematic diagram illustrating a process for completing index creation according to an embodiment of the present invention;

[0059] Figure 10 This is a schematic diagram of a specific process for step 20 in an embodiment of the present invention;

[0060] Figure 11 This is a schematic diagram of a specific process of step 201 in an embodiment of the present invention;

[0061] Figure 12 This is a schematic diagram of a process for loading incremental data provided in an embodiment of the present invention;

[0062] Figure 13 This is another specific flowchart of step 20 in an embodiment of the present invention;

[0063] Figure 14 This is a schematic diagram of a process for batch loading incremental data provided in an embodiment of the present invention;

[0064] Figure 15 This is a schematic diagram of a specific process for step 30 in an embodiment of the present invention;

[0065] Figure 16 This is a schematic diagram of the sharded storage of the current master table provided by the present invention;

[0066] Figure 17 This is a schematic diagram of a fragmented storage data node provided by the present invention;

[0067] Figure 18 This is a schematic diagram of a query execution process based on a unique index provided by the present invention;

[0068] Figure 19 This is a schematic diagram of the architecture of a storage device for a unique index of a distributed native graph database provided in an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0070] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0071] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0072] In this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium.

[0073] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0074] Example 1:

[0075] For ease of description, the descriptions of "points" and "nodes" in the embodiments of the present invention are further defined below; all "points" in the embodiments refer to points in the graph, and "graph" here refers to graphs in the graph database; for distinction, "node" refers to management nodes or data nodes in a distributed computing architecture.

[0076] Distributed native graph storage is a storage solution that extends native graph storage systems to a distributed environment. It distributes graph data across multiple nodes to achieve horizontal scaling of storage capacity and processing power, and supports parallel querying and computation.

[0077] It should be noted that, in order to distribute the pressure of processing data on each data node, this embodiment adopts a management node and partition data node (gStore) mode, separating the functions of controlling data and storing data, and not placing them on the same node. The management node can be implemented using a graph storage cluster manager (gStore ClusterManager, abbreviated as GCM). For ease of description, the partition data node will be simply referred to as a data node below. However, the storage method for the distributed native graph database unique index of this invention is not limited to this storage mode; control and storage can be implemented on a single node, and this is not a limitation here. Those skilled in the art can refer to existing technologies and, without creative effort, obtain the storage method for the distributed native graph database unique index of this embodiment using other storage modes based on this embodiment.

[0078] In existing technologies, the main table data in distributed native graph databases is distributed across multiple distributed data nodes according to the index key values ​​in each partition. Each data node has an independent data structure, and the data between data nodes has no overlap. A unique index is a database indexing mechanism used to ensure the uniqueness of one or more columns in a data table. It has the following advantages: ensuring data integrity and consistency, improving query efficiency, and providing index coverage.

[0079] In existing technologies, unique indexes are typically stored so that each node stores only its own unique index of data, excluding indexes of data on other nodes. The unique index data structure maintains a one-to-one relationship with the main table's data structure; since the main table is partitioned, each partition has its own separate unique index structure. For each unique index structure, the keys only map to the main table data within its own partition, not to the main table data in other partitions. Therefore, a given index value may exist on all partitions.

[0080] When inserting data into the main table, to ensure the data in the database conforms to uniqueness constraints, an insert request needs to be sent to all data nodes on the relevant partitions of the main table. Uniqueness checks are performed on all data nodes on the relevant partitions to prevent data leakage. Similarly, when querying data, a query request also needs to be sent to all data nodes on the relevant partitions of the main table. This significantly reduces the efficiency of CRUD operations on the database, resulting in a waste of CPU and I / O resources. Although existing technologies improve efficiency to some extent by designing partitioned parallel computing, they still require sending requests to all data nodes on the relevant partitions of the main table, thus not changing the waste of CPU and I / O resources and not fundamentally solving the problem. The unique index storage method in existing technologies has a significant impact on the performance of the database system, resulting in low overall database system performance.

[0081] Furthermore, current technologies cannot directly create unique indexes for incremental data. Because unique indexes in distributed storage clusters cannot be duplicated, if data in the database already contains an index, the existing index must be deleted and recreated, making direct creation of a unique index impossible. It's possible to create a unique index only on each data node, not across different data nodes. When unique indexes are duplicated across different data nodes, a control node must be created to extract data from all data nodes and then create the unique index there. This process significantly reduces database performance and has poor practicality.

[0082] To address the issues in existing technologies where CRUD operations on the main table require sending requests to all data nodes across all partitions of the main table, resulting in wasted CPU and I / O resources, and the inability to directly create unique indexes on incremental data, leading to low overall database system performance and poor usability, such as... Figure 1 As shown, Embodiment 1 of the present invention provides a method for storing a unique index of a distributed native graph database, including:

[0083] Step 10: Create a unique index for the existing data based on the unique attribute columns of the data already existing in the current main table.

[0084] The existing data refers to instances in the graph database that contain data such as points, edges, attributes, and labels, which are equivalent to rows in a relational database. The unique attribute column is a column in the existing data that stores unique attributes. A unique attribute means that the value of the attribute is not repeated. The unique attribute column is equivalent to the primary key in a relational database.

[0085] For example, such as Figure 2 As shown, the label of the node is User, and the label of the edge is FOLLOWS. The attribute of the node labeled User is name, with the attribute value being Johan, Peter, or Emil. The edges represent the follow relationship between them. Figure 2 Three instances are shown, named Johan, Peter, and Emil.

[0086] A unique index is a mapping between a unique attribute column and a unique identifier; the unique identifier is rowid. Here, rowid is a globally unique identifier in the distributed native graph storage.

[0087] The master table refers to a data table that has been partitioned and stored on different nodes in a distributed storage cluster; that is, a data table that is stored in shards. The current master table is the master table on which operations are currently being performed.

[0088] This invention establishes a unique index on existing data, enabling the retrieval of existing data directly based on the unique identifier by querying the unique index using only the unique attribute column.

[0089] In existing distributed storage clusters, different data nodes maintain their own rowid sequence, which can only guarantee that there will be no duplicate rowids in the current data node. This invention provides a scheme that includes parameters in the rowid, ensuring that the data can be directly located in the specified data node of the distributed cluster through the rowid. The specific scheme is described in step 201 below.

[0090] Step 20: Insert incremental data into the current main table, and create a unique index for the incremental data based on the unique attribute column of the incremental data to update the current main table.

[0091] In this embodiment of the invention, step 10 involves establishing a unique index on a large amount of existing data in the current master table, and step 20 involves directly creating a unique index on the incremental data (i.e., existing data) that is being appended when a large amount of existing data already exists in the current master table. Whether to create a unique index for the incremental data is determined based on whether the created unique index violates a uniqueness constraint, i.e., whether the same unique index already exists in the current master table.

[0092] In an optional embodiment, the index partitioning algorithm may be a hash algorithm. The specific type of index partitioning algorithm shall be selected by those skilled in the art according to the specific application, and is not limited here.

[0093] Step 30: Query the corresponding unique index based on the index key of the data to be queried, and then query the data to be queried from the current main table based on the unique index found.

[0094] This invention creates unique indexes on existing data based on unique attribute columns. By inserting incremental data into the current main table and updating the current main table using the index key of the incremental data's unique index, a unique index is directly established on the incremental data. The invention then queries the corresponding unique index based on the index key of the data to be queried, and retrieves the data from the current main table using the found unique index, enabling direct querying based on the unique index. This invention allows for CRUD operations on the current main table without sending requests to all data nodes across all partitions, optimizing communication and network overhead during computation. It also improves the efficiency of CRUD operations on the current main table using unique indexes, avoids wasting CPU and I / O resources, and significantly enhances the overall performance of the database system.

[0095] In existing distributed storage clusters, the main table is partitioned according to a preset partitioning method. The data in the main table is distributed across various partitions and stored on various data nodes. The data on each data node has an independent data structure, and there is no overlap between the data nodes.

[0096] The unique index is also partitioned according to a preset index partitioning method, distributing the unique index across different partitions. The preset main table partitioning method and the preset index partitioning method are selected by those skilled in the art based on the specific use case. The unique index is stored such that each data node only stores the unique index of the data on that data node, excluding unique indexes of data on other data nodes. The data structure of the unique index maintains a one-to-one relationship with the data structure of the main table. Since the main table has already undergone partitioning, each partition of the main table has a unique index unique to that partition, but there is no globally unique index (a unique index across all partitions).

[0097] Furthermore, since the partitioning mode of a unique index is completely independent and has no relation to the partitioning of the main table, meaning the default main table partitioning method and the default index partitioning method are unrelated, any unique index may map to different main table partitions. This means that data retrieved through the index may exist on all partitions. Therefore, when performing CRUD operations, requests need to be sent to all partition nodes, leading to excessive communication pressure and network overhead in the database system.

[0098] To reduce the number of communications between data nodes during CRUD operations and thus improve efficiency, unique indexes are created for the large amount of existing data in the current main table, such as... Figure 3 As shown, step 10 includes:

[0099] Step 101: The data node calculates the partition where the existing data is located based on the unique attribute column, extracts the unique identifier of the existing data from the partition based on the unique attribute column, and uses the mapping relationship between the unique attribute column and the corresponding unique identifier as a unique index.

[0100] The calculation method for determining the partition where existing data resides based on the unique attribute column can be chosen by those skilled in the art according to the specific use case, and is not limited here. In an optional embodiment, the partition table of the distributed native graph database is first obtained, then the specific value of the specified partition key (i.e., the unique attribute column) is obtained, and the corresponding node partition ID is queried in the partition table based on the specific value to determine the partition where existing data resides.

[0101] To facilitate load balancing across nodes in a distributed storage cluster, this invention, like existing technologies, uses a completely independent partitioning mode for the unique index, unrelated to the partitioning of the main table. Since the unique attribute columns of the current main table are globally unique, and the unique identifiers obtained after storing data from the current main table across various data nodes are also globally unique, this invention uses the mapping relationship between unique attribute columns and their corresponding unique identifiers as the unique index to obtain a globally unique index.

[0102] Step 102: The data nodes, based on the data volume and allocation of the unique index, and according to the number of all data nodes involved in the unique index, shard the unique index for storage; after storage is completed, they send status information to the management node.

[0103] The allocation details refer to the distribution of the unique index across different data nodes according to the preset index partitioning method. When sharding a unique index, it is necessary to determine the total number of data nodes involved in the unique index to be stored based on the allocation details, and then determine the specific storage method and perform the storage based on the number of data nodes.

[0104] Step 103: When a data node discovers that a unique index under creation violates the uniqueness constraint, it sends an error message to the management node; the management node receives the error message and notifies all data nodes involved in the unique index to delete all unique indexes under creation.

[0105] When creating a unique index, the management node notifies all data nodes. Each data node then sends unique indexes that are not located in its own partition (the partition where the data node resides) to the corresponding data nodes in batches to achieve sharded storage. Each data node has a worker thread responsible for processing unique indexes sent from other data nodes. Each data node periodically sends updates on the unique indexes to the management node, as well as information on whether uniqueness constraints have been violated.

[0106] If a violation of the uniqueness constraint is found during the creation process, the management node will send a cancellation message for the unique index creation task, notifying all data nodes involved in creating the unique index to roll back all transactions related to the creation of the unique index, so that the data in the database system (including existing data and unique indexes) remains consistent with that before the creation of the unique index.

[0107] Step 104: The management node integrates all status information and updates the metadata information of the unique index based on all status information to complete the creation of the unique index.

[0108] The metadata information for unique indexes is stored in the management node. This metadata includes the unique index name, index key information, creation time, status, and creation progress. Creating a unique index based on existing data, querying based on a unique index, and inserting data into a unique index all require requesting or updating the metadata information in the management node using the driver program.

[0109] In a distributed storage cluster, each data node can apply for a unique identifier, and the database uses the unique identifier to locate the vertex relationship. Therefore, this embodiment of the invention establishes a unique index by using the mapping relationship between unique attribute columns and unique identifiers. This enables the system to quickly find the corresponding unique identifier and locate the data to be queried when performing CRUD operations, by only inputting the value of a unique attribute column.

[0110] The process of storing a unique index will be further detailed below, such as... Figure 4 As shown, step 102 includes:

[0111] Step 1021: Based on the data volume and allocation, and according to the number of all data nodes involved in the unique index, determine the index partition information of the multiple data nodes to which the unique index needs to be stored.

[0112] Among them, the index partition information is the allocation of the unique index across different data nodes in different partitions.

[0113] The management node sends the index partition information to each relevant data node. The data node parses the index partition information required by its own node to determine the partition where the existing data for which a unique index needs to be built is located, as well as the amount of data in the existing data, i.e., the index partition information, which facilitates the subsequent creation of the unique index.

[0114] Step 1022: Establish a local index table and a remote index table based on the index partition information, and write the unique index to the local index table and the remote index table respectively.

[0115] Each data node involved creates a local index table and multiple remote index tables.

[0116] Step 1023: Based on the index partition information, send a synchronization request to the remote data node so that the remote data node can receive the synchronization request and synchronize the remote index table, so as to realize the sharded storage of the unique index through the local index table and the remote index table.

[0117] The remote data node refers to the data node that is not in the same partition as the current data node when storing the unique index.

[0118] In this embodiment of the invention, the unique index of a row of data is stored in shards. That is, according to the index partition information, the unique index is stored in the local index table or the remote index table of each data node in different partitions. For example, the unique index of entity 1 (a row of data in the current main table) is stored in the local index table of data node 3 in partition 1, and also in the remote index tables of data nodes 1 and 2 in partition 1.

[0119] In order to locate data in any partition with a single global query using a unique attribute column, the remote index table in this embodiment of the invention needs to be synchronized with the data nodes of other partitions, and each data node synchronously maintains both the local index table and the remote index table to ensure that the uniqueness constraint of the database is not violated.

[0120] Since existing data is stored based on the sharding results of unique identifiers, while unique indexes are stored based on the sharding results of unique attribute columns, unique attribute column information on the same data node may be randomly distributed across all data nodes. This invention simplifies the data synchronization process between data nodes by establishing a local index table and multiple remote index tables based on the number of data nodes, reducing the complexity of data synchronization and improving synchronization efficiency. When performing CRUD operations using the unique index, it is only necessary to calculate the data node using the unique attribute column to find the unique identifier of the data to be queried, and then locate the complete existing data using the unique identifier.

[0121] like Figure 5 As shown, this embodiment provides an optional process for creating a unique index on a large amount of existing data. The specific steps are as follows:

[0122] Step 1-01: Driver processes data from the data source;

[0123] Step 1-02: GCM processes the unique index creation request, analyzes the index status, and performs the operation;

[0124] Step 1-03: gStore processes unique index creation requests;

[0125] Step 1-04: Complete the creation of the unique index.

[0126] like Figure 6 As shown, this embodiment provides an optional solution for step 1-01, and the specific steps are as follows:

[0127] Step 1-01-01: Receive user request;

[0128] Step 1-01-02: Analyze the information of the index to be built, and obtain information such as vertex label name and attribute name of the index to be built;

[0129] Step 1-01-03: Send a unique index creation request to GCM.

[0130] like Figure 7 As shown, this embodiment provides an optional solution for step 1-02, and the specific steps are as follows:

[0131] Step 1-02-01: GCM resolves the unique index creation request sent by the driver;

[0132] Step 1-02-02: GCM queries the status of the index to be created. If it already exists, it returns the current status information of the index; if it does not exist, it queries the attribute type of the index to be created, the amount of data in the index to be created, and the data allocation of the index to be created.

[0133] Step 1-02-03: GCM returns an existing index status to the driver or sends a request to gStore to create a unique index.

[0134] like Figure 8 As shown, this embodiment provides an optional solution for step 1-03, and the specific steps are as follows:

[0135] Step 1-03-01: gStore parses the information of the index to be created, and determines the partition where the original data of the index is located and the amount of data;

[0136] Step 1-03-02: gStore extracts data from the unique index and performs HASH sharding according to the number of nodes in the gStore cluster;

[0137] Step 1-03-03: gStore creates multiple index tables locally based on partitions, divided into local tables and remote tables. After the unique index is written, the remote tables send synchronization requests to the remote data nodes in batches.

[0138] When performing hash calculations on the attribute values ​​of a unique index column, if synchronization with remote data nodes is required, the following steps may be involved:

[0139] Step 1-03-04: gStore completes the local unique index write and synchronizes the remote synchronization information sent by other gStore nodes to the local index table.

[0140] like Figure 9 As shown, this embodiment provides an optional solution for step 1-04, and the specific steps are as follows:

[0141] Step 1-04-01: After gStore completes the unique index write and data synchronization, it sends the current node index status information to GCM;

[0142] Step 1-04-02: GCM integrates the index status information of the gStore node. If a conflict exists, a rollback is performed and the current index creation status is set to failure; otherwise, the index status is set to success and the metadata information of the unique index is updated.

[0143] Because the unique index in this embodiment of the invention is globally unique, there is no need to delete and recreate the existing unique index in the database, nor is it necessary to establish a control node. Figure 10 As shown, step 20 includes:

[0144] Step 201: Insert the incremental data into the current main table and obtain the unique identifier of the incremental data.

[0145] In the distributed storage cluster of this invention embodiment, each data node can apply for a globally unique identifier.

[0146] Since the unique identifier of the incremental data can only be obtained after the incremental data insertion operation is performed, the incremental data is inserted first, and then the unique index is created in subsequent steps.

[0147] Step 202: Calculate the index partition to be inserted for the incremental data based on the unique attribute column of the incremental data.

[0148] Since a unique index needs to be created, the partition and data nodes required for storing the unique index of the incremental data are calculated based on the unique attribute columns in the incremental data and the specific index partitioning algorithm used in the database.

[0149] Step 203: Based on the unique identifier, determine whether the unique index created in the partition to be inserted violates the uniqueness constraint.

[0150] Since the unique identifier of existing data may not be on the same data node as the existing data, locating existing data based on the unique attribute column may involve two query processes. Each existing data has a globally unique unique identifier. Therefore, when performing a unique constraint check before insertion, it is not necessary to query the specific existing data. Instead, it is sufficient to determine whether a unique index containing the unique identifier exists in the index partition to be inserted. This ensures that the unique index of the incremental data has not been inserted into the database before the creation of the unique index can proceed.

[0151] Step 204: If the uniqueness constraint is violated, the unique index is not created.

[0152] When the same unique index already exists, during batch data loading, if it is necessary to load the existing data and create the unique index simultaneously, then the transaction that inserts the existing data and the transaction that creates the unique index are rolled back, and the incremental data is not used to update the current main table. If it is not necessary to load the existing data and create the unique index simultaneously, then only the transaction that creates the unique index is rolled back, and the incremental data is used to update the current main table, but the unique index of the incremental data is not created.

[0153] Step 205: If the uniqueness constraint is not violated, the unique attribute column is used as the index key, the mapping relationship between the index key and the corresponding unique identifier is used as the unique index, and the unique index is inserted into the index partition to be inserted.

[0154] In this embodiment, the value of the unique attribute column is the same as the value of the index key. The unique index in this embodiment is the mapping relationship between the unique attribute column and the unique identifier. In an optional embodiment, the mapping relationship can be stored using key-value pairs, where the index key is the key in the key-value pair of the unique index, and the unique identifier is the value.

[0155] To ensure that the unique identifier is globally unique, allowing direct location of the desired data node in the distributed cluster, such as... Figure 11 As shown, in step 201, when the incremental data is inserted into the current main table, the unique identifier for the incremental data includes:

[0156] Step 2011: Perform a bitwise AND operation on the set ID of the graph where the incremental data is located and the preset AND operation parameters to obtain the preset sequence value of the set ID.

[0157] The preset and operation parameters are selected by those skilled in the art based on the specific use case of the distributed storage cluster. Bitwise operation refers to operating on each bit of the binary number of the data to be calculated (such as the set ID and preset and operation parameters); operations include bitwise AND and bitwise OR operations.

[0158] It should be noted that the operational rules in existing technologies such as bitwise AND and bitwise OR operations are well known to those skilled in the art and will not be elaborated upon further. They should not be regarded as a lack of understanding in this invention.

[0159] In an optional embodiment, since the number of bits in the preset and operation parameters represents the maximum number of set IDs that can be stored in a unique identifier, the number of bits in the unique identifier is prioritized to satisfy the storage of all information of the set IDs. If an array is used to store the incremental data of this embodiment, then the set ID is an array ID.

[0160] For example, when the unique identifier in this embodiment of the invention is of type INT64, the preset AND operation parameter is 0xFFFFFFFFFF. A bitwise AND operation is performed between the set ID and 0xFFFFFFFFFF. Since 0xFFFFFFFFFF is a 64-bit binary number, it is represented as 0xFFFFFFFFFF in hexadecimal. The bitwise AND operation retains the lower 40 bits of the set ID and clears the remaining bits to zero, resulting in the preset sequence value.

[0161] Step 2012: Perform a bitwise OR operation on the node partition ID where the incremental data is located and the preset OR operation parameter to obtain the initial partition result; clear the data in the initial partition result except for the preset partition position to obtain the intermediate partition result; move the data in the intermediate partition result of the preset partition position to the high bit to obtain the preset partition value.

[0162] The preset or operational parameters and preset partition bits are selected by those skilled in the art based on the specific use case of the distributed storage cluster.

[0163] For example, when the unique identifier in this embodiment of the invention is of type INT64, the preset OR operation parameter is 0x3FFFFF. The node partition ID is bitwise ORed with 0x3FFFFF, and the result retains the lower 22 bits of the node partition ID, while the remaining bits are cleared to zero. Then, the result is shifted left by 40 bits, and the lower 22 bits are shifted to the higher bits to obtain the preset partition value.

[0164] Step 2013: Perform a bitwise OR operation on the data type of the incremental data to obtain an initial type result; clear the data in the initial type result except for the preset type bit to obtain an intermediate type result; move the data of the preset type bit in the intermediate type result to the higher bit to obtain the preset type value.

[0165] The preset type field is selected by those skilled in the art based on the specific use case of the distributed storage cluster.

[0166] For example, when the unique identifier in this embodiment of the invention is of type INT64, the data type is bitwise ANDed with 0x3 to obtain a type initialization result that retains the lower 2 bits of the data type, and the remaining bits are cleared to zero. Then, the type initialization result is shifted left by 62 bits (that is, the bits in the type initialization result other than the preset type bits), and the lower 2 bits are shifted to the higher bits.

[0167] Step 2014: Obtain the unique identifier based on the preset sequence value, the preset partition value, and the preset type value.

[0168] This invention extracts the required binary bits from the set ID (arrayID), node partition ID (segmentId), and data type (dataType) of each existing data set used for fast query and location through bitwise AND and bitwise OR operations, and stores this information in the same parameter value (i.e., a unique identifier). Under the current encoding rules, the unique identifier can accommodate trillions of vertices. The encoding rules can be further adjusted according to actual business scenarios to increase data capacity and application scope. When the sum of the number of bits in the preset sequence value, the preset partition value, and the preset type value is greater than the number of bits in the unique identifier, the number of bits in the preset sequence value remains unchanged, and the number of bits in the preset partition value and / or the number of bits in the preset type value is reduced according to a preset adjustment strategy, so that the sum of the number of bits in the preset sequence value, the preset partition value, and the preset type value equals the number of bits in the unique identifier, thus obtaining the unique identifier.

[0169] The preset adjustment strategy is selected by those skilled in the art based on the specific usage scenario.

[0170] In an optional embodiment, since the number of bits in the preset partition bit represents the maximum number of node partition IDs that can be stored in a unique identifier, and the number of bits in the preset type bit represents the maximum number of data of the specified data type that can be stored in a unique identifier, when the number of bits in the unique identifier satisfies all the information of the storage set ID, the remaining bits in the unique identifier are used by those skilled in the art to select the number of bits in the preset partition bit and the number of bits in the preset type bit according to the specific use case of the distributed storage cluster, such as the specific number of bits in the node partition ID and the data type, so as to adjust the number of bits in the preset partition value and / or the number of bits in the preset type value.

[0171] This invention provides a uniquely identified encoding / decoding example, as detailed below:

[0172] For a distributed storage cluster with 3 data nodes, when writing data to a vertex with arrayId 100000001, we know that segmentId = HASH(100000001) = 2. The vertex's default dataType is 1. Therefore, the following calculation is performed:

[0173] rowid:=(arrayId&0xFFFFFFFFFF)<<0

[0174] rowid|=(segmentId&0x3FFFFF)<<40

[0175] rowid|=(dataType&0x3)<<62

[0176] The final rowid value is: 4611688217460643457

[0177] On the storage node, decode it:

[0178] dataType:=(rowid>>62)&0x3

[0179] segmentId:=(rowid>>40)&0x3FFFFF

[0180] arrayId:=(rowid>>0)&0xFFFFFFFFFF

[0181] Through the above calculations, we can obtain the original arrayId (100000001), segmentId (2), and dataType (1).

[0182] It is worth noting that the unique identifier in the existing technology is only unique on the node storing the unique identifier. When using a non-globally unique identifier as the primary key of existing data, the unique index is built separately on each data node, and there is only one index table. When querying based on the unique index or the unique identifier, it is necessary to broadcast messages to all data nodes. As the number of data nodes in the distributed storage cluster increases, the communication pressure between data nodes becomes greater and greater, which will lead to the problem of continuous decline in read and write performance after the cluster is expanded.

[0183] Specifically, in distributed systems, to quickly build indexes, indexes can be created separately for multiple data nodes. However, when executing index queries, messages need to be broadcast to each data node, and the driver or master node then aggregates the result set, resulting in a significant performance bottleneck. Distributed clusters use unique IDs maintained independently by each data node for labeling, and the driver layer uses a unique attribute column (Key) for query location, which has the following obvious drawbacks:

[0184] 1. High query cost: When performing a query, it is first necessary to query the unique ID under the current data node through the key, and then query the array position of the existing data based on the unique ID, and finally obtain the complete existing data. The query chain is relatively long.

[0185] 2. Difficulty in creating unique indexes for data already in the database: Since each data node independently maintains its own index information and the data nodes do not communicate with each other, it is difficult to achieve data deduplication between data nodes. The creation of unique indexes can only be supported when data is written for the first time, and there is a serious performance bottleneck in multi-node deduplication.

[0186] 3. Index queries require broadcasting: Since a unique index exists only on the current node and is not distributed according to shard information, index queries in a distributed cluster need to be broadcast to all nodes and aggregated by the driver or management node, which significantly reduces query performance in distributed mode.

[0187] In this embodiment of the invention, the unique identifier is globally unique throughout the entire distributed cluster. Because it includes node partition ID information, the unique identifier allows for easy location of existing data on specific data nodes. By using a globally unique identifier, both a local index table and a remote index table are maintained to store the unique index. Whether querying based on the unique index or the unique identifier, only one query is required. The increase in the number of data nodes in the distributed storage cluster will not affect the database's read / write performance.

[0188] Specifically, in this embodiment of the invention, each data node maintains a globally unique identifier. Data node information can be obtained through this unique identifier, avoiding a series of problems caused by broadcasting, such as transmission delay, low reliability, and inability to guarantee consistency. The specific advantages are as follows:

[0189] 1. Supports creating unique indexes for data already in the database: unique identifiers are globally unique, and unique indexes are synchronized across nodes, which can more effectively achieve unique index deduplication.

[0190] 2. Eliminate message broadcasting: The unique index is stored in shards according to the number of nodes, eliminating the need for broadcasting, reducing communication volume, reducing the communication pressure between management nodes and drivers, and improving throughput.

[0191] 3. By using unique identifiers to locate array positions, unnecessary secondary queries based on unique attribute columns are reduced, shortening the query process and reducing storage load.

[0192] 4. Under the current storage model, data batch loading can be optimized to achieve batch creation of unique indexes, greatly improving data loading efficiency.

[0193] This embodiment provides an optional process for adding a new unique index row, and the specific steps are as follows:

[0194] For the following embodiment, the following vertex data row information is used to write it into a distributed native graph database containing 3 storage nodes:

[0195] 100000001 521641156412211234 Zhang San 100000002 513651198601215113 Li Si 100000003 513651197206133262 Wang Wu

[0196] Among them, rowid is a unique identifier, IDCARD is a unique attribute column, and NAME is an attribute column in the current main table.

[0197] Step 2-01: The driver sends an index creation request to the GCM node;

[0198] Step 2-02: GCM receives the index creation request, analyzes the index status, and performs the operation;

[0199] Step 2-03: The driver sends information about the newly added unique index to each storage node;

[0200] Step 2-04: The storage node processes the received information and begins writing data;

[0201] Step 2-05: The storage node sends the data write result to the driver;

[0202] This embodiment provides an optional solution for step 2-02, the specific steps of which are as follows:

[0203] Step 2-02-01: GCM parses the index creation request;

[0204] Step 2-02-02: GCM analyzes the index status. If the index already exists, it returns directly; if the index does not exist, the index creation process is executed.

[0205] Step 2-02-03: GCM sends information about creating the index table to gStore and waits for a response;

[0206] Step 2-02-04: GCM returns a response to the driver indicating that the index table has been successfully created, and the driver can start adding data;

[0207] Step 2-02-05: The driver performs a hash calculation on the read data based on the rowid to determine the graph storage node information (segmentId) where the graph array data needs to be saved; based on the data in the example table above, the graph array information will be saved to the following node partitions respectively:

[0208] 100000001 2 4611688217460643457 100000002 0 4611686018428387906 100000003 1 4611687117940015683

[0209] Step 2-02-06: Perform a hash calculation on the read data based on the index value to determine the graph storage node information (segmentId) where the unique index needs to be saved; based on the data in the table example above, the unique index will be saved to the following node partitions respectively:

[0210] 521641156412211234 0 513651198601215113 2 513651197206133262 0

[0211] This embodiment provides an optional solution for step 2-03, the specific steps of which are as follows:

[0212] Step 2-03-01: The driver sends a message to initiate a data write transaction.

[0213] Step 2-03-02: The driver sends graph array data information to the known nodes; based on the data in the table example above, the driver will send one piece of data to each of the three storage nodes for storage;

[0214] Step 2-03-03: The driver sends the unique index information to the known nodes; based on the data in the table example above, the driver can merge and send data from the same nodes to reduce the number of interactions;

[0215] This embodiment provides an optional solution for step 2-04, the specific steps of which are as follows:

[0216] Step 2-04-01: The storage node receives the graph array data write message sent by the driver;

[0217] Step 2-04-02: Determine if there are duplicate data based on rowid, and perform the data write operation;

[0218] Step 2-04-03: Collect data, write status codes, and send execution results to the driver;

[0219] Step 2-04-04: The storage node receives the unique index write message;

[0220] Step 2-04-05: Write the newly added unique index information to the index table and determine whether there are duplicate index attributes;

[0221] Step 2-04-06: Collect the unique index write status code and send the execution result to the driver;

[0222] After the write operation is complete, the index table information for node 0 is as follows:

[0223] 521641156412211234 4611688217460643457 513651197206133262 4611687117940015683

[0224] The index table information for node 2 is as follows:

[0225] 513651198601215113 4611686018428387906

[0226] This embodiment provides an optional solution for step 2-05, the specific steps of which are as follows:

[0227] Step 2-05-01: The driver receives the graph array data write status sent by the storage node;

[0228] Step 2-05-02: The driver receives the unique index write status returned by the storage node;

[0229] Step 2-05-03: Drive the integration of graph array data and unique index write status;

[0230] Step 2-05-04: The driver analyzes whether there is an exception during the write operation. If there is an exception, a rollback is performed; otherwise, the operation is returned as successful and the subsequent operations are continued.

[0231] like Figure 12 As shown, when updating the current main table with a small amount of incremental data, it is not necessary to simultaneously write the incremental data and the unique index. However, when creating a unique index during the process of batch loading incremental data, in order to ensure data consistency and the correctness of the unique index, it is necessary to simultaneously load the existing data and the unique index to ensure that the loading of each batch of data meets the uniqueness constraint. To achieve the loading of batch incremental data, step 20 of this embodiment of the invention is further refined below, as follows: Figure 13 As shown, during the batch loading of existing data, the storage method for the unique index of the distributed native graph database further includes:

[0232] Step 201: Determine the data nodes of the existing data based on the unique identifier of the existing data, and determine the data nodes to be stored for the corresponding unique index based on the unique attribute columns of the existing data.

[0233] Step 202: Group and encapsulate the existing data and / or unique indexes that need to be sent to the same data node to obtain the grouped data that need to be sent to each data node.

[0234] The specific grouping and encapsulation method can be selected by those skilled in the art based on the specific application scenario, and is not limited further. Since it is necessary to load existing data and unique indexes simultaneously, this embodiment of the invention encapsulates existing data and unique indexes that need to be written to the same data node, achieving simultaneous loading.

[0235] It should be noted that because the data nodes for existing data in sharded storage and the data nodes for unique indexes in sharded storage are determined independently, existing data and unique indexes on the same data node are not necessarily corresponding.

[0236] Step 203: Send the grouped data to the corresponding data nodes.

[0237] The globally unique identifier and unique identifier in this invention significantly improve the performance of big data loading, eliminating the need for broadcasting to all distributed nodes, reducing network interactions, and supporting unique index storage for big data. When the partition sorting rule and the index tree sorting rule are consistent, scanning is performed within the range of values ​​of the unique attribute column (index key). The index scan is executed only once in each partition in a unidirectional order, without having to shuttle back and forth between partitions involved in the index. Therefore, whether it is an index scan targeting a fixed index key or an index scan targeting a range of values ​​for an index key, it can directly locate one or several partitions that need to be scanned, instead of blindly scanning all partitions.

[0238] like Figure 14 As shown, this embodiment provides an optional process for loading batch vertices (existing data) based on a unique index. The specific steps are as follows:

[0239] Step 3-01: The driver parses the data source information and sends a vertex label and unique index creation request to GCM;

[0240] Step 3-02: GCM analyzes the driver request and analyzes the vertex label and unique index status;

[0241] Step 3-03: GCM sends a request to the gStore node to initialize the vertex labels and unique index table;

[0242] Step 3-04: The driver receives a batch of existing data and performs HASH sharding on the rowid and unique attribute columns based on the number of gStore nodes;

[0243] Step 3-05: The driver groups and encapsulates the raw data according to HASH fragmentation;

[0244] Step 3-06: The driver sends a vertex data load request and a unique index write request to the storage node;

[0245] Step 3-07: The storage node receives and processes the request;

[0246] Step 3-08: The storage node returns the execution result to the driver;

[0247] Step 3-09: The driver analyzes the current batch data loading status and performs retry / rollback / success operations.

[0248] When querying existing data based on the globally unique identifier and globally unique index in this embodiment of the invention, such as Figure 15 As shown, step 30 includes:

[0249] Step 301: Obtain the local index table and / or remote index table of the data to be queried based on the index key of the data to be queried; obtain the unique index based on the index key of the data to be queried.

[0250] Since local and remote index tables store unique indexes, the local and / or remote index tables containing the index key can be retrieved based on that index key; by matching, unique indexes containing the same index key are identified, thus enabling the search.

[0251] Step 302: Based on the unique index, find the unique identifier of the data to be queried in the local index table and / or the remote index table.

[0252] Step 303: Based on the unique identifier, obtain the data node where the data to be queried is located, and query the data to be queried from the data node.

[0253] When the unique identifier and existing data are on the same data node, the existing data can be retrieved with only one query; when the unique identifier and existing data are not on the same data node, the existing data can still be retrieved with only two queries. This avoids sending requests to all data nodes, greatly reduces the communication interaction and network overhead between data nodes during the calculation process, and improves the availability of the database system.

[0254] For example, the current main table includes four columns (attribute column 1, attribute column 2, unique identifier, and unique attribute column, respectively) and six rows (i.e., six instances); Figure 16 As shown, the data of the current main table is stored in partition 1 and partition 2 according to the horizontal sharding method (that is, the partitions involved in the current main table are partition 1 and partition 2).

[0255] like Figure 17 As shown, the management node performs unified management of the data nodes of each partition; the data to be queried for entity 1 is stored in data node 1, the data to be queried for entity 2 is stored in data node 2, the data to be queried for entity 3 is stored in data node 3, the data to be queried for entity 4 and entity 5 is stored in data node 4, and the data to be queried for entity 6 is stored in data node 5.

[0256] The data to be queried for each instance is independent of the data node storing the unique index corresponding to each instance; the unique index of entity 1 is stored in the local index table of data node 3, and also in the remote index tables of data nodes 1 and 2; the unique index of entity 2 is stored in the local index table of data node 2, and also in the remote index tables of data nodes 1 and 3; the unique index of entity 3 is stored in the local index table of data node 3, and also in the remote index tables of data nodes 1 and 2; the unique indexes of entity 4, entity 5, and entity 6 are all stored in the local index table of data node 5 (data node 4 is not involved in the storage of the unique index of the current main table).

[0257] According to the above storage method, when the data to be queried is the data of entity 3, only the index key (i.e., the unique attribute column) of the data to be queried is used to query in all partitions; in an optional embodiment, the index key is found in the remote index table of data node 1 of partition 1, that is, the corresponding unique identifier is obtained according to the unique index of the index key stored in the remote index table, and then the data to be queried is determined to be stored on data node 3 of partition 1 according to the unique identifier, thus obtaining the data to be queried.

[0258] When the data to be queried is the data of entity 6, the query is performed in all partitions using only the index key (i.e., the unique attribute column) of the data to be queried. In an optional embodiment, the index key is found in the local index table of data node 5 of partition 2. The corresponding unique identifier can be obtained based on the unique index of the index key stored in the local index table, and then the data to be queried can be obtained directly on data node 5 based on the unique identifier.

[0259] The embodiments of the present invention implement unique index constraints in a distributed system. That is, for a certain index key, its data can only fall in a fixed index partition, and there is only one unique index structure in a single index partition. Therefore, uniqueness constraints can be guaranteed across the entire table.

[0260] like Figure 18 As shown, this embodiment provides an optional process for querying existing data, and the specific steps are as follows:

[0261] Step 4-01: The driver parses the query parameter information and sends a unique index query request to the specified node;

[0262] Step 4-02: The storage node processes the received information and executes the query;

[0263] Step 4-03: The driver parses the rowid parameter information and sends a query request to the storage;

[0264] Step 4-04: The storage node processes the query request and executes the query;

[0265] This embodiment provides an optional solution for step 4-01, and the specific steps are as follows:

[0266] Step 4-01-01: Calculate the HASH distribution information based on the provided parameter information to obtain the node information that should be queried for the current data;

[0267] Step 4-01-02: Obtain the query connection for the specified node and send a message to execute the query;

[0268] This embodiment provides an optional solution for step 4-02, the specific steps of which are as follows:

[0269] Step 4-02-01: The storage node (NODE 0) parses the received query request to obtain the tag name, index name, and index parameter information of the current query;

[0270] Step 4-02-02: The storage node locates the specific index table (in Node 2) based on the tag name and index name;

[0271] Step 4-02-03: The storage node executes a query in the specified index table to obtain rowid information;

[0272] Step 4-02-04: The storage sends the retrieved rowid to the driver;

[0273] This embodiment provides an optional solution for step 4-03, the specific steps of which are as follows:

[0274] Step 4-03-01: The driver parses the message sent by the storage node and obtains the rowid information;

[0275] Step 4-03-02: The driver parses the rowid information to obtain the storage node information where the current rowid resides;

[0276] Step 4-03-03: The driver obtains a connection to the specified storage node and sends a query request based on the rowid;

[0277] This embodiment provides an optional solution for step 3-04, the specific steps of which are as follows:

[0278] Step 4-04-01: The storage node parses the rowid information sent by the driver and obtains the corresponding arrayId information for that rowid in the current node;

[0279] Step 4-04-02: The storage node queries the vertex table for complete data information based on the vertex label information and arrayId;

[0280] Step 4-04-03: The storage node packages the complete data information obtained from the query and sends it to the driver;

[0281] Step 4-04-04: The driver receives the query results returned by the storage node, and the query is complete.

[0282] Example 2:

[0283] like Figure 19 The diagram shown is an architectural schematic of a storage device for a unique index of a distributed native graph database according to an embodiment of the present invention. The storage device for the unique index of a distributed native graph database in this embodiment includes one or more processors 31 and a memory 32. Figure 19 Take a processor 31 as an example.

[0284] Processor 31 and memory 32 can be connected via a bus or other means. Figure 19 Taking the example of a connection between China and Israel via a bus.

[0285] The memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the method for storing a unique index of a distributed native graph database in Embodiment 1. The processor 31 executes the method for storing a unique index of a distributed native graph database by running the non-volatile software program and instructions stored in the memory 32.

[0286] Memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 32 may optionally include memory remotely located relative to processor 31, which can be connected to processor 31 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0287] The program instructions / modules are stored in the memory 32. When executed by one or more processors 31, they perform the storage method for the unique index of the distributed native graph database in Embodiment 1 above, for example, performing the above-described method. Figure 1 , Figures 3-4 , Figures 10-11 , Figure 13 and Figure 15 The steps shown.

[0288] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0289] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0290] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for storing a unique index in a distributed native graph database, characterized in that, include: Based on the unique attribute columns of existing data in the current master table, a unique index is created for the existing data. This includes: data nodes calculating the partition where the existing data resides based on the unique attribute columns; extracting the unique identifier of the existing data from the partition based on the unique attribute columns; using the mapping relationship between the unique attribute columns and the corresponding unique identifiers as the unique index; determining the index partition information of multiple data nodes to which the unique index needs to be stored, based on the data volume and allocation, and according to the number of all data nodes involved in the unique index; establishing a local index table and a remote index table based on the index partition information; and writing the unique index to the local index table and the remote index table respectively. The system generates an index table. Based on the index partitioning information, a synchronization request is sent to the remote data node so that the remote data node can receive the synchronization request and synchronize the remote index table to achieve sharded storage of the unique index through the local index table and the remote index table. After storage is completed, status information is sent to the management node. When a data node finds that a unique index under creation violates the uniqueness constraint, it sends an error message to the management node. The management node receives the error message and notifies all data nodes involved in the unique index to delete all unique indexes under creation. The management node integrates all status information and updates the metadata information of the stored unique index according to all status information to complete the creation of the unique index. Insert incremental data into the current main table, and create a unique index for the incremental data based on its unique attribute column to update the current main table; The corresponding unique index is retrieved based on the index key of the data to be queried, and the data to be queried is retrieved from the current main table based on the retrieved unique index.

2. The method for storing a unique index in a distributed native graph database according to claim 1, characterized in that, Each data node synchronously maintains both local and remote index tables.

3. The method for storing a unique index in a distributed native graph database according to claim 1, characterized in that, The storage method for the unique index of the distributed native graph database further includes the following during the batch loading process of existing data: The data nodes for existing data are determined based on the unique identifiers of the existing data, and the data nodes for storing the corresponding unique indexes are determined based on the unique attribute columns of the existing data. Existing data and / or unique indexes that need to be sent to the same data node are grouped and encapsulated to obtain grouped data that need to be sent to each data node; Send the grouped data to the corresponding data nodes.

4. The method for storing a unique index in a distributed native graph database according to claim 1, characterized in that, The step of inserting incremental data into the current main table and creating a unique index for the incremental data based on its unique attribute column to update the current main table includes: Insert the incremental data into the current main table and obtain the unique identifier of the incremental data; The index partition to be inserted for the incremental data is calculated based on the unique attribute column of the incremental data; Based on the unique identifier, determine whether the unique index created in the partition to be inserted violates the uniqueness constraint; If the uniqueness constraint is violated, the unique index will not be created; If the uniqueness constraint is not violated, the unique attribute column is used as the index key, the mapping relationship between the index key and the corresponding unique identifier is used as the unique index, and the unique index is inserted into the index partition to be inserted.

5. The method for storing a unique index in a distributed native graph database according to claim 4, characterized in that, The step of inserting the incremental data into the current main table and obtaining the unique identifier of the incremental data includes: Perform a bitwise AND operation on the set ID of the graph where the incremental data is located and a preset AND operation parameter to obtain the preset sequence value of the set ID; Perform a bitwise OR operation between the node partition ID where the incremental data is located and a preset OR operation parameter to obtain the initial partition result; clear the data in the initial partition result except for the preset partition position to obtain the intermediate partition result; move the data in the intermediate partition result with the preset partition position to the higher bit to obtain the preset partition value; Perform a bitwise OR operation on the data type of the incremental data to obtain an initial type result; clear the data in the initial type result except for the preset type bit to obtain an intermediate type result; shift the data of the preset type bit in the intermediate type result to the higher bit to obtain the preset type value; The unique identifier is obtained based on the preset sequence value, the preset partition value, and the preset type value.

6. The method for storing a unique index in a distributed native graph database according to claim 5, characterized in that, The process of obtaining the unique identifier based on the preset sequence value, the preset partition value, and the preset type value includes: When the sum of the number of bits of the preset sequence value, the preset partition value, and the preset type value is greater than the number of bits of the unique identifier, the number of bits of the preset sequence value remains unchanged, and the number of bits of the preset partition value and / or the number of bits of the preset type value are reduced according to a preset adjustment strategy, so that the sum of the number of bits of the preset sequence value, the preset partition value, and the preset type value is equal to the number of bits of the unique identifier, so as to obtain the unique identifier.

7. The method for storing a unique index in a distributed native graph database according to claim 1, characterized in that, The step of querying the corresponding unique index based on the index key of the data to be queried, and then retrieving the data to be queried from the current main table based on the retrieved unique index, includes: Based on the index key of the data to be queried, obtain the local index table and / or the remote index table of the data to be queried; obtain the unique index based on the index key of the data to be queried; Based on the unique index, the unique identifier of the data to be queried is found in the local index table and / or the remote index table; Based on the unique identifier, the data node where the data to be queried is located is obtained, and the data to be queried is retrieved from the data node.

8. A storage device for a unique index of a distributed native graph database, characterized in that, It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to implement the storage method for the unique index of the distributed native graph database as described in any one of claims 1-7.

Images