Graph storage method, device and system based on community structure and electronic equipment

Abstract

The invention provides a graph storage method, device and system based on a community structure and electronic equipment, and relates to the technical field of graph storage, and the method comprises the following steps: obtaining to-be-stored graph data input by a user; storing the to-be-stored graph data into the memory graph, and freezing the memory graph which reaches the capacity upper limit after storing the to-be-stored graph data into an immutable memory graph; based on the L0 community table, determining a target community to which each target node in the immutable memory graph belongs; the L0 community table is used for representing a mapping relation between all nodes or all edges in the target graph network and the community to which the nodes or the edges belong; each community in a bottom-layer community structure in the target map network is stored by adopting a community bucket, and each group in a non-bottom-layer community structure is stored by adopting a tree index bucket; and based on the target community to which each target node belongs, storing each target node into a target community bucket corresponding to the target community to which the target node belongs. Reasonable hierarchical storage of the graph network can be realized, and the acquisition difficulty of the community structure of the graph is reduced.

Description

Technical Field

[0001] The present invention relates to the field of graph storage technology, and in particular to a community structure-based graph storage method, device, system, and electronic device. Background Art

[0002] In social network analysis, graph storage supports efficient graph query processing and analysis, and is a key research issue in this field. To efficiently analyze large-scale graph data, researchers have designed a variety of disk-based graph storage systems. Disk-based graph storage systems generally include out-of-core graph processing systems (OCPs) and graph database management systems (GDBMSs). OCPs support a variety of graph traversal algorithms, such as breadth-first search (BFS) and connected component (CC) algorithms. OCPs include GraphChi, GridGraph, and DGraph. GraphChi uses a parallel sliding window (PSW) approach to partition the graph and stores each shard in the compressed sparse row (CSR) format. GridGraph uses a grid-based partitioning approach, dividing each edge in the graph into blocks based on their starting and ending points. Each block is also stored in the CSR format. DGraph uses graph-based strongly connected components (SCCs) for storage and computation, significantly improving graph processing efficiency. Graph database management systems (DBMSs) are used to persistently store graph data. Graph DBMSs, such as Neo4j and JanusGraph, support a variety of graph analysis and query tasks by providing powerful graph query languages ​​(such as Cypher provided by Neo4j).

[0003] Community structure is a prominent substructure in social networks and one of the most interesting substructures for researchers. For example, algorithms such as subgraph matching and node centrality can be optimized using community structure. However, existing graph storage systems employ inefficient graph storage methods, making it difficult to extract community structure from graphs. Therefore, finding a reasonable graph storage method to extract community structure from graphs is an urgent problem. Summary of the Invention

[0004] The present invention provides a community structure-based graph storage method, device, system and electronic device to solve the defect in the prior art that it is difficult to obtain the community structure of a graph due to an unreasonable graph storage method of the graph storage system.

[0005] The present invention provides a community structure-based graph storage method, which includes the following steps.

[0006] Get the graph data to be stored input by the user.

[0007] The graph data to be stored is stored in a memory graph, and the memory graph that has stored the graph data to be stored and has reached an upper capacity limit is frozen as an immutable memory graph.

[0008] Based on the L0 community table, the target community to which each target node in the immutable memory graph belongs is determined; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; the target graph network includes at least two layers of community structure, and the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored using a community bucket, and each group in the non-lower layer community structure is stored using a tree index bucket.

[0009] Based on the target community to which each target node belongs, each target node is stored in a target community bucket corresponding to the target community to which it belongs; the target community bucket is in the underlying community structure.

[0010] According to the community structure-based graph storage method provided by the present invention, the target community bucket includes a target key-value pair community block group and a target CSR community block group; The storing each target node in a target community bucket corresponding to the target community to which each target node belongs based on the target community to which each target node belongs includes: For each of the target nodes, a jump table index is performed based on the node identifier of the target node, and the target key-value pair community block corresponding to the target node is determined in the target key-value pair community block group corresponding to the target community; the target node is stored in the target key-value pair community block; and the updated target key-value pair community block group is merged into the target CSR community block group.

[0011] According to the community structure-based graph storage method provided by the present invention, after the underlying community structure is updated, the method further includes: For each community in the underlying community structure, when the community changes in the community meet the target community behavior, the target community behavior is executed on the community to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0012] According to the community structure-based graph storage method provided by the present invention, the target community behavior includes at least one of a moving behavior, a splitting behavior, and a merging behavior; Determining that community changes in the community satisfy target community behaviors includes: When the target community behavior is a movement behavior, determining a first edge number of edges connected to the change node in the community and a second edge number of edges connected to the change node in at least one adjacent community; if at least one second edge number is greater than the first edge number, determining that the community change in the community satisfies the movement behavior; When the target community behavior is a split behavior, determining a pre-division value and a first modularization degree corresponding to the community, wherein the first modularization degree is used to represent the rationality of the layout of the community divided into the pre-division value subcommunities; when the first modularization degree is greater than a first preset threshold, determining that the community change in the community satisfies the split behavior; When the target community behavior is a merging behavior, a modularity increment between the community and a target adjacent community is determined, where the modularity increment is used to characterize the rationality of the layout after the target adjacent community and the community are merged. When the modularity increment is greater than a second preset threshold, it is determined that the community change in the community satisfies the merging behavior.

[0013] According to the community structure-based graph storage method provided by the present invention, executing the target community behavior on the community to obtain the adjusted community includes: In a case where the target community behavior is a moving behavior, determining the adjacent community corresponding to the largest second number of edges as the target moving community, and moving the changed node from the community to the target moving community to obtain the adjusted community; In the case where the target community behavior is a split behavior, dividing the community into the pre-divided numerical number of subcommunities; In the case where the target community behavior is a merging behavior, the community is merged with the target adjacent community to obtain the merged community.

[0014] The present invention also provides a community structure-based graph storage device, comprising the following modules.

[0015] The acquisition module is used to obtain the graph data to be stored input by the user.

[0016] The first storage module is configured to store the graph data to be stored in a memory graph, and freeze the memory graph after storing the graph data to be stored and reaching an upper capacity limit as an immutable memory graph.

[0017] A determination module is configured to determine, based on an L0 community table, the target community to which each target node in the immutable memory graph belongs; the L0 community table is configured to represent the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; the target graph network includes at least two layers of community structure, the upper layer community structure is configured to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored using a community bucket, and each group in the non-lower layer community structure is stored using a tree index bucket.

[0018] The second storage module is configured to store each target node in a target community bucket corresponding to the target community to which each target node belongs, based on the target community to which each target node belongs; the target community bucket is in the underlying community structure.

[0019] The present invention also provides a community structure-based graph storage system, comprising: The query engine is used to receive the graph data to be stored from the user and perform the graph analysis tasks required by the user; A transaction manager, connected to the query engine, configured to receive a transaction request including graph data to be stored, sent by the query engine; An LSM community storage system, wherein the LSM community storage system is connected to the transaction manager and is used to execute any of the community structure-based graph storage methods described above.

[0020] According to the community structure-based graph storage system provided by the present invention, the LSM community storage system includes: a community tree storage unit and a community maintenance unit, wherein: The community tree storage unit is used to store each node in the graph data to be stored into a target community bucket corresponding to the target community to which it belongs; the target community bucket is a storage structure corresponding to the underlying community structure; The community maintenance unit is used to, after the underlying community structure is updated, execute the target community behavior on each community in the underlying community structure if the community changes in the community meet the target community behavior, so as to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0021] According to the community-structured graph storage system provided by the present invention, the LSM community storage system further includes an attribute storage unit and a manifest file, wherein: The attribute storage unit is used to store all nodes in the target graph network and some neighbor nodes corresponding to each node; The manifest file is used to store the target community behavior executed by the community maintenance unit on each community to perform fault recovery.

[0022] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the community structure-based graph storage method described above is implemented.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the community structure-based graph storage methods described above.

[0024] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the community structure-based graph storage methods described above.

[0025] The community-structure-based graph storage method, device, system, and electronic device provided by the present invention first stores the graph data to be stored input by the user into a memory graph. After the memory graph is full, all the data to be stored in the memory graph is frozen into an immutable memory graph. Based on the mapping relationship between each node and the community to which it belongs in the target graph network in the L0 community table, the target community to which each target node in the data to be stored in the immutable memory graph belongs is determined. Based on the target community, the target community bucket corresponding to the target community is determined in the community bucket corresponding to each community in the underlying community structure storing the target graph network, and each target node is stored in the corresponding target community bucket. The target graph network includes a multi-layer community structure, the upper-layer community structure is used to represent the association relationship between multiple groups in the lower-layer community structure, each community in the underlying community structure is stored using a community bucket, and each group in the non-underlying community structure is stored using a tree index bucket. In the present invention, the target graph network is divided according to the multi-layer community structure, and the multi-layer community structure is stored in layers. The graph data to be stored is stored in the corresponding target community bucket in the underlying community structure, making full use of the spatial locality of the graph network to achieve rational hierarchical storage of the graph network. At the same time, graph storage based on the multi-layer community structure can reduce the difficulty of obtaining the community structure of the graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 This is a flowchart of a community structure-based graph storage method provided by an embodiment of the present invention.

[0028] Figure 2Schematic diagram of a target graph network provided by an embodiment of the present invention.

[0029] Figure 3 It is a schematic diagram of a community tree storage unit corresponding to a target graph network provided in an embodiment of the present invention.

[0030] Figure 4 It is a structural diagram of the community bucket provided by an embodiment of the present invention.

[0031] Figure 5 It is a structural diagram of the tree index bucket in the L1 layer provided by an embodiment of the present invention.

[0032] Figure 6 is a schematic diagram of movement behavior provided by an embodiment of the present invention.

[0033] Figure 7 Schematic diagram of the splitting behavior provided by an embodiment of the present invention.

[0034] Figure 8 It is a schematic diagram of the merging behavior provided by an embodiment of the present invention.

[0035] Figure 9 4 is a schematic structural diagram of a community structure-based graph storage device provided by an embodiment of the present invention.

[0036] Figure 10 3 is a structural diagram of a community structure-based graph storage system provided by an embodiment of the present invention.

[0037] Figure 11 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0039] Existing technologies, such as GraphChi and GridGraph, partition graphs solely based on the range of node IDs (identity documents), failing to fully consider the graph's topological information. Furthermore, a node's neighbors may be stored in non-adjacent physical spaces. This generates a significant amount of extra I / O (input / output) operations when retrieving neighbor node information, impacting the performance of the graph storage system. Another example is DGraph, which stores graphs based on strongly connected components (SCCs), storing nodes within the same SCC in adjacent physical spaces. While this approach leverages the graph's spatial locality, improving cache hit rates and reducing unnecessary I / O operations, SCCs are unstable substructures. Even small graph updates can significantly alter the SCC structure, requiring significant computational resources and time to maintain. As can be seen from the above, existing graph storage methods are irrational and fail to consider community structure, making it difficult to retrieve the graph's community structure.

[0040] In order to solve the problem in the prior art that it is difficult to obtain the community structure of a graph due to an unreasonable graph storage method of the graph storage system, an embodiment of the present invention provides a graph storage method based on a community structure, and the graph storage method based on a community structure is applied to a graph storage system based on a community structure, hereinafter collectively referred to as a graph storage system. Figure 1 This is a flow chart of a community structure-based graph storage method provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps 110 to 140.

[0041] Step 110: Obtain the graph data to be stored input by the user.

[0042] Specifically, after the user performs an input operation on the system interface of the graph storage system, the graph storage system can respond to the user's input operation and obtain the graph data to be stored input by the user. The graph data to be stored may be newly added graph data. For example, the graph data to be stored may include newly added nodes, or newly added edges between existing nodes in the target graph network. The nodes and edges are both expressed in a key-value pair format, where: the key of the node represents the node identifier (i.e., the node ID), and the value of the node represents the attribute of the node, for example, the label of the node. The key of the edge represents the starting node and the ending node of the edge, and the value of the edge represents the weight. In the graph data to be stored, the newly added graph data indicates that the user needs to perform a write operation on the target graph network.

[0043] In addition, the graph data to be stored can also be graph data to be queried or graph data to be deleted. Graph data to be deleted indicates that the user needs to perform a delete operation on the target graph network, and graph data to be queried indicates that the user needs to perform a read operation on the target graph network.

[0044] It should be noted that the target graph network is a community tree constructed after importing the full graph data. The community tree is a hierarchical structure representation. Before executing step 110, a community tree storage unit of the target graph network has been constructed based on LSM-Tree. The community tree storage unit is used to represent the storage structure of the target graph network on the disk. A key-value pair is written to the community tree storage unit. The key-value pair can be persistently stored on the disk, where both the key and the value are represented in the form of a byte array. In addition, the corresponding value can be read from the community tree storage unit according to the key.

[0045] Step 120: Store the graph data to be stored in a memory graph, and freeze the memory graph after storing the graph data to be stored and reaching an upper capacity limit as an immutable memory graph.

[0046] For example, taking the newly added graph data as the graph data to be stored, after obtaining the graph data to be stored, due to the faster read and write speed of the memory, the graph data to be stored is first written to the memory graph rather than directly stored on the disk. Since the memory graph has a small capacity and is not persistent, after the capacity of the memory graph reaches its upper limit, the memory graph is frozen and converted into an immutable memory graph, which facilitates the subsequent writing of the immutable memory graph group into the community tree storage unit corresponding to the target graph network.

[0047] It should be noted that, after the memory graph is frozen and converted into an immutable memory graph, a new memory graph can be created in the memory to facilitate the storage of graph data to be stored that is subsequently input by the user.

[0048] In addition, when the graph data to be stored is graph data to be queried or graph data to be deleted, the graph data in the memory graph can be read, and then the graph data in the immutable memory graph can be read.

[0049] Step 130: Determine the target community to which each target node in the immutable memory graph belongs based on the L0 community table; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; the target graph network includes at least two layers of community structure, and the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored using a community bucket, and each group in the non-lower layer community structure is stored using a tree index bucket.

[0050] Specifically, the L0 community table is obtained synchronously when constructing the community tree storage unit of the target graph network. The L0 community table includes the mapping relationship between all nodes in the target graph network and their corresponding communities. Afterwards, each target node in the immutable memory graph can be traversed, and the L0 community table can be searched for nodes that match each target node. The community to which each matching node belongs is determined as the target community to which each target node belongs.

[0051] It should be noted that the target graph network is stored based on a multi-layer community structure. This includes at least two layers of community structures. Leaf nodes in the bottom layer represent all communities in the target graph network, while parent nodes in the upper layer recursively group groups in the lower layer. This multi-layer community structure reflects the multi-level granularity of the community structure in the target graph network.

[0052] For example, Figure 2 is a schematic diagram of a target graph network provided by an embodiment of the present invention, such as Figure 2 As shown, the community tree corresponding to the target graph network includes three layers of community structure, namely L0, L1 and L2. Through the community discovery algorithm, all nodes in the target graph network are first divided into four communities in the L0 layer (i.e., the underlying community structure) of the community tree. The nodes within each community are closely connected, but there are fewer connections between communities. The two groups in the L1 layer are obtained by aggregating the four communities in the L0 layer, and are used to characterize the association relationship between the communities in the L0 layer, and each group in the L1 layer points to the corresponding community in the L0 layer. One group in the L2 layer is obtained by aggregating the two groups in the L1 layer, and is used to characterize the association relationship between the groups in the L1 layer, and each group in the L2 layer points to each group in the L1 layer.

[0053] It's important to note that this community discovery algorithm is a network analysis technique used to identify communities within a target graph network that have dense internal connections but few external connections. These communities typically represent collections of nodes with similar characteristics or functions and are widely used in fields such as social networks, biological networks, and information dissemination. By dividing a target graph network into meaningful communities, its internal structure can be revealed, facilitating understanding of the organizational principles and functional properties of complex networks.

[0054] Figure 3 Schematic diagram of a community tree storage unit corresponding to a target graph network provided by an embodiment of the present invention, such as Figure 3 As shown in the figure, after constructing the community tree corresponding to the target graph network, for each community in the L0 layer, each community is stored on the disk using a community bucket (CB), and for each group in the L1 and L2 layers, each group is stored on the disk using a tree index bucket (TB).

[0055] Figure 4 This is a schematic diagram of the structure of the community bucket provided by an embodiment of the present invention. Figure 4 As shown, the community bucket includes a key-value pair community block group, a CSR community block group and a skip list.

[0056] (1) The key-value community block group is used to quickly store recently updated graph data, and the key-value community block is inserted in the key-value format and has a fast insertion speed. The key-value community block group includes multiple key-value community blocks (KV), each of which stores node update information within the corresponding node ID range. Each key-value community block includes header information and at least one graph key unit, wherein the header information includes a block number, a maximum ID, a minimum ID, and a usage space, wherein the block number represents the number corresponding to the key-value community block, the maximum ID represents the maximum node ID of all nodes included in the key-value community block, the minimum ID represents the minimum node ID of all nodes included in the key-value community block, and the usage space represents the size of the data space occupied by the key-value community block. Based on the usage space, it can be determined whether the key-value community block group is merged into the CSR community block group. For example, a key-value community block with a node ID range of [40,60) can include three nodes with node IDs of 40, 45, and 55 respectively. The block number for a community block with a key-value pair in the node ID range [40, 60] can be 2, with a maximum ID of 55 and a minimum ID of 40, using 8 bytes. Each graph key-value unit includes the node sub-information corresponding to each node and the node IDs of some neighboring nodes. This node sub-information includes the node ID, timestamp, direction of the connected edges, and deletion flags, facilitating subsequent concurrency control.

[0057] (2) The CSR community block group is used to store early graph data, and the insertion speed is slow. The CSR community block includes multiple CSR community blocks (CSRs), and each CSR community block stores early node data within the corresponding node ID range. Each CSR community block includes header information, node number, node array, and neighbor array. The header information in the CSR community block has the same meaning as the header information in the key-value pair community block, and the embodiment of the present invention will not be repeated here. The number of nodes represents the total number of nodes included in the CSR community block. The node array includes the node ID and the offset from the node to the neighbor node in the edge array. The edge array stores the neighbor nodes corresponding to each node. The neighbor array includes the node sub-information corresponding to each neighbor node. The node sub-information corresponding to the neighbor node is similar to the node sub-information of each node included in each graph key-value unit. The embodiment of the present invention will not be repeated here.

[0058] (3) Skip list is a multi-layer ordered linked list. Through the hierarchical structure, the query rate can be accelerated by querying in order from the upper layer to the lower layer.

[0059] Figure 5 Schematic diagram of the structure of the tree index bucket in the L1 layer provided by the embodiment of the present invention. Figure 5 As shown, the tree index bucket includes multiple community tree index blocks and jump tables, wherein: the community tree index block is used to store the basic information of each community within the community ID range in the L1 layer. It includes header information, the number of communities, the number of adjacent communities, a community array, a neighbor community array, and a bridge array, wherein: the header information has the same meaning as the header information in the key-value pair community block, and the embodiments of the present invention will not be repeated here. The number of communities indicates the number of communities included in the group corresponding to the tree index bucket. The number of adjacent communities indicates the number of communities of the group in the neighbor community array, with Figure 2 Groups shown For example, the group Internal and Community Connected communities include communities ,Community and community , the group Internal and Community Connected communities include communities , therefore, the group The corresponding number of adjacent communities is 4. The community array includes the community ID and the offset from the community in the group to the neighbor community in the neighbor community array. The neighbor community array stores the neighbor community ID and offset corresponding to each community in the group. The offset here refers to the cross-community edge between each community and the neighbor community. The bridge data includes the starting node and ending node at both ends of the cross-community edge corresponding to each offset in the neighbor community array. Similar to the community bucket, each community tree index block is indexed by a skip table, but the skip table in the community tree index block records the community ID.

[0060] Step 140: Based on the target community to which each target node belongs, each target node is stored in a target community bucket corresponding to the target community to which it belongs; the target community bucket is a storage structure corresponding to the underlying community structure.

[0061] Specifically, after determining the target community corresponding to each target node, further determine the target community bucket of the target community in the underlying community structure, first store each target node in the key-value pair community block corresponding to the target community bucket, and then merge it into the corresponding CSR community block.

[0062] The community structure-based graph storage method provided by the present invention first stores the graph data to be stored input by the user into a memory graph, freezes all the data to be stored in the memory graph into an immutable memory graph after the memory graph is full, determines the target community to which each target node in the data to be stored in the immutable memory graph belongs based on the mapping relationship between each node and the community to which it belongs in the target graph network in the L0 community table, and determines the target community bucket corresponding to the target community in the community bucket corresponding to each community in the underlying community structure storing the target graph network based on the target community, and stores each target node in the corresponding target community bucket. The target graph network includes a multi-layer community structure, the upper-layer community structure is used to characterize the association relationship between multiple groups in the lower-layer community structure, and the nodes in each community in the lower-layer community structure are closely connected. Each community in the lower-layer community structure is stored using a community bucket, and each group in the non-lower-layer community structure is stored using a tree index bucket. In the present invention, the target graph network is divided according to the multi-layer community structure, and the multi-layer community structure is stored in layers. The graph data to be stored is stored in the corresponding target community bucket in the underlying community structure, making full use of the spatial locality of the graph network to achieve rational hierarchical storage of the graph network. At the same time, graph storage based on the multi-layer community structure can reduce the difficulty of obtaining the community structure of the graph.

[0063] Furthermore, based on the target community to which each target node belongs, storing each target node in a target community bucket corresponding to the target community includes: For each of the target nodes, a jump table index is performed based on the node identifier of the target node, and the target key-value pair community block corresponding to the target node is determined in the target key-value pair community block group corresponding to the target community; the target node is stored in the target key-value pair community block; and the updated target key-value pair community block group is merged into the target CSR community block group.

[0064] For example, the node ID corresponding to the target node is 50, and the node needs to be added to the community tree storage unit corresponding to the target graph network, such as Figure 4As shown, the target key-value pair community block group corresponding to the target node includes key-value pair community blocks with node ID ranges [0, 40), [40, 60), [60, 70), [70, 80), and [80, ∞). Using the node ID, the node ID of the target node is indexed sequentially in the skip list from the top layer to the bottom layer to determine which node ID range the target node belongs to. In the first layer, the pointer in the skip list points from 0 to ∞. Since 0<50<∞, node 50 is in the range [0, ∞). Subsequently, in the second layer, the pointer in the skip list points from 0 to 80. Since 0<50<80, node 50 is in the range [0, 80). In the third layer, the pointer in the skip list points from 0 to 60. Since 0<50<60, node 50 is in the range [0, 60). In the fourth layer, the pointer in the skip list first points from 0 to 40. Since 50>40, node 50 is not in the range [0, 40). Afterwards, the pointer in the skip list points from 40 to 60, with 40 < 50 < 60. Therefore, node 50 is in the range [40, 60). That is, the target node should first be written into the target key-value community block with a node ID in the range [40, 60) in the target community bucket. If the used space of the target key-value community block exceeds a preset space threshold, the target key-value community block can be merged into the target CSR community block corresponding to the node ID range.

[0065] In the embodiment of the present invention, the target key-value pair community block corresponding to the target node can be quickly queried through the skip table index, and the query rate is relatively fast.

[0066] Furthermore, after the underlying community structure is updated, the method further includes: For each community in the underlying community structure, when the community changes in the community meet the target community behavior, the target community behavior is executed on the community to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0067] Specifically, in the prior art, when a large-scale update occurs in a graph network, for example, when a large number of nodes and edges are inserted and deleted, the community structure may also change accordingly. Existing community discovery algorithms all consider the calculation of community structure from a global perspective, which means that they must obtain full graph information in the calculation process. When the scale of the stored graph is large, a large amount of I / O overhead will be introduced. In addition, this "calculation from scratch" model does not fully utilize the community information at historical moments. Therefore, in an embodiment of the present invention, after the underlying community structure is updated, the community changes of each community are characterized with each community in the underlying community structure as the center, and it is determined whether the community changes of each community meet the target community behavior. If so, the target community behavior is executed on the community, and the division of the adjusted community is more appropriate and the storage layout is more reasonable.

[0068] In an embodiment of the present invention, in the process of processing graph structure updates, without loading the entire graph, the community structure of the target graph network stored on the disk is dynamically and efficiently maintained with the community as the center, avoiding the degradation of the community quality of the stored graph network as the community structure is updated, and without introducing a large amount of I / O overhead.

[0069] Furthermore, the target community behavior includes at least one of a moving behavior, a splitting behavior, and a merging behavior; Determining that community changes in the community satisfy target community behaviors includes: When the target community behavior is a movement behavior, determining a first edge number of edges connected to the change node in the community and a second edge number of edges connected to the change node in at least one adjacent community; if at least one second edge number is greater than the first edge number, determining that the community change in the community satisfies the movement behavior; When the target community behavior is a split behavior, determining a pre-division value and a first modularization degree corresponding to the community, wherein the first modularization degree is used to represent the rationality of the layout of the community divided into the pre-division value subcommunities; when the first modularization degree is greater than a first preset threshold, determining that the community change in the community satisfies the split behavior; When the target community behavior is a merging behavior, a modularity increment between the community and a target adjacent community is determined, where the modularity increment is used to characterize the rationality of the layout after the target adjacent community and the community are merged. When the modularity increment is greater than a second preset threshold, it is determined that the community change in the community satisfies the merging behavior.

[0070] Furthermore, executing the target community behavior on the community to obtain the adjusted community includes: In a case where the target community behavior is a moving behavior, determining the adjacent community corresponding to the largest second number of edges as the target moving community, and moving the changed node from the community to the target moving community to obtain the adjusted community; In the case where the target community behavior is a split behavior, dividing the community into the pre-divided numerical number of subcommunities; In the case where the target community behavior is a merging behavior, the community is merged with the target adjacent community to obtain the merged community.

[0071] For example, Figure 6 is a schematic diagram of the mobile behavior provided by the embodiment of the present invention, such as Figure 6As shown in the left part of , before the graph is updated, node V1 belongs to community C1. The graph is updated by deleting the edge between node V1 and node V2, and the edge between node V1 and node V3 in community C1, and adding the edge between node V1 and node V5, and the edge between node V1 and node V6 in community C2, that is, the changed node in community C1 is node V1. After the graph is updated, only node V4 is connected to node V1 in community C1, that is, the first edge number of the edge connected to node V1 in community C1 is 1, and in community C2, node V5, node V6 and node V7 are all connected to node V1, that is, the second edge number of the edge connected to node V1 in community C2 is 3. Since the second edge number is greater than the first edge number, the changed node V1 in community C1 satisfies the moving behavior. At this time, community C2 can be determined as the target moving community corresponding to node V1, and node V1 is moved (Move) from community C1 to community C2, so as to obtain the result after executing the moving behavior. Figure 6 The communities shown on the right and community .

[0072] It should be noted that if new edges are added between node V1 and nodes in multiple adjacent communities during the graph update process, the number of second edges between node V1 and each adjacent community can be determined, and the multiple second edge numbers can be sorted in descending order. The adjacent community corresponding to the second edge number with the highest sorting value is determined as the target mobile community, and then node V1 can be moved to the target mobile community.

[0073] For example, taking the first preset threshold as 0.6, Figure 7 is a schematic diagram of the splitting behavior provided by an embodiment of the present invention, such as Figure 7 As shown in the left part of the figure, before the graph update, community C1 consists of 8 nodes. The graph update deletes the edge between nodes V2 and V6, and the edge between nodes V4 and V7 in community C1. Afterwards, the Louvain algorithm is used to calculate the pre-division value corresponding to community C1, which is 2, and the first modularity is 0.65. At this time, since 0.65>0.6, it shows that the layout after dividing community C1 into two sub-communities is more reasonable. It can be determined that community C1 meets the splitting behavior and can be split into Figure 7 The communities shown on the right and community .

[0074] It should be noted that the first degree of modularity is used to measure the expected value of the internal connection density of the community in the graph network relative to the random distribution, that is, the first degree of modularity is used to measure the quality of community division, and the higher the first degree of modularity, the better the quality of community division.

[0075] For example, taking the second preset threshold as 0, Figure 8 is a schematic diagram of the merging behavior provided by an embodiment of the present invention, such as Figure 8 As shown in the left part of , before the graph is updated, it includes two communities C1 and C2, which are updated to add edges between nodes V5 and V2, between nodes V1 and V2, between nodes V1 and V3, and between nodes V7 and V4. Afterwards, the modularity increment between communities C1 and C2 is calculated to be 0.3, that is, modularity or Newman-Girvan Q. At this time, since 0.3>0, it indicates that after the merger of communities C1 and C2, a higher quality community structure can be obtained. Therefore, it can be determined that communities C1 and C2 meet the merging behavior. At this time, communities C1 and C2 can be merged (Merge) as Figure 8 The communities shown on the right .

[0076] In addition, after the graph is updated, it can be determined whether the information school student attending the international conference special fund application form moves, or if the community change of the community does not meet the movement behavior, it can be determined whether the community meets the split behavior. If so, the split behavior is performed on the community. If the community change of the community does not meet the split behavior, or after the split behavior is performed on the community, it can be determined whether the community meets the merge behavior. If so, the merge behavior is performed on the community. If not, the status quo of the community is maintained.

[0077] The community structure-based graph storage device provided by the present invention is described below. The community structure-based graph storage device described below and the community structure-based graph storage method described above can refer to each other.

[0078] The embodiment of the present invention also provides a graph storage device based on community structure, Figure 9 is a structural diagram of a community-structured graph storage device provided by an embodiment of the present invention. Figure 9 As shown, the community structure-based graph storage device 900 includes: an acquisition module 910 , a first storage module 920 , a determination module 930 and a second storage module 940 .

[0079] The acquisition module 910 is used to acquire the graph data to be stored input by the user.

[0080] The first storage module 920 is configured to store the graph data to be stored in a memory graph, and freeze the memory graph after storing the graph data to be stored and after the memory graph is fully stored with the graph data to be stored as an immutable memory graph.

[0081] Determination module 930 is used to determine the target community to which each target node in the immutable memory graph belongs based on the L0 community table; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; the target graph network includes at least two layers of community structure, and the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored using a community bucket, and each group in the non-lower layer community structure is stored using a tree index bucket.

[0082] The second storage module 940 is configured to store each target node in a target community bucket corresponding to the target community to which each target node belongs, based on the target community to which each target node belongs; the target community bucket is in the underlying community structure.

[0083] The community-structured graph storage device provided by the present invention stores the graph data to be stored input by the user into a memory graph first. After the memory graph is fully stored, all the data to be stored in the memory graph is frozen into an immutable memory graph. Based on the mapping relationship between each node and the community to which it belongs in the target graph network in the L0 community table, the target community to which each target node in the data to be stored in the immutable memory graph belongs is determined. Based on the target community, the target community bucket corresponding to the target community is determined in the community bucket corresponding to each community in the underlying community structure storing the target graph network, and each target node is stored in the corresponding target community bucket. The target graph network includes a multi-layer community structure, the upper-layer community structure is used to represent the association relationship between multiple groups in the lower-layer community structure, and the nodes in each community in the lower-layer community structure are closely connected. Each community in the lower-layer community structure is stored using a community bucket, and each group in the non-lower-layer community structure is stored using a tree index bucket. In the present invention, the target graph network is divided according to the multi-layer community structure, and the multi-layer community structure is stored in layers. The graph data to be stored is stored in the corresponding target community bucket in the underlying community structure, making full use of the spatial locality of the graph network to achieve rational hierarchical storage of the graph network. At the same time, graph storage based on the multi-layer community structure can reduce the difficulty of obtaining the community structure of the graph.

[0084] Optionally, the target community bucket includes a target key-value pair community block group and a target CSR community block group.

[0085] Optionally, the second storage module 940 is specifically configured to: For each of the target nodes, a jump table index is performed based on the node identifier of the target node, and the target key-value pair community block corresponding to the target node is determined in the target key-value pair community block group corresponding to the target community; the target node is stored in the target key-value pair community block; and the updated target key-value pair community block group is merged into the target CSR community block group.

[0086] Optionally, the community structure-based graph storage device 900 further includes a dynamic maintenance module, which is specifically configured to: For each community in the underlying community structure, when the community changes in the community meet the target community behavior, the target community behavior is executed on the community to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0087] Optionally, the target community behavior includes at least one of a moving behavior, a splitting behavior, and a merging behavior.

[0088] Optionally, the dynamic maintenance module is specifically configured to: When the target community behavior is a movement behavior, determining a first edge number of edges connected to the change node in the community and a second edge number of edges connected to the change node in at least one adjacent community; if at least one second edge number is greater than the first edge number, determining that the community change in the community satisfies the movement behavior; When the target community behavior is a split behavior, determining a pre-division value and a first modularization degree corresponding to the community, wherein the first modularization degree is used to represent the rationality of the layout of the community divided into the pre-division value subcommunities; when the first modularization degree is greater than a first preset threshold, determining that the community change in the community satisfies the split behavior; When the target community behavior is a merging behavior, a modularity increment between the community and a target adjacent community is determined, where the modularity increment is used to characterize the rationality of the layout after the target adjacent community and the community are merged. When the modularity increment is greater than a second preset threshold, it is determined that the community change in the community satisfies the merging behavior.

[0089] Optionally, the dynamic maintenance module is specifically configured to: In a case where the target community behavior is a moving behavior, determining the adjacent community corresponding to the largest second number of edges as the target moving community, and moving the changed node from the community to the target moving community to obtain the adjusted community; In the case where the target community behavior is a split behavior, dividing the community into the pre-divided numerical number of subcommunities; In the case where the target community behavior is a merging behavior, the community is merged with the target adjacent community to obtain the merged community.

[0090] The embodiment of the present invention also provides a graph storage system based on community structure, Figure 10 is a structural diagram of a community-based graph storage system provided by an embodiment of the present invention. Figure 10 As shown, the community-structured graph storage system includes: a query engine, a transaction manager, and an LSM community storage system.

[0091] The query engine is used to receive the graph data to be stored input by the user and perform the graph analysis tasks required by the user.

[0092] A transaction manager is connected to the query engine and is used to receive a transaction request sent by the query engine that includes graph data to be stored.

[0093] An LSM community storage system, wherein the LSM community storage system is connected to the transaction manager and is used to execute any of the community structure-based graph storage methods described above.

[0094] Specifically, the query engine includes a graph query processor and a graph update unit. The graph query processor is used to query all subgraphs that are isomorphic to the subgraph given by the user in the full graph data corresponding to the target graph network. The graph update unit serves as the system interface of the community-structured graph storage system and is used to interact with upper-level businesses or upper-level users. In addition, the query engine stores a variety of graph analysis algorithms, such as a depth-first search algorithm and a breadth-first search algorithm. The depth-first search algorithm and the breadth-first search algorithm are used to search for the shortest path of the unit to perform the graph analysis tasks required by the user. The transaction manager is used to ensure that read and write operations can maintain the atomicity, consistency and durability of transactions. The LSM community storage system is used to execute the above-mentioned community-structured graph storage method, and the embodiments of the present invention will not be repeated here.

[0095] Further, if Figure 10 As shown, the LSM community storage system includes: a community tree storage unit and a community maintenance unit, wherein: The community tree storage unit is used to store each node in the graph data to be stored into a target community bucket corresponding to the target community to which it belongs; the target community bucket is a storage structure corresponding to the underlying community structure; The community maintenance unit is used to, after the underlying community structure is updated, execute the target community behavior on each community in the underlying community structure if the community changes in the community meet the target community behavior, so as to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0096] The specific structure of the community tree storage unit will not be described in detail in this embodiment of the present invention.

[0097] Further, if Figure 10 As shown, the LSM community storage system also includes an attribute storage unit and a manifest file, wherein: The attribute storage unit is used to store all nodes in the target graph network and some neighbor nodes corresponding to each node; The manifest file is used to store the target community behavior executed by the community maintenance unit on each community to perform fault recovery.

[0098] Specifically, after the graph update unit receives the graph data to be stored from the user, it can write the graph data to the community tree storage unit and the attribute storage unit, respectively. The attribute storage unit includes all nodes in the target graph network and some of the neighboring nodes corresponding to each node. After the graph data to be stored is written to the community tree storage unit, after the existing community performs a graph update, the community maintenance unit dynamically maintains the corresponding community structure. At the same time, the target community behavior executed during dynamic maintenance is recorded in a manifest file to facilitate recovery in the event of a failure in the community executing the target community behavior.

[0099] Figure 11 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 11 As shown, the electronic device may include: a processor (processor) 1110 , a communication interface (Communications Interface) 1120 , a memory (memory) 1130 and a communication bus 1140 , wherein the processor 1110 , the communication interface 1120 , and the memory 1130 communicate with each other via the communication bus 1140 . The processor 1110 can call the logic instructions in the memory 1130 to execute a community structure-based graph storage method, which includes: obtaining graph data to be stored input by a user; storing the graph data to be stored in a memory graph, and freezing the memory graph after storing the graph data to be stored and reaching the capacity limit as an immutable memory graph; based on the L0 community table, determining the target community to which each target node in the immutable memory graph belongs; the L0 community table is used to characterize the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; the target graph network includes at least two layers of community structure, and the upper community structure is used to characterize the association relationship between multiple groups in the lower community structure; each community in the underlying community structure is stored in a community bucket, and each group in the non-bottom community structure is stored in a tree index bucket; based on the target community to which each target node belongs, each target node is stored in a target community bucket corresponding to the target community to which it belongs; the target community bucket is in the underlying community structure.

[0100] Furthermore, the logic instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0101] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the community structure-based graph storage method provided by the above methods, the method including: obtaining graph data to be stored input by a user; storing the graph data to be stored in a memory graph, and freezing the memory graph after storing the graph data to be stored and reaching the capacity limit as an immutable memory graph; determining the target community to which each target node in the immutable memory graph belongs based on an L0 community table; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; the target graph network includes at least two layers of community structure, the upper community structure is used to represent the association relationship between multiple groups in the lower community structure; each community in the underlying community structure is stored using a community bucket, and each group in the non-underlying community structure is stored using a tree index bucket; based on the target community to which each target node belongs, each target node is stored in a target community bucket corresponding to the target community to which it belongs; the target community bucket is in the underlying community structure.

[0102] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the community structure-based graph storage method provided by the above-mentioned methods, the method comprising: obtaining graph data to be stored input by a user; storing the graph data to be stored in a memory graph, and freezing the memory graph after storing the graph data to be stored and reaching the capacity limit as an immutable memory graph; determining the target community to which each target node in the immutable memory graph belongs based on the L0 community table; the L0 community table is used to characterize the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; the target graph network includes at least two layers of community structure, and the upper community structure is used to characterize the association relationship between multiple groups in the lower community structure; each community in the underlying community structure is stored in a community bucket, and each group in the non-bottom community structure is stored in a tree index bucket; based on the target community to which each target node belongs, each target node is stored in a target community bucket corresponding to the target community to which it belongs; the target community bucket is in the underlying community structure.

[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0104] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A graph storage method based on community structure, characterized in that: include: Get the graph data to be stored input by the user; storing the graph data to be stored in a memory graph, and freezing the memory graph after storing the graph data to be stored and reaching a capacity upper limit as an immutable memory graph; Determine, based on the L0 community table, the target community to which each target node in the immutable memory graph belongs; The L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; The target graph network includes at least two layers of community structures, the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored in a community bucket, and each group in the non-lower layer community structure is stored in a tree index bucket; Based on the target community to which each target node belongs, storing each target node in a target community bucket corresponding to the target community to which it belongs; The target community bucket is in the underlying community structure.

2. The community structure-based graph storage method according to claim 1, characterized in that: The target community bucket includes a target key-value pair community block group and a target CSR community block group; The storing each target node in a target community bucket corresponding to the target community to which each target node belongs based on the target community to which each target node belongs includes: For each of the target nodes, a jump table index is performed based on the node identifier of the target node, and the target key-value pair community block corresponding to the target node is determined in the target key-value pair community block group corresponding to the target community; the target node is stored in the target key-value pair community block; and the updated target key-value pair community block group is merged into the target CSR community block group.

3. The community structure-based graph storage method according to claim 1 or 2, characterized in that: After the underlying community structure is updated, the method further includes: For each community in the underlying community structure, when the community changes in the community meet the target community behavior, the target community behavior is executed on the community to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

4. The community structure-based graph storage method according to claim 3, characterized in that: The target community behavior includes at least one of a moving behavior, a splitting behavior, and a merging behavior; Determining that community changes in the community satisfy target community behaviors includes: When the target community behavior is a movement behavior, determining a first edge number of edges connected to the change node in the community and a second edge number of edges connected to the change node in at least one adjacent community; if at least one second edge number is greater than the first edge number, determining that the community change in the community satisfies the movement behavior; When the target community behavior is a split behavior, determining a pre-division value and a first modularization degree corresponding to the community, wherein the first modularization degree is used to represent the rationality of the layout of the community divided into the pre-division value subcommunities; when the first modularization degree is greater than a first preset threshold, determining that the community change in the community satisfies the split behavior; When the target community behavior is a merging behavior, a modularity increment between the community and a target adjacent community is determined, where the modularity increment is used to characterize the rationality of the layout after the target adjacent community and the community are merged. When the modularity increment is greater than a second preset threshold, it is determined that the community change in the community satisfies the merging behavior.

5. The community structure-based graph storage method according to claim 4, characterized in that: The performing the target community behavior on the community to obtain the adjusted community includes: In a case where the target community behavior is a moving behavior, determining the adjacent community corresponding to the largest second number of edges as the target moving community, and moving the changed node from the community to the target moving community to obtain the adjusted community; In the case where the target community behavior is a split behavior, dividing the community into the pre-divided numerical number of subcommunities; In the case where the target community behavior is a merging behavior, the community is merged with the target adjacent community to obtain the merged community.

6. A graph storage device based on community structure, characterized in that: include: The acquisition module is used to obtain the graph data to be stored input by the user; A first storage module is configured to store the graph data to be stored in a memory map, and freeze the memory map after storing the graph data to be stored and reaching a capacity upper limit as an immutable memory map; A determination module, configured to determine, based on the L0 community table, a target community to which each target node in the immutable memory graph belongs; The L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and the communities to which they belong; The target graph network includes at least two layers of community structures, the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored in a community bucket, and each group in the non-lower layer community structure is stored in a tree index bucket; A second storage module is configured to store each target node in a target community bucket corresponding to the target community to which each target node belongs based on the target community to which each target node belongs; The target community bucket is in the underlying community structure.

7. A community-structured graph storage system, characterized in that: include: The query engine is used to receive the graph data to be stored from the user and perform the graph analysis tasks required by the user; A transaction manager, connected to the query engine, configured to receive a transaction request including graph data to be stored, sent by the query engine; An LSM community storage system, wherein the LSM community storage system is connected to the transaction manager and is used to execute the community structure-based graph storage method according to any one of claims 1 to 5.

8. The community structure-based graph storage system according to claim 7, characterized in that: The LSM community storage system includes: a community tree storage unit and a community maintenance unit, wherein: The community tree storage unit is used to store each node in the graph data to be stored into a target community bucket corresponding to the target community to which it belongs; the target community bucket is a storage structure corresponding to the underlying community structure; The community maintenance unit is used to, after the underlying community structure is updated, execute the target community behavior on each community in the underlying community structure if the community changes in the community meet the target community behavior, so as to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

9. The community structure-based graph storage system according to claim 8, characterized in that: The LSM community storage system further includes an attribute storage unit and a manifest file, wherein: The attribute storage unit is used to store all nodes in the target graph network and some neighbor nodes corresponding to each node; The manifest file is used to store the target community behavior executed by the community maintenance unit on each community to perform fault recovery.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the community structure-based graph storage method according to any one of claims 1 to 5 is implemented.

Images