Community structure based graph storage method, device, system and electronic device

Abstract

The application provides a community structure-based graph storage method, device, system and electronic equipment, and relates to the technical field of graph storage. The method comprises the following steps: obtaining user-inputted to-be-stored graph data; storing the to-be-stored graph data into an in-memory graph, and freezing the in-memory graph that reaches the upper limit of capacity after storing the to-be-stored graph data into an immutable in-memory graph; determining the target community to which each target node in the immutable in-memory graph belongs based on an L0 community table; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and the communities to which the nodes or edges belong; each community in the underlying community structure in the target graph network is stored by using a community bucket, and each group in the non-underlying community structure is stored by using a tree index bucket; and each target node is stored into the target community bucket corresponding to the target community to which the target node belongs based on the target community to which each target node belongs. The application can realize rationalized hierarchical storage of a graph network and reduce the difficulty of obtaining the community structure of the graph.

Description

Technical Field

[0001] This invention relates to the field of graph storage technology, and in particular to a graph storage method, apparatus, system and electronic device based on community structure. Background Technology

[0002] In social network analysis, graph storage supports efficient graph query processing and analysis, making it an important research problem in this field. To efficiently analyze large-scale graph data, researchers have designed various disk-based graph storage systems. These systems typically include out-of-core graph processing systems (OOBS) and graph database management systems (GDBMS). OOBS support multiple graph traversal algorithms, such as breadth-first search (BFS) and connected component search (CC). Examples of OOBS include GraphChi, GridGraph, and DGraph. GraphChi uses a parallel sliding window (PSW) method to partition the graph and stores each shard using a compressed sparse row (CSR) format. GridGraph uses a grid partitioning approach, dividing each edge into blocks based on its start and end points, with each block also stored using a CSR. DGraph stores and computes graphs based on strongly connected components (SCCs), significantly improving the efficiency of graph processing. Graph database management systems are used for persistent storage of graph data. These systems include Neo4j and JanusGraph, which provide powerful graph query languages ​​(such as Cypher provided by Neo4j) to support various graph analysis and query tasks.

[0003] Community structure is a prominent substructure in social networks and is one of the substructures of greatest interest to researchers. For example, subgraph matching algorithms and node centrality algorithms can both be optimized using community structure. However, existing graph storage systems employ inadequate graph storage methods, making it difficult to obtain the community structure of a graph. Therefore, providing a reasonable graph storage method to obtain the community structure in a graph is a pressing problem that needs to be solved. Summary of the Invention

[0004] This invention provides a graph storage method, apparatus, system, and electronic device based on community structure, to address the shortcomings of existing graph storage systems where unreasonable graph storage methods lead to difficulties in obtaining the community structure of graphs.

[0005] This invention provides a graph storage method based on community structure, comprising the following steps.

[0006] Obtain the graph data to be stored, as input by the user.

[0007] The data to be stored is stored in the memory graph, and the memory graph that has reached its capacity limit after storing the data to be stored is frozen as an immutable memory graph.

[0008] Based on the L0 community table, the target community to which each target node belongs in the immutable memory graph is determined; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and their respective communities; the target graph network includes at least two layers of community structure, the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored using community buckets, and each group in the non-lower layer community structure is stored using tree index buckets.

[0009] Based on the target community to which each target node belongs, each target node is stored in the target community bucket corresponding to its target community; the target community bucket is located in the underlying community structure.

[0010] According to the community-structure-based graph storage method provided by the present invention, the target community bucket includes a target key-value pair community block group and a target CSR community block group; The step of storing each target node in the target community bucket corresponding to its respective target community, based on the target community to which each target node belongs, includes: For each target node, a skip list index is performed based on the node identifier of the target node to determine the target key-value pair community block corresponding to the target node in the target key-value pair community block group corresponding to the target community; the target node is stored in the target key-value pair community block; and the updated target key-value pair community block group is merged into the target CSR community block group.

[0011] According to the graph storage method based on community structure provided by the present invention, after the underlying community structure is updated, the method further includes: For each community in the underlying community structure, if the community changes in the community satisfy the target community behavior, the target community behavior is executed on the community to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0012] According to the graph storage method based on community structure provided by the present invention, the target community behavior includes at least one of movement behavior, splitting behavior, and merging behavior; Determining that community changes in the community satisfy the target community behavior includes: When the target community behavior is a movement behavior, determine the first number of edges connected to the changing node in the community, and the second number of edges connected to the changing node in at least one adjacent community; if there is at least one second number of edges greater than the first number of edges, determine that the community change in the community satisfies the movement behavior; When the target community behavior is a splitting behavior, a pre-division value and a first modularity degree are determined for the community. The first modularity degree is used to characterize the rationality of the layout of the community divided into the pre-division value of sub-communities. When the first modularity degree is greater than a first preset threshold, it is determined that the community changes in the community satisfy the splitting behavior. When the target community behavior is a merging behavior, the modularity increment between the community and the target adjacent community is determined. The modularity increment is used to characterize the rationality of the layout after the target adjacent community and the community are merged. When the modularity increment is greater than a second preset threshold, it is determined that the community changes in the community satisfy the merging behavior.

[0013] According to the graph storage method based on community structure provided by the present invention, the step of performing the target community behavior on the community to obtain the adjusted community includes: When the target community behavior is a movement behavior, the adjacent community corresponding to the largest second edge number is determined as the target movement community, and the changed node is moved from the community to the target movement community to obtain the adjusted community; If the target community's behavior is a splitting behavior, the community is divided into the pre-defined numerical number of sub-communities; If the target community's behavior is a merge behavior, the community is merged with the target's adjacent communities to obtain the merged community.

[0014] The present invention also provides a graph storage device based on a community structure, comprising the following modules.

[0015] The acquisition module is used to acquire the image data to be stored, which is input by the user.

[0016] The first storage module is used to store the graph data to be stored in the memory graph, and freeze the memory graph that has reached its capacity limit after storing the graph data to be stored as an immutable memory graph.

[0017] The determination module is used to determine the target community to which each target node in the immutable memory graph belongs based on the L0 community table; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and their respective communities; the target graph network includes at least two layers of community structure, the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored using community buckets, and each group in the non-lower layer community structure is stored using tree index buckets.

[0018] The second storage module is used to store each target node into the target community bucket corresponding to its target community, based on the target community to which each target node belongs; the target community bucket is located in the underlying community structure.

[0019] The present invention also provides a graph storage system based on a community structure, comprising: The query engine is used to receive graph data to be stored from user input and to perform graph analysis tasks requested by the user. A transaction manager, which is connected to the query engine, is used to receive transaction requests from the query engine that include graph data to be stored. The LSM community storage system, connected to the transaction manager, is used to execute the community-based graph storage method as described above.

[0020] According to the community-based graph storage system provided by the present invention, the LSM community storage system includes: a community tree storage unit and a community maintenance unit, wherein: The community tree storage unit is used to store each node in the graph data to be stored into the target community bucket corresponding to its target community; the target community bucket is the storage structure corresponding to the underlying community structure. The community maintenance unit is used to, after the underlying community structure is updated, execute the target community behavior on each community in the underlying community structure if the community changes in the community meet the target community behavior, and obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0021] According to the community-based graph storage system provided by the present invention, the LSM community storage system further includes an attribute storage unit and a manifest file, wherein: The attribute storage unit is used to store all nodes in the target graph network and some of the neighbor nodes corresponding to each node. The manifest file is used to store the target community actions performed by the community maintenance unit on each community for fault recovery.

[0022] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the community-based graph storage method as described above.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the community-structure-based graph storage method as described above.

[0024] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the community-structure-based graph storage method as described above.

[0025] The graph storage method, apparatus, system, and electronic device based on community structure provided by this invention first stores the graph data to be stored input by the user into a memory graph. Once the memory graph is full, all data to be stored in the memory graph is frozen into an immutable memory graph. Based on the mapping relationship between each node in the target graph network and its corresponding community in the L0 community table, the target community to which each target node in the data to be stored in the immutable memory graph belongs is determined. Then, based on the target community, the target community bucket corresponding to the target community is determined in the community buckets corresponding to each community in the underlying community structure storing the target graph network. Each target node is then stored in its corresponding target community bucket. The target graph network includes a multi-layered community structure. The upper-layer community structure represents the association relationships between multiple groups in the lower-layer community structure. Each community in the lower-layer community structure is stored using community buckets, while each group in the non-lower-layer community structure is stored using tree-indexed buckets. In this invention, the target graph network is divided according to a multi-layer community structure, and the multi-layer community structure is stored hierarchically. The graph data to be stored is stored in the target community bucket corresponding to the bottom community structure. This fully utilizes the spatial locality of the graph network to achieve rational hierarchical storage of the graph network. At the same time, storing the graph based on the multi-layer community structure can reduce the difficulty of obtaining the community structure of the graph. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the graph storage method based on community structure provided in an embodiment of the present invention.

[0028] Figure 2This is a schematic diagram of the target graph network provided in an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the community tree storage unit corresponding to the target graph network provided in the embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram of the community bucket provided in an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of the tree index bucket structure in the L1 layer provided in an embodiment of the present invention.

[0032] Figure 6 This is a schematic diagram of the movement behavior provided in an embodiment of the present invention.

[0033] Figure 7 This is a schematic diagram of the splitting behavior provided in an embodiment of the present invention.

[0034] Figure 8 This is a schematic diagram of the merging behavior provided in an embodiment of the present invention.

[0035] Figure 9 This is a schematic diagram of the structure of a community-based graph storage device provided in an embodiment of the present invention.

[0036] Figure 10 This is a schematic diagram of the structure of a community-based graph storage system provided in an embodiment of the present invention.

[0037] Figure 11 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] Existing technologies, such as GraphChi and GridGraph, partition the graph solely based on the range of node IDs (Identity Documents), failing to adequately consider the graph's topological information. Furthermore, a node's neighbors may be stored in non-adjacent physical spaces, generating significant additional I / O operations when retrieving neighbor information, thus impacting the performance of the graph storage system. Similarly, DGraph stores the graph according to SCCs (Strongly Connected Components), i.e., storing nodes within the same SCC in adjacent physical spaces. While this fully utilizes the spatial locality of the graph, improving cache hit rates and reducing unnecessary I / O operations, SCCs are unstable substructures. Even minor graph updates can cause significant changes to the SCC structure, requiring substantial computational resources and time to maintain it. Therefore, existing graph storage methods are inadequate, failing to consider community structure during graph storage, making it difficult to obtain the graph's community structure.

[0040] To address the problem in existing graph storage systems where unreasonable graph storage methods lead to difficulties in obtaining the community structure of graphs, this invention provides a community-structure-based graph storage method. This method is applied to a community-structure-based graph storage system, hereinafter referred to as the graph storage system. Figure 1 This is a flowchart illustrating the graph storage method based on community structure provided in an embodiment of the present invention, such as... Figure 1 As shown, the method includes steps 110 to 140.

[0041] Step 110: Obtain the image data to be stored input by the user.

[0042] Specifically, after a user inputs data into the graph storage system's interface, the system responds and retrieves the graph data to be stored. This graph data can be newly added. For example, it may include newly added nodes or new edges between existing nodes in the target graph network. Both nodes and edges are represented in key-value pair format, where the node's key represents its identifier (node ​​ID), and the value represents its attribute, such as its label. The edge's key represents its starting and ending nodes, and the value represents its weight. Newly added graph data indicates that the user needs to perform a write operation on the target graph network.

[0043] Furthermore, the graph data to be stored can also be graph data to be queried or graph data to be deleted. Graph data to be deleted indicates that the user needs to perform a deletion operation on the target graph network, while graph data to be queried indicates that a read operation on the target graph network needs to be performed.

[0044] It should be noted that the target graph network is a community tree constructed after importing the full graph data, and this community tree is a hierarchical structure representation. Before executing step 110, a community tree storage unit for the target graph network has been constructed based on the LSM-Tree. This community tree storage unit is used to represent the storage structure of the target graph network on disk. A key-value pair is written to this community tree storage unit, and this key-value pair can be persistently stored on disk, where both the key and value are represented as byte arrays. Furthermore, the corresponding value can be retrieved from this community tree storage unit based on the key.

[0045] Step 120: Store the graph data to be stored in the memory graph, and freeze the memory graph that has stored the graph data and reached its capacity limit as an immutable memory graph.

[0046] For example, taking newly added graph data as an example, after acquiring the graph data, due to the fast read / write speed of memory, the graph data is first written to the memory graph instead of being directly stored on the disk. Since the memory graph has a small capacity and is not persistent, once its capacity reaches its limit, the memory graph is frozen and converted into an immutable memory graph. This facilitates the subsequent grouping and writing of the immutable memory graph into the community tree storage unit corresponding to the target graph network.

[0047] It should be noted that when the memory graph is frozen and converted into an immutable memory graph, a new memory graph can be created in memory to facilitate the storage of subsequent user-inputted graph data to be stored.

[0048] Furthermore, when the graph data to be stored is graph data to be queried or graph data to be deleted, the graph data in the memory graph can be read, and the graph data in the immutable memory graph can be read subsequently.

[0049] Step 130: Based on the L0 community table, determine the target community to which each target node in the immutable memory graph belongs; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and their respective communities; the target graph network includes at least two layers of community structure, the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored using community buckets, and each group in the non-lower layer community structure is stored using tree index buckets.

[0050] Specifically, the L0 community table is obtained synchronously when constructing the community tree storage unit of the target graph network. The L0 community table includes the mapping relationship between all nodes in the target graph network and their respective communities. Then, each target node in the immutable memory graph can be traversed, and the nodes matching each target node can be queried in the L0 community table. The community to which each matching node belongs is then determined as the target community to which each target node belongs.

[0051] It should be noted that this target graph network is stored based on a multi-layered community structure. The target graph network includes at least two layers of community structure. The leaf nodes in the bottom layer community structure represent all communities in the target graph network, and the parent nodes in the upper layer community structure recursively group the groups in the lower layer community structure. The multi-layered community structure reflects the multi-level granularity of the community structure in the target graph network.

[0052] For example, Figure 2 This is a schematic diagram of the target graph network provided in an embodiment of the present invention, such as... Figure 2 As shown, the community tree corresponding to the target graph network includes a three-layer community structure: L0, L1, and L2. Using a community detection algorithm, in the L0 layer (the bottom-level community structure), all nodes in the target graph network are first divided into four communities. Nodes within each community are tightly connected, but connections between communities are fewer. The two groups in the L1 layer are aggregates of the four communities in the L0 layer, representing the relationships between communities in the L0 layer, and each group in the L1 layer points to its corresponding community in the L0 layer. Similarly, the group in the L2 layer is aggregated from the two groups in the L1 layer, representing the relationships between groups in the L1 layer, and each group in the L2 layer points to its corresponding group in the L1 layer.

[0053] It should be noted that this community detection algorithm is a network analysis technique used to identify communities in a target graph network that have tight internal connections but few external connections. These communities typically represent sets of nodes with similar characteristics or functions and are widely used in fields such as social networks, biological networks, and information dissemination. By dividing a target graph network into meaningful communities, its internal structure can be revealed, facilitating an understanding of the organizational principles and functional characteristics of complex networks.

[0054] Figure 3 This is a schematic diagram of the community tree storage unit corresponding to the target graph network provided in the embodiments of the present invention, such as... Figure 3 As shown, after constructing the community tree corresponding to the target graph network, for each community in the L0 layer, each community is stored on the disk using a Community Bucket (CB), and for each group in the L1 and L2 layers, each group is stored on the disk using a Tree Bucket (TB).

[0055] Figure 4 This is a schematic diagram of the community bucket structure provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the community bucket includes key-value community block groups, CSR community block groups, and skip lists.

[0056] (1) The key-value community block group is used to quickly store recently updated graph data. The key-value community block is inserted in key-value pair format and the insertion speed is fast. The key-value community block group includes multiple key-value community blocks (KV). Each key-value community block stores the node update information within the corresponding node ID range. Each key-value community block includes header information and at least one graph key-value unit. The header information includes block number, maximum ID, minimum ID, and used space. The block number represents the number corresponding to the key-value community block. The maximum ID represents the maximum node ID among all nodes included in the key-value community block. The minimum ID represents the minimum node ID among all nodes included in the key-value community block. The used space represents the data space occupied by the key-value community block. Based on the used space, it can be determined whether to merge the key-value community block into the CSR community block group. For example, a key-value community block with a node ID range of [40, 60) can include three nodes with node IDs of 40, 45, and 55 respectively. The key-value pair community block with node IDs in the range [40, 60) can have a block number of 2, a maximum ID of 55, a minimum ID of 40, and uses 8 bytes of space. Each graph key-value unit includes the node sub-information corresponding to each node and the node IDs of some neighboring nodes. The node sub-information includes the node ID corresponding to the node, timestamp, direction of the connected edges, and deletion flag, which facilitates subsequent concurrency control.

[0057] (2) The CSR community block group is used to store early graph data, and the insertion speed is relatively slow. This CSR community block includes multiple CSRs (CSRs), each storing early data of nodes within the corresponding node ID range. Each CSR community block includes header information, node count, node array, and neighbor array. The header information in the CSR community block has the same meaning as the header information in the key-value pair community block, and will not be repeated here. The node count represents the total number of nodes included in the CSR community block. The node array includes the node ID and the offset from the node to its neighbor nodes in the edge array, which stores the neighbor nodes corresponding to each node. The neighbor array includes node sub-information corresponding to each neighbor node. This node sub-information is similar to the node sub-information included in each graph key-value pair, and will not be repeated here.

[0058] (3) A skip list is a multi-level ordered linked list. By using the hierarchical structure, the query speed can be accelerated by querying in the order from the top level to the bottom level.

[0059] Figure 5 This is a schematic diagram of the tree index bucket structure in the L1 layer provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the tree index bucket includes multiple community tree index blocks and skip lists. The community tree index blocks store basic information for each community within the community ID range in the L1 layer. This includes header information, the number of communities, the number of adjacent communities, a community array, a neighboring community array, and a bridge array. The header information has the same meaning as the header information in the key-value pair community block, and will not be repeated here. The number of communities represents the number of communities included in the group corresponding to the tree index bucket. The number of adjacent communities represents the number of communities in the neighboring community array for that group. Figure 2 The groups shown For example, this group Internal and Community Connected communities include communities ,Community and community The group Internal and Community Connected communities include communities Therefore, the group The corresponding number of adjacent communities is 4. The community array includes the community ID and the offset from the community in the neighbor community array to the neighbor community. The neighbor community array stores the neighbor community ID and offset of each community in the group, where the offset refers to the cross-community edge between each community and its neighbor. The bridge data includes the start and end nodes of the cross-community edge corresponding to each offset in the neighbor community array. Similar to the community bucket, each community tree index block is indexed by a skip list, but the skip list in the community tree index block records the community ID.

[0060] Step 140: Based on the target community to which each target node belongs, store each target node in the target community bucket corresponding to its target community; the target community bucket is the storage structure corresponding to the underlying community structure.

[0061] Specifically, after determining the target community corresponding to each target node, the target community bucket in the underlying community structure is further determined. First, each target node is stored in the key-value pair community block corresponding to the target community bucket, and then merged into the corresponding CSR community block.

[0062] The graph storage method based on community structure provided by this invention first stores the graph data to be stored by the user into a memory graph. Once the memory graph is full, all data to be stored in the memory graph is frozen into an immutable memory graph. Based on the mapping relationship between each node in the target graph network and its corresponding community in the L0 community table, the target community to which each target node in the data to be stored in the immutable memory graph belongs is determined. Then, based on the target community, the target community bucket corresponding to the target community is determined in the community buckets corresponding to each community in the underlying community structure of the target graph network, and each target node is stored in the corresponding target community bucket. The target graph network includes a multi-layered community structure. The upper-layer community structure represents the association relationship between multiple groups in the lower-layer community structure, and the nodes within each community in the lower-layer community structure are tightly connected. Each community in the lower-layer community structure is stored using community buckets, while each group in the non-lower-layer community structure is stored using tree-indexed buckets. In this invention, the target graph network is divided according to a multi-layer community structure, and the multi-layer community structure is stored hierarchically. The graph data to be stored is stored in the target community bucket corresponding to the bottom community structure. This fully utilizes the spatial locality of the graph network to achieve rational hierarchical storage of the graph network. At the same time, storing the graph based on the multi-layer community structure can reduce the difficulty of obtaining the community structure of the graph.

[0063] Further, the step of storing each target node in the target community bucket corresponding to its respective target community based on the target community to which each target node belongs includes: For each target node, a skip list index is performed based on the node identifier of the target node to determine the target key-value pair community block corresponding to the target node in the target key-value pair community block group corresponding to the target community; the target node is stored in the target key-value pair community block; and the updated target key-value pair community block group is merged into the target CSR community block group.

[0064] For example, taking the target node with node ID 50 as an example, and needing to add this node to the community tree storage unit corresponding to the target graph network, such as... Figure 4As shown, the target key-value pair community block group corresponding to the target node includes key-value pair community blocks with node ID ranges of [0,40), [40,60), [60,70), [70,80), and [80,∞). Using the node ID, the skip list is indexed sequentially from top to bottom to determine which node ID range the target node's node ID belongs to. In the first level, the pointer in the skip list points from 0 to ∞, and since 0 < 50 < ∞, node 50 belongs to the range [0,∞). Then, in the second level, the pointer in the skip list points from 0 to 80, and since 0 < 50 < 80, node 50 belongs to the range [0,80). In the third level, the pointer in the skip list points from 0 to 60, and since 0 < 50 < 60, node 50 belongs to the range [0,60). In the fourth level, firstly, the pointer in the skip list points from 0 to 40, and since 50 > 40, node 50 does not belong to the range [0,40). Afterwards, the pointer in the skip list moves from 40 to 60, and 40 < 50 < 60. Therefore, node 50 belongs to the range [40, 60). That is, the target node should first be written into the target key-value pair community block in the target community bucket where the node ID range is [40, 60). Then, if the used space of the target key-value pair community block is greater than the preset space threshold, the target key-value pair community block can be merged into the target CSR community block corresponding to the node ID range.

[0065] In this embodiment of the invention, the target key-value pair community block corresponding to the target node can be quickly queried through the skip list index, resulting in a fast query speed.

[0066] Furthermore, after the underlying community structure is updated, the method further includes: For each community in the underlying community structure, if the community changes in the community satisfy the target community behavior, the target community behavior is executed on the community to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0067] Specifically, in existing technologies, when a large-scale update occurs in a graph network, such as the insertion and deletion of a large number of nodes and edges, the community structure may also change accordingly. Existing community detection algorithms consider the calculation of community structure from a global perspective, which requires obtaining full graph information during the calculation process. When the size of the stored graph is large, this introduces a large amount of I / O overhead. In addition, this "calculate from scratch" mode does not fully utilize the community information at historical moments. Therefore, in this embodiment of the invention, after the underlying community structure is updated, the changes in each community in the underlying community structure are characterized, and it is determined whether the changes in each community meet the target community behavior. If they do, the target community behavior is executed on that community, and the adjusted community division is more appropriate and the storage layout is more reasonable.

[0068] In this embodiment of the invention, during the process of updating the graph structure, without loading the entire graph, the community structure of the target graph network stored on the disk is dynamically and efficiently maintained with the community as the center, so as to avoid the community quality of the stored graph network from degrading with the update of the community structure, and without introducing a large amount of I / O overhead.

[0069] Furthermore, the target community behavior includes at least one of movement behavior, splitting behavior, and merging behavior; Determining that community changes in the community satisfy the target community behavior includes: When the target community behavior is a movement behavior, determine the first number of edges connected to the changing node in the community, and the second number of edges connected to the changing node in at least one adjacent community; if there is at least one second number of edges greater than the first number of edges, determine that the community change in the community satisfies the movement behavior; When the target community behavior is a splitting behavior, a pre-division value and a first modularity degree are determined for the community. The first modularity degree is used to characterize the rationality of the layout of the community divided into the pre-division value of sub-communities. When the first modularity degree is greater than a first preset threshold, it is determined that the community changes in the community satisfy the splitting behavior. When the target community behavior is a merging behavior, the modularity increment between the community and the target adjacent community is determined. The modularity increment is used to characterize the rationality of the layout after the target adjacent community and the community are merged. When the modularity increment is greater than a second preset threshold, it is determined that the community changes in the community satisfy the merging behavior.

[0070] Furthermore, the step of performing the target community behavior on the community to obtain the adjusted community includes: When the target community behavior is a movement behavior, the adjacent community corresponding to the largest second edge number is determined as the target movement community, and the changed node is moved from the community to the target movement community to obtain the adjusted community; If the target community's behavior is a splitting behavior, the community is divided into the pre-defined numerical number of sub-communities; If the target community's behavior is a merge behavior, the community is merged with the target's adjacent communities to obtain the merged community.

[0071] For example, Figure 6 This is a schematic diagram of the movement behavior provided in an embodiment of the present invention, such as... Figure 6As shown in the left part of the graph, before the graph update, node V1 belongs to community C1. The graph update involves deleting the edges between nodes V1 and V2, and between nodes V1 and V3, in community C1, and adding edges between nodes V1 and V5, and between nodes V1 and V6, in community C2. That is, the changed node in community C1 is node V1. After the graph update, only node V4 is connected to node V1 in community C1, meaning the first edge count of the edges connecting node V1 in community C1 is 1. In community C2, nodes V5, V6, and V7 are all connected to node V1, meaning the second edge count of the edges connecting node V1 in community C2 is 3. Since the second edge count is greater than the first edge count, the changed node V1 in community C1 satisfies the movement behavior. Therefore, community C2 can be determined as the target movement community for node V1, and node V1 is moved from community C1 to community C2, thus obtaining the result after the movement behavior. Figure 6 The community shown on the right and community .

[0072] It should be noted that if new edges are added between node V1 and nodes in multiple adjacent communities during the graph update process, the number of second edges between node V1 and each adjacent community can be determined. The multiple second edge numbers are sorted in descending order, and the adjacent community corresponding to the second edge number with the highest number of second edges is determined as the target mobile community. Then, node V1 can be moved to the target mobile community.

[0073] For example, taking a first preset threshold of 0.6 as an example, Figure 7 This is a schematic diagram of the splitting behavior provided in an embodiment of the present invention, such as... Figure 7 As shown in the left part, before the graph update, community C1 consisted of 8 nodes. The graph update involved deleting the edge between nodes V2 and V6, and the edge between nodes V4 and V7 in community C1. Afterwards, the Louvain algorithm was used to calculate the pre-partition value for community C1 as 2, and the first modularity was 0.65. Since 0.65 > 0.6, it indicates that dividing community C1 into two sub-communities is more reasonable. Therefore, community C1 satisfies the splitting behavior and can be split (Split) into two sub-communities. Figure 7 The community shown on the right and community .

[0074] It should be noted that the first modularity is used to measure the expected value of the internal connection density of a community in a graph network relative to a random distribution. That is, the first modularity is used to measure the quality of community partitioning, and the higher the first modularity, the better the quality of community partitioning.

[0075] For example, taking a second preset threshold of 0 as an example, Figure 8 This is a schematic diagram of the merging behavior provided in an embodiment of the present invention, such as... Figure 8 As shown in the left part of the graph, before the graph update, there are two communities, C1 and C2. The update adds edges between nodes V5 and V2, V1 and V2, V1 and V3, and V7 and V4. Then, the modularity increment between communities C1 and C2 is calculated to be 0.3, i.e., modularity or Newman-Girvan Q. Since 0.3 > 0, it indicates that merging communities C1 and C2 results in a higher-quality community structure. Therefore, it can be determined that communities C1 and C2 satisfy the merging behavior, and thus, communities C1 and C2 can be merged. Figure 8 The community shown on the right .

[0076] Furthermore, after the graph is updated, it can first determine whether the community meets the requirements for the movement behavior of the application form for the special fund for international conferences of the School of Information Science and Technology, or if the community change does not meet the requirements for the movement behavior, it can continue to determine whether the community meets the requirements for the split behavior. If it does, the split behavior is executed for the community. If the community change does not meet the requirements for the split behavior, or after the split behavior is executed for the community, it can continue to determine whether the community meets the requirements for the merge behavior. If it does, the merge behavior is executed for the community. If it does not meet the requirements, the status quo of the community is maintained.

[0077] The community-based graph storage device provided by the present invention is described below. The community-based graph storage device described below and the community-based graph storage method described above can be referred to in correspondence.

[0078] This invention also provides a graph storage device based on a community structure. Figure 9 This is a schematic diagram of the structure of the graph storage device based on community structure provided in an embodiment of the present invention, such as... Figure 9 As shown, the community-structure-based graph storage device 900 includes: an acquisition module 910, a first storage module 920, a determination module 930, and a second storage module 940.

[0079] The acquisition module 910 is used to acquire the image data to be stored input by the user.

[0080] The first storage module 920 is used to store the graph data to be stored in a memory graph, and freeze the memory graph that has been filled with the graph data to be stored as an immutable memory graph.

[0081] The determination module 930 is used to determine the target community to which each target node in the immutable memory graph belongs based on the L0 community table; the L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and their respective communities; the target graph network includes at least two layers of community structure, the upper layer community structure is used to represent the association relationship between multiple groups in the lower layer community structure; each community in the lower layer community structure is stored using community buckets, and each group in the non-lower layer community structure is stored using tree index buckets.

[0082] The second storage module 940 is used to store each target node in the target community bucket corresponding to its target community based on the target community to which each target node belongs; the target community bucket is located in the underlying community structure.

[0083] The community-structure-based graph storage device provided by this invention first stores the user-input graph data to be stored in a memory graph. Once the memory graph is full, all data to be stored in the memory graph is frozen into an immutable memory graph. Based on the mapping relationship between each node in the target graph network and its corresponding community in the L0 community table, the target community to which each target node in the data to be stored belongs is determined. Then, based on the target community, the target community bucket corresponding to the target community is determined in the community buckets corresponding to each community in the underlying community structure storing the target graph network. Each target node is then stored in its corresponding target community bucket. The target graph network includes a multi-layered community structure. The upper-layer community structure represents the association relationships between multiple groups in the lower-layer community structure, and the nodes within each community in the lower-layer community structure are tightly connected. Each community in the lower-layer community structure is stored using community buckets, while each group in the non-lower-layer community structure is stored using tree-indexed buckets. In this invention, the target graph network is divided according to a multi-layer community structure, and the multi-layer community structure is stored hierarchically. The graph data to be stored is stored in the target community bucket corresponding to the bottom community structure. This fully utilizes the spatial locality of the graph network to achieve rational hierarchical storage of the graph network. At the same time, storing the graph based on the multi-layer community structure can reduce the difficulty of obtaining the community structure of the graph.

[0084] Optionally, the target community bucket includes a target key-value pair community block group and a target CSR community block group.

[0085] Optionally, the second storage module 940 is specifically used for: For each target node, a skip list index is performed based on the node identifier of the target node to determine the target key-value pair community block corresponding to the target node in the target key-value pair community block group corresponding to the target community; the target node is stored in the target key-value pair community block; and the updated target key-value pair community block group is merged into the target CSR community block group.

[0086] Optionally, the community-based graph storage device 900 further includes a dynamic maintenance module, which is specifically used for: For each community in the underlying community structure, if the community changes in the community satisfy the target community behavior, the target community behavior is executed on the community to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0087] Optionally, the target community behavior includes at least one of movement behavior, splitting behavior, and merging behavior.

[0088] Optionally, this dynamic maintenance module is specifically used for: When the target community behavior is a movement behavior, determine the first number of edges connected to the changing node in the community, and the second number of edges connected to the changing node in at least one adjacent community; if there is at least one second number of edges greater than the first number of edges, determine that the community change in the community satisfies the movement behavior; When the target community behavior is a splitting behavior, a pre-division value and a first modularity degree are determined for the community. The first modularity degree is used to characterize the rationality of the layout of the community divided into the pre-division value of sub-communities. When the first modularity degree is greater than a first preset threshold, it is determined that the community changes in the community satisfy the splitting behavior. When the target community behavior is a merging behavior, the modularity increment between the community and the target adjacent community is determined. The modularity increment is used to characterize the rationality of the layout after the target adjacent community and the community are merged. When the modularity increment is greater than a second preset threshold, it is determined that the community changes in the community satisfy the merging behavior.

[0089] Optionally, this dynamic maintenance module is specifically used for: When the target community behavior is a movement behavior, the adjacent community corresponding to the largest second edge number is determined as the target movement community, and the changed node is moved from the community to the target movement community to obtain the adjusted community; If the target community's behavior is a splitting behavior, the community is divided into the pre-defined numerical number of sub-communities; If the target community's behavior is a merge behavior, the community is merged with the target's adjacent communities to obtain the merged community.

[0090] This invention also provides a graph storage system based on a community structure. Figure 10 This is a schematic diagram of the community-based graph storage system provided in an embodiment of the present invention, such as... Figure 10 As shown, this community-based graph storage system includes a query engine, a transaction manager, and an LSM community storage system.

[0091] The query engine receives graph data to be stored from user input and performs graph analysis tasks requested by the user.

[0092] A transaction manager, which is connected to the query engine, is used to receive transaction requests from the query engine that include graph data to be stored.

[0093] The LSM community storage system, connected to the transaction manager, is used to execute the community-based graph storage method as described above.

[0094] Specifically, the query engine includes a graph query processor and a graph update unit. The graph query processor queries all subgraphs isomorphic to the user-given subgraph in the full graph data corresponding to the target graph network. The graph update unit serves as the system interface of this community-based graph storage system, interacting with upper-layer services or users. Furthermore, the query engine stores various graph analysis algorithms, such as depth-first search and breadth-first search, to find the shortest path for the unit and execute the user-required graph analysis task. The transaction manager ensures that read and write operations maintain the atomicity, consistency, and durability of transactions. The LSM community storage system executes the aforementioned community-based graph storage method, which will not be elaborated further in this embodiment.

[0095] Furthermore, such as Figure 10 As shown, the LSM community storage system includes: a community tree storage unit and a community maintenance unit, wherein: The community tree storage unit is used to store each node in the graph data to be stored into the target community bucket corresponding to its target community; the target community bucket is the storage structure corresponding to the underlying community structure. The community maintenance unit is used to, after the underlying community structure is updated, execute the target community behavior on each community in the underlying community structure if the community changes in the community meet the target community behavior, and obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

[0096] The specific structure of the community tree storage unit will not be described in detail in this embodiment of the invention.

[0097] Furthermore, such as Figure 10 As shown, the LSM community storage system also includes attribute storage units and manifest files, wherein: The attribute storage unit is used to store all nodes in the target graph network and some of the neighbor nodes corresponding to each node. The manifest file is used to store the target community actions performed by the community maintenance unit on each community for fault recovery.

[0098] Specifically, after the graph update unit receives the graph data to be stored input by the user, it can write the graph data to be stored into the community tree storage unit and the attribute storage unit respectively. The attribute storage unit includes all nodes in the target graph network and some of the neighbor nodes corresponding to each node. After the graph data to be stored is written into the community tree storage unit, and after a graph update is performed in a community, the community maintenance unit dynamically maintains the community structure corresponding to the community. At the same time, the target community behavior executed during dynamic maintenance is recorded in the manifest file so that recovery can be performed if the community executing the target community behavior fails.

[0099] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 11 As shown, the electronic device may include: a processor 1110, a communications interface 1120, a memory 1130, and a communications bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other through the communications bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a graph storage method based on community structure. This method includes: acquiring graph data to be stored input by a user; storing the graph data to be stored in a memory graph, and freezing the memory graph, after storing the graph data and reaching its capacity limit, as an immutable memory graph; determining the target community to which each target node in the immutable memory graph belongs based on an L0 community table; the L0 community table is used to characterize the mapping relationship between all nodes or edges in the target graph network and their respective communities; the target graph network includes at least two layers of community structure, with the upper-layer community structure characterizing the association relationship between multiple groups in the lower-layer community structure; each community in the lower-layer community structure is stored using community buckets, and each group in the non-lower-layer community structure is stored using tree index buckets; based on the target community to which each target node belongs, storing each target node in the target community bucket corresponding to its target community; the target community bucket is located in the lower-layer community structure.

[0100] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the community-based graph storage method provided by the above methods. The method includes: acquiring graph data to be stored input by a user; storing the graph data to be stored in a memory graph, and freezing the memory graph after storing the graph data to be stored and reaching its capacity limit as an immutable memory graph; determining the target community to which each target node in the immutable memory graph belongs based on an L0 community table; the L0 community table is used to characterize the mapping relationship between all nodes or all edges in the target graph network and their respective communities; the target graph network includes at least two layers of community structure, the upper-layer community structure is used to characterize the association relationship between multiple groups in the lower-layer community structure; each community in the lower-layer community structure is stored using a community bucket, and each group in the non-lower-layer community structure is stored using a tree index bucket; based on the target community to which each target node belongs, storing each target node in the target community bucket corresponding to its target community; the target community bucket is located in the lower-layer community structure.

[0102] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the community-based graph storage method provided by the above methods. The method includes: acquiring graph data to be stored input by a user; storing the graph data to be stored in a memory graph, and freezing the memory graph after storing the graph data to be stored and reaching its capacity limit as an immutable memory graph; determining the target community to which each target node in the immutable memory graph belongs based on an L0 community table; the L0 community table is used to characterize the mapping relationship between all nodes or all edges in the target graph network and their respective communities; the target graph network includes at least two layers of community structure, the upper-layer community structure is used to characterize the association relationship between multiple groups in the lower-layer community structure; each community in the lower-layer community structure is stored using a community bucket, and each group in the non-lower-layer community structure is stored using a tree index bucket; based on the target community to which each target node belongs, storing each target node in the target community bucket corresponding to its target community; the target community bucket is located in the lower-layer community structure.

[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A graph storage method based on community structure, characterized in that, include: Obtain the image data to be stored, input by the user; The graph data to be stored is stored in the memory graph, and the memory graph that has reached its capacity limit after storing the graph data to be stored is frozen as an immutable memory graph. Based on the L0 community table, the target community to which each target node belongs in the immutable memory graph is determined; The L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and their respective communities; The target graph network includes at least two layers of community structure. The upper community structure is used to represent the association relationship between multiple groups in the lower community structure. Each community in the lower community structure is stored using community buckets, and each group in the non-lower community structure is stored using tree index buckets. Based on the target community to which each target node belongs, each target node is stored in the target community bucket corresponding to its target community. The target community bucket is located in the underlying community structure; The target community bucket includes the target key-value pair community block group and the target CSR community block group; The step of storing each target node in the target community bucket corresponding to its respective target community, based on the target community to which each target node belongs, includes: For each target node, a skip list index is performed based on the node identifier of the target node to determine the target key-value pair community block corresponding to the target node in the target key-value pair community block group corresponding to the target community; the target node is stored in the target key-value pair community block; and the updated target key-value pair community block group is merged into the target CSR community block group.

2. The graph storage method based on community structure according to claim 1, characterized in that, After the underlying community structure is updated, the method further includes: For each community in the underlying community structure, if the community changes in the community satisfy the target community behavior, the target community behavior is executed on the community to obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

3. The graph storage method based on community structure according to claim 2, characterized in that, The target community behavior includes at least one of the following: movement behavior, splitting behavior, and merging behavior; Determining that community changes in the community satisfy the target community behavior includes: When the target community behavior is a movement behavior, determine the first number of edges connected to the changing node in the community, and the second number of edges connected to the changing node in at least one adjacent community; if there is at least one second number of edges greater than the first number of edges, determine that the community change in the community satisfies the movement behavior; When the target community behavior is a splitting behavior, a pre-division value and a first modularity degree are determined for the community. The first modularity degree is used to characterize the rationality of the layout of the community divided into the pre-division value of sub-communities. When the first modularity degree is greater than a first preset threshold, it is determined that the community changes in the community satisfy the splitting behavior. When the target community behavior is a merging behavior, the modularity increment between the community and the target adjacent community is determined. The modularity increment is used to characterize the rationality of the layout after the target adjacent community and the community are merged. When the modularity increment is greater than a second preset threshold, it is determined that the community changes in the community satisfy the merging behavior.

4. The graph storage method based on community structure according to claim 3, characterized in that, The process of performing the target community behavior on the community to obtain the adjusted community includes: When the target community behavior is a movement behavior, the adjacent community corresponding to the largest second edge number is determined as the target movement community, and the changed node is moved from the community to the target movement community to obtain the adjusted community; If the target community's behavior is a splitting behavior, the community is divided into the pre-defined numerical number of sub-communities; If the target community's behavior is a merge behavior, the community is merged with the target's adjacent communities to obtain the merged community.

5. A graph storage device based on a community structure, characterized in that, include: The acquisition module is used to acquire the image data to be stored, which is input by the user. The first storage module is used to store the graph data to be stored into the memory graph, and freeze the memory graph that has stored the graph data to be stored and has reached the capacity limit as an immutable memory graph. The determination module is used to determine the target community to which each target node in the immutable memory graph belongs, based on the L0 community table. The L0 community table is used to represent the mapping relationship between all nodes or all edges in the target graph network and their respective communities; The target graph network includes at least two layers of community structure. The upper community structure is used to represent the association relationship between multiple groups in the lower community structure. Each community in the lower community structure is stored using community buckets, and each group in the non-lower community structure is stored using tree index buckets. The second storage module is used to store each target node into the target community bucket corresponding to its target community, based on the target community to which each target node belongs; The target community bucket is located in the underlying community structure; The target community bucket includes the target key-value pair community block group and the target CSR community block group; The second storage module is specifically used for: performing a skip list index based on the node identifier of each target node, and determining the target key-value pair community block corresponding to the target node in the target key-value pair community block group corresponding to the target community; Store the target node into the target key-value pair community block; The updated target key-value pair community blocks are combined into the target CSR community block group.

6. A graph storage system based on a community structure, characterized in that, include: The query engine is used to receive graph data to be stored from user input and to perform graph analysis tasks requested by the user. A transaction manager, which is connected to the query engine, is used to receive transaction requests from the query engine that include graph data to be stored. The LSM community storage system, connected to the transaction manager, is used to execute the community-based graph storage method as described in any one of claims 1 to 4.

7. The community-based graph storage system according to claim 6, characterized in that, The LSM community storage system includes: a community tree storage unit and a community maintenance unit, wherein: The community tree storage unit is used to store each node in the graph data to be stored into the target community bucket corresponding to its target community; the target community bucket is the storage structure corresponding to the underlying community structure. The community maintenance unit is used to, after the underlying community structure is updated, execute the target community behavior on each community in the underlying community structure if the community changes in the community meet the target community behavior, and obtain the adjusted community; the target community behavior is used to adjust the storage layout of the community.

8. The community-based graph storage system according to claim 7, characterized in that, The LSM community storage system also includes attribute storage units and manifest files, wherein: The attribute storage unit is used to store all nodes in the target graph network and some of the neighbor nodes corresponding to each node. The manifest file is used to store the target community actions performed by the community maintenance unit on each community for fault recovery.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the community-structure-based graph storage method as described in any one of claims 1 to 4.

Images