A hybrid memory-based adaptive graph computing update method and system

Abstract

The present invention discloses a hybrid memory-based adaptive graph computing and updating method and system. First, when the directed graph data is stored in an out-edge manner, the vertex interval and its corresponding out-edge data block are evenly divided according to the vertex interval; when the directed graph data is stored in an in-edge manner, the vertex interval and its corresponding in-edge data block are evenly divided according to the vertex interval; when the subgraph is constructed, the corresponding in-edge data block and out-edge data block are directly used to avoid traversing all data blocks; to improve the access efficiency of the subgraph data, the present invention stores the in-edge data and out-edge data of the graph data in two NUMA nodes respectively; secondly, a data-driven push-pull adaptive data update strategy is adopted to optimize the message update process during iterative graph calculation. The present invention solves the subgraph construction and update mode problems existing in the hybrid memory graph computing model, and greatly improves the efficiency of graph computing.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence-big data-graph computing technology, and relates to a hybrid memory-based adaptive graph computing update method and system. Background Art

[0002] In computer science, graphs are one of the most complex and widely used data structures. They use vertices V and edges E to represent relationships between objects, providing a native representation of relationships between things. Generally, a graph is denoted by G = (V, E). The vertex set V is a finite, non-empty set that can represent various objects and can contain vertex IDs, values, and other user-defined attributes. The edge set E is a finite, potentially empty set that can represent various relationships between objects. Many real-world data can be naturally represented as graphs. For example, web pages in a web network can be considered vertices, and hyperlinks within them can be considered edges; sites in a transportation network can be considered vertices, and routes between sites can be considered edges; user data in the social realm can be considered vertices, and friendships and followings between users can be considered edges. Powered by graph theory and various graph algorithms, many practical problems can be effectively solved. For example, web page value assessment can be implemented using centrality algorithms; route planning can be implemented using path-finding algorithms; and social group discovery can be implemented using community discovery algorithms. However, with the rapid development of the internet and digital technologies, the scale of graph data is growing exponentially, often with a power-law distribution. Traditional graph algorithms struggle to address numerous issues, such as unbalanced load, frequent data exchange, and inefficient caching mechanisms. This directly impacts their ability to meet the ever-increasing performance and functionality demands. In recent years, with the rapid development of digital technologies such as artificial intelligence, high-performance graph computing has garnered widespread attention from researchers in industry and academia both domestically and internationally.

[0003] Graph computing is an abstraction of classic graph algorithms, leveraging various iterative paradigms and optimization techniques within a given hardware environment to efficiently process large-scale graph data. Graph computing plays a vital role in a wide range of fields, from relationship analysis and product recommendations to fraud detection. In recent years, with the rapid development of artificial intelligence (AI), the scale of graph data has grown exponentially, leading to significant memory usage challenges when storing and processing this massive data. In addition to its enormous size, graph data often exhibits a power-law distribution. Kumar P et al. pointed out that this inherent imbalance in graph computing can lead to problems such as high memory-to-computation ratios, load imbalance, and low parallel efficiency. To address these issues, a significant amount of research has focused on improving the efficiency of graph computing models. Y. Zhang et al. utilized large-capacity memory to construct a single-machine in-memory graph computing model to avoid high disk I / O overhead. They also utilized graph partitioning techniques to construct a distributed in-memory graph computing model within a distributed cluster, processing multiple subgraphs in parallel to improve the scalability of single-machine graph computing systems. With significant progress in persistent memory (PMEM) research, this new type of memory device, combined with traditional DRAM, forms a hybrid memory system (HMS). HMS has become a viable solution for efficient, data-centric graph computing. In summary, current memory-based graph computing models can be categorized as: single-machine graph computing models based on traditional memory, graph computing models based on distributed shared memory, and graph computing models based on hybrid memory.

[0004] (1) Single-machine graph computing model based on traditional memory

[0005] Traditional memory-based single-machine graph computing models leverage large memory capacities to build lightweight graph computing models. The memory capacity of these models can accommodate the entire graph data, facilitating efficient programming and improving the parallel execution efficiency of graph algorithms. The Ligra model proposed by J. Shun et al. provides two typical programming interfaces: one for edge mapping and the other for vertex mapping. This makes the model highly suitable for implementing graph traversal algorithms on vertex sets. Furthermore, based on the parallel traversal principle employed in the breadth-first search (BFS) algorithm, the programming interface can adaptively switch between push-pull computing modes based on the density of the graph data. Vertex-program-centric programs are mapped into GraphMat, an efficient and scalable sparse matrix operation framework. Under multi-core and multi-threaded parallel conditions, this can further improve the efficiency of graph computing models. The graph computing model GraphIt proposed by Y. Zhang et al. separates graph computation from graph scheduling and can handle graph data of varying structures and sizes. Due to its large DRAM capacity, these systems can avoid high disk I / O overhead.

[0006] (2) Graph Computing Model Based on Distributed Shared Memory

[0007] This type of model is designed to achieve a high level of abstraction similar to MapReduce, enabling efficient execution of iterative graph algorithms with sparse computational dependencies while ensuring high consistency of graph data and efficient parallel computation. This open-source graph computing framework, developed in C++ by the Select Lab at Carnegie Mellon University (CMU), is designed for large-scale, streaming graph data-parallel computing and can run efficiently on multi-processor clusters. Rong Chen et al. analyzed the power-law properties of graph data and adopted a hybrid partitioning strategy. They employed a vertex-cut strategy for high-degree vertices to reduce the generation of mirror vertices, thereby alleviating the computational burden of high-degree vertices. They also employed an edge-cut approach for low-degree vertices to ensure that computations on low-degree vertices are performed locally as much as possible. Tim et al. proposed the Grade graph computing model, which leverages resource attribution to build a fine-grained, unified workload- and system-level performance view by monitoring logs and applications. It can also automatically identify resource bottlenecks and common performance issues. All of these models can process multiple subgraphs in parallel, improving the parallel computing capabilities of single-machine graph computing systems.

[0008] (3) Graph Computing Model Based on Hybrid Memory

[0009] This type of model aims to build efficient single-machine in-memory graph computing models in emerging hybrid memory systems, reducing the reliance on and construction costs of traditional DRAM and improving the scalability of traditional in-memory graph computing models. Huang et al. used a hybrid storage format combining edge lists and adjacency lists to process evolving graphs. They first looped through the edge log in DRAM and stored updated graph data in edge list format. Furthermore, they used numerous adjacency lists to store older data (edge data periodically archived from the edge log), thus supporting efficient graph storage and querying. Wang R et al. introduced XPGraph, an efficient graph storage model based on PMEM. They proposed a large-scale dynamic graph storage model, leveraging the PMEM-friendly XPLine data access program. This model features hierarchical vertex buffer management and NUMA-friendly graph data access. B. Li et al. constructed the PMEM-based graph computing model EPGraph, placing all vertices and edges in DRAM and PMEM, respectively. To reduce random access to vertices, they used DRAM as a data buffer. Furthermore, they employed a degree-aware graph data tiering approach to improve graph data access efficiency.

[0010] Existing technical solutions mainly include standalone graph computing models based on traditional memory, graph computing models based on distributed shared memory, and graph computing models based on emerging hybrid memory. The existing mainstream models in these solutions have certain shortcomings in different aspects, as follows:

[0011] 1. Traditional memory-based, single-machine graph computing models: These models avoid high disk I / O overhead and facilitate the use of multithreading to optimize the execution efficiency of graph algorithms. While these models can process graph data of a certain scale, they rely heavily on large DRAM capacities and struggle to handle exponentially growing graph data. While these models can utilize multithreaded parallelism to optimize graph algorithms, the computing power of a single machine is limited. These factors severely restrict the application of single-machine, in-memory graph computing systems within limited resources.

[0012] 2. Distributed shared memory-based graph computing models: These models leverage distributed clusters to process extremely large-scale graph data, reducing the hardware requirements for individual computing nodes and improving the scalability and parallel computing capabilities of single-machine in-memory graph computing models. However, the inherent power-law nature of graph data can lead to load imbalance during graph partitioning and task imbalance during graph computing. Furthermore, the communication overhead between distributed computing nodes becomes a performance bottleneck for these models.

[0013] Hybrid memory-based graph computing models: These models leverage the large capacity, persistence, and byte-addressability of PMEM to reduce their reliance on traditional DRAM. The read and write performance differences between DRAM and PMEM can lead to inefficient graph data access in HMS. In particular, existing hybrid memory graph computing models fail to account for memory heterogeneity between NUMA nodes. This means that local PMEM data access is far less efficient than remote memory access, preventing optimal performance. Summary of the Invention

[0014] In view of the problems existing in the prior art, the purpose of the present invention is to provide a hybrid memory-based adaptive graph computing and updating method and system.

[0015] The present invention is mainly used to solve the subgraph construction and update mode problems existing in the existing hybrid memory graph computing model; the subgraph construction is an important basic link in the graph computing model; and the update mode of the graph data G directly affects the efficiency of the graph computing. This method mainly focuses on accelerating the subgraph construction based on the Dual-Block storage method, and effectively supports the data-driven adaptive push-pull update model. Specifically, in HMS, the large-capacity characteristics of PMEM are used to store the out-edge (InBlock) and in-edge (OutBlock) data of the subgraph; first, when the directed graph data is stored in the out-edge mode, when it is evenly divided according to the vertex interval, the vertex interval interval[i] and its corresponding out-edge data block are OutBlock[i]; similarly, when the directed graph data is stored in the in-edge mode, when it is evenly divided according to the vertex interval, the vertex interval interval[i] and its corresponding in-edge data block are InBlock[i]; construct the subgraph G i When traversing the inbound data block InBlock[i] and the outbound data block OutBlock[i], the inbound data block InBlock[i] is directly used, thereby avoiding traversing all InBlocks or OutBlocks. In order to improve the access efficiency of subgraph data, the present invention stores the inbound data InBlock and the outbound data OutBlock of the graph data in two NUMA (Non-Uniform Memory Access) nodes (Node0, Node1) respectively. Secondly, a data-driven push-pull adaptive data update strategy (Date-Drivened Adaptive Push-PullUpdating Model) is adopted to optimize the message update process during iterative graph calculation. Finally, a dynamic data migration strategy based on degree awareness is used to perform swap-in and swap-out operations between DRAM and PMEM for possible subgraphs in the next round of iteration.

[0016] In summary, this method accelerates the subgraph construction process by storing the subgraph's InBlock[i] and OutBlock[i] data in a bidirectional subgraph data storage called Dual-Block. This graph storage method effectively supports a data-driven adaptive push-pull update model. This adaptive computing model directly improves graph computing efficiency, which is the key to this method's ability to efficiently process large-scale graph data.

[0017] This invention significantly improves subgraph construction efficiency, boosting the overall model efficiency at an acceptable cost in data storage space. Unlike previous work, the proposed model combines a dual-block graph storage representation with a data-driven adaptive push-pull strategy. Furthermore, it optimizes graph data access efficiency within HMS through graph data layering and dynamic graph data migration strategies. Extensive experimental results demonstrate that the proposed NPGraph model delivers superior performance.

[0018] The technical solution of the present invention is:

[0019] A hybrid memory-based adaptive graph computing update method, comprising the following steps:

[0020] 1) Set up two NUMA nodes, denoted as Node0 and Node1. Each of the NUMA nodes uses a hybrid memory system HMS, which includes persistent memory PMEM and DRAM. The node Node0 is equipped with a preprocessing module, a subgraph construction module, and a subgraph update module.

[0021] 2) The preprocessing module divides the vertex set V and edge set E of the graph data G = (V, E) into P non-intersecting vertex intervals V1 to V p and edge blocks E1~E p , used to generate P subgraphs G1~G P ; Where i = 1~P, the i-th vertex interval V i The vertex subset included is the subgraph G to be generated i The vertex set and edge block E i Including the subgraph G to be generated i The vertex interval V i The outbound data block OutBlock[i] of the source vertex and the vertex interval V i Enter the edge data block InBlock[i] for the destination vertex; then load the subgraph data in outgoing edge mode into node Node0, and load the subgraph data in incoming edge mode into node Node1;

[0022] 3) The subgraph construction module calculates the density of each subgraph to be generated based on the outgoing edge data of each subgraph read from the node Node0, and sets a proportion parameter δ based on the statistical analysis of the vertex degree information of the graph data G, and migrates the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS of the node Node0 to the memory DRAM of the hybrid memory system HMS of the node Node0; and reads the incoming edge data of the corresponding subgraph from the node Node1 to calculate the density of each subgraph to be generated, and according to the proportion parameter δ, migrates the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS of the node Node1 to the memory DRAM of the hybrid memory system HMS of the node Node1; then the subgraph construction module generates the i-th subgraph G based on the i-th subgraph data in the local memory DRAM of the node Node0 and the i-th subgraph data in the memory DRAM of the remote access Node1. i =(V i ,E i );

[0023] 4) Subgraph update module updates each subgraph G i Perform iterative updates by calculating the subgraph G at the current iteration i Activity ε [i] Determine the pair of subgraphs G i The update mode adopted; which sets a ε-based [i] and the threshold function selectModel(), when the subgraph G i Activity ε [i] When the threshold value is less than the set threshold value θ, the threshold function selectModel() will return to Push mode, otherwise it will return to Pull mode; in Push mode, the subgraph G i The vertex v in retains the target vertex set D v ; In Pull mode, it is subgraph G i The vertex v in retains the source vertex set S v ; The push mode graph algorithm pushes the updated value of vertex v to D v , Pull-mode graph algorithm from S v Pull the source vertex value from to update the value of vertex v.

[0024] Furthermore, the specific implementation method of step 1) is: first, the outbound edge pattern data of the graph data G is stored in the first storage unit and the subgraph is divided according to the vertex interval to obtain each vertex interval and its corresponding outbound edge data block, wherein the outbound edge data block corresponding to the i-th vertex interval interval[i] is OutBlock[i]; the inbound edge pattern data of the graph data G is stored in the second storage unit and the subgraph is divided according to the vertex interval to obtain each vertex interval and its corresponding inbound edge data block, wherein the inbound edge data block corresponding to the i-th vertex interval interval[i] is InBlock[i].

[0025] Furthermore, the first storage unit and the second storage unit are solid-state hard drives or mechanical hard drives.

[0026] Furthermore, in step 2), the method for constructing the subgraph is as follows: 1) Node0 and Node1 simultaneously create the state data and attribute data of all subgraph data vertices in DRAM; 2) read the outbound edge data blocks OutBlock of all subgraphs to be generated from Node0 and enter the local PMEM; read the inbound edge data blocks InBlock of all subgraphs to be generated from Node1 and enter the local PMEM; 3) Node0 and Node1 respectively calculate the density R of each subgraph to be generated i ; 4) Node0 and Node1 set a scale parameter δ based on the statistical analysis of the vertex degree information of the graph data G, and migrate the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS to the memory DRAM of the hybrid memory system HMS; 5) Generate the i-th subgraph G based on the i-th subgraph data in the DRAM or PMEM of the node Node0 and the i-th subgraph data in the DRAM or PMEM of the node Node1 i =(V i ,E i ).

[0027] Furthermore, subgraph G i The density Among them, d v represents the degree of vertex v; |V i | represents the subgraph G i The number of vertices in .

[0028] Furthermore, the access rate of DRAM Among them, N D is the number of DRAM accesses, N P is the number of visits to PMEM, 0≤δ≤1.

[0029] Furthermore, the data of each subgraph includes graph structure data, attribute data and status data; the vertices in the graph structure data are marked as Row and the edges are marked as Col; attribute data and status data are created for each vertex; the attribute data of the vertex is marked as D curr and D next , mark the vertex state data as S curr and S next .

[0030] Furthermore, for the subgraph G i The outbound data block OutBlock[i] is stored in the node Node0 after CSR compression processing; i The input edge data block InBlock[i] is compressed by CSR and stored in node Node1.

[0031] A hybrid memory-based adaptive graph computing and updating system, characterized by comprising two NUMA nodes, denoted as Node0 and Node1; each of the NUMA nodes adopts a hybrid memory system HMS, the hybrid memory system HMS including persistent memory PMEM and DRAM; the node Node0 is provided with a preprocessing module, a subgraph construction module, and a subgraph update module;

[0032] The node Node0 is used to store the subgraph data of the outgoing edge mode; the node Node1 is used to store the subgraph data of the incoming edge mode;

[0033] The preprocessing module is used to split the vertex set V and edge set E of the graph data G = (V, E) into P non-intersecting vertex intervals V1 to V p and edge blocks E1~E p , used to generate P subgraphs G1~G P ; Where i = 1~P, the i-th vertex interval V i The vertex subset included is the subgraph G to be generated i The vertex set and edge block E i Including the subgraph G to be generated i The outbound data block OutBlock[i] and the inbound data block InBlock[i];

[0034] The subgraph construction module is used to calculate the density of each subgraph to be generated based on the outgoing edge data of each subgraph read from the node Node0, and set a proportion parameter δ based on the statistical analysis of the vertex degree information of the graph data G, and migrate the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS of the node Node0 to the memory DRAM of the hybrid memory system HMS of the node Node0; read the incoming edge data of the corresponding subgraph from the node Node1 to calculate the density of each subgraph to be generated, and according to the proportion parameter δ, migrate the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS of the node Node1 to the memory DRAM of the hybrid memory system HMS of the node Node1; then generate the i-th subgraph G based on the i-th subgraph data in the memory DRAM of the node Node0 and the i-th subgraph data in the memory DRAM of the node Node1. i =(V i ,E i );

[0035] The subgraph updating module is used to update each subgraph G i Perform iterative updates by calculating the subgraph G at the current iteration i Activity ε [i] Determine the pair of subgraphs G i The update mode adopted; which sets a ε-based [i] and the threshold function selectModel(), when the subgraph G i Activity ε [i] When the threshold value is less than the set threshold value θ, the threshold function selectModel() will return to Push mode, otherwise it will return to Pull mode; in Push mode, the subgraph G i The vertex v in retains the target vertex set D v ; In Pull mode, it is subgraph G i The vertex v in retains the source vertex set S v ; The push mode graph algorithm pushes the updated value of vertex v to D v , Pull-mode graph algorithm from S v Pull the source vertex value from to update the value of vertex v.

[0036] The advantages of the present invention are as follows:

[0037] The present invention is a new hybrid memory graph computing model with effective competitiveness following the current mainstream graph computing models, namely the stand-alone graph computing model based on traditional memory, the graph computing model based on distributed shared memory, and the graph computing model based on hybrid memory, and effectively solves the problems existing in the first three graph computing models. In short, the stand-alone graph computing model based on traditional memory is heavily dependent on large-capacity DRAM, and its computing power is limited, making it difficult to process large-scale graph data; the graph computing model based on distributed shared memory is prone to load imbalance problems during the graph partitioning process, and causes the communication overhead between distributed computing nodes to become a performance bottleneck for this type of model. For graph computing models based on emerging hybrid memory, due to the difference in read and write performance between DRAM and PMEM, remote data access overhead will become the main performance bottleneck. In response to these problems, the solution provided by the present invention is a new hybrid memory graph computing model, which completely gets rid of the problems of weak data processing capabilities, insufficient computing power, low overall cost performance, and inefficient deployment. In terms of advantages, the present invention effectively combines the dual-block graph storage representation and the data-driven adaptive push-pull strategy, and optimizes the graph data access efficiency in HMS through graph data layering and dynamic graph data migration strategies; experimentally, compared with the most advanced models currently available, it can achieve optimal performance and a better balance between efficient graph data access and efficient graph computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of the method for storing representations for Dual-Block diagrams.

[0039] Figure 2 Construct a flow chart for the subgraph based on the subgraph data layering strategy of the dual-block graph storage representation.

[0040] Figure 3 Push mode and Pull mode;

[0041] (a) is Push mode, (b) is Pull mode.

[0042] Figure 4 Graphs for the forward update mode and the backward update mode;

[0043] (a) is the original graph G, (b) is the push mode based on CSR (forward update mode), and (c) is the pull mode based on CSR (backward update mode).

[0044] Figure 5 This is a specific flow chart of the adaptive update strategy.

[0045] Figure 6 A framework for efficient hybrid memory graph computing models based on NUMA.

[0046] Figure 7 is the execution time of different update strategies;

[0047] (a) is the execution time of different update patterns in Facebook, (b) is the execution time of different update patterns in soc-LiveJournal, (c) is the execution time of different update patterns in Twitter-2010, (d) is the execution time of different update patterns in Friendster, and (e) is the execution time of different update patterns in Yahoo Web.

[0048] Figure 8 Multi-threaded execution time of PageRank and WCC on Friendster.

[0049] Figure 9 Execution time of different models on five different graph datasets;

[0050] (a) is the execution time of different models in Facebook, (b) is the execution time of different models in soc-LiveJournal, (c) is the execution time of different models in Twitter-2010, (d) is the execution time of different models in Friendster, and (e) is the execution time of different models in Yahoo Web. DETAILED DESCRIPTION

[0051] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0052] The hybrid memory-based adaptive graph computing update method and system proposed in the present invention is another competitive model following the three models of traditional memory-based stand-alone graph computing model, distributed shared memory graph computing model, and emerging hybrid memory graph computing model. First, based on the large capacity, persistence, and byte addressing capability of PMEM in HMS, a dual-block storage method is constructed to accelerate the subgraph construction process; at the same time, this storage method can effectively support the data-driven adaptive push-pull update model. Next, according to Algorithm 3, an adaptive update strategy (Adaptive Push-Pull Update Strategy) is constructed to select the corresponding computing mode: Algorithm 4 Push Mode (PushModel) and Algorithm 5 Pull Mode (Push Model). Then, with the help of the algorithms of these two modes, in Algorithm 4 Push Mode (PushModel), execute Algorithm 1 (push-based graph algorithm) or execute Algorithm 2 (pull-based graph algorithm) in Algorithm 5 (Push Model). The details of the specific technical solution will be described in four parts: (1) Dual-Block graph storage representation method, (2) subgraph data stratification strategy based on Dual-Block graph storage representation, (3) adaptive push-pull update strategy and (4) NPGraph system framework.

[0053] (1) First, the Dual-Block graph storage representation method. Similar to the solution proposed by Li et al., NPGraph divides the graph G = (V, E) into P subgraphs: G1 ~ G p That is, the vertex set V and the edge set E are divided into P non-overlapping intervals V1~V p and edge blocks E1~E p . In order to reduce data storage space, graph data is generally stored in Compressed Sparse Row (CSR) compression format. The implementation process of the Dual-Block graph representation method is as follows: 1) In the preprocessing stage, the outgoing edge mode data and the incoming edge mode data of the graph data are stored simultaneously on the solid-state drive or mechanical hard disk. 2) Then, the graph data in the two storage formats are divided into subgraphs according to the vertex intervals and the corresponding edge data. 3) The subgraph data is compressed using the CSR compression method and stored on the solid-state drive or mechanical hard disk. The process design of the Dual-Block graph storage representation method is as follows: Figure 1 shown.

[0054] (2) Then comes the subgraph data tiering strategy based on Dual-Block graph storage representation: Affected by the asymmetric scalability between DRAM and PMEM in HMS, NPGraph analyzes the data access rate of DRAM in HMS. Figure 1The execution flow, graph attribute data and state data of the in-memory graph structure mentioned in (D curr 、D next 、S curr and S next ) should be placed in DRAM. NPGraph analyzes the access rate of DRAM in HMS. Assume that the number of DRAM accesses is N D , the number of visits to PMEM is N P , then the memory access rate R D It can be expressed as:

[0055] R D =N D / (N D +N P )#(1)

[0056] Among them, according to Figure 2 The execution flow of the in-memory graph structure mentioned in N D and N p It can be expressed as:

[0057]

[0058] and

[0059]

[0060] Among them, R D Indicates the ratio of access to DRAM; Represents vertex v i Degree; N D Indicates the number of times DRAM is accessed in one iteration, which is the same as N p similar.

[0061]

[0062] Among them, R i Represents subgraph G i =(V i ,E i ) density; d v represents the degree of vertex v; |V i | represents the subgraph G i The number of vertices in .

[0063] According to the above formula (4), according to the subgraph G i =(V i ,E i ) and the degree of each vertex v in the subgraph i The number of subgraphs can be used to calculate the density R of all subgraphs. i , then R iAccording to the statistical analysis of the vertex degree information of graph data by Li et al., the ratio parameter δ loaded into memory is set to 0.2, that is, the first 20% of the densest subgraph data in PMEM is loaded into DRAM; the remaining 80% of the subgraph data is still stored in PMEM.

[0064] In the case of layered subgraph data, according to the density R i and the scale parameter δ, the subgraph structure data is selectively migrated from PMEM to DRAM. At this time, the DRAM access rate R in formula (1) is D It can be expressed as

[0065]

[0066] Among them, δ (0≤δ≤1) represents the part of the sub-image data loaded into DRAM; Indicates the DRAM access ratio when graph data is layered between DRAM and PMEM.

[0067] Formulas (2)-(5) demonstrate the effectiveness of the hierarchical placement strategy based on subgraph density in HMS. Based on the dual-block graph representation and hierarchical placement strategy, NPGraph can achieve better performance in HMS. In addition, given that remote memory access is faster than local PMEM access, NPGraph converts local PMEM data access to remote memory access when accessing data across NUMA nodes.

[0068] In the subgraph construction phase, the subgraph data of the outbound mode is loaded into the node Node0; the subgraph data of the inbound mode is loaded into the node Node1; and the outbound data of each subgraph is read from the node Node0, and the inbound data of each subgraph is read from the node Node1 to complete the construction of the subgraph. The specific process is as follows: 1) Node Node0 and node Node1 simultaneously create the status data and attribute data of all subgraph data vertices in DRAM. 2) If the creation is successful, node Node0 reads the OutBlock data of all subgraphs to be generated into the local PMEM; at the same time, node Node1 reads the InBlock data of all subgraphs to be generated into the local PMEM. 3) In nodes Node0 and Node1, the density R of each subgraph to be generated is calculated according to formula (4) i , and sort them in descending order. 4) In nodes Node0 and Node1, according to formula (4) and the ratio parameter δ of loading into memory, some dense subgraphs stored in PMEM are migrated to local DRAM. 5) Next, the subgraph G can be iterated. iThe construction operation is: read the OutBlock[i] data block from the DRAM or PMEM in the node Node0, and read the InBlock[i] data block from the DRAM or PMEM in the node Node1 through remote memory access, completing the subgraph G i The process design of the subgraph data layering strategy based on Dual-Block graph storage representation in the subgraph construction phase is as follows: Figure 2 shown.

[0069] (3) The third is the adaptive push-pull update strategy. Due to the power-law distribution of graph data, asymmetric convergence is common in iterative graph calculations. This is why sparse subgraphs usually converge quickly while dense subgraphs converge slowly. Figure 3 As shown in Figure 2, in the push model, each vertex distributes (writes) changes to its neighbors through its outgoing edges. Conversely, in the pull model, each vertex collects (reads) information from its neighbors and then uses the collected information to update its own value.

[0070] In DRAM, the constructed subgraph data includes graph structure data, attribute data, and status data. Among them, the vertices and edges in the graph structure data (that is, vertex and edge data) are marked as Row and Col; in order to facilitate the execution of graph algorithms, the attribute data and status data of the vertices need to be created. Since the attribute data and status data of the vertex correspond one-to-one to each vertex, the attribute data and status data can be directly represented by the available array. Among them, the attribute data of the vertex before this round of iterative calculation is marked as D curr The attribute data of the vertex after this round of iterative calculation is marked D next ; The state data of the vertex before this round of iterative calculation is marked as S curr The state data of the vertex after this round of iterative calculation is marked as S next Therefore, different graph algorithms, such as PageRank or WCC algorithm, can be iteratively executed on each constructed subgraph.

[0071] like Figure 4 As shown, each subgraph G i =(V i ,E i ) are associated with the forward mode (Forward manner) and the backward mode (Backward manner). i The outgoing edge storage method, after CSR compression processing, is suitable for push mode update and is marked as forward mode. Figure 4 (b) is shown. Subgraph G iThe input edge storage method of , after CSR compression processing, is suitable for pull mode update and is marked as forward mode (Backwardmanner), such as Figure 4 (c) shown.

[0072] like Figure 4 As shown in (a), the original graph G with 6 vertices and 13 edges is divided into G0 = (V0, E0) and G1 = (V1, E1). Among them, V0 = {0, 1, 2} and V1 = {3, 4, 5}. Its forward manner and backward manner are as follows Figure 4 (b) and (c) show that the execution flow starts from the active vertex, and either updates its outgoing edge neighbor data or updates its own data. Figure 4 (b) In the forward approach, during one iteration, there are two execution flows, represented by solid arrows and dashed arrows. Specifically, since S curr [0] = 1 and S curr [5] = 1, v0 and v5 are active vertices. Vertex v0 has an outgoing edge set E0 = Col[Row[0], Row[1]) = {(0,1), (0,2)}. Vertex v5 has an outgoing edge set E5 = Col[Row[5], sizeof(Row)) = {(5,2), (5,3)}. In the push model, the D corresponding to v1, v2 and v3 next [1,2,3] and S next [1,2,3] will be updated in the subsequent process. However, in the backward approach, vertices v0 and v5 need to update their own data by collecting information from their incoming edge neighbors. Specifically, v0 needs to collect information D from v3. curr [3] and S curr [3], and update its own D next [0] and S next [0]. Similarly, for vertex v5, it needs to collect information from source vertices v2 and v4, and then update its D next [5] and S next [5].

[0073] Obviously, the push update model and the pull update model are suitable for different scenarios, which depends on the subgraph G i The number of active vertices and edges in:

[0074] sum[i]=∑ v∈Vi d v #(6)

[0075] Among them, sum[i] represents the subgraph G i The degree of all vertices in ; dv Indicates the out-degree or in-degree of vertex v. Subgraph G i The number of active vertices and the degree of active vertices in the subgraph G can be described i The overall activity of can be expressed as ε[i]:

[0076] ε[i]=∑ v∈Vi S curr [v]×d v #(7)

[0077] Among them, ε[i] represents the subgraph G i Activity level; S curr [v] represents the subgraph G i All active vertices in d v Indicates the out-degree or in-degree information of the active vertex v. And the subgraph G i The relative activity of can be normalized as

[0078]

[0079] The specific process of the adaptive update strategy is as follows Figure 5 shown.

[0080] Algorithm 1 Adaptive push-pull update process of the present invention

[0081]

[0082] In an iterative graph computation process, Algorithm 1 describes the adaptive push-pull update strategy. selectModel() is a ε-based [i] and the threshold function that sets the threshold θ. [i] If θ is less than θ, it returns Push. Otherwise, it returns Pull. For each subgraph, NPGraph adaptively selects a push or pull update model to suit different graph computation tasks. The choice between push and pull modes is based on the number of active vertices and a method for predicting data access performance.

[0083] Algorithm 2 describes the execution process of the data-driven push model. It processes the vertex interval V in a forward manner. i Once a vertex v appears in the worklist, all its out-edge neighbors are traversed and updates are pushed to its out-edge neighbors D using a user-defined update function. v In the process, it will curr Read vertex values from D and write updates to D next If the target vertex D vOnce activated, they are added to the worklist and continue to be calculated in the next iteration. Algorithm 3 describes the execution process of the data-driven pull model. Similarly, the algorithm processes the vertex interval V in the corresponding Backward manner. i The entry edge.

[0084] Algorithms 2 and 3: Forward manner / Backward manner update mode of the present invention

[0085]

[0086] According to the way data is activated, graph algorithms can be divided into two categories: topology-driven graph algorithms and data-driven graph algorithms. For topology-driven graph algorithms, all vertices need to be processed in each iteration. In contrast, in data-driven graph algorithms, vertices are dynamically activated by their neighbors, that is, vertices are made active or inactive through user-defined functions. Data-driven graph algorithms allow us to focus more on "hot-vertex" in the graph that need to be updated more frequently. Therefore, in many applications, data-driven graph algorithms are more efficient than topology-driven graph algorithms. NPGraph takes the PageRank algorithm as an example to comprehensively analyze the data-driven push-pull model, and its execution process is shown in Algorithms 4 and 5. Algorithms 4 and 5 are for each subgraph G i The vertices v in each retain the target vertex set D v and the source vertex set S v As shown in Algorithm 4, the Push-based PageRank updates the value r of vertex v v Push to target vertex set D v All vertices in for the next iteration. Pull-based PageRank reads the source vertex set S when calculating the current value of vertex v. v All vertex data in to complete its own data update.

[0087] Algorithms 4 and 5: Push-based / pull-based graph algorithms of the present invention (taking PageRank as an example)

[0088]

[0089] Usually the benefit of filtering active vertices is greater than the cost of calculating all vertices. For graph algorithms, the execution order of active vertices is crucial. For example, in the PageRank algorithm based on push mode, whenever a vertex v has a new residual r v , its pr v and will be updated. The total residual r v Will reduce (1-α)*rv This suggests that the PageRank algorithm may converge faster if vertices with larger residuals are processed first.

[0090] (4) Finally, the system framework of NPGraph. In the HMS architecture, the working mode of PMEM can be switched between Memory mode and App-Drive mode. However, in Memory mode, as an extension of DRAM, PMEM is used as a volatile memory. At the same time, DRAM becomes a hardware-managed cache, which results in a read / write latency of 10ns. In addition, all DRAM cache misses and page faults flow to PMEM, which results in a read / write latency of μs. This means that Memory mode is not suitable for all application scenarios, especially large-scale graph data processing. However, in App-Drive mode, PMEM is byte-addressable like DRAM. At the same time, it provides two memory access functions: DMA (direct memory access) and RDMA (remote direct memory access). More importantly, it can read / write data in blocks (e.g., 512B, 4KB) like a hard disk. Compared with traditional HDDs and SSDs, this mode can provide better performance, lower latency, and better durability. Therefore, App-Drive mode is particularly suitable for large-scale graph processing, and it provides a more cost-effective solution for graph computing.

[0091] The model framework is as follows Figure 6As shown in the figure, it is designed for NUMA-based HMS. It consists of two NUMA nodes, denoted as Node0 and Node1. Each NUMA node contains 16GB of memory (DRAM) and 256GB of PMEM, and the external memory is composed of HDD or SSD. It uses two NUMA nodes (Node) to store outbound (OutBlock) and inbound (InBlock) graph data respectively. Specifically, in the preprocessing stage, the subgraph is divided according to the vertex interval; and the Compressed Sparse Row (CSR) data compression method is used to compress the outbound (OutBlock) data and inbound (InBlock) data; then the external memory (HDD or SSD) simultaneously stores the forward manner compressed graph data (based on the push update method of OutBlock) and the backward manner compressed graph data (based on the pull update method of InBlock); in the data loading stage, node Node0 loads all forward manner outbound data, and node Node1 loads all backward manner inbound data. In each node, DRAM mainly stores all vertices, vertex state data, and some dense subgraph edge block data; PMEM mainly stores all other subgraph data edge block data.

[0092] In addition, the execution time of graph algorithms is proportional to the number of memory accesses for active vertices and edges. Therefore, the dynamic migration of graph data between DRAM and PMEM should be considered in HMS. curr [v], D v and S v It can reflect the activity of different subgraphs. During the iteration of the graph algorithm, the migration strategy calculates the relative activity ε of each subgraph in DRAM and PMEM respectively. By using the memcpy() function, NPGraph can perform swap operations between DRAM and PMEM. By adopting an adaptive push-pull update strategy and dynamic data migration strategy, NPGraph can maximize the cache hit rate in DRAM. In addition, binding processes to specific cores and disabling NUMA Balance can help improve the efficiency of graph data access between different NUMA nodes.

[0093] The following describes the experimental data in detail. First, we introduce the experimental environment configuration, experimental dataset, and comparison algorithms used for the NPGraph experiment, and then present the experimental results and analysis.

[0094] (1) First, let’s configure the experimental environment. All experiments were conducted on two NUMA nodes. Each node includes an Intel Xeon Gold 5218R CPU with 20 cores, 40 threads, 32KB L2 cache, and 27.5MB L3 cache. To achieve experimental conditions with limited DRAM resources, each NUMA node is equipped with 16GB of DRAM and 265GB of Optane DC PMEM modules. The experimental platform runs on an Ubuntu 18.04LTS system, and experiments on large-scale graph data processing are conducted.

[0095] Table 1 Brief introduction to experimental environment configuration

[0096] Device Name Specific parameters CPU 2×Intel Xeon Gold 5218R@2.10GHz L1 Cache 32KB, 64B cache block L2 Cache 32KB, 64B cache block L3 Cache 28160KB, 64B cache block DRAM 4×8GB DDR4, 2666MHz PMEM 4×128GB Intel Optane DC,2666MHz

[0097] (2) The datasets and comparison algorithms are then introduced. All graph datasets used in the experiments are public, real-world graph data. The social network graph datasets Facebook and soc-LiveJournal are selected to verify NPGraph. In addition, the large-scale graph datasets Twitter-2010, Friendster, and Yahoo Web are social networks, game websites, and websites, respectively. They are composed of billions of vertices and edges and have large diameters. Specifically, the storage scale of the following three graph datasets based on the dual-block graph representation method is 1.56 times, 1.94 times, and 6.81 times the DRAM capacity, respectively.

[0098] Table 2 Introduction to benchmark datasets

[0099] Dataset Number of vertices Number of sides type Facebook 4,039 88,234 Social Graphs soc-LiveJournal 4,847,571 68,993,773 Social Graphs Twitter-2010 61,578,416 246,313,664 Social Graphs Friendster 65,608,366 1,806,067,135 Game Graphs Yahoo Web 1,413,511,424 5,654,045,696 Web Graphs

[0100] The basic graph algorithms used in the evaluation include PageRank, a representative sparse matrix multiplication algorithm, and WCC, a graph traversal-based algorithm. In the experiments, PageRank was set to run ten iterations, while WCC was run until convergence. These two algorithms exhibit different computational properties, allowing NPGraph to be evaluated in terms of both computation and traversal. Finally, NPGraph was compared with state-of-the-art systems, GraphOne and XPGraph.

[0101] (3) Impact of Adaptive Push-Pull Strategy: Using the WCC algorithm and PageRank algorithm, we conduct a comprehensive comparison of the Forward-Push model, Backward-Pull model, and Dual-Adaptive model on the five public datasets mentioned above. The single-thread execution time of the three update models is shown in the figure below. Figure 7 shown.

[0102] Clearly, the Dual-Adaptive model outperforms both the Forward-Push and Backward-Pull models overall. More specifically, for WCC, the Dual-Adaptive model outperforms the Forward-Push model by 15.3%-27.6%. For PageRank, the Dual-Adaptive model outperforms the Forward-Push model by 14.7%-28.9%.

[0103] It's not hard to see that the Forward-Push model outperforms the Backward-Pull model in this experiment. Generally speaking, the Backward-Pull model has more read characteristics and is friendlier to graph data caching. However, in real-world graph data, because graph data often exhibits small-world properties, a large amount of random data access occurs during the iterative computation of graphs. This can lead to the fact that optimizing for caching behavior (the Backward-Pull model) isn't necessarily more efficient than optimizing for fast propagation (the Forward-Push model). In other words, when it comes to updating information, the Forward-Push model has an advantage over the Backward-Pull model. The additional write operation in the Forward-Push model isn't just another way to implement vertex updates; it also affects task scheduling. It makes it easier to transition vertex states, and this improved scheduling compensates for the time overhead of the write load.

[0104] (4) Impact of data tiering strategy: To verify the effectiveness of the data tiering strategy in HMS, NPGraph conducted a multi-threaded (1 to 64 threads) comparative experiment on Friendster. As mentioned above, a graph G is divided into P subgraphs. According to the offset of Row and the size of Col, it is easy to determine the sparseness or density of the subgraph. Therefore, it is a natural method to load dense subgraphs into DRAM first. The comparative experiment is divided into two cases: one is to load all the structural data of the subgraph into PMEM. In this case, Figure 8 As shown, PageRank and WCC are labeled PR and WCC. An alternative data placement strategy is presented for comparison. Based on the DRAM and dataset size, NPGraph loads the top 20% of dense subgraphs into memory. In this case, PageRank and WCC are labeled PR-L and WCC-L, respectively. Importantly, NPGraph employs an adaptive push-pull strategy for optimal performance. Furthermore, exchange primitives such as atomic_compare_exchange() are used in the code.

[0105] from Figure 8As can be seen from the comparison experiment of the two data placement strategies, the performance of PR-L and WCC-L is 1.51-1.83 times that of PR and WCC. And as the number of threads increases, the execution time gradually decreases. This trend proves that the graph data tiering strategy can make full use of DRAM and PMEM in HMS. Therefore, the overall performance of NPGraph can be significantly improved by the data tiering strategy. Based on the above observations, we can draw two conclusions: (1) Compared with WCC, PageRank achieves a greater performance improvement due to its sensitivity to heterogeneous memory. (2) From the trend of changes in multi-threaded execution time, it can be seen that NPGraph has good parallel scalability.

[0106] (5) Impact of Dynamic Data Migration Strategies: To evaluate the effectiveness of data migration strategies in HMS, NPGraph ran the WCC and PageRank algorithms on the five datasets mentioned above. NPGraph labels the adaptive push-pull mode, data tiering mode, and data migration mode as NPGraph-A, NPGraph-L, and NPGraph-M. To achieve optimal performance, 64 worker threads were used to accelerate the NPGraph graph computation model.

[0107] Table 3 Execution time of WCC and PageRank (seconds)

[0108]

[0109] As shown in Table 3, NPGraph-M outperforms NPGraph-L overall, while NPGraph-L outperforms NPGraph-A. Specifically, in the WCC and PageRank algorithms, NPGraph-M outperforms NPGraph-A by 14.3%-34.6%. Furthermore, in WCC, NPGraph-M outperforms NPGraph-L by 14.1%-23.5%. Furthermore, in PageRank, NPGraph-M outperforms NPGraph-L by 12.9%-18.5%.

[0110] By analyzing the experimental results in Table 3, we confirm the effectiveness of the dynamic data migration strategy in NPGraph. This strategy further improves NPGraph's performance. This further demonstrates that combining data stratification and dynamic data migration can simultaneously improve both spatial and temporal locality of graph computations in HMS.

[0111] (6) Comparison with other models: NPGraph is compared with state-of-the-art memory models: GraphOne and XPGraph. Both models support graph computation in parallel mode. To achieve optimal performance, NPGraph is provided with 16GB of DRAM, 256GB of PMEM, and two NUMA nodes. To ensure fairness, GraphOne, XPGraph, and NPGraph are run in the same environment with 64 threads. Figure 9 Execution times for five different graph datasets are shown.

[0112] like Figure 9 As shown, NPGraph significantly accelerates GraphOne and XPGraph. Specifically, NPGraph achieves 27.36% to 43.8% improvements over GraphOne on Pagerank and WCC algorithms. Furthermore, it achieves 21.67% to 32.03% improvements over XPGraph on Pagerank and WCC algorithms. This performance improvement is attributed to NPGraph's dual-block graph storage representation and its supporting data-driven adaptive push-pull update model.

[0113] To store extremely large-scale graph data, GraphOne uses a hybrid storage format that combines two of the most commonly used in-memory graph storage formats: edge lists and adjacency lists. It employs a vertex-centric random access pattern to traverse graph-structured data. However, because it ignores the dynamic nature of graph computation, it must traverse all in-memory graph data, resulting in inefficiencies. Furthermore, it generates a large number of intermediate result writes, incurring significant I / O overhead. The NUMA-based XPGraph develops an XPLine-friendly graph access model with a vertex-centric graph buffer. However, it primarily addresses the storage issues of large-scale dynamic graphs. It ignores the load imbalance between multiple threads during each iteration. This is precisely the greatest advantage of NPGraph. In a NUMA-based hybrid memory system, NPGraph, leveraging a dual-block graph storage representation and a data-driven adaptive push-pull strategy, reduces graph data access costs and improves the efficiency of parallel computations on graphs.

[0114] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A hybrid memory-based adaptive graph computing update method, comprising the following steps: 1) Set up two NUMA nodes, denoted as Node0 and Node1. Each of the NUMA nodes uses a hybrid memory system HMS, which includes persistent memory PMEM and DRAM. The node Node0 is equipped with a preprocessing module, a subgraph construction module, and a subgraph update module. 2) The preprocessing module divides the vertex set V and edge set E of the graph data G = (V, E) into P non-intersecting vertex intervals V1 to V p and edge blocks E1~E p , used to generate P subgraphs G1~G P ; Where i = 1~P, the i-th vertex interval V i The vertex subset included is the subgraph G to be generated i The vertex set and edge block E i Including the subgraph G to be generated i The vertex interval V i The outbound data block OutBlock[i] of the source vertex and the vertex interval V i Enter the edge data block InBlock[i] for the destination vertex; then load the subgraph data in outgoing edge mode into node Node0, and load the subgraph data in incoming edge mode into node Node1; 3) The subgraph construction module calculates the density of each subgraph to be generated based on the outgoing edge data of each subgraph read from the node Node0, and sets a proportion parameter δ based on the statistical analysis of the vertex degree information of the graph data G, and migrates the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS of the node Node0 to the memory DRAM of the hybrid memory system HMS of the node Node0; and reads the incoming edge data of the corresponding subgraph from the node Node1 to calculate the density of each subgraph to be generated, and according to the proportion parameter δ, migrates the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS of the node Node1 to the memory DRAM of the hybrid memory system HMS of the node Node1; then the subgraph construction module generates the i-th subgraph G based on the i-th subgraph data in the local memory DRAM of the node Node0 and the i-th subgraph data in the memory DRAM of the remote access Node1. i =(V i ,E i ); 4) Subgraph update module updates each subgraph G i Perform iterative updates by calculating the subgraph G at the current iteration i Activity ε [i] Determine the pair of subgraphs G i The update mode adopted; which sets a ε-based [i] and the threshold function selectModel(), when the subgraph G i Activity ε [i] When the threshold value is less than the set threshold value θ, the threshold function selectModel() will return to Push mode, otherwise it will return to Pull mode; in Push mode, the subgraph G i The vertex v in retains the target vertex set D v ; In Pull mode, it is subgraph G i The vertex v in retains the source vertex set S v ; The push mode graph algorithm pushes the updated value of vertex v to D v , Pull-mode graph algorithm from S v Pull the source vertex value from to update the value of vertex v.

2. The method according to claim 1, characterized in that The specific implementation method of step 1) is: first, the outbound edge pattern data of the graph data G is stored in the first storage unit and the subgraph is divided according to the vertex interval to obtain each vertex interval and its corresponding outbound edge data block, wherein the outbound edge data block corresponding to the i-th vertex interval interval[i] is OutBlock[i]; the inbound edge pattern data of the graph data G is stored in the second storage unit and the subgraph is divided according to the vertex interval to obtain each vertex interval and its corresponding inbound edge data block, wherein the inbound edge data block corresponding to the i-th vertex interval interval[i] is InBlock[i].

3. The method according to claim 2, characterized in that The first storage unit and the second storage unit are solid-state hard drives or mechanical hard drives.

4. The method according to claim 1, 2 or 3, characterized in that: In step 2), the method for constructing the subgraph is as follows: 1) Node 0 and Node 1 simultaneously create the state data and attribute data of all subgraph data vertices in DRAM; 2) Read all the outbound data blocks OutBlock of the subgraph to be generated from node Node0 and store them in the local PMEM; read all the inbound data blocks InBlock of the subgraph to be generated from node Node1 and store them in the local PMEM; 3) Node Node0 and node Node1 calculate the density R of each subgraph to be generated respectively. i 4) Node 0 and Node 1 each set a scaling parameter δ based on statistical analysis of the vertex degree information of the graph data G, and migrate the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS to the memory DRAM of the hybrid memory system HMS; 5) Generate the i-th subgraph G according to the i-th subgraph data in the DRAM or PMEM of node Node0 and the i-th subgraph data in the DRAM or PMEM of Node1 i =(V i ,E i ).

5. The method according to claim 1, 2 or 3, characterized in that: Subgraph G i The density Among them, d v represents the degree of vertex v; |V i | represents the subgraph G i The number of vertices in .

6. The method according to claim 1, 2 or 3, characterized in that: DRAM access rate Among them, N D is the number of DRAM accesses, N P is the number of visits to PMEM, 0≤δ≤1.

7. The method according to claim 1, 2 or 3, characterized in that: The data of each subgraph includes graph structure data, attribute data and status data; the vertices in the graph structure data are marked as Row and the edges are marked as Col; an attribute data and status data are created for each vertex; the attribute data of the vertex is marked as D curr and D next , mark the vertex state data as S curr and S next .

8. The method according to claim 1, characterized in that Pair graph G i The outbound data block OutBlock[i] is stored in the node Node0 after CSR compression processing; i The input edge data block InBlock[i] is compressed by CSR and stored in node Node1.

9. An adaptive graph computing and updating system based on hybrid memory, characterized in that: It includes two NUMA nodes, denoted as Node0 and Node1; each of the NUMA nodes uses a hybrid memory system HMS, which includes persistent memory PMEM and memory DRAM; the node Node0 is provided with a pre-processing module, a subgraph construction module, and a subgraph update module; The node Node0 is used to store the subgraph data of the outgoing edge mode; the node Node1 is used to store the subgraph data of the incoming edge mode; The preprocessing module is used to split the vertex set V and edge set E of the graph data G = (V, E) into P non-intersecting vertex intervals V1 to V p and edge blocks E1~E p , used to generate P subgraphs G1~G P ; Where i = 1~P, the i-th vertex interval V i The vertex subset included is the subgraph G to be generated i The vertex set and edge block E i Including the subgraph G to be generated i The outbound data block OutBlock[i] and the inbound data block InBlock[i]; The subgraph construction module is used to calculate the density of each subgraph to be generated based on the outgoing edge data of each subgraph read from the node Node0, and set a proportion parameter δ based on the statistical analysis of the vertex degree information of the graph data G, and migrate the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS of the node Node0 to the memory DRAM of the hybrid memory system HMS of the node Node0; read the incoming edge data of the corresponding subgraph from the node Node1 to calculate the density of each subgraph to be generated, and according to the proportion parameter δ, migrate the densest δ*P subgraph data to be generated in the persistent memory PMEM of the hybrid memory system HMS of the node Node1 to the memory DRAM of the hybrid memory system HMS of the node Node1; then generate the i-th subgraph G based on the i-th subgraph data in the memory DRAM of the node Node0 and the i-th subgraph data in the memory DRAM of the node Node1. i =(V i ,E i ); The subgraph updating module is used to update each subgraph G i Perform iterative updates by calculating the subgraph G at the current iteration i Activity ε [i] Determine the pair of subgraphs G i The update mode adopted; which sets a ε-based [i] and the threshold function selectModel(), when the subgraph G i Activity ε [i] When the threshold value is less than the set threshold value θ, the threshold function selectModel() will return to Push mode, otherwise it will return to Pull mode; in Push mode, the subgraph G i The vertex v in retains the target vertex set D v ; In Pull mode, it is subgraph G i The vertex v in retains the source vertex set S v ; The push mode graph algorithm pushes the updated value of vertex v to D v , Pull-mode graph algorithm from S v Pull the source vertex value from to update the value of vertex v.

Images