A method and apparatus for data recovery after server crash restart based on ultra-large graph computing

Abstract

This invention relates to the field of graph computing technology, and provides a method and apparatus for data recovery after server crash restart based on ultra-large graph computing. Multi-threaded concurrent access to pagination is employed, with each thread traversing the vertex IDs of one pagination, iterating to obtain intermediate results for the i-th round. These intermediate results are stored in a TABLE. After each i-th round of computation, the intermediate computation results are sent to other nodes uniformly via the TABLE.<PARTITION_INDEX,HOST> Obtain the network address (HOST) of the node corresponding to the page number, and then obtain the loading data (MAP) on the compute node of that HOST.<ID,VALUE> The system loads server memory data according to the adjacency relationship. This invention optimizes how to achieve rapid memory data reorganization in a distributed architecture when server data needs to be loaded from different network nodes, even during system crashes and restarts.

Description

[Technical Field]

[0001] This invention relates to the field of graph computing technology, and in particular to a method and apparatus for data recovery after server crash restart based on ultra-large graph computing. [Background Technology]

[0002] Distributed graph computing has become a popular method for network structure data analysis and computation in recent years, offering more complex relational processing capabilities compared to traditional MapReduce. Currently, the most popular distributed graph computing engines include Spark GraphX ​​and Flink Gelly, both of which feature high-performance in-memory computing capabilities. Spark provides distributed RDD datasets, while Flink provides distributed DataSet datasets.

[0003] The biggest problem for databases during operation is server downtime. Once a server goes down, it takes a long time to restore the data state. The most common method is to read historical state data from the operation logs. However, when the data is stored in a distributed manner, the efficiency bottleneck becomes even more apparent.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. [Summary of the Invention]

[0005] The technical problem this invention aims to solve is that the most feared event during database operation is server downtime. Once a downtime occurs, the process of restoring the data state of the corresponding server is time-consuming. The most conventional method is to read historical state data according to the operation log. However, when the corresponding data content is distributed storage, the efficiency bottleneck becomes even more apparent.

[0006] The present invention adopts the following technical solution:

[0007] Firstly, this invention provides a method for data recovery after a server crash and restart based on ultra-large graph computing. The server's logs record paging information of the currently loaded data in memory. This paging information includes the network address of the distributed computing node where the loaded data resides, and also includes the adjacency relationships between the paging objects. When the server restarts from a crash state, the method includes:

[0008] The server sends a cluster status information request (request_cluster_infomation()) to the gateway node and retrieves the returned cluster status information; wherein, the cluster status information includes the number of compute nodes (N-COMPUTATION) and the node routing table (TABLE).<PARTITION,HOST> ;

[0009] The server retrieves the number of graph vertices N-VERTICES from the graph storage engine using get_vertex_count(); it then calculates the number of nodes by taking the modulo of the total number of vertices, and calculates the number of graph vertices N-LENGTH that each compute node needs to load.

[0010] Load the adjacency relationships and traverse the graph vertex IDs;

[0011] Multi-threaded concurrent access to pagination: each thread traverses the vertex IDs of one pagination, iterating to obtain intermediate results for the i-th iteration. These intermediate results are stored in a TABLE. <PARTITION,MAP<ID,VALUE> After each i-th round of computation, intermediate computation results are sent to other nodes via TABLE.<PARTITION_INDEX,HOST> Obtain the network address (HOST) of the node corresponding to the page number, and then obtain the loading data (MAP) on the compute node of that HOST.<ID,VALUE> And load the server memory data according to the adjacency relationship; where i is a natural number.

[0012] Preferably, before traversing the graph vertex IDs and loading the adjacency relationships, the method further includes:

[0013] To analyze whether N-LENGTH vertices require multi-page memory processing, divide N-LENGTH by the paging threshold. If the value is greater than 1, the server requests multiple pages, with the maximum number of pages being the value + 1; if the value is less than 1, the server directly requests a single page of memory space.

[0014] Preferably, the paging threshold size is 2GB, which can be used to accommodate a Long type array with a length of 268435456.

[0015] Preferably, during the process of traversing the vertex IDs of the graph, the method further includes:

[0016] To ensure load balancing across all compute nodes, a discrete hop-point approach is used to read vertex IDs. Each compute node loads the ID adjacency sequence PARTITI_INDEX + (N*N-COMPUTE) to avoid data skew caused by super vertices within a certain ID range. The adjacent point ID array of the vertex ID is queried from the storage engine using get_v_neighbors(ID), and recorded in the server's memory page until all compute nodes have loaded all adjacency data into the memory page.

[0017] Preferably, loading the adjacency relationships and traversing the graph vertex IDs specifically includes:

[0018] After loading the adjacency relationships, the closed-loop relationships in the corresponding adjacency relationships are used to form the basis for the vertex ID traversal order.

[0019] Preferably, after each i-th round of calculation, intermediate calculation results are sent to other nodes in a unified manner, specifically including:

[0020] After receiving the intermediate calculation results and clarifying the adjacency relationship between itself and the node currently interacting with the data, other nodes pre-read the data to be returned to the server into a buffer so that they can complete the response process in a timely manner when a data request is received.

[0021] Preferably, after completing a round of data loading following a system crash, if statistics show that data in a page is distributed across more than a preset threshold number of nodes, the method further includes:

[0022] Data objects that were originally scattered across multiple nodes but correspond to data loaded in the same page are transferred according to load balancing parameters, so that the number of nodes where the data is scattered in the corresponding page is less than the preset threshold after the transfer.

[0023] Preferably, the load balancing specifically includes one or more of the following: network latency, node data access response speed, remaining storage space, historical access frequency and peak value, and historical CPU and memory utilization.

[0024] Secondly, the present invention also provides a server crash restart data recovery device based on ultra-large graph computing, used to implement the server crash restart data recovery method based on ultra-large graph computing described in the first aspect, the device comprising:

[0025] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the server crash restart data recovery method based on ultra-large graph computing as described in the first aspect.

[0026] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the server crash restart data recovery method based on ultra-large graph computing described in the first aspect.

[0027] This invention optimizes how to achieve rapid in-memory data reorganization in a distributed architecture when server data needs to be loaded from different network nodes, even during system crashes and restarts. This method applies a graph computing engine to in-memory data loading, representing the graph data in memory while ensuring balanced data paging, enabling in-memory representation of hundreds of billions of graph data points. [Attached Image Description]

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0029] Figure 1 This is a schematic diagram of a server crash restart data recovery method based on ultra-large graph computing provided in an embodiment of the present invention;

[0030] Figure 2 This is an example of a pagination structure in a server crash restart data recovery method based on ultra-large graph computing provided in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of a server crash restart data recovery method based on ultra-large graph computing provided in an embodiment of the present invention;

[0032] Figure 4 This is a data partitioning diagram in a server crash restart data recovery method based on ultra-large graph computing provided in an embodiment of the present invention;

[0033] Figure 5 This is a data partitioning diagram in a server crash restart data recovery method based on ultra-large graph computing provided in an embodiment of the present invention;

[0034] Figure 6 This is a routing representation schematic in a server crash restart data recovery method based on ultra-large graph computing provided in an embodiment of the present invention;

[0035] Figure 7 This is a schematic diagram of the adjacency loading process in a server crash restart data recovery method based on ultra-large graph computing provided in an embodiment of the present invention;

[0036] Figure 8 This is a schematic diagram of the message sending process in a server crash restart data recovery method based on ultra-large graph computing provided in an embodiment of the present invention;

[0037] Figure 9 This is a schematic diagram of a server crash restart data recovery device based on ultra-large graph computing, provided in an embodiment of the present invention.

Detailed Implementation Methods

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0040] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] The main computational model for graph computing is a vertex-centered Pregel iterative model. This model is characterized by using vertices as the smallest unit of computation. During the iterative computation process, vertices are used to send and receive intermediate values ​​of the computation results, and these intermediate values ​​are used to carry out the next round of iteration.

[0042] In large-scale graph computing scenarios, single-machine computing models and distributed computing models differ significantly. Single-machine in-memory computing suffers from RAM capacity bottlenecks; distributed in-memory computing, by load balancing graph data, can overcome the RAM usage bottleneck of single-machine computing. Furthermore, a distributed adjacency list is used to represent the native graph model G.<V,E> The adjacency list method is used to reduce the number of messages sent in a distributed network. However, existing adjacency list methods are mostly used for relational physical storage tables, which are not effective in server crash and restart scenarios.

[0043] In the Spark Distributed Graph dataset, a set of VertexRDD and EdgeRDD triples is used to represent the graph. Each VertexRDD contains...<VID,VTYPE,PARTITION> Elements, contained in each EdgeRDD<SRCID,DSTID,ATTR> The elements in this dataset are represented sequentially in a two-dimensional table structure of DataFrames at each node. This invention uses an adjacency list approach based on native arrays to represent distributed graph data in memory, balancing in-memory data paging with fast lookup of vertex index IDs using PARTITION_INDEX.

[0044] Example 1:

[0045] Embodiment 1 of this invention provides a data recovery method for server crash restart based on ultra-large graph computing. The corresponding logs in the server record the paging information of the currently loaded data in memory, and the corresponding paging information includes the network address of the distributed computing node where the loaded data is located (e.g., ...). Figure 3 The diagram shows an example of the data structure for paging information. Here, PAGE_INDEX is the paging index number, VERTEX_COUNT is the vertex number corresponding to the paging, and ADJACENT_LIST is the list of adjacent vertices. The paging information also includes the adjacency relationships between the paging objects (see reference). Figure 2 As shown, it represents the adjacency relationship between pages, represented by vertices. Taking the first row as an example, the adjacency of vertex 1 is vertex 2, vertex 3, and vertex 4); when the server restarts from a crash state, as... Figure 1 As shown, it includes:

[0046] In step 201, the server sends a cluster status information request (request_cluster_infomation()) to the gateway node and obtains the returned cluster status information; wherein, the cluster status information includes the number of compute nodes N-COMPUTATION and the node routing table TABLE.<PARTITION,HOST> .

[0047] In step 202, the server obtains the number of graph vertices N-VERTICES = get_vertex_count() from the graph storage engine; it calculates the number of nodes by taking the modulo of the total number of vertices, and calculates the number of graph vertices N-LENGTH that each computing node needs to load.

[0048] In step 203, the adjacency relationship is loaded, and the vertex IDs of the graph are traversed.

[0049] In step 204, multiple threads concurrently access the pagination. Each thread traverses the vertex IDs in one pagination, iterating to obtain the intermediate results of the values ​​in the i-th round. These intermediate results are stored in a TABLE. <PARTITION,MAP<ID,VALUE> After each i-th round of computation, intermediate computation results are sent to other nodes via TABLE.<PARTITION_INDEX,HOST> Obtain the node network address HOST corresponding to the page number (see reference). Figure 6 (See the network address list diagram shown). Obtain the loading data MAP on this HOST compute node.<ID,VALUE> The server memory data is loaded according to the adjacency relationship; where i is a natural number. The corresponding paginated data content presentation can be found in the following reference. Figure 4 and Figure 5 .

[0050] This invention optimizes how to achieve rapid in-memory data reorganization in a distributed architecture when server data needs to be loaded from different network nodes, even during system crashes and restarts. This method applies a graph computing engine to in-memory data loading, representing the graph data in memory while ensuring balanced data paging, enabling in-memory representation of hundreds of billions of graph data points.

[0051] In conjunction with embodiments of the present invention, there is a preferred extended implementation scheme, in which, before traversing the graph vertex IDs and loading the adjacency relationships, the method further includes:

[0052] To analyze whether N-LENGTH vertices require multi-page memory processing, divide N-LENGTH by a paging threshold. If the value is greater than 1, the server requests multiple pages, with the maximum number of pages being the value + 1. If the value is less than 1, the server directly requests a single page of memory. The paging threshold is set to 2GB, which can accommodate a Long array of length 268435456.

[0053] In conjunction with embodiments of the present invention, there is a preferred extended implementation scheme in which the method further includes the following during the traversal of graph vertex IDs:

[0054] To ensure load balancing across all compute nodes, a discrete hop-point approach is used to read vertex IDs. Each compute node loads the ID adjacency sequence PARTITI_INDEX + (N*N-COMPUTE) to avoid data skew caused by super vertices within a certain ID range. The adjacent point ID array of the vertex ID is queried from the storage engine using get_v_neighbors(ID), and recorded in the server's memory page until all compute nodes have loaded all adjacency data into the memory page.

[0055] In conjunction with embodiments of the present invention, there is a preferred extended implementation scheme, which loads the adjacency relationship and traverses the graph vertex IDs, specifically including:

[0056] After loading the adjacency relationships, the closed-loop relationships in the corresponding adjacency relationships are used to form the basis for the vertex ID traversal order.

[0057] In conjunction with the embodiments of the present invention, there is a preferred extended implementation scheme in which intermediate calculation results are uniformly sent to other nodes after each i-th round of calculation, specifically including:

[0058] After receiving the intermediate calculation results and clarifying the adjacency relationship between itself and the node currently interacting with the data, other nodes pre-read the data to be returned to the server into a buffer so that they can complete the response process in a timely manner when a data request is received.

[0059] In conjunction with embodiments of the present invention, there is a preferred extended implementation scheme. After completing a round of data loading following a system crash, if statistics show that data in a page is distributed across more than a preset threshold number of nodes, the method further includes:

[0060] Data objects that were originally scattered across multiple nodes but correspond to data loaded in the same page are transferred according to load balancing parameters, so that the number of nodes where the data is scattered in the corresponding page is less than the preset threshold after the transfer.

[0061] The load balancing specifically includes one or more of the following: network latency, node data access response speed, remaining storage space, historical access frequency and peak value, and historical CPU and memory utilization.

[0062] Example 2:

[0063] Currently, most mainstream computing engines use a two-dimensional table data structure when processing graph model data.<VID,VTYPE,PARTITION> The set is used to represent the vertex data of the entire graph, in order to<SRCID,DSTID,ATTR> This model represents the edge data of the entire graph. It stores the data in distributed data shards across different computing machine memories. Typically, a vertex-partitioned model reduces the amount of message communication at critical points during computation. It performs well on sparse graphs, achieving load balancing. However, computation on dense graphs can lead to severe data skew, where dense vertices are distributed across shards on the same machine, placing a heavy burden on the machine's memory and hardware resources such as CPU and memory. During computation, each vertex needs to request adjacent edge data shards to find its corresponding output vertex ID and send the intermediate computation results to that output vertex.

[0064] To reduce data skew and data sharding requests at output points, this invention provides a distributed adjacency list method for ultra-large graphs, including: dividing vertices and loading them onto different computing nodes, and recording routing information; traversing the adjacency relationships of vertices and loading them into node data pages (adjacency list) while simultaneously paging the data pages (adjacency list); and sending messages to the nodes where adjacent points are located during computation.

[0065] Step 1: Distribute vertices to different computing nodes for loading. In the initial stages of computation, computing nodes load vertices into memory in parallel, storing them in a one-dimensional array. The loading rule is to distribute loading evenly according to the total number of vertices and the number of computing nodes. Computing nodes use an auto-incrementing page index from 0 to N. During vertex loading, the vertex ID is modulo the computing node's page index; if the index is integer, the vertex is placed in the one-dimensional array. The computing node's page number and host address information are recorded in the routing table. This discrete, hop-point loading method helps resolve data skew issues in the subsequent adjacency list.

[0066] Step 2: Traverse vertex adjacency relationships. After step 1, each computing node sequentially traverses the one-dimensional array of vertex IDs in parallel. Based on the number of vertices, it allocates memory pages. If the number of vertices is less than the maximum number of pages, data is stored in a single page; otherwise, multiple pages are allocated. The graph storage engine is accessed via I / O, and adjacent vertices are written to the corresponding memory pages according to the adjacency list. This paging method breaks down the large contiguous memory space allocation into discrete memory page blocks. Simultaneously, the memory page containing the adjacent data can be found using the vertex ID index. (See reference...) Figure 7 The process is shown.

[0067] Step 3: During computation, messages are sent to the vertices of adjacent nodes. Vertices within each computation page begin iterative computation in parallel. In algorithms requiring adjacency relationships, the memory page for adjacency data is located using the vertex ID index, the set of adjacent vertex IDs is obtained, and the actual algorithm computation is performed. The computation message is then sent to the computation node where the adjacent node is located. Since this embodiment uses a jump-point loading method in Step 1, the page index of that point can be obtained directly using modulo. Then, the address of the computation node corresponding to that page can be found through the HOST mapping in the routing table, thus sending the computation message. (See reference...) Figure 8 The process is shown.

[0068] Example 3:

[0069] This embodiment of the invention will provide a more complete description of the method described in Embodiment 1, combined with a specific example scenario. There is a distributed system consisting of a graph computing gateway node, a cluster of 10 graph computing nodes, and a graph storage engine cluster. The graph storage engine cluster stores an offline attribute graph with 50 billion vertices and 100 billion edges on disk. The vertices in this graph use an auto-incrementing ID strategy of type Long, so the vertex ID range is (0, 50, 000, 000, 000).

[0070] The graph computing gateway node receives a request for the PageRank algorithm on the current very large graph. The graph computing process is distributed to computing nodes 1-10, with the PARTITION of the computing nodes between [1, 10].

[0071] After each computing node receives the algorithm request sent by the graph computing gateway, it performs the first step:

[0072] The current number of vertices loaded on the compute node is calculated. The compute node sends a cluster status information request (request_cluster_infomation()) to the gateway node, providing a set of cluster status information including the number of compute nodes (N-COMPUTATION) and a compute node routing table (TABLE).<PARTITION,HOST> The compute node requests the number of graph vertices N-VERTICES from the graph storage engine. Then, by taking the total number of vertices modulo the number of compute nodes, the number of graph vertices N-LENGTH that each compute node needs to load can be calculated.

[0073] After calculating the number of vertices loaded into the compute node, perform the second step:

[0074] The current compute node's initial memory paging is determined by first calculating whether N-LENGTH vertices require multi-page memory processing. This is done by dividing N-LENGTH by the paging threshold. If the value is greater than 1, multiple pages need to be allocated, with the maximum number of pages being the value plus 1. If the value is less than 1, a single page of memory is allocated directly. Typically, a page size is 2GB, which can accommodate a Long array of length 268435456.

[0075] After memory paging is completed, the third step is performed:

[0076] Traverse vertex IDs and load adjacency relationships. To ensure load balancing across all compute nodes, a discrete hop-point approach is used to read vertex IDs. That is, each compute node loads the ID adjacency sequence PARTITI_INDEX + (N*N-COMPUTE) to avoid data skew caused by super vertices within a certain ID range. The adjacent point ID array of the vertex ID is queried from the storage engine using get_v_neighbors(ID), and recorded in the memory page until all compute nodes have loaded all adjacency data into the memory page.

[0077] After loading the adjacency relationships, proceed to the fourth step:

[0078] Distributed algorithm computation. Computational nodes begin superstep iterative computation, with multiple threads concurrently accessing the pagination. Each thread traverses the vertex IDs within a page, iterating to obtain intermediate results for the i-th round. These intermediate results are stored in a TABLE. <PARTITION,MAP<ID,VALUE> In this process, after each i-th iteration of computation, intermediate computation results are sent to other nodes via a TABLE.<PARTITION_INDEX,HOST> Obtain the HOST address corresponding to the page number, and send the MAP to the compute node of that HOST.<ID,VALUE> .

[0079] Example 4:

[0080] like Figure 9 The diagram shown is an architectural schematic of a server crash restart data recovery device based on ultra-large graph computing according to an embodiment of the present invention. This embodiment of the server crash restart data recovery device based on ultra-large graph computing includes one or more processors 21 and a memory 22. Wherein, Figure 9 Take a processor 21 as an example.

[0081] Processor 21 and memory 22 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0082] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the server crash restart data recovery method based on ultra-large graph computing in Embodiment 1. The processor 21 executes the server crash restart data recovery method based on ultra-large graph computing by running the non-volatile software programs and instructions stored in the memory 22.

[0083] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0084] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the server crash restart data recovery method based on ultra-large graph calculation described in Embodiment 1 above. For example, they perform the above-described... Figure 1 , Figure 7 and Figure 8 The steps shown.

[0085] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0086] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for data recovery after server crash restart based on ultra-large graph computing, characterized in that, The corresponding logs in the server record the paging information of the data currently loaded into memory. The paging information includes the network address of the distributed computing node where the data is loaded, as well as the adjacency relationship between paging objects. When the server restarts from a crashed state, it includes: The server sends a cluster status information request (request_cluster_infomation()) to the gateway node and retrieves the returned cluster status information; wherein, the cluster status information includes the number of compute nodes (N-COMPUTATION) and the node routing table (TABLE).<PARTITION,HOST> ; The server retrieves the number of graph vertices in the graph storage engine, N-VERTICES = get_vertex_count(); it then calculates the number of nodes by taking the modulo of the total number of vertices, and calculates the number of graph vertices N-LENGTH that each compute node needs to load; Load the adjacency relationships and traverse the graph vertex IDs; Multi-threaded concurrent access to pagination: each thread traverses the vertex IDs of one pagination, iterating to obtain intermediate results for the i-th iteration. These intermediate results are stored in a TABLE. <PARTITION,MAP<ID,VALUE> After each i-th round of calculation, intermediate calculation results are sent to other nodes via TABLE.<PARTITION_INDEX,HOST> Obtain the network address (HOST) of the node corresponding to the page number, and then obtain the loading data (MAP) on the compute node of that HOST.<ID,VALUE> And load the server memory data according to the adjacency relationship; where i is a natural number.

2. The server crash restart data recovery method based on ultra-large graph computing according to claim 1, characterized in that, Before traversing the graph vertex IDs and loading the adjacency relationships, the method further includes: To analyze whether N-LENGTH vertices require multi-page memory processing, divide N-LENGTH by the paging threshold. If the value is greater than 1, the server requests multiple pages, with the maximum number of pages being the value + 1; if the value is less than 1, the server directly requests a single page of memory space.

3. The server crash restart data recovery method based on ultra-large graph computing according to claim 2, characterized in that, The pagination threshold size is set to 2GB to accommodate a Long type array with a length of 268435456.

4. The server crash restart data recovery method based on ultra-large graph computing according to claim 1, characterized in that, The method also includes the following during the traversal of graph vertex IDs: To ensure load balancing across all compute nodes, a discrete hop-point approach is used to read vertex IDs. Each compute node loads the ID adjacency sequence PARTITION_INDEX + (N × N-COMPUTE) to avoid data skew caused by super vertices within a certain ID range. The adjacent point ID array of the vertex ID is queried from the storage engine using get_v_neighbors(ID) and recorded in the server's memory page until all compute nodes have loaded all adjacency data into the memory page.

5. The server crash restart data recovery method based on ultra-large graph computing according to any one of claims 1-4, characterized in that, Loading the adjacency relationships and traversing the graph vertex IDs specifically includes: After loading the adjacency relationships, the closed-loop relationships in the corresponding adjacency relationships are used to form the basis for the vertex ID traversal order.

6. The server crash restart data recovery method based on ultra-large graph computing according to claim 5, characterized in that, After each i-th round of calculation, intermediate calculation results are sent to other nodes, specifically including: After receiving the intermediate calculation results and clarifying the adjacency relationship between itself and the node currently interacting with the data, other nodes pre-read the data to be returned to the server into a buffer so that they can complete the response process in a timely manner when a data request is received.

7. The server crash restart data recovery method based on ultra-large graph computing according to any one of claims 1-4, characterized in that, After completing a round of data loading following a system crash, if statistics show that data in a page is distributed across more than a preset threshold of nodes, the method further includes: Data objects that were originally scattered across multiple nodes but correspond to data loaded in the same page are transferred according to load balancing parameters, so that the number of nodes where the data is scattered in the corresponding page is less than the preset threshold after the transfer.

8. The server crash restart data recovery method based on ultra-large graph computing according to claim 7, characterized in that, The load balancing specifically includes one or more of the following: network latency, node data access response speed, remaining storage space, historical access frequency and peak value, and historical CPU and memory utilization.

9. A server crash restart data recovery device based on ultra-large graph computing, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the server crash restart data recovery method based on ultra-large graph computing as described in any one of claims 1-8.

Images