Method, device and system for processing distributed graph data structure

Abstract

The invention provides a processing method, device and system for a distributed graph data structure, the method is applied to a target process in a distributed system, the distributed system further comprises other processes communicating with the target process, one edge in the graph data structure is distributed to one process in the distributed system, and in the ith iteration, one edge is distributed to one process in the distributed system. The method comprises the steps that sub-graphs propagated by all processes in the distributed system are obtained, edge positioning is conducted on edges in any sub-graph in the obtained sub-graphs, a target edge is obtained, edge positioning results of the same sub-graph under different processes are the same, edge positioning results of different sub-graphs under the same process are different, and the target edge is obtained. And if the target edge is not the edge allocated to the target process, the target process does not need to process any sub-graph, any sub-graph is constructed by the process allocated with the target edge, and the sub-graph constructed by the ith round of iteration is propagated.

Description

Technical Field

[0001] The present specification relates to the field of data processing technology, and in particular to a method, device and system for processing distributed graph data structures. Background Art

[0002] Distributed computing is applied to graph computing. In the process of distributed graph computing, temporary subgraphs are usually generated. Multiple processes in a distributed system may perform repeated operations on temporary subgraphs.

[0003] In order to avoid repeated operations, in the related art, when a process propagates a subgraph, it can add its own process number, so that each process can respectively construct a mapping table of the corresponding relationship between the subgraph and the process by propagating and aggregating the process number. Accordingly, a specific process can process the subgraph when the minimum process number corresponding to the subgraph in the mapping table is its own process number.

[0004] However, the above scheme requires the propagation of process numbers and aggregated process numbers, which results in relatively large communication and computational overheads. Moreover, the subgraph is always processed by the process with the smallest process number, which results in a load imbalance problem.

[0005] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the invention

[0006] This specification provides a method, device and system for processing a distributed graph data structure to avoid at least one of the above-mentioned technical problems.

[0007] In a first aspect, the present specification provides a method for processing a distributed graph data structure, the method being applied to a target process in a distributed system, the distributed system also including other processes communicating with the target process, an edge in the graph data structure being assigned to a process in the distributed system, the method including multiple rounds of iterations, and in the i-th round of iterations, the method including:

[0008] Obtain a subgraph propagated by each process in the distributed system, wherein the subgraph includes multiple nodes and at least one edge, a node represents an entity, an edge connecting two nodes is used to represent an association relationship between the two nodes, and i is an integer greater than or equal to 3;

[0009] For any subgraph in each obtained subgraph, edge positioning is performed on the edge in the arbitrary subgraph to obtain a target edge, wherein edge positioning results of the same subgraph in different processes are the same, and edge positioning results of different subgraphs in the same process are different; and

[0010] If the target edge is not an edge assigned to the target process, the target process does not need to process the arbitrary subgraph, wherein the arbitrary subgraph is constructed by the process assigned with the target edge, and the subgraph constructed in the i-th round of iteration is propagated.

[0011] In a second aspect, the present specification provides a distributed graph data structure processing device, the device being a target process in a distributed system, comprising:

[0012] at least one storage medium storing at least one instruction set to perform processing of a distributed graph data structure;

[0013] At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, it reads the at least one instruction set and executes the method as described in any one of the first aspects according to the instructions of the at least one instruction set.

[0014] In a third aspect, the present specification provides a system for processing a distributed graph data structure, wherein the system is a distributed system, and the distributed system includes multiple processes, and at least one of the multiple processes is the device as described in the second aspect.

[0015] It can be seen from the above technical solutions that the processing method, device and system of the distributed graph data structure provided by this specification, wherein the process corresponds to the edge assigned to it one by one, such as an edge is only assigned to one process. Correspondingly, with respect to a specific process, the process can locate the edge in the subgraph obtained by propagation, so that when the located edge is the edge assigned to the process, the subgraph is processed, while other processes do not need to process the subgraph. In other words, an edge is only assigned to one process, so there can only be one process, not multiple processes, to process a specific subgraph. Therefore, redundant calculations of the same subgraph by multiple processes can be avoided. And relatively speaking, there is no need to propagate process numbers and aggregate process numbers to reduce communication and computing overhead. In addition, there is no need to always process the subgraph by processes with small process numbers, which can avoid the problem of load imbalance. Moreover, each process can independently determine whether it needs to process the corresponding subgraph by itself to avoid interference and improve reliability.

[0016] Other functions of the distributed graph data structure processing method, device and system provided in this specification will be partially listed in the following description. The creative aspects of the distributed graph data structure processing method, device and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of an application scenario of the method for processing a distributed graph data structure of this specification;

[0019] Figure 2 A schematic diagram of a distribution edge provided for an embodiment of this specification;

[0020] Figure 3 A schematic diagram of sub-graph propagation provided for one embodiment of this specification;

[0021] Figure 4 A schematic diagram of a mapping table between subgraphs and processes provided in one embodiment of this specification;

[0022] Figure 5 A schematic diagram of a mapping table between subgraphs and processes provided in another embodiment of this specification;

[0023] Figure 6 A schematic diagram of the hardware structure of a distributed graph data structure processing device provided in an embodiment of this specification;

[0024] Figure 7 A schematic diagram of a process flow of a method for processing a distributed graph data structure provided in an embodiment of this specification;

[0025] Figure 8 A schematic diagram of sub-graph propagation provided in another embodiment of the present specification;

[0026] Fig. 9 This is a schematic diagram of the principle of the method for processing the distributed graph data structure of this specification. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0028] It should be understood that the terms "including" and "having" and any variations thereof in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0029] In the embodiments of this specification, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0030] The term "plurality" in the embodiments of the present specification refers to two or more than two, and other quantifiers are similar thereto.

[0031] The terms "first", "second", "third", etc. in this specification are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this specification.

[0032] The term "unit / module" as used in this specification refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0033] To facilitate the reader's understanding of this specification, at least some of the terms in this specification are explained as follows:

[0034] Graph data structure (also referred to as graph data) is a nonlinear data structure used to represent the relationship between objects. In a graph, a node (also called a vertex or point) represents a data entity (also referred to as an entity), and an edge (also called an arc or line) connecting two entities represents the connection or relationship (also referred to as an association relationship) between the two entities.

[0035] Graph data structures include directed property graphs, which are graph models in which the edges of the graph are directional (i.e., the edges point from one vertex to another), and both vertices and edges can have additional attribute information (such as weights, labels, descriptions, etc.).

[0036] A connected subgraph, or simply a subgraph, is a subgraph in graph theory, such as in graph data structures.

[0037] Distributed graph computing refers to the computation of graph data structures performed on multiple machines (or nodes / processes). The distributed environment allows the processing of graph data structures that far exceed the memory limit of a single machine. In distributed graph computing, the graph data structure is divided into multiple parts (such as subgraphs) and assigned to different machines for processing.

[0038] Redundant computing, in distributed graph computing, the same computing work is performed on different nodes. Among them, the "redundant computing" introduced in this specification mainly refers to multiple processes performing the same computing work on the same subgraph.

[0039] Process, in distributed graph computing, a process refers to an independent computing entity running on a single machine or different machines. In distributed graph computing, a large graph (such as a complete graph data structure) is divided into multiple parts (such as subgraphs), which are processed by different processes.

[0040] Temporaray Subgraph is a smaller graph segment generated from the original graph. It can be generated during the computation process for a specific computation. In distributed graph computing, multiple processes may independently generate and process the same temporary subgraph.

[0041] Subgraph enumeration refers to identifying all subgraphs that meet certain conditions in a graph data structure. These subgraphs are connected, can be used to satisfy certain structural patterns, or have specific properties.

[0042] Iteration refers to an algorithm that relies on repeated computational steps to gradually solve a problem. For example, gradient descent is an iterative algorithm for solving optimization problems, which updates the parameters step by step in the opposite direction of the gradient to find the minimum of the function.

[0043] To facilitate readers' understanding of this specification, the application scenarios of this specification are now introduced.

[0044] The technical solution provided in this specification is applicable to scenarios where the same temporary subgraph is generated on multiple processes in distributed graph computing and redundant computing of the temporary subgraph needs to be reduced. The application scenarios include but are not limited to social scenarios, recommendation scenarios, and financial risk prevention and control scenarios.

[0045] For example, see Figure 1 , Figure 1 A schematic diagram of an application scenario of the distributed graph data structure processing method of this specification.

[0046] like Figure 1 As shown, the application scenario includes a distributed system. A distributed system includes multiple processes, such as Figure 1The process 1 is shown to the process K. K is an integer greater than or equal to 2. Accordingly, by performing distributed graph computing on the graph data structure through multiple processes, corresponding processing results may be obtained and output.

[0047] Take the financial risk prevention and control scenario as an example: Figure 1 The graph data structure in is a directed attribute graph, which shows 5 nodes and 6 edges as an example. A node represents a user, and a node stores the characteristics of the user it represents. For example, the characteristics may include financial accounts, transfer tendencies (such as transfer time tendencies and transfer location tendencies, etc.), education, height, occupation, weight, etc. The edge connecting two nodes can represent the transaction behavior information between the two connected nodes. For example, the transaction behavior information includes transfer method, transfer amount, transfer frequency, etc.

[0048] Among them, distributed graph calculation can be used to calculate the average age, the total transfer amount, and so on, which will not be listed here one by one.

[0049] However, when distributed graph computing is performed on a graph data structure based on K processes, the same temporary subgraph may be generated on multiple processes. For example, both process 1 and process K generate a temporary subgraph p.

[0050] In practical applications, both process 1 and process K will perform subgraph processing on the temporary subgraph p, so redundant calculations of process 1 and process K on the temporary subgraph P will be stored.

[0051] Exemplarily, a distributed pure memory-based graph computing framework can first distribute the edges of the input graph (ie, the graph data structure) to each process, and then perform subgraph enumeration.

[0052] like Figure 2 To simplify the description, Figure 2 In the example, two processes, three nodes, and two edges are shown. The two processes include process 1 and process 2. The three nodes include node 1, node 2, and node 3. The two edges include the edge between node 1 and node 2, and the edge between node 2 and node 3.

[0053] All edges are evenly distributed to each process. For example, process 1 owns edge "1->2", and process 2 owns edge "2->3". For nodes, process 1 owns nodes 1 and 2, and process 2 owns nodes 2 and 3.

[0054] After the input graph is allocated, multiple rounds of subgraph enumeration are started.

[0055] In each round of iteration, each process traverses the subgraph received in the previous round, and tries to add another edge around the node to the subgraph according to the node in the subgraph to construct a new subgraph, and then propagates (also called broadcasting) the constructed new subgraph.

[0056] In the next iteration, we continue to add an edge to these subgraphs until no new subgraphs are generated. This method, similar to Depth-First Search (DFS), can fully enumerate all subgraphs, and the number of edges in each enumerated subgraph will be one more than the previous round.

[0057] The specific process is as follows: each round of iteration mainly includes three steps: "subgraph construction", "subgraph processing" and "subgraph propagation".

[0058] Iteration 1:

[0059] 1. Subgraph construction:

[0060] Since this is the first iteration, process 1 constructs a unilateral subgraph "1->2" based on the local edge (i.e., the edge connecting node 1 and node 2). Similarly, process 2 constructs a unilateral subgraph "2->3".

[0061] 2. Sub-image processing:

[0062] Process 1 performs relevant calculations on subgraph "1->2" (for example, calculating, filtering, displaying, etc. the attributes of each node or edge of the subgraph) and outputs the results. Similarly, process 2 performs relevant calculations on the new subgraph "2->3" and outputs the results.

[0063] 3. Subgraph propagation:

[0064] Each subgraph needs to be propagated to the process where each node is located, such as Figure 3 As shown, the next iteration can continue to expand from other edges around these points.

[0065] For example, nodes 1 and 2 of subgraph "1->2" are in processes 1 and 2, respectively, so subgraph "1->2" is propagated to processes 1 and 2. Similarly, nodes 2 and 3 of subgraph "2->3" are in processes 2 and 1, respectively, so subgraph "2->3" is propagated to processes 1 and 2.

[0066] Second iteration:

[0067] 1. Subgraph construction:

[0068] Process 1 has no edge to join the subgraph "1->2", so it skips it. However, for subgraph "2->3", process 1 has an edge "1->2" locally on node 2 that is not in the subgraph, so it joins and generates a new subgraph "1->2->3". Similarly, process 2 can expand subgraph "1->2" and generate a new subgraph "1->2->3".

[0069] 2. Sub-image processing:

[0070] Process 1 performs relevant calculations on the new subgraph "1->2->3" and outputs the results. Similarly, process 2 performs relevant calculations on the new subgraph "1->2->3" and outputs the results.

[0071] 3. Subgraph propagation:

[0072] Process 1 propagates the subgraph "1->2->3" to processes 1 and 2. Similarly, process 2 propagates the subgraph "1->2->3" to processes 1 and 2.

[0073] Iteration 3:

[0074] 1. Subgraph construction:

[0075] Neither process 1 nor process 2 adds any new edges to the subgraph "1->2->3", so no new subgraph is generated and the process ends.

[0076] Based on the above process, it can be seen that in the "sub-graph processing" step of the second iteration, process 1 and process 2 both process the sub-graph "1->2->3", and there is repeated operation on the sub-graph "1->2->3".

[0077] In some embodiments, in order to avoid redundant operations of multiple processes on the same subgraph as described in the above example, the following improvements can be made based on the above example:

[0078] When each process propagates a subgraph, it carries its own process number.

[0079] After each process collects the subgraphs broadcast by each process, it sorts each subgraph according to the node ID to obtain a subgraph representation. In this way, the same subgraph always gets the same subgraph representation. That is, the subgraph representations of the same subgraph in different processes are the same.

[0080] The same subgraph is represented as the key value, the corresponding process number is aggregated, and the process number itself is added to finally obtain a mapping table between each subgraph and the corresponding process. That is, the mapping table is used to represent the corresponding relationship between the subgraph and the process.

[0081] Each process traverses the subgraph, and only processes the subgraph if its own process number is the smallest in the mapping table.

[0082] To facilitate understanding, the above example of two processes, three nodes, and two edges is still used as an example for explanation.

[0083] In this embodiment, the order of steps in each iteration is changed. "Subgraph processing" is the first step in each iteration, so it can be performed after the subgraph propagation of the previous iteration. That is, each iteration mainly includes three steps: "subgraph processing", "subgraph construction" and "subgraph propagation".

[0084] Iteration 1:

[0085] 1. Sub-image processing:

[0086] Since this is the first iteration, there is no subgraph propagated from the previous round. Each process is skipped.

[0087] 2. Subgraph construction:

[0088] Since this is the first iteration, process 1 constructs a single-edge subgraph "1->2" based on the local edge. Similarly, process 2 constructs a single-edge subgraph "2->3".

[0089] 3. Subgraph propagation:

[0090] This is consistent with the "1st iteration - subgraph propagation" in the above example and will not be repeated here.

[0091] Second iteration:

[0092] 1. Sub-image processing:

[0093] For example, based on the description of the above improvements, each process is as follows: Figure 4 The mapping table shown.

[0094] Accordingly, process 1 first tries to process subgraph "1->2". As the corresponding minimum process, it calculates and outputs the result for this subgraph. Process 1 tries to process subgraph "2->3". As the corresponding minimum process, it calculates and outputs the result for this subgraph. When process 2 tries to process, it does not process either subgraph because it is not the minimum process.

[0095] 2. Subgraph construction:

[0096] This is consistent with the “Second Iteration - Subgraph Construction” in the above example and will not be repeated here.

[0097] 3. Subgraph propagation:

[0098] Process 1 propagates the subgraph "1->2->3" and its own process number 1 to process 1 and process 2. Similarly, process 2 propagates the subgraph "1->2->3" and its own process number 1 to process 1 and process 2.

[0099] Iteration 3:

[0100] 1. Sub-image processing:

[0101] Similarly, based on the description of the above improvements, each process is as follows Figure 5 The mapping table shown.

[0102] Accordingly, process 1 tries to process the subgraph "1->2->3" and it is the corresponding minimum process, so it calculates the subgraph and outputs the result. When process 2 tries to process, it does not process the subgraph because it is not the minimum process.

[0103] 2. Subgraph construction:

[0104] Processes 1 and 2 do not add new edges to the subgraph "1->2->3", so no new subgraph is generated and the process ends.

[0105] Based on the above analysis, it can be seen that in this embodiment, each process propagates each subgraph and its process number so that the subgraph is processed by the smallest process, thereby avoiding the disadvantage of multiple processes repeatedly processing the same subgraph.

[0106] However, although the above scheme can avoid redundant operations of multiple processes on the same subgraph, it requires the propagation of process numbers and aggregated process numbers, which results in relatively large communication and computational overheads. Moreover, the subgraph is always processed by the process with the smallest process number, which results in a load imbalance problem.

[0107] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification.

[0108] In order to avoid at least one of the above technical problems, the present specification proposes a technical concept through creative labor: a process corresponds to an edge assigned to it one by one, such as an edge is only assigned to one process. Accordingly, with respect to a specific process, the process can locate an edge in a subgraph obtained by propagation, so that when the located edge is an edge assigned to the process, the subgraph is processed, while other processes do not need to process the subgraph.

[0109] Based on the above analysis, it can be known that in the technical solution provided in this specification, an edge is only assigned to one process, so there can only be one process, not multiple, to process a specific subgraph. Therefore, redundant calculations of the same subgraph by multiple processes can be avoided. And relatively speaking, there is no need to propagate process numbers and aggregate process numbers to reduce communication and computing overhead. In addition, there is no need for processes with small process numbers to always process subgraphs, which can avoid the problem of load imbalance. Moreover, each process can independently determine whether it needs to process the corresponding subgraph itself to avoid interference and improve reliability.

[0110] Combined with the above analysis, it can be seen that the execution subject of the technical solution provided in this specification can be a processing device for a distributed graph data structure (hereinafter referred to as a processing device), and the processing device can specifically be a process in a distributed system. Figure 6 The hardware structure diagram of a processing device 600 provided according to an embodiment of this specification is shown. The processing device 600 can execute the processing method of the distributed graph data structure described in this specification. The processing method of the distributed graph data structure is introduced in other parts of this specification.

[0111] like Figure 6 As shown, the processing device 600 may include at least one storage medium 603 and at least one processor 602. In some embodiments, the processing device 600 may also include a communication port 604 and an internal communication bus 601. The processing device 600 may also include an I / O component 605.

[0112] The internal communication bus 601 can connect different system components. For example, the internal communication bus 601 can connect the storage medium 603, the processor 602, the communication port 604 and the I / O component 605.

[0113] I / O component 605 supports input / output between processing device 600 and other components.

[0114] The communication port 604 is used for data communication between the processing device 600 and the outside world. For example, the communication port 604 can be used for data communication between the processing device 600 and a network. The communication port 604 can be a wired communication port or a wireless communication port.

[0115] The storage medium 603 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 6031, a read-only storage medium (ROM) 6032, or a random access storage medium (RAM) 6033. The storage medium 603 also includes at least one instruction set stored in the data storage device. The instruction set includes computer program code, and the computer program code may include programs, routines, objects, components, data structures, processes, modules, etc. for executing the distributed graph data structure processing method provided in this specification.

[0116] At least one processor 602 can be connected to at least one storage medium 603 in communication. At least one processor 602 is used to execute the above-mentioned at least one instruction set. When the processing device 600 is running, at least one processor 202 reads the at least one instruction set and executes the processing method of the distributed graph data structure provided in this specification according to the instructions of the at least one instruction set. The processor 602 can execute all the steps included in the processing method of the distributed graph data structure. The processor 602 can be in the form of one or more processors. In some embodiments, the processor 602 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0117] For illustration purposes only, only one processor 602 is shown in the processing device 600 in the figure. However, it should be noted that the processing device 600 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor or by multiple processors in combination. For example, if the processor 602 of the processing device 600 is described in this specification as performing step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 602 (for example, the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0118] See also Figure 7 , Figure 7A flowchart of a method for processing a distributed graph data structure provided in an embodiment of the present specification. The method is applied to a target process in a distributed system, the distributed system also includes other processes that communicate with the target process, an edge in the graph data structure is assigned to a process in the distributed system, and the method includes multiple rounds of iterations. For example, the method is applied to Figure 1 Process 1 or process K, etc. in the distributed system shown in .

[0119] In the i-th iteration, if Figure 7 As shown, the method includes the following S701 to S703:

[0120] S701: Obtain a subgraph propagated by each process in the distributed system, wherein the subgraph includes multiple nodes and at least one edge, a node represents an entity, an edge connecting two nodes is used to represent an association relationship between the two nodes, and i is an integer greater than or equal to 3.

[0121] It is understandable that, in order to avoid redundant statements, the present embodiment will not repeat the same or similar technical features as those in the above embodiments. For the description of the distributed system, process, subgraph, etc., please refer to the above examples.

[0122] For example, Figure 8 As shown in the figure, the subgraph constructed by process 1 in round i-1 is Figure 1 , and so on, the subgraph constructed by process 4 in round i-1 is sub Figure 4 . son Figure 1 Until the child Figure 4 The target process is process 1. Process 1 propagates subprocesses to process 1 and process 4 respectively. Figure 1 , and so on, process 4 propagates the child to process 1 and then to process 4. Figure 4 ( Figure 8 In the figure, the propagation of subgraph from process 1 to process 4 to process 1 is mainly taken as an example for demonstration).

[0123] Correspondingly, process 1 obtains the subgraph propagated by process 1 to process 4, that is, the subgraph Figure 1 Until the child Figure 4 Four sub-graphs.

[0124] S702: For any subgraph in each obtained subgraph, edge positioning is performed on the edge in any subgraph to obtain a target edge, wherein the edge positioning results of the same subgraph in different processes are the same, and the edge positioning results of different subgraphs in the same process are different.

[0125] Combining the above examples with Figure 8 , each subgraph includes Figure 1 Until the child Figure 4 . Any subgraph is a subgraph Figure 1 Until the child Figure 4 Any subgraph in Figure 1 Alternatively, it can be understood that, for each subgraph in each subgraph, process 1 will execute S702 and S703 described below. This embodiment is just an example of an arbitrary subgraph for exemplary explanation.

[0126] In this embodiment, process 1 is used to Figure 1 There is no limit on the way to perform edge positioning to ensure that the child Figure 1 The results obtained by the four processes for edge positioning are consistent. That is, the same subgraph will be positioned to the same edge in different processes.

[0127] In some embodiments, S702 may include the following steps 1 and 2:

[0128] Step 1: Generate a unique representation value that is the same for any subgraph in each process.

[0129] The unique representation value can be understood as representing any subgraph by a numerical value, and the numerical value corresponding to any subgraph in each process is the same. In other words, the same subgraph can be represented by the same numerical value in different processes.

[0130] In combination with the above examples, it can be seen that each node has its own corresponding identifier. Different nodes can be distinguished based on their corresponding identifiers. In some embodiments, step 1 may include the following sub-steps 1 and 2:

[0131] Sub-step 1: Sort the identifiers of the nodes in any subgraph to obtain the same node sequence of any subgraph under each process.

[0132] Combining the above examples with Fig. 9 , if any subgraph is a sub Figure 1 ,son Figure 1 The target process includes nodes 1, 2, and 3. The identifier of node 1 is 1, the identifier of node 2 is 2, and the identifier of node 3 is 3. Figure 1 Sort by identifier to get the node sequence (1,2,3).

[0133] Since different nodes have different identifiers, the same subgraph has the same node sequence in different processes, while different subgraphs have different node sequences in the same process.

[0134] Sub-step 2: Generate a hash value of any subgraph according to the node sequence, wherein the unique representation value includes the hash value.

[0135] Accordingly, combining the above examples and Fig. 9, the target process can generate children based on the node sequence (1,2,3) Figure 1 The unique representation value may be a hash value.

[0136] Based on the above analysis of sub-steps 1 and 2, it can be seen that in this embodiment, for each subgraph, the target process first determines the node sequence of the subgraph based on the identifiers of the nodes in the subgraph, and then determines the hash value of the subgraph according to the node sequence, so as to determine the hash value as the unique representation value of the subgraph. Since the identifier of the node is unique, that is, different nodes have different identifiers, and therefore, different subgraphs have different node sequences. In turn, the hash value determined on the basis of the node sequence is unique, that is, different subgraphs have different hash values. Thereby achieving the effectiveness and reliability of determining the unique representation value corresponding to each subgraph.

[0137] Step 2: Locate the edge of any subgraph according to the unique representation value to obtain the target edge.

[0138] Combined with the above example, in process 1, get the child Figure 1 After the unique representation value of Figure 1 Perform edge positioning to obtain the target edge.

[0139] In this embodiment, the target process performs edge positioning by combining the unique representation value, which can improve the efficiency of edge positioning. In addition, since the unique representation values ​​of the same subgraph in different processes are the same, and the unique representation values ​​of different subgraphs in the same process are different, the accuracy and reliability of edge positioning can be achieved.

[0140] In some embodiments, step 2 may include the following sub-steps 1 and 2:

[0141] Sub-step 1: Sort the edges of any subgraph according to the identifiers of the two nodes connected by the edge to obtain the edge sequence of any subgraph.

[0142] Continuing with the above example and Fig. 9 , the target process will be Figure 1 The edges are sorted according to the node identifiers at both ends of the edges to obtain the edge sequence [(1,2),(2,3)].

[0143] Sub-step 2: Perform a modulo operation on the number of edges in the edge sequence according to the hash value to determine the target edge according to the modulo result.

[0144] Continuing with the above example and Fig. 9 , the number of edges in the edge sequence [(1,2),(2,3)] is 2. The target process performs a modulo operation on 2 based on the hash value to locate the target edge.

[0145] For example, Fig. 9As shown, the target process performs a modulo operation on 2 based on the hash value, and the modulo result is 0. The target edge located by the target process is edge (1,2). That is, the target edge is the edge connecting node 1 and node 2.

[0146] Combined with the above analysis of the first and second sub-steps, it can be known that in this embodiment, the target process can first sort the identifiers according to the nodes at both ends of the edge to obtain an edge sequence, and then take the modulus of the number of edges in the edge sequence according to the hash value to obtain a modulus value (i.e., a modulus result), and determine the target edge from the edge sequence according to the modulus value. Since the identifier is unique, the edge sequence of the subgraph is also unique. On this basis, taking the modulus of the unique hash value can make the modulus result unique. And then the target edge determined on the basis of the uniqueness of the modulus result is also unique. Therefore, the accuracy and reliability of edge positioning can be achieved.

[0147] S703: If the target edge is not an edge assigned to the target process, the target process does not need to process the arbitrary subgraph, wherein the arbitrary subgraph is constructed by the process assigned with the target edge based on the arbitrary subgraph, and the subgraph constructed in the i-th round of iteration is propagated.

[0148] In another embodiment, the target edge may also be an edge assigned to the target process. Exemplarily, if the target edge is an edge assigned to the target process, the target process performs subgraph processing and subgraph construction on any subgraph, and propagates the subgraph constructed in the i-th round of iteration.

[0149] Combining the above example, process 1 Figure 1 After locating the target edge, the target edge and the edge assigned to process 1 can be compared to determine whether they are consistent (i.e., whether they are the same edge). If so, process 1 Figure 1 Perform the subgraph construction of the i-th round of iteration, obtain the subgraph constructed in the i-th round of iteration, and propagate the subgraph constructed in the i-th round of iteration.

[0150] For example, combining the above example and Fig. 9 , the target edge is edge (1,2). If the edge assigned to process 1 is the edge used to connect node 1 and node 2, then the target edge and the edge assigned to process 1 are the same edge. Then process 1 assigns Figure 1 Subgraph processing and subgraph construction are performed to obtain a new subgraph through subgraph construction. Accordingly, process 1 propagates the new subgraph to the processes including the nodes of the new subgraph.

[0151] For example, if the target edge is edge (1,2) and the edge assigned to the target process is edge (2,2), then the target process does not need to Figure 1 Perform subgraph processing and subgraph construction.

[0152] Correspondingly, when a process other than the target process locates an edge, it will locate the edge (1,2). Figure 8 It can be seen that when process 2 locates the edge, it may locate the edge (1,2). And the edge assigned to process 2 may be the edge (1,2). Then, process 2 will Figure 1 Perform subgraph processing and subgraph construction.

[0153] That is, through the above example, different processes will not perform subgraph processing and subgraph construction on the same subgraph, thereby avoiding redundant calculation of temporary subgraphs.

[0154] Based on the above analysis, it can be seen that i is an integer greater than or equal to 3. That is, the minimum value of i is 3. In some embodiments, i may also have a maximum value.

[0155] Exemplarily, the maximum value of i is n, and n is determined based on a preset condition. The preset condition is: the subgraph construction of the nth iteration does not generate a new subgraph; or, in the nth iteration, the target edge is not an edge assigned to the target process.

[0156] That is to say, if the target process is in a round of iteration and the round of iteration meets the preset conditions, then the round of iteration is the nth round of iteration. The target process does not need to enter the next round of iteration, but ends the distributed graph computing.

[0157] Among them, n can be determined by two conditions. Assume that the two conditions are condition A and condition B. Condition A is that the subgraph construction of the nth iteration does not generate a new subgraph. Condition B is that in the nth iteration, the target edge is not the edge assigned to the target process. Then, as long as one of the conditions is met, it will be fine.

[0158] For example, taking condition A as an example: if in a certain round of iteration, the target process does not generate a new subgraph through the subgraph construction of this round, that is, there is no edge that can be added to the current subgraph, then the target process does not need to enter the next round of iteration, that is, this round of iteration is the nth round of iteration.

[0159] For example, taking condition B as an example: if in a certain round of iteration, the target edge located by the target process is not the same as the edge assigned to the target process, then the target process does not need to perform subgraph processing and subgraph construction, and does not need to enter the next round of iteration, that is, this round of iteration is the nth round of iteration.

[0160] Combined with the above analysis, it can be seen that in this embodiment, the target process can determine whether to enter the next round of iteration based on whether the current round of iteration generates a new subgraph, or it can determine whether to enter the next round of iteration based on whether the target edge located by the current round of iteration is consistent with the assigned edge, so as to improve the reliability and effectiveness of distributed graph computing.

[0161] Combined with the above analysis of S701 to S703, it can be seen that the above example mainly describes the third and subsequent iterations. In some embodiments, if it is the first iteration, the target process can construct a subgraph for the first iteration based on the edge assigned to the target process, and propagate the subgraph constructed for the first iteration.

[0162] Exemplarily, in the first round of iteration, the target process has not received the subgraph propagated by other processes. Therefore, the target process can only consider the assigned edges to construct the subgraph based on the edges. Accordingly, the target process propagates the subgraph constructed in the first round of iteration based on the process including the nodes in the subgraph constructed in the first round of iteration.

[0163] In some embodiments, if it is the second round of iteration, a subgraph including the edge assigned to the target process is determined from the subgraph constructed in the first round of iteration obtained through propagation, and subgraph processing is performed on the determined subgraph, and the processing result is output. In addition, a subgraph construction is performed on the subgraph constructed in the first round of iteration obtained through propagation, and the subgraph constructed in the second round of iteration is obtained and propagated.

[0164] For example, combining the above examples and Figure 8 In the second iteration, processes 1 to 4 propagate their corresponding subgraphs to process 1. Because each of the four subgraphs has only one edge, and the subgraph constructed by each process is the subgraph that includes the edge assigned to the process. Therefore, process 1 obtains the subgraph that includes the edge assigned to process 1 from the four subgraphs, and performs subgraph construction on the subgraph to obtain a new subgraph, and propagates the new subgraph to the processes that include the nodes in the new subgraph.

[0165] To facilitate the reader's understanding, the technical solution provided in this specification is now described in detail by taking the distributed graph computing with the same graph data structure as the above-mentioned technology as an example.

[0166] In this embodiment, "subgraph processing" is the first step in each iteration, so it can be performed after the subgraph propagation of the previous iteration. That is, each iteration mainly includes three steps of "subgraph processing", "subgraph construction" and "subgraph propagation".

[0167] Iteration 1:

[0168] 1. Sub-image processing:

[0169] Since this is the first iteration, there is no subgraph propagated from the previous round. Each process is skipped.

[0170] 2. Subgraph construction:

[0171] Since this is the first iteration, process 1 constructs a single-edge subgraph "1->2" based on the local edge. Similarly, process 2 constructs a single-edge subgraph "2->3".

[0172] 3. Subgraph propagation:

[0173] This is consistent with the "1st round iteration - subgraph propagation" in the above example of redundant calculation, and will not be repeated here.

[0174] Second iteration:

[0175] 1. Sub-image processing:

[0176] Since subgraphs "1->2" and "2->3" each have only one edge, the processes that need to process the subgraphs are the processes where the only edge in the subgraphs is located.

[0177] For example, process 1 calculates the subgraph "1->2" and outputs the result. Similarly, process 2 calculates the subgraph "2->3" and outputs the result.

[0178] 2. Subgraph construction:

[0179] This is consistent with the “second round of iteration - subgraph construction” in the above example of redundant calculation, and will not be repeated here.

[0180] 3. Subgraph propagation:

[0181] This is consistent with the "second round of iteration - subgraph propagation" in the above example of redundant calculation, and will not be repeated here.

[0182] Iteration 3:

[0183] 1. Sub-image processing:

[0184] Exemplarily, process No. 1 and process No. 2 respectively use the methods described in the above examples to perform edge positioning and other operations to implement subgraph processing, which will not be repeated here.

[0185] For example, process 1 and process 2 each locate the edge of the subgraph "1->2->3" locally, and both locate the edge at the 0th edge, that is, "1->2". Accordingly, only process 1 where edge "1->2" is located needs to calculate the subgraph "1->2->3" and output the result.

[0186] 2. Subgraph construction:

[0187] Neither process adds any new edges to the subgraph "1->2->3", so no new subgraph is generated and the process ends.

[0188] It is worth noting that the above examples are only used to illustrate possible implementations of the distributed graph data structure processing method of this specification, and cannot be understood as limiting the implementation of the distributed graph data structure processing method of this specification. Exemplarily, based on the above technical concept, some of the above technical features can be combined to obtain a new embodiment; new technical features can be added on the basis of the above examples to obtain a new embodiment; some technical features can be reduced on the basis of the above examples to obtain a new embodiment; some technical features in the above examples can be replaced with other technical features; some technical features and the order of the above examples can be adjusted to obtain a new embodiment, etc., which will not be listed here one by one.

[0189] Based on the above technical concept, this specification also provides a computer-readable non-temporary storage medium, which stores at least one instruction set. When the at least one instruction set is executed by a processor, the steps of the distributed graph data structure processing method described in this specification are implemented.

[0190] In some possible implementations, various aspects of this specification may also be implemented in the form of a program product, which includes a program code. When the program product is run on a processing device 600, the program code is used for the processing device 600 to execute the steps of the processing method for the distributed graph data structure described in this specification. The program product for implementing the above method may include program code using a portable compact disk read-only memory (CD-ROM) and may be run on the processing device 600. However, the program product of this specification is not limited to this. In this specification, a readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system or used in combination with it. The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the processing device 600, partially on the processing device 600, as a stand-alone software package, partially on the processing device 600 and partially on a remote processing device 600, or entirely on the remote processing device 600.

[0191] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of relevant information of users (such as financial accounts, education, height, etc.) involved in the technical solution of this specification are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0192] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0193] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0194] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0195] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0196] Each patent, patent application, publication of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except to the extent that it is inconsistent or conflicting with this document or that has a limiting effect on the broadest scope of the claims, may be incorporated herein by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0197] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A method for processing a distributed graph data structure, the method being applied to a target process in a distributed system, the distributed system also including other processes communicating with the target process, an edge in the graph data structure being assigned to a process in the distributed system, the method including multiple rounds of iterations, and in the i-th round of iterations, the method including: Obtain a subgraph propagated by each process in the distributed system, wherein the subgraph includes multiple nodes and at least one edge, a node represents an entity, an edge connecting two nodes is used to represent an association relationship between the two nodes, and i is an integer greater than or equal to 3; For any subgraph in each obtained subgraph, edge positioning is performed on the edge in the arbitrary subgraph to obtain a target edge, wherein edge positioning results of the same subgraph in different processes are the same, and edge positioning results of different subgraphs in the same process are different; and If the target edge is not an edge assigned to the target process, the target process does not need to process the arbitrary subgraph, wherein the arbitrary subgraph is constructed by the process assigned with the target edge, and the subgraph constructed in the i-th round of iteration is propagated.

2. The method according to claim 1, wherein: Positioning the edges in the arbitrary subgraph to obtain the target edges includes: Generate a unique representation value of the arbitrary subgraph that is the same under each process; and The edge of the arbitrary subgraph is located according to the unique representation value to obtain the target edge.

3. The method according to claim 2, wherein: Each node has its own corresponding identifier; generating the same unique representation value of the arbitrary subgraph under each process includes: Sort the identifiers of the nodes in the arbitrary subgraph to obtain the same node sequence of the arbitrary subgraph under the processes; and A hash value of the arbitrary subgraph is generated according to the node sequence, wherein the unique representation value includes the hash value.

4. The method according to claim 3, wherein: The performing edge positioning on the arbitrary subgraph according to the unique representation value to obtain the target edge includes: Sort the edges of the arbitrary subgraph according to the identifiers of the two nodes connected by the edges to obtain an edge sequence of the arbitrary subgraph; and The number of edges in the edge sequence is modulo-ed according to the hash value to determine the target edge according to the modulo-ed result.

5. The method according to claim 1, wherein: The method further comprises: If the target edge is an edge assigned to the target process, the target process performs subgraph processing and subgraph construction on the arbitrary subgraph, and propagates the subgraph constructed in the i-th round of iteration.

6. The method according to claim 1, wherein: The method further comprises: In the first iteration, a subgraph of the first iteration is constructed according to the edges assigned to the target process, and the subgraph constructed in the first iteration is propagated.

7. The method according to claim 6, wherein: The method further comprises: In the second round of iteration, from the subgraphs constructed in the first round of iteration obtained through propagation, a subgraph including the edge assigned to the target process is determined, and subgraph processing is performed on the determined subgraph, and a processing result is output; and The subgraph constructed in the first round of iteration is constructed through propagation, and the subgraph constructed in the second round of iteration is obtained and propagated.

8. The method according to claim 1, wherein: The maximum value of i is n, and n is determined based on a preset condition; the preset condition is: The subgraph construction of the nth iteration does not generate a new subgraph; or, In the nth iteration, the target edge is not an edge assigned to the target process.

9. A processing device for a distributed graph data structure, the device being a target process in a distributed system, comprising: at least one storage medium storing at least one instruction set to perform processing of a distributed graph data structure; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, the at least one instruction set is read, and the method as described in any one of claims 1 to 8 is executed according to the instructions of the at least one instruction set.

10. A processing system for a distributed graph data structure, wherein the system is a distributed system, and the distributed system includes multiple processes, and at least one of the multiple processes is the device as described in claim 9.

Images