A graph data processing system, method, device, and storage medium

Abstract

This specification discloses a graph data processing system, method, device, and storage medium. It can extract various set operations required for subgraphs matching a specified graph pattern from target graph data according to a preset graph model matching algorithm. For each set operation, it determines the cost value corresponding to the performance of the processing unit occupied by executing the set operation according to different execution strategies based on the number of elements in the two sets involved in executing the set operation. Furthermore, it can select the execution strategy that minimizes the performance occupation of the processing unit to execute the set operation, thereby improving the graph data processing efficiency.

Description

Technical Field

[0001] This specification relates to the field of graph computing technology, and particularly to a graph data processing system, method, device, and storage medium. Background Art

[0002] With the development of big data technology, the scale of graph data is continuously increasing, the types of graph data are also continuously increasing, and the relationships between the entity objects corresponding to each node in the graph data have become more complex. How to analyze and mine the complex relationships contained in graph data has become a current research hotspot.

[0003] In the prior art, a graph pattern matching algorithm is usually used to extract subgraphs that match a specified graph pattern from graph data, and tasks are executed based on the extracted subgraphs. Here, the graph pattern is used to represent the association rules presented among specific entities in an actual scenario. For example, in the medical field, there is a specific molecular structure that can be used for medical treatment, and the connection relationships among the molecules contained in this characteristic molecular structure can be designed as a graph pattern, and subgraphs that match this graph pattern are queried from the graph data corresponding to other macromolecular substances to determine whether this specific molecular structure is contained in the other macromolecular substances. However, due to the low execution efficiency of the current graph pattern matching algorithm, it cannot meet the processing requirements of large-scale graph data.

[0004] Therefore, how to improve the execution efficiency of the graph pattern matching algorithm is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a graph data processing method, device, device, and storage medium to partially solve the above problems existing in the prior art.

[0006] This specification adopts the following technical solutions:

[0007] This specification provides a graph data processing system, and the graph data processing system includes: each processing unit and a memory. Among them, each processing unit further includes: a decision module;

[0008] The processing unit is configured to, for the obtained target graph data, determine each set operation required when extracting a subgraph that matches a specified graph pattern from the target graph data according to a preset graph pattern matching algorithm. The set operation is used to represent performing a specified type of operation on the neighbor node sets of two nodes in the target graph data, and the specified type of operation includes at least one of taking an intersection and taking a difference set;

[0009] Through the decision-making module, for each set operation, the processing unit determines the cost value corresponding to executing the set operation according to each execution strategy based on the number of nodes included in the two node sets involved in the set operation and a preset cost function, and selects a target strategy from each execution strategy according to the cost value, so that the processing unit executes the set operation according to the target strategy, obtains the execution result corresponding to the set operation, and stores it in the memory; in response to obtaining the execution result corresponding to each set operation, reads the execution results corresponding to each set operation from the memory, and determines the subgraph in the target graph data that matches the specified graph pattern according to the execution results corresponding to each set operation, so as to execute a task according to the subgraph;

[0010] The memory is used to store the execution results corresponding to each set operation.

[0011] Optionally, a detection module is further provided in the processing unit;

[0012] The detection module is used to, for each set operation, judge whether the number of times the set operation is executed exceeds a preset threshold. If so, determine that the set operation is a target set operation, and persistently save the execution result of the target set operation for reuse when the set operation needs to be executed again.

[0013] Optionally, the graph data processing system further includes a dynamic partitioning module;

[0014] The dynamic partitioning module is used to obtain the original graph data, for each node in the original graph data, judge whether the degree of the node exceeds a preset threshold. If so, determine that the node is a central node, and for each central node, determine, through multiple rounds of neighbor node traversal, each node having a connection relationship with the central node as an associated node of the central node;

[0015] Determine each graph data block according to each central node and the associated nodes of each central node, and for each graph data block, use the graph data block as the target graph data, so that the processing unit processes the graph data block.

[0016] Optionally, the dynamic partitioning module is used to, for each graph data block, generate a processing task for processing the graph data block, and add the processing task to a preset task queue, so that the processing unit obtains the processing task from the task queue and uses the graph data block corresponding to the processing task as the target graph data.

[0017] Optionally, for each central node, the dynamic partitioning module is configured to determine whether the central node is an accessed node. If not, through multiple rounds of neighbor node traversal, determine the nodes that have a connection relationship with the central node as the associated nodes of the central node, and set the central node as an accessed node.

[0018] Optionally, for each set operation, the detection module is configured to determine whether there is a unique identifier corresponding to the set operation. If not, generate and save the unique identifier corresponding to the set operation according to the two sets involved in the set operation and the specified type corresponding to the set operation.

[0019] Optionally, for each set operation, the decision module is configured to determine the calculation time and memory access time required to execute the set operation according to each execution strategy based on the number of nodes included in the two node sets involved in the set operation and the processing data of the processing unit that executes the set operation. According to the calculation time and the memory access time, determine the cost value corresponding to each execution strategy. The processing data of the processing unit includes: the bandwidth of the processing unit and the memory access latency of the processing unit.

[0020] This specification provides a graph data processing method. The graph data processing system includes: each processing unit and a memory. Among them, each processing unit further includes: a decision module. The method includes:

[0021] For the obtained target graph data, the processing unit determines each set operation required when extracting a subgraph matching a specified graph pattern from the target graph data according to a preset graph pattern matching algorithm. The set operation is used to represent an operation of a specified type on the neighbor node sets of two nodes in the target graph data. The specified type of operation includes at least one of: taking the intersection and taking the difference set;

[0022] Through the decision module, for each set operation, determine the cost value corresponding to each execution strategy according to the number of nodes included in the two node sets involved in the set operation and a preset cost function, and select a target strategy from each execution strategy according to the cost value, so that the processing unit executes the set operation according to the target strategy, obtains the execution result corresponding to the set operation, and stores it in the memory;

[0023] After obtaining the execution result corresponding to each set operation, read the execution results corresponding to each set operation from the memory, and determine the subgraph in the target graph data that matches the specified graph pattern according to the execution results corresponding to each set operation, so as to execute a task according to the subgraph.

[0024] Optionally, the processing unit further includes: a detection module, and the method further includes:

[0025] Through the detection module, for each set operation, determine whether the number of times the set operation is executed exceeds a preset threshold. If so, determine that the set operation is a target set operation, and persistently save the execution result of the target set operation for reuse when the set operation needs to be executed again.

[0026] Optionally, the graph data processing system further includes: a dynamic partitioning module;

[0027] For the obtained target graph data, according to a preset graph pattern matching algorithm, determine each set operation required when extracting a subgraph matching a specified graph pattern from the target graph data, specifically including:

[0028] Take the obtained graph data in chunks as the target graph data. According to a preset graph pattern matching algorithm, determine each set operation required to extract a subgraph matching a specified graph pattern from the target graph data. The graph data chunks are obtained by the dynamic partitioning module from the original graph data. For each node in the original graph data, determine whether the degree of the node exceeds a preset threshold. If so, determine that the node is a central node. For each central node, through multiple rounds of neighbor node traversal, determine each node having a connection relationship with the central node as the associated node of the central node, which is determined according to each central node and the associated nodes of each central node.

[0029] Optionally, taking the obtained graph data in chunks as the target graph data specifically includes:

[0030] Obtain a processing task from a preset task queue, and take the graph data chunk corresponding to the processing task as the target graph data. The processing task is generated by the dynamic partitioning module for each graph data chunk and added to the preset task queue.

[0031] Optionally, before determining each node having a connection relationship with each central node through multiple rounds of neighbor node traversal for each central node, the method further includes:

[0032] For each central node, determine whether the central node is an already visited node;

[0033] For each central node, determining each node having a connection relationship with the central node through multiple rounds of neighbor node traversal specifically includes:

[0034] If the central node is not an accessed node, then through multiple rounds of neighbor node traversal, determine each node that has a connection relationship with the central node as the associated nodes of the central node, and set the central node as an accessed node.

[0035] Optionally, before determining whether the number of times a set operation is executed exceeds a preset threshold for each set operation, the method further includes:

[0036] For each set operation, determine whether there is a unique identifier corresponding to the set operation. If not, generate and save a unique identifier corresponding to the set operation according to the two sets involved in the set operation and the specified type corresponding to the set operation.

[0037] Optionally, according to the number of elements included in the two sets involved in the set operation and a preset cost function, determine the cost value corresponding to executing the set operation according to each execution strategy, specifically including:

[0038] For each set operation, according to the number of nodes included in the two node sets involved in the set operation and the processing data of the processing unit that executes the set operation, determine the computing time and memory access time required to execute the set operation according to each execution strategy, and determine the cost value corresponding to each execution strategy according to the computing time and the memory access time. The processing data of the processing unit includes: the bandwidth of the processing unit and the memory access latency of the processing unit.

[0039] This specification provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above graph data processing method is implemented.

[0040] This specification provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above graph data processing method is implemented.

[0041] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0042] In the graph data processing method provided in this specification, the processing unit determines, for the acquired target graph data, the various set operations required to extract a subgraph matching a specified graph pattern from the target graph data according to a preset graph pattern matching algorithm. The set operation is used to represent the execution of a specified type of operation on the neighbor node set of two nodes in the target graph data. The specified type of operation includes: taking at least one of taking the intersection and taking the difference. Through the decision module, for each set operation, based on the number of nodes included in the two node sets involved in the set operation and a preset cost function, the cost value corresponding to executing the set operation according to each execution strategy is determined, and based on the cost value, a target strategy is selected from each execution strategy so that the processing unit executes the set operation according to the target strategy, obtains the execution result corresponding to the set operation, and stores it in a memory. In response to obtaining the execution result corresponding to each set operation, the execution result corresponding to each set operation is read from the memory, and based on the execution result corresponding to each set operation, the subgraph matching the specified graph pattern in the target graph data is determined, so as to execute the task according to the subgraph.

[0043] It can be seen from the above method that, according to the preset graph model matching algorithm, the various set operations required for the subgraph matching the specified graph pattern can be extracted from the target graph data. For each set operation, according to the number of elements in the two sets involved in executing the set operation, the cost value corresponding to the performance of the processing unit occupied by executing the set operation according to different execution strategies is determined. Then, the execution strategy with the least performance occupation of the processing unit can be selected to execute the set operation, thereby improving the efficiency of graph data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0045] Figure 1 A schematic diagram of a graph data processing system provided in this specification;

[0046] Figure 2 A schematic diagram of the graph pattern matching algorithm provided in this specification;

[0047] Figure 3 A schematic diagram of the dynamic partitioning module provided in this specification;

[0048] Figure 4 A schematic diagram of a graph data processing method provided in this specification;

[0049] Figure 5 This manual provides a corresponding Figure 4Schematic diagram of an electronic device. Detailed implementation manners

[0050] To make the objectives, technical solutions and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0051] The following will detail the technical solutions provided by each embodiment of this specification in conjunction with the drawings.

[0052] This specification provides a graph data processing system, as Figure 1 shown:

[0053] Figure 1 Schematic diagram of a graph data processing system provided in this specification.

[0054] From Figure 1 It can be seen that the graph data processing system includes: each processing unit and a memory. Among them, each processing unit further includes: a decision-making module.

[0055] Among them, the processing unit is used to, for the obtained target graph data, according to a preset graph pattern matching algorithm, determine each set operation required to extract a subgraph matching a specified graph pattern from the target graph data. Here, the set operation is used to perform a specified type of operation on the neighbor node sets of two nodes in the target graph data. The specified type of operation here includes: taking the intersection, taking the difference set, and other operations.

[0056] Further, the decision-making module is used to, for each set operation, according to the number of nodes included in the two node sets involved in the set operation, and a preset cost function, determine the cost value corresponding to executing the set operation according to each execution strategy, and according to the cost value, select a target strategy from each execution strategy (such as: determining the execution strategy with the smallest cost value as the target strategy), so that the processing unit executes the set operation according to the target strategy, obtains the execution result corresponding to the set operation, and stores it in the memory. In response to obtaining the execution result corresponding to each set operation, read the execution results corresponding to each set operation from the memory, and according to the execution results corresponding to each set operation, determine the subgraph in the target graph data that matches the specified graph pattern, so as to execute a task according to the subgraph.

[0057] Among them, the above execution strategy can be set according to actual requirements, such as: an execution strategy based on a merging algorithm, an execution strategy based on a binary search algorithm, an execution strategy based on a hash algorithm, and an execution strategy based on a bit array algorithm.

[0058] In different application scenarios, the tasks to be executed according to the subgraph are also different. For example: in an e-commerce scenario, based on the commonalities among users, the graph pattern corresponding to the association relationship between users with similar needs can be determined. Furthermore, from the user graph data, the user subgraph that matches the above graph pattern can be determined, and based on the user subgraph, product recommendations can be made for each user in the user subgraph.

[0059] It should be noted that the graph pattern matching algorithms required to extract different graph patterns from the target graph data are also different. The processing unit can determine each set operation required to extract the subgraph that matches the specified graph pattern from the target graph data according to the preset graph pattern matching algorithm, specifically as Figure 2 shown.

[0060] Figure 2 This is a schematic diagram of the graph pattern matching algorithm provided in this specification.

[0061] Combined with Figure 2 it can be seen that when the graph pattern to be matched is a triangular structure composed of three nodes connected by an edge to each other, the corresponding graph pattern matching algorithm can be to judge whether there is an intersection between the neighbor node set of each node in the target graph data and the neighbor node sets of each neighbor node of this node. If so, it means that there is the above graph pattern of the triangular structure; if not, it means that there is no above graph pattern of the triangular structure.

[0062] For example: assume that the neighbor node set of node A includes node B and node C, and the neighbor node set of node B includes node C, node A, and node D. At this time, both the neighbor node set of node A and the neighbor node set of node B contain node C. At this time, it can be said that node C is the common neighbor node of node A and node B. That is to say, node A, node B, and node C are all connected by an edge to each other, that is, it satisfies the above graph pattern of the triangular structure. At this time, node A, node B, and node C can be used as the subgraph matched by the graph pattern matching algorithm.

[0063] Furthermore, it can be seen from the above that there are a large number of redundant set operations in the process of executing the graph pattern matching algorithm. For example, in practical application scenarios, the intersection between the neighbor node set of node A and the neighbor node set of neighbor node B of node A is first taken to obtain the subgraph corresponding to nodes A, B, and C. In addition, since the graph pattern matching algorithm needs to start from each node to determine the intersection between the neighbor node set of the node and the neighbor node set of the neighbor node of the node, in addition to starting from node A to determine the intersection between the neighbor node set of node A and the neighbor node set of neighbor node B of node A, it will also start from node B to determine the intersection between the neighbor node set of node B and the neighbor node set of neighbor node A of node B to obtain the subgraph corresponding to nodes A, B, and C. Therefore, the execution efficiency of the graph pattern matching algorithm is reduced.

[0064] Based on this, a detection module can also be provided in the above processing unit. The detection module is used to determine whether the number of executions of each set operation exceeds a preset threshold for each set operation. If so, it is determined that the set operation is a target set operation, and the execution result of the target set operation is persistently saved (which can be understood as saving the execution result of the target set operation to a specified memory space to avoid overwriting the execution result of the target combination operation during the execution of the graph pattern matching algorithm) for reuse when the set operation needs to be executed again.

[0065] Of course, before determining whether the number of executions of the set operation exceeds the preset threshold, the detection module can also determine whether there is a unique identifier corresponding to the set operation for each set operation. If not, a unique identifier corresponding to the set operation is generated and saved according to the two sets involved in the set operation and the specified type corresponding to the set operation. If so, the number of executions of the set operation can be determined according to the unique identifier corresponding to the set operation, and it can be determined whether the number of executions of the set operation exceeds the preset threshold.

[0066] It should be noted that for each node, the neighbor node set of the node can be identified using the unique identifier corresponding to the node.

[0067] For the node set composed of intermediate data (that is, the result after taking the intersection or difference of two sets), since the frequency of performing set operations on the intermediate data and other node sets is relatively low, and since the number of nodes in the node set composed of intermediate data is small, the detection module does not detect the set operation corresponding to the node set composed of intermediate data.

[0068] Further, for each set operation, the detection module can determine whether the set operation is a target set operation. If so, it can return the address in the memory of the execution result of the target set operation saved in advance. If not, it can transmit the set operation to the decision module for execution.

[0069] In an actual application scenario, the scale of the original graph data obtained by the graph data processing system is often large. Therefore, when directly using the original graph data as the target graph data for graph pattern matching, it often leads to the fact that due to the large and scattered data scale, the execution of a set operation will be performed by multiple processing units. As a result, the statistical values of the number of times the set operation is executed among the processing units need to be synchronized, leading to a large amount of additional performance overhead.

[0070] Based on this, the above graph data processing system may further include a dynamic partitioning module. The dynamic partitioning module is used to obtain the original graph data, and for each node in the original graph data, determine whether the degree of the node exceeds a preset threshold. If so, determine that the node is a central node. For each central node, through multiple rounds of neighbor node traversal, determine the nodes that have a connection relationship with the central node as the associated nodes of the central node. According to each central node and the associated nodes of each central node, determine each graph data block, and for each graph data block, use the graph data block as the target graph data so that the processing unit processes the graph data block, specifically as Figure 3 shown.

[0071] Figure 3 It is a schematic diagram of the dynamic partitioning module provided in this specification.

[0072] Combined Figure 3 It can be seen that the dynamic partitioning module can, for each central node, determine whether the central node is an accessed node. If not, through multiple rounds of neighbor node traversal, determine the nodes that have a connection relationship with the central node as the associated nodes of the central node, and set the central node as an accessed node.

[0073] Among them, the above method of neighbor node traversal can be, for example, depth - first traversal, breadth - first traversal. For each central node, when the depth of accessing the neighbor nodes of the central node reaches the preset threshold, or when there are no nodes to access, the above multiple rounds of neighbor node traversal end, and the graph data blocks are obtained.

[0074] Further, the dynamic partitioning module can, for each graph data block, generate a processing task for processing the graph data block, and add the processing task to a preset task queue, so that the processing unit obtains the processing task from the task queue and uses the graph data block corresponding to the processing task as the target graph data.

[0075] It should be noted that since the dynamic partitioning module can partition the original graph data into several graph data chunks and assign the processing tasks of each graph data chunk to a processing unit for execution, a set operation often concentrates in one processing unit, thereby avoiding the performance loss of the processing unit caused by data synchronization between processing units.

[0076] In addition, the above graph data processing system can allocate independent storage spaces for each processing unit to store the execution results of set operations and data such as the number of times the set operation is executed.

[0077] It should be noted that by setting task queues between the dynamic partitioning module and each processing unit, each processing unit can execute each processing task, and the dynamic partitioning module can partition the original graph data and generate each processing task to be executed in parallel, thereby improving the processing efficiency of graph data.

[0078] Furthermore, when the decision-making module determines the target execution strategy corresponding to each set operation, for each set operation, according to the number of elements included in the two sets involved in the set operation and the processing data of the processing unit that executes the set operation, determine the calculation time and memory access time required to execute the set operation according to each execution strategy. According to the calculation time and memory access time, determine the cost value corresponding to each execution strategy. The processing data of the processing unit here includes: the bandwidth of the processing unit and the memory access latency of the processing unit.

[0079] In addition, in this specification, for an execution strategy with a large computation-to-memory ratio (such as: an execution strategy based on a merge algorithm), a dedicated logic circuit module can be set in the processing unit for calculation. For an execution strategy with a small computation-to-memory ratio (such as: an execution strategy based on a bit array algorithm), a near-memory architecture can be used, that is, bit operations are set in the memory to execute set operations, which can be understood as the set operation is completed inside the memory, reducing the memory access time required for data transmission between the memory and the processing unit, thereby improving the processing efficiency of graph data.

[0080] It can be seen from the above content that according to the preset graph model matching algorithm, each set operation required to extract the subgraph matching the specified graph pattern from the target graph data can be determined. For each set operation, according to the number of elements in the two sets involved in the execution of the set operation, determine the cost value corresponding to the performance of the processing unit occupied by executing the set operation according to different execution strategies, and then the execution strategy with the smallest occupancy of the processing unit performance can be selected to execute the set operation, thereby improving the processing efficiency of graph data.

[0081] In this specification, the above-mentioned processing unit may refer to each processing core of a multi-core processor, and the above-mentioned detection module, decision module, and dynamic partitioning module may be hardware units provided on the multi-core processor.

[0082] To further elaborate on the above-mentioned graph data processing system, this specification also provides a method for processing graph data through the above-mentioned graph data processing system, specifically as Figure 4 shown.

[0083] Figure 4 It is a schematic diagram of a graph data processing method provided in this specification, including the following steps:

[0084] S401: For the obtained target graph data, according to a preset graph pattern matching algorithm, determine each set operation required when extracting a subgraph matching a specified graph pattern from the target graph data. The set operation is used to represent performing a specified type of operation on the neighbor node sets of two nodes in the target graph data, and the specified type of operation includes at least one of taking the intersection and taking the difference set;

[0085] S402: Through the decision module, for each set operation, according to the number of nodes included in the two node sets involved in this set operation and a preset cost function, determine the cost value corresponding to performing this set operation according to each execution strategy, and based on the cost value, select a target strategy from each execution strategy, so that the processing unit executes this set operation according to the target strategy, obtains the execution result corresponding to this set operation, and stores it in the memory;

[0086] S403: After obtaining the execution result corresponding to each set operation, read the execution results corresponding to each set operation from the memory, and according to the execution results corresponding to each set operation, determine the subgraph in the target graph data that matches the specified graph pattern, so as to execute a task according to the subgraph.

[0087] For the obtained target graph data, according to a preset graph pattern matching algorithm, determine each set operation required when extracting a subgraph matching a specified graph pattern from the target graph data. The set operation is used to represent performing a specified type of operation on the neighbor node sets of two nodes in the target graph data, and the specified type of operation includes at least one of taking the intersection and taking the difference set;

[0088] Through a decision-making module, for each set operation, based on the number of nodes included in the two node sets involved in the set operation and a preset cost function, determine the cost value corresponding to executing the set operation according to each execution strategy, and based on the cost value, select a target strategy from each execution strategy, so that a processing unit executes the set operation according to the target strategy to obtain an execution result corresponding to the set operation and store it in a memory;

[0089] After obtaining the execution result corresponding to each set operation, read the execution results corresponding to each set operation from the memory, and based on the execution results corresponding to each set operation, determine a subgraph in the target graph data that matches a specified graph pattern, so as to execute a task based on the subgraph.

[0090] Through a detection module, for each set operation, determine whether the number of times the set operation is executed exceeds a preset threshold. If so, determine the set operation as a target set operation, and persistently save the execution result of the target set operation for reuse when the set operation needs to be executed again.

[0091] For the obtained target graph data, according to a preset graph pattern matching algorithm, determine each set operation required for extracting a subgraph that matches a specified graph pattern from the target graph data. Specifically, it includes:

[0092] Use the obtained graph data in chunks as the target graph data. According to a preset graph pattern matching algorithm, determine each set operation required for extracting a subgraph that matches a specified graph pattern from the target graph data. The graph data in chunks is obtained by a dynamic partitioning module from the original graph data. For each node in the original graph data, determine whether the degree of the node exceeds a preset threshold. If so, determine the node as a central node. For each central node, through multiple rounds of neighbor node traversal, determine each node having a connection relationship with the central node as an associated node of the central node, which is determined based on each central node and the associated nodes of each central node.

[0093] Obtain a processing task from a preset task queue, and use the graph data in chunks corresponding to the processing task as the target graph data. The processing task is generated by the dynamic partitioning module for each graph data in chunk and added to the preset task queue.

[0094] For each central node, determine whether the central node is an accessed node;

[0095] For each central node, through multiple rounds of neighbor node traversal, determine each node having a connection relationship with the central node. Specifically, it includes:

[0096] If the central node is not an accessed node, then through multiple rounds of neighbor node traversal, determine each node that has a connection relationship with the central node as the associated nodes of the central node, and set the central node as an accessed node.

[0097] For each set operation, determine whether there is a unique identifier corresponding to the set operation. If not, generate and save the unique identifier corresponding to the set operation according to the two sets involved in the set operation and the specified type corresponding to the set operation.

[0098] For each set operation, according to the number of nodes included in the two node sets involved in the set operation, and the processing data of the processing unit that executes the set operation, determine the computing time and memory access time required to execute the set operation according to each execution strategy. According to the computing time and memory access time, determine the cost value corresponding to each execution strategy. The processing data of the processing unit includes: the bandwidth of the processing unit and the memory access latency of the processing unit.

[0099] It can be seen from the above that, according to the preset graph model matching algorithm, each set operation required to extract the subgraph matching the specified graph pattern from the target graph data can be obtained. For each set operation, according to the number of elements in the two sets involved in executing the set operation, determine the cost value corresponding to the performance of the processing unit occupied by executing the set operation according to different execution strategies. Furthermore, the execution strategy that occupies the least performance of the processing unit can be selected to execute the set operation, thereby improving the efficiency of graph data processing.

[0100] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the Figure 1 method provided above.

[0101] This specification also provides Figure 5 a schematic structural diagram of an electronic device corresponding to Figure 1 . As Figure 5 shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the Figure 1 method provided above.

[0102] Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logical devices or the combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or logical devices.

[0103] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many method flow improvements today can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented using a hardware entity module. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is an integrated circuit whose logic function is determined by the user programming the device. Designers can program themselves to "integrate" a digital system onto a single PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, today, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a Hardware Description Language (HDL). There is not just one type of HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow using the above-mentioned several hardware description languages and programming it into an integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0104] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium that stores computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0105] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0106] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0107] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) that contain computer-usable program code.

[0108] This specification is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the specification. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable graphic data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable graphic data processing devices generate means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks

[0109] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable graphic data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks

[0110] These computer program instructions can also be loaded onto a computer or other programmable graphic data processing device, such that a series of operational steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks

[0111] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0112] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0113] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0114] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0115] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, system or computer program product. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0117] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, reference can be made to the corresponding description in the method embodiment.

[0118] The above description is only for the embodiments of this specification and is not intended to limit this specification. For those skilled in the art, various modifications and changes can be made to this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this specification.

Claims

1. A graph data processing system, characterized in that: The graph data processing system includes: various processing units and a memory, wherein each processing unit is further provided with: a decision module; The processing unit is configured to determine, based on a preset graph pattern matching algorithm and with respect to the acquired target graph data, various set operations required for extracting a subgraph matching a specified graph pattern from the target graph data, wherein the set operations are configured to represent performing a specified type of operation on a set of neighbor nodes of two nodes in the target graph data, the specified type of operation comprising at least one of: taking an intersection and taking a difference; Through the decision module, the processing unit determines, for each set operation, the computing time and memory access time required to execute the set operation according to each execution strategy based on the number of nodes included in the two node sets involved in the set operation and the processing data of the processing unit that executes the set operation, determines the cost value corresponding to executing the set operation according to each execution strategy based on the computing time and the memory access time, and selects a target strategy from each execution strategy based on the cost value so that the processing unit executes the set operation according to the target strategy, obtains the execution result corresponding to the set operation, and stores it in the memory; in response to obtaining the execution result corresponding to each set operation, reads the execution result corresponding to each set operation from the memory, and determines the subgraph in the target graph data that matches the specified graph pattern based on the execution result corresponding to each set operation, so as to execute the task according to the subgraph; the processing data of the processing unit includes: the bandwidth of the processing unit and the memory access delay of the processing unit; The memory is used to store the execution results corresponding to each set operation.

2. The graph data processing system according to claim 1, wherein: The processing unit is further provided with: a detection module; The detection module is used to determine, for each collective operation, whether the number of times the collective operation is executed exceeds a preset threshold. If so, the collective operation is determined to be a target collective operation, and the execution result of the target collective operation is persisted and saved for reuse when the collective operation needs to be executed again.

3. The graph data processing system according to claim 1, wherein: The graph data processing system further includes: a dynamic partitioning module; The dynamic partitioning module is used to obtain original graph data, and for each node in the original graph data, determine whether the degree of the node exceeds a preset threshold. If so, the node is determined to be a central node. For each central node, through multiple rounds of neighbor node traversal, each node that has a connection relationship with the central node is determined as the associated node of the central node; According to each central node and the associated nodes of each central node, each graph data block is determined, and for each graph data block, the graph data block is used as the target graph data, so that the processing unit processes the graph data block.

4. The graph data processing system according to claim 3, wherein: The dynamic partitioning module is used to generate a processing task for processing each graph data block, and add the processing task to a preset task queue so that the processing unit obtains the processing task from the task queue and uses the graph data block corresponding to the processing task as the target graph data.

5. The graph data processing system according to claim 3, wherein: The dynamic partitioning module is used to determine, for each central node, whether the central node is a visited node. If not, through multiple rounds of neighbor node traversal, determine the nodes that have a connection relationship with the central node as the associated nodes of the central node, and set the central node as a visited node.

6. The graph data processing system according to claim 2, wherein: The detection module is used to determine, for each set operation, whether there is a unique identifier corresponding to the set operation; if not, generate and save the unique identifier corresponding to the set operation based on the two sets involved in the set operation and the specified type corresponding to the set operation.

7. A graph data processing method, characterized in that: The graph data processing method is applied to a graph data processing system, which includes: various processing units and a memory, wherein each processing unit is further provided with: a decision module, and the method includes: The processing unit determines, based on the acquired target graph data and according to a preset graph pattern matching algorithm, various set operations required when extracting a subgraph matching a specified graph pattern from the target graph data, wherein the set operations are used to represent performing a specified type of operation on a set of neighbor nodes of two nodes in the target graph data, and the specified type of operation includes at least one of taking an intersection and taking a difference; The decision module determines, for each collective operation, the computation time and memory access time required to execute the collective operation according to each execution strategy based on the number of nodes included in the two node sets involved in the collective operation and the processing data of the processing unit executing the collective operation. A cost value corresponding to executing the collective operation according to each execution strategy is determined based on the computation time and the memory access time. A target strategy is selected from each execution strategy based on the cost value so that the processing unit executes the collective operation according to the target strategy, obtains an execution result corresponding to the collective operation, and stores the result in the memory. The processing data of the processing unit includes: the bandwidth of the processing unit and the memory access latency of the processing unit. In response to obtaining the execution result corresponding to each set operation, the execution result corresponding to each set operation is read from the memory, and based on the execution result corresponding to each set operation, the subgraph in the target graph data that matches the specified graph pattern is determined to execute the task according to the subgraph.

8. The method according to claim 7, wherein The processing unit is further provided with a detection module, and the method further comprises: Through the detection module, for each collective operation, it is determined whether the number of times the collective operation is executed exceeds a preset threshold. If so, the collective operation is determined to be a target collective operation, and the execution result of the target collective operation is persisted and saved for reuse when the collective operation needs to be executed again.

9. The method according to claim 7, wherein: The graph data processing system further includes: a dynamic partitioning module; For the acquired target graph data, according to a preset graph pattern matching algorithm, the set operations required to extract a subgraph matching the specified graph pattern from the target graph data are determined, specifically including: The obtained graph data blocks are used as target graph data, and according to the preset graph pattern matching algorithm, the various set operations required to extract the subgraph matching the specified graph pattern from the target graph data are determined. The graph data blocks are obtained by the dynamic partitioning module. For each node in the original graph data, it is determined whether the degree of the node exceeds the preset threshold. If so, the node is determined to be the central node. For each central node, through multiple rounds of neighbor node traversal, the nodes that have a connection relationship with the central node are determined as the associated nodes of the central node, and are determined based on each central node and the associated nodes of each central node.

10. The method according to claim 9, wherein The obtained graph data is divided into blocks as the target graph data, specifically including: A processing task is obtained from a preset task queue, and the graph data block corresponding to the processing task is used as the target graph data. The processing task is generated by the dynamic partitioning module for each graph data block and added to the preset task queue.

11. The method according to claim 9, wherein For each central node, before determining each node connected to the central node through multiple rounds of neighbor node traversal, the method further includes: For each central node, determine whether the central node is a visited node; For each central node, multiple rounds of neighbor node traversal are performed to determine the nodes that are connected to the central node, including: If the central node is not a visited node, multiple rounds of neighbor node traversal are performed to determine the nodes that are connected to the central node as associated nodes of the central node, and the central node is set as a visited node.

12. The method according to claim 8, wherein Before determining, for each collective operation, whether the number of times the collective operation has been executed exceeds a preset threshold, the method further includes: For each set operation, determine whether there is a unique identifier corresponding to the set operation. If not, generate and save the unique identifier corresponding to the set operation based on the two sets involved in the set operation and the specified type corresponding to the set operation.

13. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 7 to 12 is implemented.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 7 to 12 is implemented.

Images