Method and apparatus for partitioning graph data, electronic device and program product

Abstract

The invention relates to a method and device for dividing graph data, electronic equipment and a program product. The method includes determining a total number related to both vertices and edges in the graph data. Further, the method further includes determining an upper limit value of the number of both vertices and edges to which the processing threads can be allocated based on the total number and the number of the plurality of processing threads in the distributed cluster. In this way, the multiple vertexes can be allocated to the multiple processing threads according to the number of the edges of the multiple vertexes in the graph data, and the number of the vertexes allocated to the processing threads and the number of the corresponding edges are smaller than or equal to the upper limit value.

Description

Technical Field

[0001] The present invention relates to the field of graph technology, and more particularly, to a method, an apparatus, an electronic device, and a program product for partitioning graph data. Background Art

[0002] Graph data has a wide range of applications because it is easy to represent complex dependencies between different entities. For example, in social network analysis, graph data can be used to represent the relationship between users. In route planning, graph data can be used to represent the connection between road networks and cities. In the field of artificial intelligence, graph data can be used to represent knowledge graphs and recommendation systems.

[0003] With the development of big data and the Internet, the scale of graph data has shown an exponential growth. There may be billions of points and trillions of edges in the graph data. Such a huge amount of data exceeds the processing capacity of a single machine. The processing of graph data usually requires the participation of multiple machines. Summary of the invention

[0004] Embodiments of the present invention provide a method, an apparatus, an electronic device, a storage medium, and a program product for partitioning graph data.

[0005] According to a first aspect of the present invention, a method for partitioning graph data is provided. The method includes determining the total number of both vertices and edges in the graph data. The method also includes determining the upper limit of the number of both vertices and edges to which the processing threads can be assigned based on the total number and the number of multiple processing threads in the distributed cluster. In addition, the method also includes assigning multiple vertices to multiple processing threads according to the number of edges of each of the multiple vertices in the graph data, and the number of both vertices and corresponding edges assigned to the processing threads is less than or equal to the upper limit.

[0006] According to a second aspect of the present invention, there is provided a device for partitioning graph data, the device comprising a total number determination module configured to determine the total number of both vertices and edges in the graph data. The device further comprises an upper limit value determination module configured to determine the upper limit value of the number of both vertices and edges to which a processing thread can be assigned based on the total number and the number of multiple processing threads in a distributed cluster. In addition, the device further comprises a vertex assignment module configured to assign multiple vertices to multiple processing threads according to the number of edges of each of the multiple vertices in the graph data, wherein the number of both vertices and corresponding edges assigned to the processing threads is less than or equal to the upper limit value.

[0007] According to a third aspect of the present invention, an electronic device is provided, which includes at least one processor and a memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the processor, causes the electronic device to execute the method according to the first aspect.

[0008] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program product stored thereon, the computer program product comprising machine-executable instructions which, when executed, cause a machine to perform the steps of the method of the first aspect of the present invention.

[0009] According to a fifth aspect of the present invention, there is provided a computer program product tangibly stored on a non-volatile computer readable medium and comprising machine executable instructions which, when executed, cause a machine to perform the steps of the method of the first aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0011] Figure 1 A schematic diagram illustrating an example scenario in which an apparatus and / or method according to an embodiment of the present invention may be implemented;

[0012] Figure 2 A schematic flow chart of a method for partitioning graph data in an embodiment of the present invention is illustrated;

[0013] Figure 3 A schematic flow chart of a method for allocating graph vertices in an embodiment of the present invention is illustrated;

[0014] Figure 4A A schematic diagram of a scenario for allocating graph vertices in the first round in an embodiment of the present invention is illustrated;

[0015] Figure 4B A schematic diagram of a scenario for allocating graph vertices in the second round in an embodiment of the present invention is illustrated;

[0016] Figure 5 A schematic block diagram of an apparatus for dividing graph data provided by an embodiment of the present invention is illustrated; and

[0017] Figure 6 A schematic block diagram of an example apparatus suitable for implementing embodiments of the present invention is illustrated.

[0018] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0019] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] As mentioned above, the processing of graph data often requires the participation of multiple computing nodes. During the processing, the graph data needs to be divided and the graph tasks are assigned to different computing nodes for parallel computing. This division often leads to load imbalance. Some computing nodes may bear too many computing tasks, resulting in reduced computing efficiency and even node overload.

[0022] At least to solve the above and other potential problems, an embodiment of the present invention provides a method for partitioning graph data. In this method, the total number of vertices and edges related to the graph data can be determined, and based on the total number and the number of multiple processing threads in the distributed cluster, the upper limit of the number of vertices and edges to which the processing thread can be assigned is determined. On the basis of determining the upper limit, multiple vertices can be assigned to multiple processing threads according to the number of edges of each of the multiple vertices in the graph data, and the number of vertices and corresponding edges assigned to the processing threads is less than or equal to the upper limit. Through this method, the vertices in the graph data can be evenly assigned to the processing threads of the distributed cluster according to the number of edges of the vertices, so that the total number of vertices and edges in each processing thread is basically the same, so that each processing thread can achieve load balancing in the process of processing graph data. In this way, the overall performance of the processing threads can be improved and the efficiency of processing graph data can be increased.

[0023] To facilitate understanding, first, combine Figure 1 Describe the scenarios to which the embodiments of the present invention are applicable. Figure 1The scene 100 shown includes graph data 101, a control device 102, and a distributed cluster 103 composed of multiple machines. The graph data 101 is a data structure that uses vertices and edges to represent entities and relationships. The graph data 101 may include multiple vertices, each of which may represent an entity or object, such as a person, a place, or a transaction. The graph data 101 may include an edge, which is used to connect two vertices and represents a relationship or dependency between the two vertices. The edge in the graph data 101 may be undirected or directed. In some embodiments, the graph data may be represented as G=(V, E), where G represents a graph, V represents the set of all vertices in the graph G, and E represents the set of all edges in the graph G. In the graph data, an edge e pointing from vertex u to vertex v may be represented by e=(u, v). The graph data may be pre-stored in the memory of the control device 102, or may be obtained by the control device 102 from other devices, and the present invention is not limited to this.

[0024] In some embodiments, the distributed cluster 103 may include multiple machines with the same configuration, and the machine refers to each independent computer or server in a distributed system, which is connected together through a network to complete a task or service together. These machines are generally referred to as computing nodes (Node), and each computing node is an independent computer system that can run applications and services. The machine may include various general and / or special processing components with processing and computing capabilities, such as but not limited to arithmetic processing units (CPUs), graphics processing units (GPUs), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc.

[0025] In some embodiments, the distributed cluster 103 may include machine 111, machine 112, machine 113, and more machines not shown in the figure. Each machine may include multiple processing threads for processing training tasks, for example, machine 111 may include processing thread 121 and processing thread 122, machine 112 may include processing thread 123 and processing thread 124, and machine 113 may include processing thread 125 and processing thread 126. It should be understood that each of the multiple machines included in the distributed cluster 103 may have more processing threads. In some embodiments, the processing threads of the machines in the distributed cluster 103 can be used to process vertices and edges in the graph data 101, for example, to perform graph computing and / or graph learning.

[0026] Graph processing is a computing model for processing graph structured data, in which graph computing can process and analyze graph data, and determine the patterns, relationships and structures in the graph data based on the calculation and analysis of graph data. Graph learning is a method of performing subsequent analysis and prediction by learning the complex node relationships and topological structures of graphs in graph data. Analysis means mapping the nodes and edges in the graph data into a continuous vector space, so that the discrete information in the graph data is converted into a continuous feature vector, so as to perform subsequent machine learning tasks, such as classification and clustering. Prediction means using the existing vertices and edges in the graph data to predict the edges or associations that may exist in the graph data in the future. It should be understood that graph computing aims to analyze and process graph data, while graph learning is to map the vertices or edges in the graph to a low-dimensional vector space through learning the representation in the graph data, so as to perform subsequent learning tasks. Therefore, compared with graph computing, graph learning involves large-scale data training and reasoning of graph data, and the number of vertices and edges involved is much larger than graph computing.

[0027] The control device 102 may distribute the graph data 101 to the distributed cluster 103 for processing. In some embodiments, the control device 102 may distribute the graph data 101 to multiple machines in the distributed cluster 103 for processing. For example, the control device 102 may distribute the graph data 101 to machines 111, 112, and 113 for processing. Further, the control device 102 may distribute the vertices in the graph data 101 to multiple processing threads of machines 111, 112, and 113 for processing. For example, when there are 12 vertices in the graph data 101, the control device 102 may evenly distribute the 12 vertices to the processing threads 121, 122, 123, 124, 125, and 126 included in machines 111, 112, and 113 for processing. In some embodiments, the edges associated with the vertices in the graph data 101 may be assigned to the same processing thread as the vertices. For example, the control device 102 may assign the edges pointing to the vertices together with the vertices to the same processing thread.

[0028] It should be understood that Figure 1 The illustrated scenario 100 is only an example of the present invention and cannot be a limitation of the present invention. For example, in some embodiments, the control device 102 may be deployed in the distributed cluster 103, for example, it may be software or a module in a machine in the distributed cluster 103. It should also be understood that the control device 102 in the embodiment of the present invention is only illustrative, and the control device 102 may be any suitable device and may be implemented in software and / or hardware.

[0029] Combined with the above Figure 1Describes example scenarios in which the devices and / or methods of embodiments of the present invention may be implemented. Figure 2 The process of dividing graph data involved in the embodiment of the present invention is described.

[0030] Figure 2 The method 200 for partitioning graph data in an embodiment of the present invention is exemplarily shown. The method 200 may be executed by a control device, which may be, for example, the control device 102 in the environment 100, or may be software or a module in a machine in the distributed cluster 103. The control device may be implemented in software and / or hardware. For ease of explanation, the method 200 is schematically described below using the control device 102 as an example of the execution subject. Figure 2 , method 200 may include steps 202 to 206, and the steps included in process 200 will be described in detail below.

[0031] In box 202, the total number associated with both the vertices and the edges in the graph data is determined. In some embodiments, the control device 102 can determine a vertex in the graph data 101, and based on the vertex, determine the number of all edges connected to the vertex, wherein the number value of all edges connected to the vertex can be referred to as the degree of the vertex. In some embodiments, the control device 102 can determine the degree of each vertex of all vertices included in the graph data 101 based on the above-mentioned method of determining the degree of a vertex. In some embodiments, the control device 102 can determine the sum of the edges included in all vertices based on the degrees of each vertex of all vertices in the graph data 101. Based on the sum of the edges and the number of all vertices in the graph data 101, the total number of all vertices in the graph data 101 and the edges included in all vertices is determined.

[0032] In block 204, an upper limit value of the number of both vertices and edges to which a processing thread can be assigned is determined based on the total number and the number of multiple processing threads in the distributed cluster. In some embodiments, the upper limit value represents an upper limit of the number of edges and the number of vertices that can be assigned to each processing thread of a machine in the distributed cluster 103. In some embodiments, the control device 102 can determine an upper limit value of the number of both vertices and edges that can be assigned to each processing thread based on the number of all vertices in the graph data 101 and the sum of the edges corresponding to all vertices and the number of multiple processing threads included in the distributed cluster 103.

[0033] In block 206, multiple vertices are allocated to multiple processing threads according to the number of edges of each of the multiple vertices in the graph data, and the number of both the vertices and the corresponding edges allocated to the processing threads is less than or equal to the upper limit value. In some embodiments, the control device 102 can determine the vertices to be allocated to the multiple processing threads based on the number of edges of each of the multiple vertices. For example, the control device 102 can sort the vertices based on the size values ​​of the respective degrees of each vertex, and determine the vertices to be allocated to the multiple processing threads based on the sorting order. In some embodiments, the control device 102 can determine whether to allocate vertices to multiple processing threads based on the upper limit value.

[0034] In this way, when using a distributed cluster to process graph data, the vertices in the graph data can be evenly distributed to the processing threads of the distributed cluster according to the number of edges of the vertices, so that the total number of vertices and edges in each processing thread is basically the same, thereby achieving load balancing for each processing thread in the process of processing graph data.

[0035] In some embodiments, in the aforementioned block 206, the control device 102 may sort the plurality of vertices according to the number of edges of each vertex to determine the vertex sequence to be assigned to the processing thread of the distributed cluster 103. In some embodiments, the control device 102 may obtain the vertex sequence by sorting in ascending order or descending order based on the number of edges of each vertex. In some embodiments, the control device 102 may assign the plurality of vertices in the vertex sequence to the plurality of processing threads according to the acquired vertex sequence.

[0036] In some embodiments, the control device 102 may determine a first group of processing threads among the plurality of processing threads, wherein the first group of processing threads corresponds to a group of machines in the distributed cluster. Exemplarily, the control device 102 may determine that the first group of processing threads among the plurality of processing threads may include processing thread 121, processing thread 123, and processing thread 125, wherein the first group of processing threads corresponds to a group of machines in the distributed cluster 103, wherein the group of machines may include machine 111, machine 112, and machine 113.

[0037] In some embodiments, the control device 102 may allocate the first group of vertices in the vertex sequence to the first group of processing threads according to the first order of the group of machines. In some embodiments, the first order of the group of machines may be the order of machine 111, machine 112, and machine 113. In some embodiments, the control device 102 may allocate the first group of vertices in the sorted vertex sequence to the first group of processing threads included in the group of machines according to the order of machine 111, machine 112, and machine 113. Exemplarily, the control device 102 may allocate the first group of vertices to processing thread 121, processing thread 123, and processing thread 125 according to the order of machine 111, machine 112, and machine 113.

[0038] In some embodiments, the control device 102 may determine a second group of processing threads among the plurality of processing threads, wherein the second group of processing threads corresponds to a group of machines in the distributed cluster. Exemplarily, the control device 102 may determine that the second group of processing threads among the plurality of processing threads may include processing thread 122, processing thread 124, and processing thread 126, wherein the second group of processing threads corresponds to a group of machines in the distributed cluster 103, wherein the group of machines may include machine 111, machine 112, and machine 113.

[0039] In some embodiments, the control device 102 may allocate the second group of vertices in the vertex sequence to the second group of processing threads according to the second order of a group of machines, wherein the first order is opposite to the second order. In some embodiments, the second group of vertices may be the next group of vertices after the first group of vertices in the vertex sequence. In some embodiments, the second order of a group of machines may be the order of machine 113, machine 112, and machine 111. In some embodiments, the control device 102 may allocate the second group of vertices to the second group of processing threads included in a group of machines according to the order of machine 113, machine 112, and machine 111. Exemplarily, the control device 102 may allocate the second group of vertices to processing thread 126, processing thread 124, and processing thread 122 according to the order of machine 113, machine 112, and machine 111. By executing the above method, each machine may be allocated substantially the same number of vertices and edges after two allocations in opposite order, thereby balancing the load of machines in the distributed cluster.

[0040] In some embodiments, in the current iteration, the control device 102 may determine the number of both vertices and corresponding edges to which the first processing thread in the first group of processing threads has been assigned. Exemplarily, the control device 102 may determine the number of both vertices and corresponding edges to which the first processing thread in the first group of processing threads, for example, the processing thread 121, has been assigned. In some embodiments, when it is determined that the number of both vertices and corresponding edges to which the processing thread 121 has been assigned is less than an upper limit value, the control device 102 may assign the first vertex in the first group of vertices to the processing thread 121. In some embodiments, the control device 102 may determine the second processing thread after the first processing thread in the first group of processing threads in a first order, for example, the processing thread 123, and determine the second vertex after the first vertex in the first group of vertices, and the processing thread 123 and the second vertex are used for the next iteration.

[0041] In some embodiments, in the first iteration, the control device 102 may determine the number of vertices and corresponding edges that have been assigned to the first processing thread in the first group of processing threads, such as the processing thread 121. In some embodiments, when it is determined that the number of vertices and corresponding edges that have been assigned to the processing thread 121 is greater than or equal to the upper limit value, the control device 102 determines the second processing thread after the first processing thread in the first group of processing threads according to the first order, which may be, for example, the processing thread 123. In some embodiments, in the second iteration, the control device 102 may determine the number of vertices and corresponding edges that have been assigned to the processing thread 123. In some embodiments, when it is determined that the number of vertices and corresponding edges that have been assigned to the processing thread 123 is less than the upper limit value, the control device 102 may assign the first vertex to the processing thread 123. In some embodiments, the control device 102 may determine the third processing thread after the second processing thread in the first group of processing threads according to the first order, and illustratively, the control device 102 may determine that the third processing thread may be the processing thread 125. In some embodiments, the control device 102 may determine the second vertex after the first vertex in the first group of vertices, wherein the third processing thread and the second vertex are used for the third iteration. By executing the above method, the number of vertices and edges allocated to each processing thread in each machine is equal to the upper limit value, so that load balancing is achieved among the processing threads.

[0042] In some embodiments, the control device 102 may determine a second group of processing threads among the plurality of processing threads. For example, the control device 102 may determine a second group of processing threads among the plurality of processing threads, including processing thread 122, processing thread 124, and processing thread 126. In some embodiments, the control device 102 may allocate a third group of vertices in the vertex sequence to the second group of processing threads according to a first order of a group of machines, wherein the third group of vertices is the next group of vertices after the second group of vertices in the vertex sequence. In some embodiments, the control device 102 may determine a first group of processing threads among the plurality of processing threads, wherein the first group of processing threads may include processing thread 121, processing thread 123, and processing thread 125. In some embodiments, the control device 102 may allocate a fourth group of vertices in the vertex sequence to the first group of processing threads according to a second order of a group of machines, wherein the fourth group of vertices is the next group of vertices after the third group of vertices in the vertex sequence. Through the above method, the vertices in the graph data can be evenly allocated to the processing threads of the distributed cluster according to the number of edges of the vertices.

[0043] In some embodiments, in the aforementioned box 204, the control device 102 may perform an integer division operation on the number of multiple processing threads based on the total number to determine the integer division result. The control device 102 may determine the upper limit value of the number of both vertices and edges to which the processing threads can be assigned based on the integer division result. Exemplarily, when the total number of vertices and edges in the graph data 101 is 12 and the number of processing threads is 6, the control device 102 may determine the upper limit value to be 2. When the total number of vertices and edges in the graph data 101 is 20 and the number of processing threads is 6, the control device 102 may determine the upper limit value to be 3. Alternatively or additionally, the upper limit value may be any selected suitable value, which is not limited here.

[0044] In some embodiments, in the aforementioned block 202, the control device 102 may determine the number of multiple vertices in the graph data 101. In some embodiments, the control device 102 may determine the number of multiple outgoing edges and multiple incoming edges corresponding to the multiple vertices. The control device 102 may determine the total number based on the number of multiple vertices, the number of multiple outgoing edges, and the number of multiple incoming edges.

[0045] Combined with the above Figure 2 The process for dividing graph data involved in the embodiment of the present invention is described. Figure 3 The following is a schematic flow chart of a method 300 for allocating graph vertices in an embodiment of the present invention. The method 300 may be executed by a control device, which may be the control device 102 in the environment 100, or may be software or a module in a machine in the distributed cluster 103. Figure 3 , method 300 may include blocks 302 to 312 .

[0046] In box 302, the control device 102 may determine a target processing thread and a target vertex, wherein the target vertex represents a vertex to be assigned to the target processing thread. In box 304, the control device 102 may determine a first number of vertices and a second number of degrees in the target processing thread. In box 305, the control device 102 determines whether the sum of the first number and the second number is less than an upper limit value, and if so, enters box 308, and if not, enters box 312. If so, in box 308, the control device 102 may assign the target vertex to the target processing thread. In box 310, the control device 102 may use the next processing thread as the target processing thread and the next vertex as the target vertex. After executing box 310, the control device 102 returns to box 304 and continues to execute related steps. If not, the control device 102 enters box 312, and the control device 102 may use the next processing thread as the target processing thread, and returns to box 304 and continues to execute related steps.

[0047] By executing the above steps, the vertices in the graph data can be evenly distributed to the processing threads of the distributed cluster according to the number of edges of the vertices, so that the total number of vertices and edges in each processing thread is substantially the same.

[0048] Next, in conjunction with FIG4 , the process of allocating vertices is described by taking at least one round of allocation as an example. Figure 4A A schematic diagram of a scenario for allocating graph vertices in a first round in an embodiment of the present invention is illustrated, wherein scenario 410 includes the first round of allocation.

[0049] In some embodiments, the control device 102 may sort the vertices included in the graph data 101 according to the degrees of the vertices, thereby obtaining a vertex sequence. In the first round of allocation, the control device 102 may determine a first group of processing threads among multiple processing threads, and the first group of processing threads corresponds to a group of machines in the distributed cluster 103. Among them, the first group of processing threads may include processing thread 121, processing thread 123, and processing thread 125. The control device 102 may determine a target processing thread, for example, processing thread 121 of machine 111. The control device 102 may determine a first number of vertices and a second number of degrees in the target processing thread 121. When the sum of the first number of all vertices and the second number of degrees of all vertices in the processing thread 121 is less than the upper limit value of the processing thread, the control device 102 may allocate the first vertex to the processing thread 121. When the sum of the first number of vertices and the second number of edges in the processing thread 121 is greater than or equal to the upper limit value, the control device 102 may determine the next processing thread 123 as the target processing thread, and determine whether the processing thread 123 meets the condition that the first vertex can be allocated.

[0050] In some embodiments, the control device 102 may determine the next vertex after the first vertex in the vertex sequence (referred to as the second vertex for ease of distinction and identification) as the target vertex to be assigned to the processing thread 123. The control device 102 may determine a first number of all vertices in the target processing thread 123 and a second number of degrees of all vertices. When the sum of the first number and the second number in the processing thread 123 is less than an upper limit value, the control device 102 may assign the second vertex to the processing thread 123. When the sum of the first number and the second number in the processing thread 123 is greater than or equal to the upper limit value, the control device 102 may determine the next processing thread 125 as the target processing thread, and determine whether the processing thread 125 meets the condition that the second vertex can be assigned.

[0051] In some embodiments, the control device 102 may determine the next vertex after the second vertex in the vertex sequence (referred to as the third vertex for ease of distinction and identification) as the target vertex to be assigned to the processing thread 125. The control device 102 may determine a first number of all vertices and a second number of degrees of all vertices in the target processing thread 125. When the sum of the first number and the second number in the processing thread 125 is less than an upper limit value, the control device 102 may assign the third vertex to the processing thread 125. When the sum of the first number and the second number in the processing thread 125 is greater than or equal to the upper limit value, the control device 102 may determine the next processing thread 126 as the target processing thread, and determine whether the processing thread 126 meets the condition that the third vertex can be assigned.

[0052] In some embodiments, in the first round of allocation, the control device 102 may continuously allocate vertices in the vertex sequence to corresponding target processing threads one by one until the processing thread 122 according to the above execution steps, thereby completing the first round of allocation. Figure 4B A schematic diagram of a scenario for allocating graph vertices in a second round according to an embodiment of the present invention is shown. In scenario 420, a second round of allocation is included.

[0053] In some embodiments, in the second round of allocation, after the control device 102 allocates the graph vertex (referred to as the fourth vertex for the convenience of distinction and identification) to the processing thread 122, the control device 102 may determine the next vertex after the fourth vertex in the vertex sequence (referred to as the fifth vertex for the convenience of distinction and identification) as the target vertex to be allocated to the processing thread 124. The control device 102 may determine the first number of all vertices in the processing thread 124 and the second number of degrees of all vertices. When the sum of the first number and the second number in the processing thread 124 is less than the upper limit value, the control device 102 may allocate the fifth vertex to the processing thread 124. When the sum of the first number and the second number in the processing thread 124 is greater than or equal to the upper limit value, the control device 102 may determine the next processing thread 126 as the target processing thread, and determine whether the processing thread 126 meets the condition that the fifth vertex can be allocated.

[0054] In some embodiments, the control device 102 may determine the next vertex after the fifth vertex in the vertex sequence (referred to as the sixth vertex for ease of distinction and identification) as the target vertex to be assigned to the processing thread 126. The control device 102 may determine a first number of all vertices and a second number of degrees of all vertices in the target processing thread 126. When the sum of the first number and the second number in the processing thread 126 is less than an upper limit value, the control device 102 may assign the sixth vertex to the processing thread 126. When the sum of the first number and the second number in the processing thread 126 is greater than or equal to the upper limit value, the control device 102 may determine the next processing thread 125 as the target processing thread, and determine whether the processing thread 125 meets the condition that the sixth vertex can be assigned.

[0055] In some embodiments, the control device 102 may continuously perform the first round and the second round of vertex allocation steps alternately performed above until all vertices in the graph data 101 are allocated.

[0056] It should be understood that the above steps of allocating vertices to processing threads are only examples, and more or fewer machines may be deployed in the distributed cluster 103. In addition, a machine may include more or fewer processing threads, which is not limited here.

[0057] Through the method of the present invention, the vertices in the graph data can be evenly distributed to the processing threads of the distributed cluster according to the number of edges of the vertices, so that the total number of vertices and edges in each processing thread is basically the same, so that each processing thread can achieve load balancing in the process of processing the graph data. In this way, the overall performance of the processing threads can be improved and the efficiency of processing the graph data can be increased.

[0058] Figure 5 FIG. 5 is a schematic diagram of an apparatus 500 for dividing graph data provided by an embodiment of the present invention. Figure 5 As shown, the device 500 may include a total number determination module 502, which is configured to determine the total number of both vertices and edges in the graph data. The device 500 also includes an upper limit value determination module 504, which is configured to determine the upper limit value of the number of both vertices and edges that can be assigned to the processing thread based on the total number and the number of multiple processing threads in the distributed cluster. In addition, the device 500 also includes a vertex allocation module 506, which is configured to allocate multiple vertices to multiple processing threads according to the number of edges of each of the multiple vertices in the graph data, and the number of both the vertices and the corresponding edges allocated to the processing thread is less than or equal to the upper limit value.

[0059] In some embodiments, the vertex allocation module 506 includes: a vertex sequence determination module configured to sort multiple vertices according to the number of edges of each of the multiple vertices to determine a vertex sequence; and a first allocation module configured to allocate multiple vertices to multiple processing threads according to the vertex sequence.

[0060] In some embodiments, the first allocation module includes: a first group of processing thread determination module, configured to determine a first group of processing threads among multiple processing threads, the first group of processing threads corresponding to a group of machines in a distributed cluster; a first group of vertex allocation module, configured to allocate a first group of vertices in a vertex sequence to the first group of processing threads according to a first order among the group of machines; a second group of processing thread determination module, configured to determine a second group of processing threads among multiple processing threads, the second group of processing threads corresponding to a group of machines; and a second group of vertex allocation module, configured to allocate a second group of vertices in a vertex sequence to a second group of processing threads according to a second order among the group of machines, the first order being opposite to the second order.

[0061] In some embodiments, the first group of vertex allocation modules includes: a quantity determination module, configured to determine, in a current iteration, the number of vertices and corresponding edges to which a first processing thread in the first group of processing threads has been allocated; a first vertex allocation module, configured to allocate a first vertex in the first group of vertices to the first processing thread in response to the number of vertices and corresponding edges to which the first processing thread has been allocated being less than an upper limit value; and a first determination module, configured to determine, in a first order, a second processing thread following the first processing thread in the first group of processing threads, and to determine a second vertex following the first vertex in the first group of vertices, the second processing thread and the second vertex being used for the next iteration.

[0062] In some embodiments, the first group of vertex allocation modules includes: a first quantity determination module, configured to determine, in a first iteration, the number of vertices and corresponding edges to which a first processing thread in a first group of processing threads has been allocated; a second processing thread determination module, configured to determine, in response to the number of vertices and corresponding edges to which the first processing thread has been allocated being greater than or equal to an upper limit value, a second processing thread in the first processing thread following the first processing thread in a first order; a second quantity determination module, configured to determine, in a second iteration, the number of vertices and corresponding edges to which the second processing thread has been allocated; a second allocation module, configured to allocate the first vertex to the second processing thread in response to the number of vertices and corresponding edges to which the second processing thread has been allocated being less than an upper limit value; and a second determination module, configured to determine, in a group of processing threads, a third processing thread following the second processing thread, and to determine, in the first group of vertices, a second vertex following the first vertex, and the third processing thread and the second vertex are used for a third iteration.

[0063] In some embodiments, the device 500 also includes: a first processing thread determination module, configured to determine the second group of processing threads among the multiple processing threads; a third group of vertex allocation module, configured to allocate the third group of vertices in the vertex sequence to the second group of processing threads according to a first order of a group of machines; a second processing thread determination module, configured to determine the first group of processing threads among the multiple processing threads; and a fourth group of vertex allocation module, configured to allocate the fourth group of vertices in the vertex sequence to the first group of processing threads according to a second order of a group of machines.

[0064] In some embodiments, the upper limit value determination module 504 includes: a divisibility result determination module, configured to perform an integer division operation on the number of multiple processing threads based on the total number; and a first upper limit value determination module, configured to determine the upper limit value of the number of vertices and edges to which the processing threads can be assigned based on the integer division result.

[0065] In some embodiments, the edges of the vertices include outgoing edges and incoming edges, and the total number determination module 502 includes: multiple vertex number determination modules configured to determine the number of multiple vertices in the graph data; edge number determination modules configured to determine the number of both the multiple outgoing edges and the multiple incoming edges corresponding to the multiple vertices; and a first total number determination module configured to determine the total number based on the number of vertices and the sum of the number of both the multiple outgoing edges and the multiple incoming edges.

[0066] Figure 6 A schematic block diagram of an example device 600 is shown that may be used to implement embodiments of the present invention. Figure 1 The detection device 102 in the embodiment can be implemented by using the device 600. Figure 6As shown, device 600 includes a processor 601, which can perform various appropriate actions and processes according to program instructions stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of device 600 can also be stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0067] The various processes and processing described above, such as method 200 and method 300, may be performed by processor 601. For example, in some embodiments, method 200 and method 300 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602. When the computer program is loaded into RAM 603 and executed by processor 601, one or more actions of method 200 and method 300 described above may be performed.

[0068] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0069] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0070] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0071] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0072] Various aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of the boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions. The various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0073] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0074] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0075] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0076] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0078] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for partitioning graph data, comprising: determining a total quantity associated with both vertices and edges in the graph data; Determining an upper limit value of both the number of vertices and the number of edges to which the processing threads can be assigned based on the total number and the number of the plurality of processing threads in the distributed cluster; as well as The plurality of vertices are allocated to the plurality of processing threads according to the number of edges of each of the plurality of vertices in the graph data, and the number of both the vertices allocated to the processing threads and the corresponding edges is less than or equal to the upper limit value.

2. The method according to claim 1, wherein allocating the plurality of vertices to the plurality of processing threads according to the number of edges of each of the plurality of vertices in the graph data comprises: Sort the plurality of vertices according to the number of edges of each of the plurality of vertices to determine a vertex sequence; as well as The plurality of vertices are distributed to the plurality of processing threads according to the vertex sequence.

3. The method according to claim 2, wherein allocating the plurality of vertices to the plurality of processing threads according to the vertex sequence comprises: Determining a first group of processing threads among the plurality of processing threads, the first group of processing threads corresponding to a group of machines in the distributed cluster; assigning a first group of vertices in the vertex sequence to the first group of processing threads in a first order of the group of machines; determining a second group of processing threads from the plurality of processing threads, the second group of processing threads corresponding to the group of machines; A second group of vertices in the sequence of vertices is assigned to the second group of processing threads in a second order of the group of machines, the first order being opposite to the second order.

4. The method of claim 3, wherein assigning a first group of vertices in the vertex sequence to the first group of processing threads in a first order of the group of machines comprises: determining, in a current iteration, the number of both vertices and corresponding edges to which a first processing thread in the first group of processing threads has been assigned; In response to the number of both the vertices and the corresponding edges to which the first processing thread has been assigned being less than the upper limit value, assigning a first vertex in the first group of vertices to the first processing thread; as well as A second processing thread following the first processing thread is determined in the first group of processing threads and a second vertex following the first vertex is determined in the first group of vertices in the first order, and the second processing thread and the second vertex are used for a next iteration.

5. The method of claim 3, wherein assigning a first group of vertices in the vertex sequence to the first group of processing threads in a first order of the group of machines comprises: In a first iteration, determining a number of both vertices and corresponding edges to which a first processing thread in the first set of processing threads has been assigned; In response to the number of both the vertices and the corresponding edges to which the first processing thread has been assigned being greater than or equal to the upper limit value, determining a second processing thread following the first processing thread in the first group of processing threads in the first order; in a second iteration, determining the number of both vertices and corresponding edges to which the second processing thread has been assigned; In response to the number of both the vertices and the corresponding edges to which the second processing thread has been assigned being less than the upper limit value, assigning the first vertex to the second processing thread; as well as A third processing thread following the second processing thread is determined in the first group of processing threads and a second vertex following the first vertex is determined in the first group of vertices in the first order, the third processing thread and the second vertex being used for a third iteration.

6. The method according to claim 3, further comprising: determining the second group of processing threads from among the plurality of processing threads; assigning a third group of vertices in the sequence of vertices to the second group of processing threads in the first order of the group of machines; determining the first group of processing threads among the plurality of processing threads; as well as A fourth group of vertices in the sequence of vertices is assigned to the first group of processing threads in the second order of the group of machines.

7. The method according to claim 1, wherein determining an upper limit value of the number of both vertices and edges to which a processing thread can be assigned based on the total number and the number of multiple processing threads in the distributed cluster comprises: Performing an integer division operation on the number of the plurality of processing threads based on the total number to determine an integer division result; as well as Based on the integer division result, the upper limit value of the number of both vertices and edges to which the processing thread can be assigned is determined.

8. The method of claim 1 , wherein the edges of a vertex include outgoing edges and incoming edges, and determining the total number associated with both the vertex and the edges in the graph data comprises: Determining the number of the plurality of vertices in the graph data; Determine the number of both the plurality of outgoing edges and the plurality of incoming edges corresponding to the plurality of vertices; as well as The total number is determined based on the number of the vertices and the sum of the number of the plurality of outgoing edges and the number of incoming edges.

9. A device for partitioning graph data, comprising: a total quantity determination module configured to determine the total quantity associated with both vertices and edges in the graph data; an upper limit value determination module configured to determine an upper limit value of both the number of vertices and the number of edges to which a processing thread can be assigned based on the total number and the number of multiple processing threads in the distributed cluster; as well as The vertex allocation module is configured to allocate the multiple vertices to the multiple processing threads according to the number of edges of each of the multiple vertices in the graph data, and the number of both the vertices and the corresponding edges allocated to the processing threads is less than or equal to the upper limit value.

10. An electronic device comprising: at least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, the instructions causing the apparatus to perform the method according to any one of claims 1 to 8 when executed by the at least one processor.

11. A computer program product comprising machine executable instructions which, when executed, cause the method according to any one of claims 1 to 8 to be implemented.

Images