A data processing method and related device

Abstract

Embodiments of the present application disclose a data processing method and related equipment, which are used for reducing communication overhead between computing nodes. The method of the embodiments of the present application comprises: partitioning a plurality of edges in a graph distribution by a target partitioning mode; wherein the target partitioning mode comprises a Steiner edge partitioning, a folded edge partitioning or a rectangular edge partitioning.

Description

Technical Field

[0001] This application relates to the field of graph decomposition, and more particularly to a data processing method and related equipment. Background Technology

[0002] In some large-scale data computations, it is necessary to obtain data from multiple nodes for calculation. As the scale of data continues to increase and the number of nodes grows, it becomes impossible to compute data from all nodes using a single computing node. This requires transmitting data from different nodes to the computing node, resulting in significant communication overhead.

[0003] Therefore, by using graph partitioning, the topological relationship formed by multiple nodes can be regarded as a graph distribution. The nodes in the graph distribution and the edges connecting the nodes are divided into multiple partitions. The data of the nodes in the partition can be calculated by the computing nodes corresponding to the partitions, which can greatly reduce the communication overhead.

[0004] Algorithms such as edge partitioning can be used to automatically partition graph distributions in software. However, due to the limitations of edge partitioning algorithms, the upper limit of the number of node replications in a graph distribution is relatively high. Computational nodes in different partitions need to replicate the data of the same node multiple times, resulting in significant communication overhead. Summary of the Invention

[0005] This application provides a data processing method and related equipment for reducing communication overhead between computing nodes.

[0006] A first aspect of this application provides a data processing method, the method comprising:

[0007] The graph distribution can be partitioned using target partitioning methods, including Steiner edge partitioning, folded edge partitioning, or rectangular edge partitioning.

[0008] Existing graph partitioning algorithms have an upper limit on the number of node replications in the graph distribution. Where nump represents the partition fraction of the graph distribution. In this embodiment, the upper limit of the node replication number of the Steiner edge partition is... For folded edge partitioning or rectangular edge partitioning, the maximum number of node replicas is [value missing]. The method in this application reduces the upper limit of node replication, thus reducing the number of computing nodes corresponding to a node. To ensure data synchronization between different computing nodes for the same node, communication between computing nodes is necessary. Reducing the number of computing nodes corresponding to a node reduces the number of times data from the same node is replicated to different computing nodes, thereby reducing communication overhead between computing nodes. Because the upper limit of node replication for partitioning results is low, the number of computing nodes requiring synchronization is reduced, thereby reducing the latency of computing node synchronization.

[0009] In one optional implementation, the target partitioning method is Steiner edge partitioning. The target partitioning method partitions multiple edges in the graph distribution, including: determining the mapping positions of the start and end points of the target edge among the multiple edges according to the node numbering matrix, wherein the order of the node numbering matrix is ​​m, and m is a prime number; determining (m+1) partitioning feature matrices corresponding to the node numbering matrix; and determining the target partition of the target edge according to the (m+1) partitioning feature matrices and the mapping positions of the start and end points of the target edge.

[0010] In this embodiment, the partitions of an edge are determined using (m+1) partition feature matrices, and the partitions of the target edge are determined by mapping the start and end points of the target edge to the (m+1) partition feature matrices. Since the number of possible partitions for the start or end point of the target edge is equal to the number of partition feature matrices (m+1), in the EPS scheme, the number of partitions nump = m(m+1) ≈ (m+1) 2 ,so That is, the starting or ending point of the target edge will only be assigned to In each partition, the maximum number of node replications is 1. Compared to existing graph partitioning algorithms, this reduces the number of partitions a node is assigned to, thereby reducing the upper limit of node replication and lowering the communication overhead between computing nodes.

[0011] In one optional implementation, each of the (m+1) partition feature matrices includes m 2 There are elements, m 2 Each element corresponds to m distinct partitions, and each partition corresponds to m elements; the (m+1) partition feature matrices include: a matrix with identical row elements, a matrix with identical column elements, and a target matrix; wherein, the matrix with identical row elements, the matrix with identical column elements, and the target matrix are all square matrices of order m; in the matrix with identical row elements, elements in the same row correspond to the same partition; in the matrix with identical column elements, elements in the same column correspond to the same partition; in the target matrix, elements in the same row correspond to m distinct partitions; in the target matrix, elements in the same column correspond to m distinct partitions.

[0012] In this embodiment, the node numbering matrix reflects the coordinate information of the start and end points in the partition feature matrix. The relationship between the coordinate information of two points can be used to quickly map to a unique partitioning result, thereby improving partitioning efficiency.

[0013] In one optional implementation, the target partitioning method is folded edge partitioning. The target partitioning method partitions multiple edges in the graph distribution, including: sparsely hashing the adjacency matrix of the graph distribution to obtain a hash adjacency matrix, which is used to reflect the correspondence between the target edge and the target edge's start and end points among the multiple edges of the graph distribution; mapping the hash adjacency matrix to a partitioning matrix, wherein the partitioning matrix is ​​a symmetric matrix; and determining the partition of the target edge based on the partitioning matrix.

[0014] In this embodiment, by using a symmetrical partitioning matrix, the number of partitions to which a node is assigned is reduced, thereby reducing the upper limit of node replication and lowering the communication overhead between computing nodes.

[0015] In one optional implementation, the partitioning matrix has a triangular number characteristic. Before mapping the hash adjacency matrix to the partitioning matrix, the method further includes: determining the lower triangular elements and upper triangular elements of the partitioning matrix, which have a triangular number characteristic, wherein the lower triangular elements and upper triangular elements are mutually symmetrical; determining the diagonal elements A of the partitioning matrix. ii Where 1 ≤ i ≤ the order of the partition matrix; A ii The corresponding partition, and A in the i-th row of the partition matrix. ij The corresponding partitions are the same, where 1≤j≤order of the partition matrix, and i and j are different.

[0016] In this embodiment of the application, since the operation logic of the triangle number feature is simpler, the computer can quickly locate the partition to which the target edge belongs.

[0017] In one alternative implementation, the order of the partitioning matrix is ​​x+1, where x is the number of triangles.

[0018] In one optional implementation, the target partitioning method is rectangular edge partitioning. Partitioning multiple edges in the graph distribution using the target partitioning method includes: determining a partitioning matrix, where the number of rows x and y in the partitioning matrix satisfy the following relationship: y = 2x + 1, or x = 2y + 1; where x and y are both positive integers; determining the mapping positions of the start and end points of the target edge among the multiple edges; determining the starting feature region and the ending feature region in the partitioning matrix based on the mapping positions of the starting and ending points of the target edge; and determining the target partition of the target edge from the intersection region of the starting feature region and the ending feature region.

[0019] In this embodiment, the target partition of the edge is determined in the intersection region of the starting feature region and the ending feature region. Since the number of elements in the intersection region is limited, not greater than 2y (when x = 2y + 1) or not greater than 2x (when y = 2x + 1), the number of partitions to which a node is assigned can be reduced, thereby reducing the upper limit of the node replication number and reducing the communication overhead between computing nodes.

[0020] In an optional implementation, before partitioning multiple edges in the graph distribution using the target partitioning method, the method may further include: determining the target partitioning number and the target partitioning method of the graph distribution; partitioning multiple edges in the graph distribution using the target partitioning method includes: dividing the multiple edges in the graph distribution into several partitions based on the target partitioning number using the target partitioning method.

[0021] In this embodiment, graph partitioning is performed using a determined target partitioning method, reducing the number of partitions in the graph distribution and thus reducing the number of computing nodes. Furthermore, by using the target partitioning method, the number of partitions assigned to a node is reduced, thereby decreasing the upper limit of node replication and lowering the communication overhead between computing nodes.

[0022] In one optional implementation, determining the target partitioning fraction of the graph distribution includes: determining the desired partitioning fraction; determining the optimal partitioning fraction corresponding to the candidate partitioning schemes based on the desired partitioning fraction, wherein the candidate partitioning schemes include at least one of Steiner edge partitioning, rectangular edge partitioning, and folded edge partitioning; and determining the target partitioning scheme and the target partitioning fraction matching the target partitioning scheme based on the optimal partitioning fraction.

[0023] In this embodiment of the application, by using the method corresponding to the expected partitioning fraction, the most suitable partitioning fraction and partitioning method can be quickly determined, thereby improving the efficiency of graph partitioning. Furthermore, based on the appropriate partitioning method, the efficiency of graph computation can be improved, and the communication overhead between computing nodes can be reduced.

[0024] In one alternative implementation, determining the desired segmentation score includes: performing test graph calculations on the graph distribution based on the default segmentation score and the default segmentation method to obtain the desired segmentation score.

[0025] In the embodiments of this application, the test graph distribution can predict the overall effect by the local test results of the graph distribution. It can determine the test parameters (expected split score, GC tuning parameters, speculative execution parameters, executor, etc.) that are more suitable for the graph distribution at a lower cost, thereby reducing computing power consumption and latency.

[0026] In one alternative implementation, after partitioning multiple edges in the graph distribution using a target partitioning method, the method further includes sending the partitioning results to the computing nodes.

[0027] In this embodiment, the partitioning result is sent to the compute nodes, which can then perform calculations on the data of the nodes within the corresponding partition based on the partitioning result. Because the upper limit of the number of node replicas in the partitioning result is low, the communication overhead caused by synchronizing node data between compute nodes is reduced, thereby improving the speed of graph computation and reducing the memory usage of graph computation. The low upper limit of the number of node replicas in the partitioning result also reduces the number of compute nodes requiring synchronization, thus reducing the latency of compute node synchronization.

[0028] A second aspect of this application provides a data processing apparatus, including a processor and a memory; the processor is coupled to the memory;

[0029] Memory, used to store programs;

[0030] A processor for executing a program in memory, causing the processor to perform the data processing method described in the first aspect.

[0031] A third aspect of this application provides a computer-readable storage medium for storing a computer program that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0032] A fourth aspect of this application provides a computer program product, which includes: computer program code;

[0033] When the computer program code is run, it implements the method described in the first aspect.

[0034] A fifth aspect of this application provides a chip including at least one processor and an interface;

[0035] An interface is used to provide program instructions or data to at least one processor;

[0036] At least one processor is used to execute program instructions to implement the method described in the first aspect.

[0037] A fifth aspect of this application provides a server including the chip described in the fourth aspect. Attached Figure Description

[0038] Figure 1 A schematic diagram of graph segmentation;

[0039] Figure 2 This is a schematic diagram of a point segmentation method;

[0040] Figure 3 This is another schematic diagram of a point segmentation method;

[0041] Figure 4A schematic diagram of the hardware architecture of the data processing method provided in the embodiments of this application;

[0042] Figure 5a A software architecture diagram of the data processing method provided in the embodiments of this application;

[0043] Figure 5b Another software architecture diagram of the data processing method provided in the embodiments of this application;

[0044] Figure 5c Another software architecture diagram of the data processing method provided in the embodiments of this application;

[0045] Figure 6 A flowchart illustrating the data processing method provided in this application embodiment;

[0046] Figure 7 A schematic diagram of the Steiner edge partitioning process provided in an embodiment of this application;

[0047] Figure 8a A schematic diagram of a Steiner edge partition provided in an embodiment of this application;

[0048] Figure 8b Another schematic diagram of the Steiner edge partition provided in the embodiments of this application;

[0049] Figure 8c Another schematic diagram of the Steiner edge partition provided in the embodiments of this application;

[0050] Figure 9 A schematic diagram of the folded edge partitioning process provided in an embodiment of this application;

[0051] Figure 10a A schematic diagram of the folded edge partition provided in an embodiment of this application;

[0052] Figure 10b Another schematic diagram of the folded edge partition provided in the embodiments of this application;

[0053] Figure 11 A schematic diagram of the rectangular side partitioning process provided in an embodiment of this application;

[0054] Figure 12a A schematic diagram of a rectangular side partition provided in an embodiment of this application;

[0055] Figure 12b A schematic diagram of a rectangular side partition provided in an embodiment of this application;

[0056] Figure 13 Another schematic diagram of the data processing method provided in the embodiments of this application;

[0057] Figure 14 Another schematic diagram of the data processing method provided in the embodiments of this application;

[0058] Figure 15 A schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0059] Figure 16 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0060] This application provides a data processing method and related equipment for reducing communication overhead between computing nodes.

[0061] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems. The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so as to include, but not limited to, a series of unit processes, methods, systems, products, or devices, and may also include processes, methods, products, or devices not explicitly listed.

[0062] First, let's describe the technical terms that appear in the embodiments of this application:

[0063] Node: The provider of data; there are multiple nodes.

[0064] Graphs are used to represent relationships between objects, using vertices and edges to describe them: nodes represent objects, and edges represent relationships between objects. The degree of a node is the number of edges connected to that node.

[0065] Subgraph (Partition): A subgraph is formed by one or more nodes in a graph distribution and the edges connecting those nodes. In the embodiments of this application, the subgraph is also referred to as a partition.

[0066] Graph partitioning patterns: The basic rules for dividing a graph distribution into multiple subgraphs (partitions), including vertex-cut, edge-cut, and hybrid-cut. The mainstream graph partitioning pattern is vertex-cut.

[0067] Load: refers to the balance of the number of nodes distributed to each subgraph in graph partitioning. Uneven load will lead to an uneven computational burden on each computing node, resulting in low graph computation efficiency.

[0068] Computation node: One computation node corresponds to one subgraph (partition). The computation node is used to perform calculations on the data of the nodes and edges in the subgraph.

[0069] Communication cost: refers to the communication overhead incurred by synchronizing nodes or edges allocated to different computing nodes, including costs such as latency and bandwidth.

[0070] Locality: The degree of association between nodes within a subgraph (partition). The higher the proportion of nodes connected by edges within a subgraph (partition), the stronger the locality. Higher locality means that the graph partitioning process is more affected by the association between nodes, leading to uneven partition load and uneven computational burden on each computing node, thus affecting graph computation performance.

[0071] I. Application scenarios of the embodiments of this application.

[0072] The data processing method in this application provides several vertex partitioning methods (edge ​​partitioning methods) for graph distributions, used to distribute edges in a graph distribution into multiple partitions. First, a brief introduction to the graph distribution partitioning methods is given:

[0073] 1. Methods for dividing graph distributions.

[0074] Large-scale data often exhibits relational characteristics, thus requiring a graph distribution. Nodes in a graph distribution provide the data, while edges represent the connections between nodes. Dividing the graph distribution yields multiple subgraphs (partitions), with nodes in each subgraph used to perform data operations on the nodes within that subgraph.

[0075] For example, in a weather forecasting scenario, nodes can be meteorological monitoring points used to acquire meteorological data such as temperature, humidity, and air pressure. The map distribution, including multiple meteorological monitoring points, can be segmented, assigning geographically proximate nodes to the same partition. The data processing center of this partition is called a computing node. Each node sends its detected data to the computing node in that partition, which then performs calculations on this data to obtain a weather forecast for that geographical location.

[0076] It is worth noting that in the embodiments of this application, compute nodes and nodes are different concepts. A node is an object that provides data within a partition, while a compute node is used to compute the data of each node within the partition.

[0077] It is worth noting that the above weather forecast scenario is only an example of the application scenarios of graph distribution. In addition to weather forecasting, graph distribution can be applied to many other scenarios, such as social network analysis, web page ranking, community discovery, design of very large-scale integrated circuits, transportation route planning, and power network simulation, etc., which are not limited here.

[0078] The example above uses the geographical location of the nodes as the basis for partitioning the graph distribution. In fact, in addition to geographical location, partitioning can be based on many other criteria, such as the correlation between nodes (whether they are connected by edges), etc., which are not limited here.

[0079] It is worth noting that, in the embodiments of this application, segmenting the graph distribution is the same concept as partitioning the graph distribution.

[0080] Based on the aforementioned partitioning requirements for graph distributions, the graph distribution can be partitioned using different graph partitioning modes to obtain multiple partitions, such as... Figure 1 As shown, graph segmentation modes include vertex segmentation, edge segmentation, and hybrid segmentation. Since this application embodiment provides a vertex segmentation method, vertex segmentation will be the focus of this description.

[0081] In point partitioning, nodes are partitioned. A node may be assigned to different partitions, but an edge will not be assigned to different partitions.

[0082] If a node is assigned to different partitions, since each partition's compute nodes need to perform calculations based on that node's data, multiple compute nodes must store that node's data, consuming significant storage resources. Furthermore, since data on a single node is used by multiple compute nodes, these nodes need to ensure synchronized usage of that data, leading to data consistency issues between different compute nodes. Different compute nodes need to interact to synchronize the same node's data, incurring communication overhead. If the node's data is updated frequently, or if a node is assigned to a large number of partitions, it will further burden the communication between compute nodes.

[0083] To minimize the communication burden between nodes, the number of partitions to which nodes are assigned should be reduced during graph partitioning. In this embodiment, the number of partitions to which a node is assigned is called the replication factor (RF). That is, by reducing the maximum replication factor (RF)... max This can reduce the communication burden of different computing nodes synchronizing data on the same node.

[0084] Most node partitioning algorithms do not have a theoretical upper limit on the number of node replicas, making it impossible to guarantee the actual range of node replicas and thus control the communication overhead between nodes. Furthermore, while some node partitioning algorithms have a theoretical upper limit on the number of node replicas, where a node's replica count is less than or equal to this upper limit, current algorithms have relatively large upper limits on the number of node replicas, resulting in significant communication overhead between nodes.

[0085] Next, we will use two-dimensional edge partitioning (EP2D) as an example to illustrate the upper limit of the number of node replications. EP2D is a typical node partitioning method.

[0086] In EP2D, the first step is to obtain the edge file of the graph distribution. The edge file represents the connection relationships between the nodes within the graph distribution. (See also...) Figure 2 For example, to facilitate data processing, different nodes can be represented by node IDs. The edge file includes information about each edge; edges are directed, pointing from the starting point to the ending point, therefore each edge is represented by the node ID of its starting point and the node ID of its ending point. For example, Figure 2 An edge pointing from node 1 to node 2 is represented as (001, 002) in the edge file. The edge file can be a file from the Hadoop Distributed File System (HDFS). Besides HDFS files, edge files can also be other types of files, such as Lustre files, FastDFS files, etc., without limitation here.

[0087] Once the edge file is determined, the edges can be partitioned according to the starting point ID, a default block ID can be set, a partition ID can be configured for each edge, and the information representing that edge in the edge file can be filled into the computation node corresponding to the partition ID. Through the above steps, the default blocks for each edge in the graph distribution can be obtained.

[0088] A default block corresponds to a partition. Once the default block for each edge is determined, the edge data, as well as the node data of the edge's start and end points, can be copied to the computing node corresponding to the partition.

[0089] Typically, edge files are sorted by either the start ID or the end ID of the edges. Since the step of configuring a partition ID for each edge is based on the order of the start IDs of the edges in the edge file, if the locality of the edge files is weak, the default chunking result may place many edges without common nodes into the same partition. For example, Figure 2 There are 3 edges connected to node 4, but in the default partitioning, these 3 edges are divided into 3 partitions. The information of node 4 needs to be copied to the nodes of these 3 partitions, resulting in excessive data copying of nodes, high resource consumption and communication costs, and reduced graph computing performance.

[0090] Therefore, after the default partitioning, a common improvement method is to perform repartitioning. In the EP2D scheme, the adjacency matrix of directed edges is transformed into a hash adjacency matrix, and the partition of the target edge (an element with 1 in the hash adjacency matrix) is determined based on the start ID and end ID of the hash matrix.

[0091] The partition matrix of EP2D is of order . A square matrix can guarantee that the maximum number of node replicas for all nodes is [value missing]. Where nump is the number of partitions. For example Figure 3 Edges originating from or ending at node 001 include (001, 002), (001, 003), (001, 004), and (003, 001), which are assigned to the four partitions with partition IDs 0, 1, and 2, respectively. Therefore, the node replication number of node 001 is 3, which is less than its upper bound.

[0092] This application provides a data processing method that reduces the upper bound of node replication number through a novel edge partitioning method. The architecture for implementing the data processing method provided in this application is described below.

[0093] 2. Implementation architecture of the embodiments of this application.

[0094] The hardware architecture of this application embodiment is as follows: Figure 4 As shown, the architecture includes: client, resource manager (RM), compute nodes, and distributed file system.

[0095] Each compute node includes at least one executor, which performs computations on the node data within the partition corresponding to that compute node. A graph distribution may also include an application master (AM) on one compute node, which is used for scheduling and controlling the compute nodes.

[0096] A distributed file system is a file management system used to manage data nodes. A distributed file system consists of multiple data nodes, which store data for partitions, such as attribute information that identifies nodes within a partition. Besides node attribute information, data nodes can also store other data, such as attribute information used in calculations (e.g., region, rainfall), etc., which are not limited here.

[0097] Optionally, the data node can be a hard disk. In addition to hard disks, the data node can also be other forms of storage media, such as flash memory, floppy disks, etc., which are not limited here.

[0098] For example, multiple data nodes in a distributed file system can include nodes corresponding to multiple partitions in a graph distribution. The distributed file system can store information related to the graph distribution, such as information about the edges or nodes included in each partition.

[0099] For example, the distributed file system can be HDFS. In addition to HDFS, the distributed file system can also be other systems, such as Lustre, FastDFS, etc., without limitation here.

[0100] The client can transfer applications (APPs) to the resource manager RM, which are used to perform calculations on the distributed file system.

[0101] Resource Manager (RM) can launch the Application Master (AM) on a specific compute node based on the application.

[0102] After the Application Manager (AM) starts, it can request the computing resources required for the application from the Resource Manager (RM). Based on the request, the Resource Manager allocates multiple compute nodes as executors for the application.

[0103] After allocating executors to multiple compute nodes, the Application Manager (AM) can schedule and allocate tasks to these executors. This task scheduling and allocation includes partitioning the graph distribution of the distributed file system, matching the partitioned partitions with corresponding compute node executors, and issuing computation tasks for the corresponding partitions to the appropriate compute node executors.

[0104] Each compute node's executor undertakes the computation task for the corresponding partition in the graph distribution according to the task assigned by AM, and stores the computation results on the corresponding data node in the distributed file system.

[0105] The hardware architecture of the embodiments of this application has been described above. The software architecture of the embodiments of this application will be described below.

[0106] like Figure 5a As shown, the software architecture includes an algorithm flow, which includes processes such as graph construction, repartition graph construction, and graph computation.

[0107] Graph construction involves reading the graph distribution from the distributed file system (disk read) and performing default partitioning on the graph distribution. Re-splitting graph construction re-splitting the edge data in the graph distribution optimizes the storage structure of the graph distribution across different nodes, and then rebuilds the computation graph. Graph computation executes graph algorithms on the re-splitting computation graph (disk write).

[0108] The Application Manager (AM) serves as the control center, controlling various processes such as graph repartitioning, graph repartitioning construction, and graph computation.

[0109] The graph repartitioning construction process in the above algorithm constructs a computational model of the graph distribution by partitioning the graph distribution. To improve the computational efficiency of this graph distribution, embodiments of this application propose... Figure 5b The software architecture shown is used to implement Figure 5a The process of graph repartitioning construction in algorithm flow.

[0110] The software architecture includes: a parameter selection module, a graph repartitioning construction module, and a graph decomposition module.

[0111] The parameter selection module selects appropriate graph partitioning parameters based on the default parameters of graph partitioning, primarily the partitioning expectation. The graph repartitioning construction module selects an appropriate partitioning method and the corresponding number of partitions based on the graph partitioning parameters confirmed by the parameter selection module. Optionally, a suitable partitioning method can be selected from the edge partitioning fold (EPF), edge partition rectangle (EPR), or edge partition Steiner (EPS) methods proposed in this embodiment. The graph decomposition module partitions the graph distribution based on the partitioning method and the number of partitions determined by the graph repartitioning construction module.

[0112] The parameter selection module can be as follows: Figure 5c As shown, through single-round graph computation and graph prefetching optimization, the system performs partition expectation adaptive configuration, executor parameter selection, garbage collection (GC) tuning parameter selection, speculative execution parameter selection, etc., and finally uses graph persistence.

[0113] Based on the above architecture, this application proposes a data processing method and three graph decomposition algorithms: Steiner edge partitioning (EPS), folded edge partitioning (EPF), and rectangular edge partitioning (EPR). These three algorithms and data processing methods reduce the maximum node replication number (RF) in the graph distribution. max This reduces the communication overhead between nodes in synchronizing data at the same node, as well as the storage and computational overhead of different nodes. The data processing method provided in the embodiments of this application will be described below.

[0114] II. Data processing methods provided in the embodiments of this application.

[0115] Please see Figure 6The data processing method provided in this application includes:

[0116] 601. Partition multiple edges in a graph distribution using a target partitioning method.

[0117] The target partitioning methods include Steiner edge partitioning, folded edge partitioning, or rectangular edge partitioning.

[0118] In this embodiment, the upper limit of node replication for folded edge partitioning, rectangular edge partitioning, or Steiner edge partitioning is less than the upper limit of node replication for EP2D.

[0119] It is worth noting that, in the embodiments of this application, the point partitioning for the graph distribution is the same as the edge partitioning for the graph distribution; therefore, the point partitioning method for the graph distribution is the same as the edge partitioning method for the graph distribution, and this is not limited here.

[0120] The following describes these three graph decomposition methods:

[0121] 1. Steiner side partition EPS.

[0122] Please see Figure 7 The Steiner edge partitioning scheme includes the following steps:

[0123] 701. Determine the mapping positions of the start and end points of the target edge based on the node numbering matrix.

[0124] In the EPS scheme, the segmentation is m(m+1), where m is a prime number. Corresponding to m, a node numbering matrix of order m can be determined. For example... Figure 8a In the node numbering matrix of example A in the middle figure, m is included. 2 Each element.

[0125] Every point in the graph distribution can be uniquely mapped to an element in the node numbering matrix. Each element in the node numbering matrix represents an ordered set of two data points, for example, (a, b), indicating the set of data points a and b, where a comes before b. The element in the node numbering matrix can also be its coordinate (i, j) in the node label matrix. Besides coordinates, elements in the node numbering matrix can also have other values, such as (j, i), etc., which are not limited here.

[0126] Optionally, the range of values ​​for the element (i, j) in the m×m node numbering matrix is: 0 ≤ i or j ≤ (m-1). In addition to 0 ≤ i or j ≤ (m-1), the element (i, j) can also be in other ranges, such as 1 ≤ i or j ≤ m, etc., which are not limited here.

[0127] In this embodiment, 'a' can be used as the identifier of the starting point of the target edge, and 'b' can be used as the identifier of the ending point of the target edge. For example, if multiple edges in a graph distribution are represented by a sorting method, then 'a' and 'b' are the sorting of the starting and ending points of the target edge, respectively. For instance, this sorting can be an ID, such as 4, 2, etc.

[0128] For example, one method of mapping from points in a graph distribution to element positions in a node numbering matrix is: pair the point indices with m... 2 Take the remainder, then divide the remainder by m to get the quotient and remainder. Use the quotient as the row coordinate i of the point; use the remainder as the column coordinate j of the point. This gives the element with coordinates (i, j) in the node numbering matrix corresponding to the point. That is, the correspondence from node number a to the element position (i, j) in the node numbering matrix is: i = floor{[a%(m...]} 2 )] / m},j=[a%(m 2 )]%m. Here, floor represents rounding down. Since computers automatically round down, this formula reflects the essence of the correspondence, but it does not represent all aspects of the relationship.

[0129] For example, a 3×3 node numbering matrix can be represented as follows: Figure 8a As shown, assuming the index of a point in the graph distribution is 3, then substituting a = 3 into i = floor{[a%(m 2 )] / m} and j=[a%(m 2 By determining the correspondence between )]%m, we can obtain the mapping position of this point in the node numbering matrix as (1,0).

[0130] It is worth noting that the above i = floor{[a%(m 2 )] / m},j=[a%(m 2 The mapping method )]%m is merely an example of how to map points in a graph distribution to elements in the node numbering matrix. It is not a limitation and other mapping methods can be used, as long as they can uniquely map any point in the graph distribution to the node numbering matrix. For example, it could also be i = [a%(m)]. 2 )]%m,j=floor{[a%(m 2 The mapping relationship of )] / m}, or the mapping relationship obtained by transforming the above mapping relationship, etc. (e.g., for Figure 8a The elements of the node numbering matrix in the matrix can be swapped (e.g., swapping (0, 1) with (2, 1), etc.), but this is not limited here.

[0131] The above illustrates the mapping from a point in a graph distribution to the element positions in the node numbering matrix. To determine the partition of a target edge in a graph distribution, we can determine the mapping of the start and end points of the target edge to the element positions in the node numbering matrix.

[0132] In the real-time example of this application, the target edge represents any edge in the graph distribution, and there is no limitation here.

[0133] For example, if the starting index of an edge is 3 and the ending index is 10, then in Figure 8a In the node numbering matrix shown, the starting point is mapped to position (1, 0), and the ending point is mapped to position (0, 1). The method for determining this has been explained previously and will not be repeated here.

[0134] 702. Determine the (m+1) partition feature matrices corresponding to the node numbering matrix.

[0135] In the EPS scheme, corresponding to the node numbering matrix, (m+1) partition feature matrices can be determined.

[0136] In this embodiment, the elements in the partition feature matrix are also called partition identification (PID), representing the partitioning of edges. Different edge partitions can be represented by different identifiers in the partition feature matrix; for example, 0 and 1 in the partition matrix represent different edge partitions.

[0137] Each of the (m+1) partition feature matrices contains m different PIDs, and the number of each PID is m.

[0138] (m+1) partition feature matrices, including matrices with identical row elements, matrices with identical column elements, and the target matrix.

[0139] In matrices with identical row elements, elements in the same row have the same PID, such as... Figure 8b The first matrix in the diagram is shown below;

[0140] In matrices with the same column elements, elements in the same column have the same PID, such as... Figure 8b The second matrix in the diagram is shown below;

[0141] In the target matrix, m elements in the same row have m different PIDs, and m elements in the same column have m different PIDs, such as... Figure 8b The third and fourth matrices are shown in the diagram.

[0142] In this context, any two partition feature matrices among the (m+1) partition feature matrices correspond to different PIDs. Therefore, the (m+1) partition feature matrices correspond to m(m+1) PIDs. Thus, the EPS segmentation number nump = m(m+1).

[0143] In one optional implementation, the target matrix includes a principal direction matrix and a secondary direction matrix. In the principal direction matrix, elements on the same straight line have the same PID (Process Indicator). In the secondary direction matrix, elements on the same straight line have the same PID.

[0144] The primary direction extends from the upper left to the lower right, while the secondary direction extends from the upper right to the lower left.

[0145] For example Figure 8c In the matrix at the top left, the direction of the main diagonal is the main direction, and elements on the same straight line in this direction have the same PID.

[0146] For example, such as Figure 8b As shown, if m = 3, then it can be determined as follows: Figure 8b The four partition feature matrices are shown. It is worth noting that... Figure 8b This is an example of a partition feature matrix. The order of the PIDs in the partition matrix does not have to be in any particular order. Figure 8b The arrangement shown can be varied. For example, in the first matrix, the PID of the elements in the first row can be set to 2, and the PID of the elements in the third row can be set to 0, etc. There are no restrictions here.

[0147] 703. Determine the target partition of the target edge based on the m+1 partition feature matrices and the mapping positions of the start and end points of the target edge.

[0148] In step 701, once the mapping positions of the target edge's start and end points in the node numbering matrix are determined, the mapping positions of the target edge's start and end points in each of the (m+1) partition matrices can be determined. Then, among the (m+1) partition matrices, the matrices whose elements have the same value for the starting and end point mapping positions are selected, and these matrices are used as the partitions for the target edge.

[0149] For example, if the starting point number of the target edge is 3 and the ending point number is 10, then in step 701 it is explained that the mapping position of the starting point of the target edge in the node numbering matrix is ​​(1,0), and the mapping position of the ending point of the target edge in the node numbering matrix is ​​(0,1). Figure 8b As shown, in the four partition matrices, lighter-colored squares represent the mapping position of the starting point in the partition matrix, and darker-colored squares represent the mapping position of the ending point in the partition matrix.

[0150] like Figure 8b As shown, in the first three partition matrices, the elements corresponding to the starting point and the ending point have different values. However, in the fourth partition matrix, the elements corresponding to the starting point and the ending point both have a value of 10. Therefore, the partition PID of the target edge is determined to be 10.

[0151] In EPS, since a point maps to the same mapping position in (m+1) partition feature matrices, and the PID of the mapping position in the partition feature matrix determines the partition of that point, a point can be assigned to at most (m+1) partitions. In other words, in EPS, the upper limit of node replication is RF. max = (m+1).

[0152] As explained in step 702, the segmentation of EPS is nump = m(m+1).

[0153] Therefore, in EPS, the upper limit of node replication number RF max The relationship between the segmentation nump and the segmentation is as follows:

[0154] nump = m(m+1)

[0155] RF max =m+1

[0156]

[0157]

[0158] Compared to existing graph partitioning algorithms, EPS reduces the number of partitions a node is assigned to, thereby reducing the upper limit of node replication and lowering the communication overhead between computing nodes.

[0159] 2. Folded edge partition EPF.

[0160] Please see Figure 9 The folded edge partitioning scheme includes the following steps:

[0161] 901. Perform sparse hashing on the adjacency matrix of the graph distribution to obtain the hashed adjacency matrix.

[0162] The adjacency matrix of a graph distribution is used to represent the correspondence between edges and nodes in the graph distribution. However, the distribution of the matrix may be uneven, resulting in higher density in some areas and lower density in others. For example... Figure 10a In the adjacency matrix, the density of the lower right part of the matrix is ​​significantly higher than that of the upper left part.

[0163] To make the matrix distribution more uniform, a hash operation is performed on the adjacency matrix. This results in a sparse hash matrix with a more uniform distribution than the original adjacency matrix, reducing the locality of the adjacency matrix. The more uniform the matrix distribution and the lower the locality, the more evenly the graph will be partitioned. This hash operation is also called sparse hashing.

[0164] For example, such as Figure 10bAs shown, if the adjacency matrix is ​​concentrated on nodes 1 and 2, due to high locality, edges related to nodes 1 and 2 are easily assigned to the same partition during graph partitioning. Consequently, the number of edges in partitions excluding nodes 1 and 2 will be significantly less than in partitions including nodes 1 and 2. This results in a heavy computational burden on the nodes in partitions including nodes 1 and 2, while the nodes in partitions excluding nodes 1 and 2 have a light computational burden. This unbalanced load on computational nodes leads to wasted performance on some nodes (too few tasks) and excessive workload on others (too many tasks).

[0165] This application embodiment reduces the impact of graph distribution locality by using sparse hashing, improves the load balance among different computing nodes, and can make fuller and more reasonable use of the computing resources of multiple computing nodes.

[0166] 902. Map the hash adjacency matrix to the partition matrix.

[0167] In this embodiment, the elements in the partition matrix are also called partition identification (PID), representing the partitioning of edges. Different edge partitions can be represented by different identifiers in the partition matrix; for example, 0 and 1 in the partition matrix represent different edge partitions.

[0168] In EPF, the partition matrix corresponding to the segmentation fraction can be determined as positively symmetric.

[0169] Optionally, the partition matrix may include lower triangular elements and upper triangular elements with triangular number characteristics, and the lower triangular elements and upper triangular elements are symmetrical to each other. In the embodiments of this application, the partition fraction is also called the number of partitions (nump), which is not limited here.

[0170] In EPF, the number of partitions (nump) and the order of the partition matrix are matched using the eigenvalue x. Optionally, the number of partitions can conform to the following correspondence: nump = x(x+1) / 2. The order of the partition matrix is ​​equal to x+1. In the embodiments of this application, the number of partitions may not exactly conform to the correspondence of nump = x(x+1) / 2, as long as the value of the number of partitions nump is approximately x(x+1) / 2, which is not limited here.

[0171] For example, the PID values ​​in the partitioning matrix range from 0 to (x 2 +x-2) / 2, and the x(x+1) / 2 elements within the triangular region below the main diagonal of the partition matrix, taking values ​​{0,1,……,[(x+1) / 2}. 2The region is defined as [x-2) / 2]}, and this triangular region exhibits the characteristic of a triangular number. Elements within the triangular region above the main diagonal are symmetrical to elements within the triangular region below the main diagonal.

[0172] It is worth noting that the triangular region below the main diagonal of the partitioned matrix may not necessarily have the characteristic of a number of triangles, as long as the x(x+1) / 2 elements in this region take values ​​{0, 1, ..., [(x...]}. 2 The expression +x-2) / 2]} is sufficient, and it is necessary to ensure that the elements in the triangle region above the main diagonal are symmetrical with the elements in the triangle region below the main diagonal.

[0173] It is worth noting that the PIDs in the partition matrix do not necessarily have to start numbering from 0, as long as the partition matrix includes x(x+1) / 2 different PIDs. There is no restriction here.

[0174] For example, such as Figure 10a As shown, if the eigenvalue x is 5, then the number of partitions is 15, and the partition matrix is ​​determined to be a 6×6 square matrix. The elements of the triangular region below the main diagonal of the partition matrix are determined to have PIDs ranging from 0 to 14, exhibiting a triangular number characteristic. The elements of the triangular region above the main diagonal are symmetrical to the elements of the triangular region below the main diagonal.

[0175] Given the partition matrix, the edges in the graph distribution can be mapped to the partition matrix according to the following mapping relationship:

[0176] a%(x+1) = the row number of the edge's PID in the partition matrix;

[0177] b%(x+1) = the column number of the edge's PID in the partition matrix;

[0178] Here, %(x+1) represents the result of taking the remainder when (x+1) is used, for example, x%(x+1) = x. Here, x is a positive integer.

[0179] For example, in Figure 10a In the partition matrix shown, (x+1) = 6. If the specific information of an edge (a, b) in the graph distribution is (5, 6), this edge is mapped to the element in the 5th row and 0th column of the partition matrix.

[0180] It is worth noting that the partition matrix can be arranged starting from row 0 and column 0 (from A...). 00 You can start arranging the layout from row 1 to column 1 (from row A). 11 (Start arranging), no restrictions are made here. If the partition matrix starts from row 0 and column 0, since the order of the partition matrix is ​​(x+1), the elements in the partition matrix are A. ij, 0≤i≤x, 0≤j≤x. If the partition matrix starts from row 1 and column 1, since the order of the partition matrix is ​​(x+1), the elements in the partition matrix are A. ij , 1≤i≤x+1, 0≤j≤x+1.

[0181] For elements on the main diagonal, it is sufficient to ensure that the PID of that element is the same as the PID of any element in the same row as it. That is, for the diagonal element A of the partition matrix... ii The PID, and the A in the i-th row of the partition matrix. ij The PIDs are the same, where 1≤j≤(x+1), 1≤i≤(x+1), and i and j are not the same.

[0182] Diagonal element A ii In the i-th row, the partition matrix divided by A ii Besides the x elements, there are x different PIDs, A ii The probability that the PID value is any of these x different PIDs is the same.

[0183] by Figure 10a A in 11 For example, if the PID in the second row of the partition matrix is ​​any one of 0, 2, 4, 7, 11, then A 11 The probabilities of taking values ​​of 0, 2, 4, 7, and 11 are equal, all being 20%. In other words, if an edge in the graph distribution is assigned to partition matrix A... 11 For the partition corresponding to the position, the probability that the PID of this edge is any one of 0, 2, 4, 7, 11 is 20%.

[0184] Optionally, if the segmentation does not satisfy the correspondence of nump = x(x+1) / 2, the method of folded edge partitioning EPF can also be implemented.

[0185] If the number of partitions is greater than x(x+1) / 2 in the partition number correspondence, a new column or row can be added to the partition matrix to place the overflowing blocks (partitions greater than the number indicated by the correspondence). In this case, the partition matrix will be a (x+1)×(x+2) or (x+2)×(x+1) matrix, and will no longer be a square matrix.

[0186] 903. Determine the partition of the target edge based on the partition matrix.

[0187] In step 902, the hash adjacency matrix is ​​mapped to the partition matrix. Therefore, based on this mapping relationship, it can be determined that the target edge is mapped to a corresponding element in the partition matrix, and the PID of that element is used as the partition of that edge.

[0188] For example, in step 902, the edge (5,6) in the graph distribution is mapped to the partition matrix, which is the element in the 5th row and 0th column of the partition matrix. Since the PID of this element is 10, it is determined that the edge (5,6) is assigned to the partition with PID 10.

[0189] Element A in the partition matrix ij The corresponding edge starts at point i and ends at point j. Since the partition matrix is ​​symmetric, all edges related to the starting point i can be assigned to a partition number of x+1.

[0190] Taking edge (5,6) in the graph distribution as an example, as explained earlier, this edge is mapped to the position in the 5th row and 0th column of the partition matrix, that is, the element A with PID 10 in the lower left corner. 50 In the partition matrix, all edges associated with the endpoint 0, including the elements in column 0 and row 0 (i.e., the portion indicated by the cross-shaped arrows in the diagram), are considered. Since the feature matrix is ​​positively symmetric, the elements in column 0 and row 0 will only correspond to the five partitions with PIDs of 0, 1, 3, 6, and 10, with a partition count of x. A 00 The PID value is also taken from these five partitions, so the edges related to point 0 can only be assigned to these five partitions.

[0191] In other words, the upper limit of the number of node replicas at point 0 is RF. max =x. Similarly, the upper limit of the number of node replicas for all points in the partition matrix is ​​RF. max =x.

[0192] The nodes in the partition matrix are uniquely mapped from the hashed adjacency matrix, reflecting the relationships between edges and nodes in the adjacency matrix. Therefore, the upper limit of the node replication number RF in the partition matrix is... max The upper limit of the number of node replications in a graph distribution is RF. max .

[0193] In other words, in EPF, the upper limit of node replication is RF. max =x

[0194] In EPF, the splitting fraction nump = x(x+1) / 2, and the upper limit of node replication RF. max Given x, the upper limit of node replication RF is... max The relationship between the segmentation nump and the segmentation is as follows:

[0195] nump = x(x+1) / 2

[0196] RF max =x

[0197]

[0198]

[0199]

[0200]

[0201]

[0202] In this embodiment, by using a symmetrical partitioning matrix, the number of partitions to which a node is assigned is reduced, thereby reducing the upper limit of node replication and lowering the communication overhead between computing nodes.

[0203] 2. Rectangular side partitioning EPR.

[0204] Please see Figure 11 The rectangular side partitioning scheme includes the following steps:

[0205] 1101. Determine the partition matrix.

[0206] Determine the partition matrix with x rows and y columns, where x and y satisfy the following relationship:

[0207] y = 2x + 1, or x = 2y + 1; where x and y are both positive integers.

[0208] For example, such as Figure 12a As shown, a 9x4 partition matrix can be determined. That is, y = 4, x = 2y + 1 = 9.

[0209] like Figure 12a As shown in Figure A, the partitioning matrix includes 4 × 9 = 36 elements, that is, the partitioning matrix from A... 00 To A 83 The problem involves identifying 36 elements. The goal is to determine the unique PID (Process ID) for each of these 36 elements. For example... Figure 12a As shown in Figure B, the PIDs numbered from 0 to 35 are determined for the 36 elements in the partition matrix.

[0210] It is worth noting that, Figure 12a Figure B in the diagram is merely an example of an EPR partition matrix. The PIDs corresponding to the elements in the partition matrix may not be assigned according to the partition matrix. Figure 12a The order shown in Figure B can also be arranged as follows, for example, as shown in Figure B. Figure 12a As shown in Figure C, the PID values ​​of the elements in the partition matrix are all different; and the PID values ​​are from 0 to x(2x+1)-1 when y = 2x+1 and from 0 to y(2y+1)-1 when x = 2y+1. No restrictions are imposed here.

[0211] It is worth noting that the PIDs in the partitioning matrix do not necessarily have to be numbered starting from 0, as long as the partitioning matrix includes x(2x+1) different PIDs. There is no restriction here.

[0212] 1102. In the partition matrix, determine the starting and ending feature regions of the target edge.

[0213] By taking the target edge in the distribution graph, we can determine the starting and ending index of the target edge.

[0214] If the number of rows in the partition matrix is ​​x = 2y + 1, then the PID values ​​corresponding to the y(2y + 1) elements in the partition matrix are any values ​​from 0 to y(2y + 1) - 1.

[0215] By taking the remainder of the starting point index with respect to y(2y+1), the starting point can be mapped to a certain element in the partition matrix.

[0216] For example, with Figure 12a Taking the partition matrix PID shown in Figure B as an example, if the starting point index of the target edge is 8, then 8%36 = 8, which corresponds to the element in the 2nd row and 0th column of the partition matrix, i.e., element A. 20 In the embodiments of this application, the element mapped to the starting point number is called the starting point feature element.

[0217] For example, with Figure 12a Taking the partition matrix PID shown in Figure C as an example, if the starting index of the target edge is 8, then 8%36 = 8, which corresponds to the element in the 0th row and 0th column of the partition matrix, i.e., element A. 00 . ElementA 00 This refers to the starting point feature element in Figure C that corresponds to the starting point number 8.

[0218] Alternatively, in addition to the modulo method mentioned above, other methods can be used to map the starting point to the starting point feature element in the partition matrix, such as hashing. As long as the mapping method is uniform, it is acceptable and is not limited here.

[0219] Once the starting point feature element corresponding to the starting point index is determined in the partition matrix, the starting point feature region can be determined within the partition matrix. Figure 12a Taking Figure B as an example, if the starting point number is 8, then the starting point feature element A 20 , with the starting feature element A 20 Element A in the same row 21 A 22 and A 23 All belong to the starting point feature region, and are related to the starting point feature element A. 20 In the same column, and located at the starting feature element A 20 The four elements below: A30 A 40 A 50 and A 60 This also belongs to the starting point feature region.

[0220] Optionally, if there are fewer than four elements below the starting feature element in the partitioning matrix, continue taking elements from the 0th row of the column containing the starting feature element. In Figure C, PID is 14, meaning the starting feature element is A. 62 For example, with the starting feature element A 62 A in the same column 72 A 82 and A selected from row 0 of that column. 02 and A 12 All of these belong to the starting point feature region.

[0221] It is worth noting that Figures B and C are merely examples of the starting point feature region. The starting point feature region can also take other forms, as long as it includes all elements in the row containing the starting point feature element, and the y elements in the column containing the starting point feature element, excluding the starting point feature element itself. For example... Figure 10a As shown in Figure A, the starting feature element A 31 The corresponding starting point feature region includes A 31 All four elements in the current row, and the starting feature element A. 31 The two elements above and the starting feature element A 31 The two elements below are not specified here.

[0222] In other words, if the number of rows in the partitioning matrix is ​​x = 2y + 1, then the starting feature region includes:

[0223] On the partition matrix, the elements mapped to the starting point index are the starting point feature elements; and,

[0224] In the partition matrix, the other y-1 elements in the same row as the starting feature element; and,

[0225] In the partitioning matrix, there are y elements in the same column as the starting feature element, excluding the starting feature element; or, there are y or more elements in the same column as the starting feature element, excluding the starting feature element.

[0226] Similarly, if the number of columns in the partitioning matrix is ​​y = 2x + 1, then the starting feature region includes:

[0227] On the partition matrix, the elements mapped to the starting point index are the starting point feature elements; and,

[0228] In the partition matrix, the other x-1 elements in the same column as the starting feature element; and,

[0229] In the partitioning matrix, x elements in the same row as the starting feature element, excluding the starting feature element; or x or more elements in the same row as the starting feature element, excluding the starting feature element.

[0230] In addition to determining the starting point feature region, it is also necessary to map the ending point number to the partition matrix to obtain the ending point feature matrix. This step is similar to the aforementioned step of uniformly mapping the starting point number to the partition matrix to obtain the starting point feature region, and will not be repeated here.

[0231] Similar to the starting point feature region, the range of the ending point feature region is:

[0232] If the number of rows in the partitioning matrix is ​​x = 2y + 1, then the endpoint feature region includes:

[0233] In the partition matrix, the elements mapped to the endpoint indices are the endpoint feature elements; and,

[0234] In the partition matrix, the other y-1 elements in the same row as the endpoint feature element; and,

[0235] In the partitioning matrix, y elements in the same column as the endpoint feature element, excluding the endpoint feature element; or y or more elements in the same column as the endpoint feature element, excluding the endpoint feature element.

[0236] If the number of columns in the partitioning matrix is ​​y = 2x + 1, then the endpoint feature region includes:

[0237] In the partition matrix, the elements mapped to the endpoint indices are the endpoint feature elements; and,

[0238] In the partition matrix, the other x-1 elements in the same column as the endpoint feature element; and,

[0239] In the partitioning matrix, x elements in the same row as the endpoint feature element, excluding the endpoint feature element; or x or more elements in the same row as the endpoint feature element, excluding the endpoint feature element.

[0240] The relationship between the endpoint feature region and the endpoint feature element is the same as the shape (coverage) between the starting feature region and the starting feature element. For example, if the starting feature region is L-shaped, then the endpoint feature region is an L-shaped region with the same shape as the starting feature region; or, if the starting feature region is a shape formed by arbitrarily selecting y elements from 2y elements, then the endpoint feature region has the same shape as the starting feature region. For example, if the endpoint feature region includes y elements below the endpoint feature element, then the starting feature region also includes y elements above the endpoint feature element, etc., without limitation here.

[0241] 1103. Determine the target partition of the target edge from the intersection region of the starting point feature region and the ending point feature region.

[0242] In the embodiments of this application, the edges in the graph distribution are represented by (a, b), where a represents the node at the starting point of the edge, also known as source, or simply s; and b represents the node at the ending point of the edge, also known as destination, or simply d.

[0243] In the starting feature region, s1 represents the row containing the starting feature element, and s2 represents the column containing the starting feature element. For example... Figure 10a In graph B, if the starting feature element of an edge in the graph distribution is A... 20 If the edge s1 is in the 2nd row and s2 is in the 0th column, then the edge s1 is in the 2nd row and s2 is in the 0th column.

[0244] Similarly, in the endpoint feature region, d1 represents the row where the endpoint feature element is located, and d2 represents the column where the endpoint feature element is located.

[0245] If the starting feature region and the ending feature region are determined in the partition matrix in step 1102, then the intersection region of the starting feature region and the ending feature region can be determined in the partition matrix, and an element can be determined in the intersection region. The PID corresponding to the element is used as the partition of the edge.

[0246] In this embodiment of the application, the partition of the determined edge is referred to as the target partition.

[0247] Among them, the intersection area has Figure 12b The six possibilities are shown below. Each is described in turn:

[0248] In the same row and column: if the starting feature element and the ending feature element are in the same row and column in the partition matrix, then s1 = d1 and s2 = d2. In this case, the starting feature region and the ending feature region overlap, and the intersection region is either the starting feature region or the ending feature region. The intersection region contains 2y elements.

[0249] In the same row but different columns: if the starting feature element and the ending feature element are in the same row but different columns in the partitioning matrix, then s1 = d1, s2 ≠ d2. In this case, the intersection of the starting feature region and the ending feature region is the row containing the starting feature element and the ending feature element, and the intersection region contains y elements.

[0250] Different rows, same column: If the starting feature element and the ending feature element are in the same column but different rows in the partitioning matrix, then s1 ≠ d1, s2 = d2. In this case, the intersection of the starting feature region and the ending feature region lies in the column where the starting feature element and the ending feature element are located.

[0251] Different rows and columns: If the starting feature element and the ending feature element are in different rows and columns in the partition matrix, then s1≠d1 and s2≠d2.

[0252] The rectangular edge partitioning EPR includes two methods: EPR1 and EPR2. The difference between the two lies in step 1103, which involves determining the elements that decide the edge partitioning from the intersection region. These will be explained separately below:

[0253] EPR1: Randomly select an element from the intersection region and use the PID corresponding to that element as the partition for that edge. For example... Figure 12b In the case of elements in the same row but different columns, the intersection area includes element A. 10 A 11 A 12 and A 13 Choose any one of these four elements, for example, choose element A. 10 Then in Figure 12a In Figure B, determine element A. 10 If the corresponding PID is 4, then the target partition for this edge is determined to be 4.

[0254] EPR2: In all possible intersection regions, set a default element. If an edge takes the shape of a certain intersection region, then determine the corresponding default element according to the shape of the intersection region, and directly use the PID corresponding to the element as the partition of the edge.

[0255] Since the target partition of an edge is determined from the intersection of the starting and ending feature regions, it is necessary to ensure that there is always an intersection between the starting and ending feature regions. The necessity of this intersection is guaranteed by the structure of the starting / ending feature regions.

[0256] Specifically, the starting feature region includes all elements in the row where the starting feature element is located, and the ending feature region includes all elements in the row where the ending feature element is located. As long as the starting feature element and the ending feature element are in the same row, there must be an intersection between the two feature regions.

[0257] If the starting feature element and the ending feature element are not in the same row, the number of elements in the starting feature region in the column where the starting feature element is located, and the number of elements in the ending feature region in the column where the ending feature element is located, are used to ensure that there is an intersection between the two feature regions.

[0258] As previously stated, the starting feature region must contain at least y+1 elements in the column containing its starting feature element. Similarly, the ending feature region must also contain at least y+1 elements in the column containing its ending feature element. To ensure that at least y+1 elements in the same column of the starting feature region and at least y+1 elements in the same column of the ending feature region are all in different rows, the partitioning matrix must have at least 2(y+1) rows. As previously stated, the number of rows in the partitioning matrix x = 2y+1, which is less than twice y+1. Therefore, the starting and ending feature regions must necessarily contain elements in the same row.

[0259] It is worth noting that, for the same edge, the partitioning process includes steps 1101 to 1103. Steps 1101 to 1103 need to be based on the same partitioning matrix to ensure that the partitions are determined based on the characteristics of that edge. For the same graph distribution, the partitioning process also needs to be based entirely on the same partitioning matrix to ensure that all edges in the graph distribution are partitioned according to the same partitioning principle, resulting in a more uniform partitioning outcome.

[0260] It is worth noting that the explanation of the method for determining the target partition of the edge in step 1103 is based on the fact that the number of rows in the partition matrix is ​​x = 2y + 1. If the number of columns in the partition matrix is ​​y = 2x + 1, then the corresponding method can be determined by referring to the explanation in step 1103, which will not be repeated here.

[0261] In EPR, when x = 2y + 1, the number of partitions that a target edge can potentially be assigned to, either at its starting or ending point, is equal to the number of elements in the starting or ending feature regions of the partition matrix, which is 2y. In other words, for any given point, the number of partitions that can be assigned is equal to 2y. Therefore, in EPR, the upper limit of node replication is RF. max =2y.

[0262] As can be seen from step 1101, in EPR, when x = 2y + 1, the number of elements in the partition matrix is ​​y(2y + 1), so the partitioning fraction nump = y(2y + 1).

[0263] The above explains the upper limit of the replication factor RF when x = 2y + 1. max Similar to the case of the segmentation nump; x = 2y + 1, it will not be elaborated here.

[0264] Therefore, the upper limit of node replication RF max The relationship between the segmentation nump and the nump is as follows:

[0265] RF max =2y

[0266] nump = y(2y+1)

[0267]

[0268]

[0269]

[0270]

[0271]

[0272] In this embodiment, the target partition of the edge is determined in the intersection region of the starting feature region and the ending feature region. Since the number of elements in the intersection region is limited, not greater than 2y (when x = 2y + 1) or not greater than 2x (when y = 2x + 1), the number of partitions to which a node is assigned can be reduced, thereby reducing the upper limit of the node replication number and reducing the communication overhead between computing nodes.

[0273] 4. The method for determining the target partitioning method provided in the embodiments of this application.

[0274] See Figure 13 The method for determining the target partitioning method provided in this application includes:

[0275] 1301. Determine the target segmentation fraction and target partitioning method for the graph distribution.

[0276] In the process of partitioning a graph distribution, it is necessary to determine the target number of partitions and the target partitioning method, so as to partition the graph distribution into the target number of partitions according to the target partitioning method.

[0277] Optionally, the process of determining the target segmentation and target partitioning method can be roughly divided into: a: determining the expected partitioning of the graph distribution (nump). exp b: Determine the optimal number of matches for the three partitioning methods mentioned above based on the partitioning expectation; c: Determine the target partitioning method and target segmentation score. These will be described in detail below:

[0278] a: Determine the partition expectation of the graph distribution using NumPy. exp :

[0279] like Figure 14 As shown, the graph distribution data is input into the parameter selection module, which can determine the expected partitioning of the graph distribution using NumPy. exp Where nump represents the number of partitions and exp represents the expected value.

[0280] Optionally, the process by which the parameter selection module determines the partitioning expectation may include:

[0281] The parameter selection module performs test graph calculations on the graph distribution based on the default partitioning fraction and default partitioning method to obtain the partition expectation.

[0282] It is worth noting that, in addition to the expected partitioning, the parameter selection module can also calculate and determine other parameters related to the distribution of the test graph in this step, such as... Figure 5c The parallelism, executor, GC tuning, speculative execution, etc. shown are not limited here.

[0283] b: Determine the optimal number of matches for the three partitioning methods mentioned above based on the partitioning expectation:

[0284] Once the partition expectation is determined, the graph repartitioning construction module can determine the optimal matching number based on the partition expectation. The optimal matching number is the number of partitions that best match the partition expectation across the three partitioning methods described above, as determined based on the partition expectation. In this embodiment, the optimal matching number is also referred to as the optimal segmentation number.

[0285] For example, if the partition expectation is 10, then the splitting fraction nump corresponding to the collapsed edge partition EPF is x(x+1) / 2. The nump closest to the partition expectation is then determined. EPF =10; the corresponding partition fraction nump = x(2x+1) for the rectangular side partition EPR determines the nump closest to the partition expectation. EPR =10; corresponding to the segmentation fraction nump=m(m+1) of the Steiner edge partition EPS, determine the nump closest to the partition expectation. EPS =12.

[0286] c: Determine the target partitioning method and target segmentation:

[0287] Having determined the optimal matching number for the three partitioning methods mentioned above, the graph repartitioning construction module can then build upon this. Figure 14 The determination method shown determines which partitioning method to use. Once the partitioning method is determined, the optimal number of matches matching that partitioning method can be determined, and this optimal number of matches is used as the segmentation fraction of the graph distribution. In the embodiments of this application, the partitioning method confirmed by the parameter selection module is also called the target partitioning method, and the segmentation fraction confirmed by the parameter selection module is also called the target segmentation fraction.

[0288] 1302. By using the target partitioning method, the multiple edges in the graph distribution are divided into target partitions.

[0289] Once the graph repartitioning construction module confirms the target partitioning method and the target number of partitions, it can inform the graph decomposition module of these parameters. The graph decomposition module then partitions the multiple edges in the graph distribution according to the target partitioning method, resulting in the target number of partitions.

[0290] Optionally, after the graph repartitioning module partitions multiple edges in the graph distribution using the target partitioning method, it can send the partitioning results to the computing nodes.

[0291] Optionally, there can be multiple computing nodes, each corresponding to a partition of the graph distribution, used to perform calculations on the node data within the corresponding partition.

[0292] In this embodiment, the partitioning result is sent to the compute nodes, which can then perform calculations on the data of the nodes within the corresponding partition based on the partitioning result. Because the upper limit of the number of node replicas in the partitioning result is low, the communication overhead caused by synchronizing node data between compute nodes is reduced, thereby improving the speed of graph computation and reducing the memory usage of graph computation. The low upper limit of the number of node replicas in the partitioning result also reduces the number of compute nodes requiring synchronization, thus reducing the latency of compute node synchronization.

[0293] III. The device provided in the embodiments of this application.

[0294] Please see Figure 15 This application provides a data processing device 1500, which includes a processor 1501 and a memory 1502.

[0295] Memory 1502 is used to store programs;

[0296] Processor 1501 is used to execute programs in memory, causing the processor to perform the aforementioned... Figures 6 to 14 The data processing method of the embodiment shown.

[0297] Please see Figure 16 This application also provides a chip 1600, which includes at least one processor 1610 and a communication interface 1620. The communication interface 1620 and the at least one processor 1610 are interconnected via a line. The at least one processor 1610 is used to run computer programs or instructions to perform the aforementioned functions. Figures 6 to 14 Data processing methods.

[0298] The communication interface 1620 in the chip can be an input / output interface, pins, or circuits.

[0299] In one possible implementation, the chip 1600 described above in this application further includes at least one memory 1630, which stores instructions. The memory 1630 can be an internal storage unit of the chip, such as a register, cache, etc., or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0300] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0301] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0302] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0303] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0304] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data processing method, characterized in that, The method is applied to a data computing scenario, which includes computing nodes and a graph distribution consisting of multiple nodes. The nodes are the data providers for the computing nodes to perform data computing. The method includes: partitioning multiple edges in the graph distribution using a target partitioning method; wherein the target partitioning method includes Steiner edge partitioning. The characteristic is that the target partitioning method is Steiner edge partitioning, and the partitioning of multiple edges in the graph distribution using the target partitioning method includes: Based on the node numbering matrix, determine the mapping positions of the start and end points of the target edge in the multiple edges, wherein the node is the data provider for large-scale data computation, the number of the nodes is multiple, the order of the node numbering matrix is ​​m, and m is a prime number; Determine (m+1) partition feature matrices corresponding to the node numbering matrix, wherein the elements of the partition feature matrices include edge partitions; The target partition of the target edge is determined based on the (m+1) partition feature matrices and the mapping positions of the start and end points of the target edge; Its characteristic is that each of the (m+1) partition feature matrices includes m 2 The elements, m 2 Each element corresponds to m distinct partitions, and each partition corresponds to m elements; The (m+1) partition feature matrices include: a matrix with identical row elements, a matrix with identical column elements, and a target matrix; wherein, the matrix with identical row elements, the matrix with identical column elements, and the target matrix are all square matrices of order m; In matrices with identical row elements, elements in the same row correspond to the same partition; In matrices with identical column elements, elements in the same column correspond to the same partition; In the target matrix, elements in the same row correspond to m different partitions; in the target matrix, elements in the same column correspond to m different partitions. When the data calculation scenario is a weather forecast scenario, the node is a meteorological monitoring point, and the graph distribution includes multiple meteorological monitoring points, with multiple meteorological monitoring points that are geographically close to each other assigned to the same partition.

2. The method according to claim 1, characterized in that, The target partitioning method also includes edge folding partitioning. Partitioning multiple edges in the graph distribution using the target partitioning method includes: The adjacency matrix of the graph distribution is sparsely hashed to obtain a hashed adjacency matrix. The hashed adjacency matrix is ​​used to reflect the correspondence between the target edge and the start and end points of the target edge among the multiple edges. The hash adjacency matrix is ​​mapped to a partition matrix, wherein the partition matrix is ​​a symmetric matrix; The partition of the target edge is determined based on the partition matrix.

3. The method according to claim 2, characterized in that, Prior to mapping the hash adjacency matrix to the partition matrix, the method further includes: The partition matrix is ​​determined by the lower triangular elements and the upper triangular elements having triangular number characteristics, wherein the lower triangular elements and the upper triangular elements are mutually symmetrical; Determine the diagonal elements of the partition matrix. Where 1 ≤ i ≤ the order of the partition matrix; The corresponding partition, and the partition in the i-th row of the partition matrix. The corresponding partitions are the same, where 1≤j≤the order of the partition matrix, and i and j are not the same.

4. The method according to claim 1, characterized in that, The target partitioning method also includes rectangular edge partitioning. Partitioning multiple edges in the graph distribution using the target partitioning method includes: Determine a partition matrix, wherein the number of rows x and the number of columns y of the partition matrix satisfy the following relationship: y = 2x + 1, or x = 2y + 1; where x and y are both positive integers; Determine the mapping positions of the start and end points of the target edge among the multiple edges; In the partitioning matrix, the starting point feature region and the ending point feature region are determined according to the mapping positions of the starting and ending points of the target edge; The target partition of the target edge is determined from the intersection region of the starting point feature region and the ending point feature region.

5. The method according to any one of claims 1 to 4, characterized in that, Before partitioning multiple edges in the graph distribution using the target partitioning method, the method further includes: Determine the target segmentation number and target partitioning method of the graph distribution; The step of partitioning multiple edges in the graph distribution using a target partitioning method includes: The target partitioning method divides multiple edges in the graph distribution into several partitions based on the target partitioning.

6. The method according to claim 5, characterized in that, The determination of the target slice number and target partitioning method of the graph distribution includes: Determine the expected segment; Based on the desired partitioning fraction, the optimal partitioning fraction corresponding to the candidate partitioning methods is determined, wherein the candidate partitioning methods include at least one of the Steiner edge partitioning, rectangular edge partitioning, and folded edge partitioning; Based on the optimal partitioning fraction, the target partitioning method and the target partitioning fraction that matches the target partitioning method are determined.

7. The method according to claim 6, characterized in that, The determination of the expected segmentation includes: Based on the default segmentation score and default segmentation method, a test graph calculation is performed on the graph distribution to obtain the expected segmentation score.

8. The method according to any one of claims 1 to 4 or 6 to 7, characterized in that, After partitioning multiple edges in the graph distribution using the target partitioning method, the method further includes: Send the partitioning results to the compute nodes.

9. A data processing apparatus, characterized in that, It includes a processor and a memory; the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, causing the processor to perform the data processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 8.

11. A computer program product, characterized in that, The computer program product includes: computer program code; When the computer program code is run, it implements the method as described in any one of claims 1 to 8.

12. A chip, characterized in that, Includes at least one processor and interface; The interface is used to provide program instructions or data to the at least one processor; The at least one processor is configured to execute the program instructions to implement the method as described in any one of claims 1 to 8.

13. A server, characterized in that, Includes the chip described in claim 12.

Images