Image data processing methods, devices, equipment, and storage media

Abstract

This application provides a graph data processing method, apparatus, electronic device, and computer-readable storage medium. The method includes: segmenting graph data to be processed to obtain multiple segmented graphs; updating the labels of nodes in the multiple segmented graphs in batches to obtain updated labels for the nodes in the multiple segmented graphs; compressing the graph data to be processed based on the updated labels of the nodes in the multiple segmented graphs to obtain a compressed graph of the graph data to be processed; performing label mining on the nodes in the compressed graph to obtain mined labels corresponding to the nodes in the compressed graph; updating the nodes in the graph data to be processed based on the mined labels corresponding to the nodes in the compressed graph, and taking the graph data composed of nodes with the same updated labels as the connected components of the graph data to be processed. This application can improve the performance of connected component mining.

Description

Technical Field

[0001] This application relates to computer data processing technology, and more particularly to a graph data processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Graphs are powerful analytical tools with a long history of helping humans communicate, understand the world, and solve scientific problems. Connected component mining is an important research direction in data mining, helping users gain insights into data through connected components. Today, connected component mining is widely used in fields such as social networks, financial risk control, and cybersecurity.

[0003] During the implementation process, the applicant discovered that connected component mining in related technologies requires many iterative processes of label updates. These multiple iterative calculations cause unnecessary consumption of resources (including hardware and computing resources), thus increasing the complexity of connected component mining. Summary of the Invention

[0004] This application provides a graph data processing method, apparatus, electronic device, and computer-readable storage medium that can improve the performance of connected component mining.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a graph data processing method, including:

[0007] The graph data to be processed is segmented to obtain multiple segmented graphs of the graph data to be processed.

[0008] The nodes in the multiple segmentation graphs are updated in batches to obtain the updated labels of the nodes in the multiple segmentation graphs.

[0009] The graph data to be processed is compressed based on the updated labels of the nodes in the multiple segmentation graphs to obtain a compressed graph of the graph data to be processed.

[0010] Perform label mining on the nodes in the compressed graph to obtain the mining labels corresponding to the nodes in the compressed graph;

[0011] Based on the mining labels corresponding to the nodes in the compressed graph, the nodes in the graph data to be processed are updated, and the graph data consisting of the updated nodes with the same labels in the graph data to be processed is taken as the connected component of the graph data to be processed.

[0012] This application provides a graph data processing apparatus, including:

[0013] The segmentation module is used to segment the graph data to be processed, thereby obtaining multiple segmented graphs of the graph data to be processed.

[0014] The execution module is used to perform label update processing on nodes in the multiple segmentation graphs based on different batches to obtain updated labels for the nodes in the multiple segmentation graphs; to compress the graph data to be processed based on the updated labels of the nodes in the multiple segmentation graphs to obtain a compressed graph of the graph data to be processed; and to perform label mining processing on the nodes in the compressed graph to obtain the mining labels corresponding to the nodes in the compressed graph.

[0015] The update module is used to update the nodes in the graph data to be processed based on the mining labels corresponding to the nodes in the compressed graph, and to take the graph data composed of the updated nodes with the same labels as the connected components of the graph data to be processed.

[0016] In the above technical solution, the execution module includes multiple sub-execution modules, each sub-execution module corresponding to at least one of the segmentation maps; the sub-execution module is used to perform batch label update processing on the nodes in the corresponding segmentation map to obtain the updated labels of the nodes in the segmentation map;

[0017] The updated labels of the nodes in the segmentation graph are sent to the parameter module.

[0018] In the above technical solution, the sub-execution module is further used to iteratively execute the following processes:

[0019] Batch allocation processing is performed on the nodes in the corresponding segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph;

[0020] Obtain from the parameter module the label of any node included in the batch, and the labels of the neighboring nodes of the included node;

[0021] Based on the labels of the neighboring nodes, the labels of any node included in the batch are updated to obtain the updated labels of any node included in the batch.

[0022] The iteration process stops when the number of iterations reaches the set number.

[0023] Send the update tags of any node contained in the batch to the parameter module.

[0024] In the above technical solution, the sub-execution module is further configured to perform batch random allocation processing on the corresponding nodes in the segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph; or,

[0025] The nodes in the corresponding segmentation graph are uniformly distributed in batches to obtain the batches corresponding to the nodes in the segmentation graph.

[0026] In the above technical solution, the sub-execution module is further configured to perform the following processing for any node included in any batch:

[0027] Determine the smallest or largest label from the label of the node and the labels of the node's neighboring nodes;

[0028] The smallest or the largest label is used as the updated label of the node.

[0029] In the above technical solution, the sub-execution module is further configured to obtain from the parameter module the update tags of multiple nodes included in the batch, and the update tags of the neighboring nodes of the included nodes;

[0030] Based on the update tags of the nodes included in the multiple batches and the update tags of the neighboring nodes, the graph data to be processed is compressed to obtain a compressed graph of the graph data to be processed.

[0031] In the above technical solution, the sub-execution module is further used to perform node combination processing based on the update tags of the nodes included in the multiple batches and the update tags of the neighboring nodes to obtain the segmentation map with updated tags corresponding to the execution module;

[0032] When the updated labels of adjacent nodes in the segmentation graph corresponding to the execution module are different, the edge between the adjacent nodes is used as the edge of the compressed graph of the graph data to be processed, and the adjacent node is used as the node of the compressed graph of the graph data to be processed.

[0033] In the above technical solution, the execution module includes a main execution module, which is used to obtain the nodes and corresponding edges in the compressed graph from the plurality of execution modules;

[0034] Based on the nodes and corresponding edges in the compressed graph, a tree structure is constructed to obtain the tree structure corresponding to the compressed graph;

[0035] From the multiple tags in the tree structure, determine the target tag of the tree structure;

[0036] The target label is used as the mining label corresponding to the node in the compressed graph;

[0037] The main execution module is one of the pre-configured sub-execution modules, or the sub-execution modules are generated through an election operation.

[0038] In the above technical solution, the main execution module is further configured to obtain the root node of the tree structure from multiple nodes in the compressed graph;

[0039] The nodes in the compressed graph that have an edge with the root node are designated as leaf nodes of the root node.

[0040] The node in the compressed graph that has an edge with the leaf node is designated as the leaf node of the leaf node;

[0041] The tree structure corresponding to the compressed graph is constructed based on the root node, the leaf node, and the leaf nodes of the leaf nodes.

[0042] In the above technical solution, the main execution module is further configured to randomly select a label from multiple labels in the tree structure and use it as the target label of the tree structure; or,

[0043] The label corresponding to the root node in the tree structure is used as the target label of the tree structure.

[0044] In the above technical solution, the update module is further used to obtain the node corresponding to the update label from the graph data to be processed based on the update label of the node in the compressed graph;

[0045] Update the labels of the nodes in the graph data to be processed that correspond to the updated labels to the mining labels.

[0046] In the above technical solution, the segmentation module is further configured to randomly segment the graph data to be processed based on the initial labels of the nodes in the graph data to be processed, thereby obtaining multiple segmented graphs of the graph data to be processed; or,

[0047] Based on the initial labels of the nodes in the graph data to be processed, the graph data to be processed is uniformly divided to obtain multiple segmented graphs of the graph data to be processed.

[0048] In the above technical solution, the segmentation module is also used to display the image data to be processed;

[0049] In response to a segmentation operation on the graph data to be processed, the segmented portion of the graph data to be processed is taken as the segmented graph.

[0050] In the above technical solution, the segmentation module is also used to display the segmentation trigger entry for the graph data to be processed;

[0051] In response to the trigger operation for the segmentation trigger entry, based on the characteristics of the nodes in the graph data to be processed, the nodes in the graph data to be processed are clustered, and the graph data composed of nodes belonging to the same category is used as the segmentation graph.

[0052] This application provides a distributed system, including:

[0053] Multiple actuators; among which,

[0054] The main executor is used to perform segmentation processing on the graph data to be processed, resulting in multiple segmented graphs of the graph data to be processed. The main executor is one of the multiple executors that are pre-configured, or it is generated by the multiple executors through an election operation.

[0055] The plurality of executors are used to perform label update processing on nodes in the plurality of segmentation graphs based on different batches to obtain updated labels for nodes in the plurality of segmentation graphs; and to compress the graph data to be processed based on the updated labels for nodes in the plurality of segmentation graphs to obtain a compressed graph of the graph data to be processed.

[0056] The main executor is also used to perform label mining processing on the nodes in the compressed graph to obtain the mining labels corresponding to the nodes in the compressed graph;

[0057] The plurality of actuators are further configured to update the nodes in the graph data to be processed based on the mining labels corresponding to the nodes in the compressed graph, and to use the graph data consisting of the updated nodes with the same labels in the graph data to be processed as the connected components of the graph data to be processed.

[0058] This application provides an electronic device for graph data processing, the electronic device comprising:

[0059] Memory, used to store executable instructions;

[0060] The processor, when executing executable instructions stored in the memory, implements the graph data processing method provided in the embodiments of this application.

[0061] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the graph data processing method provided in this application.

[0062] The embodiments of this application have the following beneficial effects:

[0063] By updating the labels of nodes in the segmentation graph in batches, asynchronous label updates are achieved, thereby reducing the number of iterations. By compressing the graph data to be processed and performing label mining based on the compressed graph, the size of the graph data is reduced, the performance of connected component mining is improved, and the resource consumption of related computations is significantly saved. Attached Figure Description

[0064] Figure 1This is a schematic diagram of the structure of the distributed system provided in the embodiments of this application;

[0065] Figure 2 This is a schematic diagram of the structure of an electronic device for graph data processing provided in an embodiment of this application;

[0066] Figures 3-5 This is a schematic flowchart of the graph data processing method provided in the embodiments of this application;

[0067] Figure 6 This is a schematic diagram of graph segmentation provided in an embodiment of this application;

[0068] Figure 7 This is a schematic diagram of the node update process provided in the embodiments of this application;

[0069] Figure 8 This is a schematic diagram of tag propagation provided in an embodiment of this application;

[0070] Figure 9 This is a schematic diagram of the graph compression process provided in the embodiments of this application;

[0071] Figure 10 This is a schematic diagram of the tree structure provided in the embodiments of this application;

[0072] Figure 11 This is a schematic diagram of the node update process provided in the embodiments of this application;

[0073] Figure 12 This is a flowchart illustrating the asynchronous connected component mining algorithm provided in an embodiment of this application;

[0074] Figure 13 This is a schematic diagram of the asynchronous mining process of connected components provided in the embodiments of this application. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0078] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0079] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0080] 1) Graph: An abstract data structure consisting of nodes and the connections between them. In real-world scenarios, nodes can represent people, organizations, goods, etc., while the connections between nodes, or edges, can represent social relationships between people, transactional relationships between goods, etc. In graph data, nodes can have their own attributes, given in the form of feature vectors, and edges are divided into directed and undirected.

[0081] 2) Connected graph: If there exists at least one path between any two nodes in an undirected graph, then the undirected graph is a connected graph.

[0082] 3) Connected Component (CC): If an undirected graph has a subgraph that satisfies the property of a connected graph, then the subgraph is a connected component of the original undirected graph.

[0083] 4) Parameter Server (PS): In a distributed architecture, this is a node (server or server cluster) that is responsible for creating the cluster, aggregating, updating, distributing and storing parameters.

[0084] 5) Angel: A high-performance distributed machine learning platform designed and developed based on the parameter server concept. It rationally divides high-dimensional large models into multiple parameter server nodes and easily implements various efficient machine learning algorithms through efficient model update interfaces, operation functions, and flexible synchronization protocols.

[0085] 6) Computation Engine (Spark): A fast and general-purpose computing engine designed for large-scale data processing. It enables in-memory distributed datasets and, in addition to providing interactive queries, optimizes iterative workloads.

[0086] 7) Spark on Angel (SONA): A high-performance distributed computing platform that combines Angel's powerful parameter server capabilities with Spark's large-scale data processing capabilities, supporting traditional machine learning, deep learning, and various graph algorithms.

[0087] 8) Disjoint set: A tree-like data structure used to handle the merging and querying of disjoint sets. In practice, it is represented as a forest.

[0088] The connected component mining methods in this application include a single-machine connected component mining method based on disjoint-set data structure (DFS) and a distributed connected component mining method based on label propagation. The single-machine DFS method is efficient, requiring only one traversal of all edges in the graph and the addition of relational edges to the DFS; ultimately, points belonging to the same set belong to the same connected component. In the distributed connected component mining method based on label propagation, each node sends its label to its neighbors in each iteration. In the next iteration, each node traverses all the labels received from its neighbors and, combined with its own label, selects the smallest label as its new label.

[0089] However, both of the above connected component mining methods have their own problems, as shown below:

[0090] 1) The single-machine connected component mining method based on disjoint set storage stores the intermediate results of all node traversal in memory and maintains a disjoint set. This method is limited by memory space and cannot handle large-scale graphs.

[0091] 2) In the label-based distributed connected component mining method, each node communicates with its neighbor nodes once per iteration, and each node's label expands outward by at most one step in each iteration. Therefore, when the diameter of the graph is large, the number of iterations will become very large, affecting the overall computation time.

[0092] 3) The label-based distributed connected component mining method is a synchronous algorithm. Each information processing step processes the information sent in the previous iteration. The information updated and calculated in the current iteration can only be used in the next iteration.

[0093] To address the aforementioned issues, embodiments of this application provide a graph data processing method, apparatus, system, and computer-readable storage medium that can improve the performance of connected component mining.

[0094] The graph data processing method provided in this application is implemented by a distributed system. The servers in this distributed system can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services.

[0095] See Figure 1 , Figure 1 This is a schematic diagram of the structure of the distributed system 10 provided in the embodiments of this application. The executors (executors 100-1 and 100-2 are shown as examples), the parameter server 200, and the terminal 300 are connected through a network 400. The network 400 can be a wide area network or a local area network, or a combination of the two.

[0096] Terminal 300 (running a client, such as an instant messaging client, a network security client, or an investment client) can be used to acquire graph data to be processed, and can also segment the graph data to be processed to obtain multiple segmented graphs of the graph data to be processed. For example, for an instant messaging client, a graph data representing social relationships can be constructed based on various social relationships, and the nodes in the graph data represent users in the social network.

[0097] The executor performs batch label update processing on nodes in multiple segmentation graphs, obtaining updated labels for the nodes in the multiple segmentation graphs. Based on the updated labels of the nodes in the multiple segmentation graphs, it compresses the graph data to be processed, obtaining a compressed graph of the graph data to be processed. For example, executor 100-1, based on the node labels obtained from parameter server 200, performs batch label update processing on the nodes in the segmentation graphs. Figure 1 The nodes in the data undergo batch label update processing to obtain the segmentation. Figure 1 Update the labels of the nodes in the data and split the data. Figure 1 The updated labels of the nodes are sent to parameter server 200. Executor 100-2, based on the node labels obtained from parameter server 200, performs segmentation... Figure 2 The nodes in the data undergo batch label update processing to obtain the segmentation. Figure 2 Update the labels of the nodes in the data and split the data. Figure 1 The updated tags of the nodes are sent to parameter server 200. Executors 100-1 and 100-2 work together based on the segmentation... Figure 1 and segmentation Figure 2 The update labels of the nodes in the graph are used to compress the graph data to be processed, resulting in a compressed graph of the graph data to be processed.

[0098] When executor 100-1 is the master executor, executor 100-1 acquires the compressed graph of the graph data to be processed, and performs label mining processing on the nodes in the compressed graph to obtain the mining labels corresponding to the nodes in the compressed graph. Based on the mining labels corresponding to the nodes in the compressed graph, the nodes in the graph data to be processed are updated, so that the graph data composed of the updated nodes with the same labels in the graph data to be processed is taken as the connected component of the graph data to be processed.

[0099] The parameter server 200 is used for tasks such as tag aggregation, updating, distribution, and storage. For example, it receives node tags sent by executors 100-1 and 100-2. When executors 100-1 and 100-2 retrieve node tags, the parameter server 200 sends the node tags to executors 100-1 and 100-2.

[0100] The embodiments of this application can be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0101] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0102] The graph data processing method provided in this application embodiment is implemented by a distributed system. The server in the distributed system can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The cloud service can be a graph data processing service, which is invoked by the terminal 300. The terminal 300 can be a smartphone, tablet, laptop, desktop computer, smart TV, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment.

[0103] It should be noted that the executor and parameter server in the embodiments of this application can be implemented in software (e.g., in the form of applications, software, software modules, scripts, or code) and deployed in electronic devices (i.e., the various forms of servers mentioned above) to achieve a distributed system architecture. The structure of the electronic device for graph data processing provided in the embodiments of this application is described below; see [link to documentation]. Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 for graph data processing provided in an embodiment of this application. The example given is of the electronic device 500 being a server. Figure 2 The illustrated electronic device 500 for graph data processing includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0104] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0105] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0106] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0107] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0108] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0109] In some embodiments, the graph data processing apparatus provided in this application can be implemented in software, for example, as the graph data processing service in the server described above. Of course, this is not the limitation; the graph data processing apparatus provided in this application can be provided in various software embodiments, including various forms such as applications, software, software modules, scripts, or code.

[0110] Figure 2 A graph data processing apparatus 555 stored in a memory 550 is shown. This apparatus can be software in the form of programs or plug-ins, and includes a series of modules, including a segmentation module 5551, an execution module 5552 (including multiple sub-execution modules), and an update module 5553. The segmentation module 5551, execution module 5552, and update module 5553 are used to implement the graph data processing functions provided in the embodiments of this application. When the execution module is deployed in a server, the corresponding server can be called an executor; similarly, a server with a parameter module deployed is implemented as a parameter server.

[0111] The following description, in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application, illustrates the graph data processing method provided in the embodiments of this application. See also... Figure 3 , Figure 3 This is a schematic flowchart of the graph data processing method provided in the embodiments of this application, combined with... Figure 3 The steps shown are explained.

[0112] In the following steps, the graph data to be processed includes multiple nodes and corresponding edges. Connectivity mining of graph data has been widely applied in fields such as social networks, financial risk control, and cybersecurity. For example, in social networks, nodes represent users within the network, and edges represent relationships between users corresponding to nodes. By mining connected components from the graph data, it can be determined that users corresponding to these connected components have social relationships. In the field of financial risk control, nodes represent product information (such as stock information), and edges represent relationships between products corresponding to nodes. By mining connected components from the graph data, it can be determined that products corresponding to these connected components have risk-sharing relationships.

[0113] In step 101, the graph data to be processed is segmented to obtain multiple segmented graphs of the graph data to be processed.

[0114] For example, if the graph data to be processed is a large graph with many nodes and corresponding edges, the graph data can be segmented to obtain multiple segmented graphs. Subsequently, node updates can be performed on each of the multiple segmented graphs, thereby speeding up the update process.

[0115] In some embodiments, the graph data to be processed is segmented to obtain multiple segmented graphs of the graph data to be processed, including: randomly segmenting the graph data to be processed based on the initial labels of the nodes in the graph data to obtain multiple segmented graphs of the graph data to be processed; or uniformly segmenting the graph data to be processed based on the initial labels of the nodes in the graph data to obtain multiple segmented graphs of the graph data to be processed.

[0116] For example, the random partitioning process is as follows: In order to obtain the first partitioned graph, multiple nodes are randomly selected from the graph data to be processed. These multiple nodes and their corresponding neighbor nodes constitute the first partitioned graph. These multiple nodes are then removed from the graph data to be processed. The above process is repeated until there are 0 nodes in the graph data to be processed, thereby obtaining multiple partitioned graphs of the graph data to be processed.

[0117] For example, uniform partitioning is performed as follows: The set number of partition graphs is obtained. Based on this number, the number of nodes in the graph data to be processed is evenly distributed. The evenly distributed nodes and their corresponding neighboring nodes form the partition graph. For example, if the set number of partition graphs is 3, and the graph data G to be processed has 30 nodes, with initial labels of node identifiers (id) ranging from 1 to 30, then nodes with IDs from 1 to 10 and their corresponding neighboring nodes are used as partitions. Figure 1 Nodes with IDs 11-20 and their corresponding neighbor nodes are used as partitions. Figure 2 Nodes with IDs between 21 and 30 and their corresponding neighbor nodes are used as partitions. Figure 3 .

[0118] In some embodiments, segmenting the graph data to be processed to obtain multiple segmented graphs of the graph data to be processed includes: displaying the graph data to be processed; and, in response to the segmentation operation on the graph data to be processed, using the segmented portion of the graph data to be processed as a segmented graph.

[0119] For example, after the terminal acquires the image data to be processed, it displays the image data on the display interface. Through user operations on the image data (such as swiping, clicking, etc.), the segmented portion of the image data is used as the segmented image. Figure 6 As shown, after manually splitting the graph data to be processed, three split graphs are formed. Through manual splitting, the graph data to be processed can be initially clustered so that the connected components of the graph data to be processed can be quickly queried in subsequent iterations.

[0120] In some embodiments, segmenting the graph data to be processed to obtain multiple segmented graphs of the graph data to be processed includes: displaying a segmentation trigger entry for the graph data to be processed; in response to a triggering operation for the segmentation trigger entry, clustering the nodes in the graph data to be processed based on the characteristics of the nodes in the graph data to be processed, and using the graph data composed of nodes belonging to the same category as the segmented graphs.

[0121] For example, after the terminal acquires the graph data to be processed, it displays the segmentation trigger entry point for the graph data on the display interface. Based on the user's trigger operation on the graph data to be processed (such as single click, double click, swipe, etc.), the classifier is invoked. The classifier combines the features of the nodes in the graph data to be processed to cluster the nodes. The graph data composed of nodes belonging to the same category is used as the segmentation graph. For example, in a social network, the nodes are users in the social network, and the features of the nodes are user profiles; in the field of financial risk control, the nodes are product information (such as stock information), and the features of the nodes are historical behavior data of the products, such as purchase volume, return volume, etc.

[0122] In step 102, the nodes in multiple segmentation graphs are updated in batches to obtain the updated labels of the nodes in multiple segmentation graphs.

[0123] For example, by performing batch label update processing on nodes in multiple segmentation graphs, updated labels for nodes in multiple segmentation graphs can be obtained, thereby achieving asynchronous update of node labels. In the process of iteratively updating labels, parameters that would otherwise only be updated in the next iteration can be obtained in advance during the current iteration, thus reducing the number of iterations.

[0124] See Figure 4 , Figure 4 This is an optional flowchart illustrating the graph data processing method provided in an embodiment of this application. Figure 4 Show Figure 3 Step 102 can be implemented through step 1021: the label update process is implemented through a distributed system, which includes a parameter server and multiple executors, each executor corresponding to at least one segmentation graph; in step 1021, the executor performs batch label update processing on the nodes in the corresponding segmentation graph to obtain the updated labels of the nodes in the segmentation graph; and sends the updated labels of the nodes in the segmentation graph to the parameter server.

[0125] For example, the following process is performed iteratively: the executor performs batch allocation processing on the nodes in the corresponding segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph; the labels of the nodes included in any batch and the labels of the neighboring nodes of the included nodes are obtained from the parameter server; based on the labels of the neighboring nodes, the labels of the nodes included in any batch are updated to obtain the updated labels of the nodes included in any batch; the iterative processing stops when the number of iterations reaches a set number; the updated labels of the nodes in the segmentation graph are sent to the parameter server, including: sending the updated labels of the nodes included in any batch to the parameter server.

[0126] like Figure 7 As shown, when the labels of nodes in the nth batch need to be updated, the labels of the nodes included in the nth batch and the labels of their neighboring nodes are obtained from the parameter server. Label update processing is then performed based on these labels to obtain the updated labels for the nodes in the nth batch. These updated labels are then sent to the parameter server. Similarly, when the labels of nodes in the (n+1)th batch need to be updated, the labels of the nodes included in the (n+1)th batch and the labels of their neighboring nodes are obtained from the parameter server. Since the nodes in the (n+1)th batch share the same node 701 as the nodes in the nth batch, the executor may be able to retrieve labels that have already been updated in the current iteration (e.g., node 701 has already been updated). This allows the executor to obtain labels that will only need to be updated in the next round under synchronous conditions in advance. This method reduces the number of iteration rounds. Therefore, the more times asynchronous fetching occurs, the significantly fewer iteration rounds will be required.

[0127] In some embodiments, the executor performs batch allocation processing on the nodes in the corresponding segmentation graph to obtain the batch corresponding to the nodes in the segmentation graph, including: the executor performs random allocation processing on the nodes in the corresponding segmentation graph to obtain the batch corresponding to the nodes in the segmentation graph; or, the executor performs uniform allocation processing on the nodes in the corresponding segmentation graph to obtain the batch corresponding to the nodes in the segmentation graph.

[0128] For example, random batches are assigned to nodes in the partition graph. For instance, a random generator assigns a random positive integer (the random positive integer is less than the total number of batches) to node 1 in the partition graph, and the random positive integer is used as the batch of node 1.

[0129] For example, the uniform distribution process is as follows: obtain the set number of batches, and distribute the nodes in the segmentation graph evenly based on the set number of batches. For example, if the set number of batches is 3 and the segmentation graph has 30 nodes, then the batch of nodes with node IDs 1-10 is set to 1, the batch of nodes with node IDs 11-20 is set to 2, and the batch of nodes with node IDs 21-30 is set to 3.

[0130] In some embodiments, based on the labels of neighboring nodes, the labels of nodes included in any batch are updated to obtain updated labels of nodes included in any batch, including: performing the following processing for any node included in any batch: determining the smallest or largest label from the node's label and the labels of the node's neighboring nodes; and using the smallest or largest label as the updated label of the node.

[0131] like Figure 8 As shown, for node 801, obtain the neighboring nodes of node 801 (node ​​802, node 803, and node 804). Among them, the label of node 801 is 4, the label of node 802 is 2, the label of node 803 is 6, and the label of node 804 is 3. From nodes 801, 802, 803, and 804, select the smallest or largest label. The smallest label 2 or the largest label 6 is used as the update label of node 801, thereby updating the label through label propagation.

[0132] In step 103, the graph data to be processed is compressed based on the update labels of nodes in multiple segmentation graphs to obtain a compressed graph of the graph data to be processed.

[0133] For example, after batch label updates, the nodes in the graph data to be processed have been fully updated. Therefore, the graph data to be processed can be compressed based only on the updated labels of the nodes in multiple segmented graphs to reduce the size of the graph data to be processed, so that it can be processed on a single machine later, saving computing resources.

[0134] See Figure 5 , Figure 5 This is an optional flowchart illustrating the graph data processing method provided in an embodiment of this application. Figure 5 Show Figure 4 Step 103 can be implemented through steps 1031-1032: In step 1031, the executor obtains the update tags of the nodes included in multiple batches and the update tags of the neighboring nodes of the included nodes from the parameter server; In step 1032, based on the update tags of the nodes included in multiple batches and the update tags of the neighboring nodes, the graph data to be processed is compressed to obtain a compressed graph of the graph data to be processed.

[0135] For example, the compression process is as follows: The executor performs node combination processing based on the updated labels of nodes contained in multiple batches and the updated labels of neighboring nodes to obtain the segmented graph with updated labels corresponding to the executor; when the updated labels of adjacent nodes in the segmented graph with updated labels corresponding to the executor are different, the edges between adjacent nodes are used as the edges of the compressed graph of the graph data to be processed, and the adjacent nodes are used as the nodes of the compressed graph of the graph data to be processed.

[0136] For example, when compressing the graph data to be processed, the update labels in each segmented graph are obtained to obtain the update labels (CCid) of each node in the graph data to be processed. Nodes with the same CCid are compressed into one node. The edges between these compressed nodes can be defined by the edges in the graph data to be processed where the update labels of the two ends of the nodes are different. If the update label of a node is inconsistent with the update label of its neighboring node, the edge is recorded in the form of <node label, neighboring node label> and used as an edge in the compressed graph.

[0137] like Figure 9 As shown, nodes with label 2 are merged into node 901, nodes with label 5 are merged into node 902, nodes with label 3 are merged into node 903, nodes with label 1 are merged into node 904, and nodes with label 4 are merged into node 905.

[0138] In step 104, label mining is performed on the nodes in the compressed graph to obtain the mining labels corresponding to the nodes in the compressed graph.

[0139] For example, after the distributed label update through multiple executors, the nodes in the compressed graph are labeled using a single machine combined with a disjoint-set data structure method to obtain the corresponding mined labels (i.e., disjoint-set labels) of the nodes in the compressed graph. This combines the advantages of distributed parallel computing and single-machine computing to improve the performance of connected component mining.

[0140] In some embodiments, label mining processing is performed on nodes in the compressed graph to obtain mining labels corresponding to the nodes in the compressed graph, including: the main executor obtaining nodes and corresponding edges in the compressed graph from multiple executors; constructing a tree structure based on the nodes and corresponding edges in the compressed graph to obtain a tree structure corresponding to the compressed graph; determining the target label of the tree structure from multiple labels in the tree structure; and using the target label as the mining label corresponding to the node in the compressed graph; wherein the main executor is one of multiple pre-configured executors, or multiple executors are generated through an election operation.

[0141] For example, the process of constructing a tree structure using the disjoint-set data structure method is as follows: obtain the root node of the tree structure from multiple nodes in the compressed graph; take the nodes in the compressed graph that have edges with the root node as the leaf nodes of the root node; take the nodes in the compressed graph that have edges with the leaf nodes as the leaf nodes of the leaf nodes; construct the tree structure corresponding to the compressed graph based on the root node, leaf nodes, and leaf nodes of the leaf nodes.

[0142] like Figure 10 As shown, the node with label 1 is taken as the root node, the nodes with labels 2, 3, 4, and 5 are taken as the leaf nodes of the root node, and the node with label 6 is taken as the leaf node of the leaf node with label 3.

[0143] In some embodiments, determining the target label of the tree structure from multiple labels in the tree structure includes: randomly selecting a label from multiple labels in the tree structure and using it as the target label of the tree structure; or, using the label corresponding to the root node in the tree structure as the target label of the tree structure.

[0144] like Figure 10 As shown, a label 5 is randomly selected from multiple labels in the tree structure and used as the target label of the tree structure; or, the label 1 corresponding to the root node in the tree structure is used as the target label of the tree structure.

[0145] In step 105, the nodes in the graph data to be processed are updated based on the mining labels corresponding to the nodes in the compressed graph. The graph data consisting of the nodes with the same labels after the update is taken as the connected components of the graph data to be processed.

[0146] For example, after obtaining the mining labels corresponding to the nodes in the compressed graph, the nodes in the graph data to be processed can be matched and updated based on the mining labels corresponding to the nodes in the compressed graph to obtain the connected components of the graph data to be processed.

[0147] In some embodiments, updating nodes in the graph data to be processed based on mining labels corresponding to nodes in the compressed graph includes: obtaining nodes corresponding to the updated labels from the graph data to be processed based on the updated labels of nodes in the compressed graph; and updating the labels of the nodes in the graph data to be processed corresponding to the updated labels to mining labels.

[0148] like Figure 11As shown, if label 1 in the compressed graph is used as the mining label, then the mining labels of nodes 901, 903, 904, and 905 are also 1. Query the nodes corresponding to nodes 901, 903, 904, and 905 from the graph data to be processed, and update the labels of the corresponding nodes to 1. Then, the graph data formed by the nodes with label 1 in the graph data to be processed is the connected component of the graph data to be processed.

[0149] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0150] Connected component mining has been widely applied in fields such as social networks, financial risk control, and cybersecurity. For example, when applied to social networks, nodes in graph data represent users in the social network. By mining connected components from the graph data, it can be determined that the users corresponding to the connected components have social relationships. In financial risk control products, connected component mining is used to detect abnormal tasks such as illegal lending, cash-out schemes, multiple borrowing, and gambling. Nodes in graph data represent user accounts. By mining connected components from the graph data, it can be determined that the users corresponding to the connected components have close relationships. When a user in a connected component exhibits abnormal behavior, other users in the connected component can be quickly located, accelerating the detection of abnormal tasks.

[0151] Connected component mining methods include single-machine connected component mining methods based on disjoint-set data structures and distributed connected component mining methods based on label propagation. However, both of these methods have their own problems, as shown below:

[0152] 1) The single-machine connected component mining method based on disjoint set storage stores the intermediate results of all node traversal in memory and maintains a disjoint set. This method is limited by memory space and cannot handle large-scale graphs.

[0153] 2) Transforming the single-machine connected component mining method based on disjoint set into a distributed method results in a single device (driver) needing to process a large amount of data. Due to the single-point bottleneck of the driver, it is still unable to process large-scale graphs.

[0154] 3) In the label-based distributed connected component mining method, each node communicates with its neighbor nodes once per iteration, and each node's label expands outward by at most one step in each iteration. Therefore, when the diameter of the graph is large, the number of iterations will become very large, affecting the overall computation time.

[0155] 4) The label-based distributed connected component mining method is a synchronous algorithm. Each information processing step processes the information sent in the previous iteration. The information updated and calculated in the current iteration can only be used in the next iteration.

[0156] To address the aforementioned issues, this application proposes a graph data processing method. First, to solve the problem of large-scale connected component mining, a distributed parallel computing logic is used. Asynchronous distributed computing is implemented based on Spark On Angel, and batch processing is added. This allows data to be processed in batches (label updates) in each iteration, and the asynchronous nature allows for the early acquisition of labels that will be available in the next iteration, thereby accelerating label propagation. When the original graph is compressed to a suitable size, the distributed computing transitions to single-machine computing, invoking an efficient single-machine disjoint-set data structure algorithm, which significantly improves computational performance. During graph compression, the intermediate results of distributed label propagation during the iteration process are utilized, allowing previous computation results to be used. By leveraging the Spark On Angel platform, the single-point limitation of the Spark driver is eliminated, significantly improving the performance of connected component mining.

[0157] The graph data processing method proposed in this application is described in detail below. The connected component mining is divided into three parts: the overall distributed mining achieved through steps 3-5, the single-machine mining of the compressed graph achieved through step 6, and the final result matching achieved through step 7. The overall steps are as follows:

[0158] Step 1: Split the original graph G (i.e., the graph data to be processed) into multiple partitions based on the node identifier (id). Each partition consists of a node and all its neighbors.

[0159] For example, if the original graph G has 30 nodes, then the nodes with IDs 1-10 and their corresponding neighbors are set as partition 1, the nodes with IDs 11-20 and their corresponding neighbors are set as partition 2, and the nodes with IDs 21-30 and their corresponding neighbors are set as partition 3.

[0160] Step 2: Initialize PS. The initial label for each node is the node ID.

[0161] Step 3: The executor in the distributed system retrieves the labels of all nodes in the current batch and the labels of all neighboring nodes from the PS for each node in the partition graph in batches. Each node in the batch calculates its minimum or maximum label based on its own label and the labels of all its neighboring nodes, and uses this as its new label for label updates. After each batch is completed, the new label is updated on the PS.

[0162] To facilitate understanding, we will first introduce the asynchronous connected component mining algorithm on Spark On Angel:

[0163] Spark On Angel implements distributed connected component algorithms by combining Angel's parameter server (PS), storing temporary node labels on the PS, and retrieving labels from the PS to the local machine when needed for iteration. For example... Figure 12 As shown, at the start of an iteration, the executor corresponding to the partition first pulls the updated labels from the input information block (inMessages) of the PS (Segmentation Graph). After completing the computation locally, it pushes the computation result (updated labels) of this round onto the PS for use in the next iteration. Synchronous updates are achieved by resetting the labels in the PS. After one iteration, the output information block (outMessages) pushed onto the PS in this round replaces the content of the input information block (inMessages) of the previous round, and resets the content of outMessages.

[0164] The connected component mining algorithm based on label propagation is asynchronously compatible because the processing of each piece of information from neighboring nodes in each iteration is a simple comparison. As long as the information is true, the smallest or largest node label will always be retained.

[0165] To achieve asynchronous updates, based on the characteristics of the PS (Physical Component), the logic for updating information on the PS is modified. This application embodiment implements asynchronous mining of connected components. For example... Figure 13 As shown, the original synchronization of inMessages and outMessages is canceled, and only a single message block (Messages) is used to store tags. Whether pushing or pulling messages (i.e., tags), everything is handled through this single data structure. For example, as... Figure 13 As shown, after Executor0 completes the calculation and pushes up the updated label, Executor1 is still in the step of needing to pull the label. At this time, Executor1 may be able to pull the updated label from Executor0, thus enabling it to pull the next round of updated labels in the synchronous state in advance.

[0166] To enhance the asynchronous effect, this embodiment modifies the computation process by updating tags in batches. Each executor processes tags in batches per round, and pushes the processed tags directly to the PS after each batch. This gives each executor multiple opportunities to interact with the PS, increasing the chances of triggering asynchronous tag retrieval. If PS updates are performed in batches, with each batch updating the PS, then when other executors perform PS retrieval operations in parallel, they may retrieve the tags of the latest updated node, effectively retrieval the next round's tags in advance under synchronous conditions. This reduces the number of iteration rounds. Therefore, the more frequent the asynchronous retrieval, the significantly fewer the number of iteration rounds.

[0167] Step 4: Repeat step 3 until the specified number of repetitions is reached.

[0168] Step 5: The executor in the distributed system performs compression. Each partition pulls the labels of all nodes in the current batch and the labels of the node's neighboring nodes from the PS in batches. Iterates through all nodes in the current batch; if a node's label does not match the label of its neighboring node, record this edge in the form of <node label, neighboring node label>, and add this edge as an edge in the new graph (i.e., the compressed graph). After all batches have been executed, the edges contained in the new graph are obtained.

[0169] In this embodiment, graph compression is performed based on intermediate results from iterations (e.g., intermediate results obtained after 3 rounds of iterations). After a finite number of distributed iterations (e.g., 3 rounds), each node in the original graph G has a new label CCid, and many different nodes already have the same CCid. At this point, nodes with the same CCid are treated as a single node, and edges with different CCids at their endpoints are organized into a new graph Gn with CCid as the node label. The size of the new graph Gn is smaller than the size of the original graph G. If the size of the new graph Gn can support single-machine computation, then label updates can be removed from the PS to free it from distributed computing.

[0170] like Figure 9As shown, the disjoint-set data structure (DFS) algorithm is applied to the edges of the compressed graph. Since nodes with the same label CCI d are compressed into a single node, the edges between these compressed nodes can be defined using the edges in the original graph G with different labels at both ends. The new edges are represented in the DFS as a new relation, indicating that the two labels belong to the same DFS. The purpose of compression is to free up the cluster and allow computation to be performed on a single machine, thereby accelerating the computation speed. Therefore, compression is only performed when the size of the compressed graph is small enough to be computed on a single machine, and an efficient disjoint-set data structure-based connected component mining algorithm is directly executed on the new graph.

[0171] Step 6: One of the multiple executors, the master executor, collects the edges of the new graph locally and executes the disjoint-set data structure algorithm locally to obtain the disjoint-set label corresponding to each label (i.e., the disjoint-set label obtained by the disjoint-set data structure algorithm for each node in the new graph).

[0172] Step 7: The executor in the distributed system performs the matching. The executor in the distributed system retrieves the labels of all nodes in the current batch, along with the labels of the node's neighboring nodes, from the PS for each node in the partition graph in batches. Iterates through all nodes in the current batch, updating the node labels to the corresponding disjoint-set data structure labels.

[0173] Step 8: Program ends.

[0174] As can be seen from the above steps, this application's embodiment, for the distributed connected component mining algorithm based on label propagation, achieves asynchronous label updates by adding batch processing. Furthermore, by summarizing and utilizing the intermediate results of the iterative algorithm (e.g., the calculation result of the 3rd iteration in 100 iterations), the graph size is compressed. Combining the final result of the compressed graph (i.e., the calculation result of the compressed graph using the disjoint-set data structure algorithm) with the intermediate results before compression (i.e., the intermediate results of the iterative algorithm), the connected component to which each node ultimately belongs can be quickly obtained. When the input original graph is compressed to a certain extent (e.g., a set threshold for the number of nodes, a set threshold for the number of edges, etc.), the overall computation logic can be switched from distributed computing mode to single-machine computing mode, thereby accelerating algorithm iteration and further reducing resource overhead.

[0175] In summary, the graph data processing method proposed in this application has the following beneficial effects:

[0176] 1. Based on the iterative characteristics of connected component mining, a real-time compression method can be used to compress the graph during iteration, thereby reducing the graph size;

[0177] 2. Based on the distributed mining method of connected components, a batch processing mechanism is added in each round, and an asynchronous mining method is proposed to reduce the number of iterations by utilizing the asynchronous characteristics.

[0178] 3. Improve connected component mining performance by combining the advantages of distributed parallel computing and single-machine computing;

[0179] 4. A weakly connected component mining method based on compression was implemented on the Spark on Angel high-performance graph computing platform. It can support ultra-large-scale networks of hundreds of billions or trillions, with low resource overhead and high performance.

[0180] The exemplary application and implementation of the electronic device provided in the embodiments of this application have been used to describe the graph data processing method provided in the embodiments of this application. The following describes the scheme for the cooperation of various modules in the graph data processing device 555 provided in the embodiments of this application to implement graph data processing.

[0181] The segmentation module 5551 is used to segment the graph data to be processed to obtain multiple segmented graphs of the graph data to be processed.

[0182] Execution module 5552 is used to perform label update processing on nodes in the multiple segmentation graphs based on different batches to obtain updated labels for the nodes in the multiple segmentation graphs; to compress the graph data to be processed based on the updated labels for the nodes in the multiple segmentation graphs to obtain a compressed graph of the graph data to be processed; to perform label mining processing on the nodes in the compressed graph to obtain mining labels corresponding to the nodes in the compressed graph; update module 5553 is used to update the nodes in the graph data to be processed based on the mining labels corresponding to the nodes in the compressed graph, and to take the graph data composed of nodes with the same updated labels as the connected components of the graph data to be processed.

[0183] In some embodiments, the execution module includes multiple sub-execution modules, each sub-execution module corresponding to at least one of the segmentation maps; the sub-execution module is further configured to perform batch label update processing on the nodes in the corresponding segmentation map to obtain updated labels of the nodes in the segmentation map; and send the updated labels of the nodes in the segmentation map to the parameter module.

[0184] In some embodiments, the sub-execution module is further configured to iteratively execute the following processes: perform batch allocation processing on the nodes in the corresponding segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph; obtain the label of any node included in the batch and the labels of the neighboring nodes of the included node from the parameter module; perform label update processing on the label of any node included in the batch based on the labels of the neighboring nodes to obtain the updated label of any node included in the batch; stop the iterative processing when the number of iterations is a set number; and send the updated label of any node included in the batch to the parameter module.

[0185] In some embodiments, the sub-execution module is further configured to perform random batch allocation processing on the nodes in the corresponding segmentation graph to obtain the batch corresponding to the nodes in the segmentation graph; or, to perform uniform batch allocation processing on the nodes in the corresponding segmentation graph to obtain the batch corresponding to the nodes in the segmentation graph.

[0186] In some embodiments, the sub-execution module is further configured to perform the following processing for any node included in any batch: determine the smallest or largest label from the label of the node and the labels of the node's neighboring nodes; and use the smallest or largest label as the updated label of the node.

[0187] In some embodiments, the sub-execution module is further configured to obtain from the parameter module the update tags of multiple nodes included in the batch and the update tags of the neighboring nodes of the included nodes; and to compress the graph data to be processed based on the update tags of the multiple nodes included in the batch and the update tags of the neighboring nodes to obtain a compressed graph of the graph data to be processed.

[0188] In some embodiments, the sub-execution module is further configured to perform node combination processing based on the updated labels of the nodes included in the multiple batches and the updated labels of the neighboring nodes to obtain the segmentation graph with updated labels corresponding to the execution module; when the updated labels of adjacent nodes in the segmentation graph with updated labels corresponding to the execution module are different, the edge between the adjacent nodes is used as the edge of the compressed graph of the graph data to be processed, and the adjacent node is used as the node of the compressed graph of the graph data to be processed.

[0189] In some embodiments, the execution module includes a main execution module, which is configured to obtain nodes and corresponding edges in the compressed graph from the plurality of execution modules; perform tree structure construction processing based on the nodes and corresponding edges in the compressed graph to obtain a tree structure corresponding to the compressed graph; determine the target label of the tree structure from a plurality of labels in the tree structure; and use the target label as the mining label corresponding to the node in the compressed graph; wherein the main execution module is one of the plurality of execution modules that are pre-configured, or the plurality of execution modules are generated through an election operation.

[0190] In some embodiments, the main execution module is further configured to obtain the root node of the tree structure from a plurality of nodes in the compressed graph; take the nodes in the compressed graph that have an edge with the root node as the leaf nodes of the root node; take the nodes in the compressed graph that have an edge with the leaf node as the leaf nodes of the leaf nodes; and construct the tree structure corresponding to the compressed graph based on the root node, the leaf nodes, and the leaf nodes of the leaf nodes.

[0191] In some embodiments, the main execution module is further configured to randomly select a label from a plurality of labels in the tree structure and use it as the target label of the tree structure; or, to use the label corresponding to the root node in the tree structure as the target label of the tree structure.

[0192] In some embodiments, the update module 5553 is further configured to obtain the node corresponding to the update label from the graph data to be processed based on the update label of the node in the compressed graph; and update the label of the node corresponding to the update label in the graph data to be processed to the mining label.

[0193] In some embodiments, the segmentation module 5551 is further configured to perform random segmentation processing on the graph data to be processed based on the initial labels of the nodes in the graph data to be processed, to obtain multiple segmented graphs of the graph data to be processed; or, to perform uniform segmentation processing on the graph data to be processed based on the initial labels of the nodes in the graph data to be processed, to obtain multiple segmented graphs of the graph data to be processed.

[0194] In some embodiments, the segmentation module 5551 is further configured to display graph data to be processed; in response to a segmentation operation on the graph data to be processed, the segmented portion of the graph data to be processed is used as the segmented graph.

[0195] In some embodiments, the segmentation module 5551 is further configured to display a segmentation trigger entry for the graph data to be processed; in response to the trigger operation for the segmentation trigger entry, based on the characteristics of the nodes in the graph data to be processed, cluster the nodes in the graph data to be processed, and use the graph data composed of nodes belonging to the same category as the segmented graph.

[0196] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the graph data processing method described above in this application.

[0197] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the graph data processing method provided in this application. For example, ... Figure 1 The diagram shows a method for processing graph data.

[0198] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0199] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0200] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0201] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0202] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A graph data processing method, characterized in that, The method includes: The graph data to be processed is segmented to obtain multiple segmented graphs of the graph data to be processed. The nodes in the multiple segmentation graphs are updated in batches to obtain the updated labels of the nodes in the multiple segmentation graphs. Obtain the update tags of the nodes included in multiple batches, as well as the update tags of the neighboring nodes of the included nodes, from the parameter server; Based on the update tags of the nodes included in the multiple batches and the update tags of the neighboring nodes, the graph data to be processed is compressed to obtain a compressed graph of the graph data to be processed. The compression process includes at least compressing nodes with the same update tag into one node. The main executor obtains the nodes and corresponding edges in the compressed graph from multiple executors, and performs tree structure construction based on the nodes and corresponding edges in the compressed graph to obtain the tree structure corresponding to the compressed graph; wherein, the main executor is one of multiple pre-configured executors, or the multiple executors are generated through an election operation; From the multiple labels in the tree structure, determine the target label of the tree structure, and use the target label as the mining label corresponding to the node in the compressed graph; Based on the update labels of the nodes in the compressed graph, obtain the nodes corresponding to the update labels from the graph data to be processed; Update the labels of the nodes in the graph data to be processed that correspond to the updated labels to the mining labels; The graph data consisting of nodes with the same mining label after the update in the graph data to be processed is taken as the connected component of the graph data to be processed.

2. The method according to claim 1, characterized in that, The label update process is implemented through a distributed system, which includes the parameter server and the plurality of executors, each executor corresponding to at least one of the segmentation graphs; The step of performing batch label update processing on the nodes in the multiple segmentation graphs to obtain the updated labels of the nodes in the multiple segmentation graphs includes: The executor performs batch label update processing on the corresponding nodes in the segmentation graph to obtain the updated labels of the nodes in the segmentation graph; The method further includes: The updated labels of the nodes in the segmentation graph are sent to the parameter server.

3. The method according to claim 2, characterized in that, The executor performs batch label update processing on the corresponding nodes in the segmentation graph to obtain the updated labels of the nodes in the segmentation graph, including: The following processing is performed iteratively: The executor performs batch allocation processing on the corresponding nodes in the segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph; Obtain the label of any node included in the batch, and the labels of the neighboring nodes of the included node, from the parameter server; Based on the labels of the neighboring nodes, the labels of any node included in the batch are updated to obtain the updated labels of any node included in the batch. The iteration process stops when the number of iterations reaches the set number. Sending the updated labels of the nodes in the segmentation graph to the parameter server includes: Send the update tags of any node included in the batch to the parameter server.

4. The method according to claim 3, characterized in that, The executor performs batch allocation processing on the corresponding nodes in the segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph, including: The actuator performs batch random allocation processing on the corresponding nodes in the segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph; or, The actuator performs batch uniform distribution processing on the nodes in the corresponding segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph.

5. The method according to claim 3, characterized in that, The step of updating the labels of nodes in any batch based on the labels of the neighboring nodes to obtain updated labels for the nodes in any batch includes: Perform the following processing for any of the nodes contained in any of the batches: Determine the smallest or largest label from the label of the node and the labels of the node's neighboring nodes; The smallest or the largest label is used as the updated label of the node.

6. The method according to claim 1, characterized in that, The step of compressing the graph data to be processed based on the update tags of the nodes included in the multiple batches and the update tags of the neighboring nodes to obtain a compressed graph of the graph data to be processed includes: The executor performs node combination processing based on the updated labels of the nodes included in the multiple batches and the updated labels of the neighboring nodes to obtain the segmentation map with updated labels corresponding to the executor. When the updated labels of adjacent nodes in the segmentation graph corresponding to the executor are different, the edge between the adjacent nodes is used as the edge of the compressed graph of the graph data to be processed, and the adjacent node is used as the node of the compressed graph of the graph data to be processed.

7. The method according to claim 1, characterized in that, The process of constructing a tree structure based on the nodes and corresponding edges in the compressed graph to obtain the tree structure corresponding to the compressed graph includes: Obtain the root node of the tree structure from multiple nodes in the compressed graph; The nodes in the compressed graph that have an edge with the root node are designated as leaf nodes of the root node. The node in the compressed graph that has an edge with the leaf node is designated as the leaf node of the leaf node; The tree structure corresponding to the compressed graph is constructed based on the root node, the leaf node, and the leaf nodes of the leaf node.

8. The method according to claim 1, characterized in that, Determining the target label of the tree structure from multiple labels in the tree structure includes: Randomly select one label from multiple labels in the tree structure and use it as the target label of the tree structure; or, The label corresponding to the root node in the tree structure is used as the target label of the tree structure.

9. The method according to claim 1, characterized in that, The process of segmenting the graph data to be processed yields multiple segmented graphs of the graph data to be processed, including: Based on the initial labels of the nodes in the graph data to be processed, the graph data to be processed is randomly segmented to obtain multiple segmented graphs; or... Based on the initial labels of the nodes in the graph data to be processed, the graph data to be processed is uniformly divided to obtain multiple segmented graphs of the graph data to be processed.

10. The method according to claim 1, characterized in that, The process of segmenting the graph data to be processed yields multiple segmented graphs of the graph data to be processed, including: Display the graph data to be processed; In response to a segmentation operation on the graph data to be processed, the segmented portion of the graph data to be processed is taken as the segmented graph.

11. The method according to claim 1, characterized in that, The process of segmenting the graph data to be processed yields multiple segmented graphs of the graph data to be processed, including: Displays the segmentation trigger entry for the graph data to be processed; In response to the trigger operation for the segmentation trigger entry, based on the characteristics of the nodes in the graph data to be processed, the nodes in the graph data to be processed are clustered, and the graph data composed of nodes belonging to the same category is used as the segmentation graph.

12. A graph data processing apparatus, characterized in that, The device includes: The segmentation module is used to segment the graph data to be processed, thereby obtaining multiple segmented graphs of the graph data to be processed. An execution module is configured to perform batch-based label update processing on nodes in the multiple segmentation graphs to obtain updated labels for the nodes in the multiple segmentation graphs; obtain updated labels for nodes included in multiple batches and updated labels for neighboring nodes of the included nodes from a parameter server; compress the graph data to be processed based on the updated labels for nodes included in multiple batches and the updated labels for neighboring nodes to obtain a compressed graph of the graph data to be processed, wherein the compression processing includes at least compressing nodes with the same updated label into one node; obtain nodes and corresponding edges in the compressed graph from multiple executors through a main executor, and construct a tree structure based on the nodes and corresponding edges in the compressed graph to obtain a tree structure corresponding to the compressed graph; wherein the main executor is one of multiple pre-configured executors, or the multiple executors are generated through an election operation; determine the target label of the tree structure from multiple labels in the tree structure, and use the target label as the mining label corresponding to the node in the compressed graph; An update module is used to obtain nodes corresponding to the update labels from the graph data to be processed based on the update labels of the nodes in the compressed graph; update the labels of the nodes corresponding to the update labels in the graph data to be processed to the mining labels; and use the graph data composed of nodes with the same mining labels after the update as the connected components of the graph data to be processed.

13. The apparatus as claimed in claim 12, characterized in that, The tag update process is implemented through a distributed system, which includes a parameter server and the plurality of executors, each executor corresponding to at least one of the segmentation graphs; The execution module includes multiple sub-execution modules, which are used to: perform batch label update processing on the corresponding nodes in the segmentation graph to obtain the updated labels of the nodes in the segmentation graph; and send the updated labels of the nodes in the segmentation graph to the parameter server.

14. The apparatus as claimed in claim 13, characterized in that, The sub-execution module is further configured to: The following processes are performed iteratively: the executor performs batch allocation processing on the corresponding nodes in the segmentation graph to obtain the batches corresponding to the nodes in the segmentation graph; and obtains the labels of the nodes included in any batch and the labels of the neighboring nodes of the included nodes from the parameter server. Based on the labels of the neighboring nodes, the labels of any node included in the batch are updated to obtain the updated labels of any node included in the batch; the iteration process is stopped when the number of iterations reaches a set number; and the updated labels of any node included in the batch are sent to the parameter server.

15. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the graph data processing method according to any one of claims 1 to 11.

16. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the graph data processing method according to any one of claims 1 to 11 when executed by a processor.

17. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the graph data processing method according to any one of claims 1 to 11.

Images