HIDE and search set data processing method based on distributed architecture

Abstract

The application discloses a kind of HIDE and inquiry set data processing methods based on distributed architecture, comprising, according to the atomization condition such as date and entity type from data source acquisition and processing data, generate the atomization DAG edge set;Edge set is formed by HashBucket algorithm and is divided into bucket, forms pre-computation cache;Based on distributed main node and cache, generate atomization condition set, and it is balancedly divided into n task groups using knapsack algorithm, and is distributed to working node;Each node locally calculates and generates simplified DAG equivalent subgraph, is sent to main node and is connected, forms global connection DAG;Again, perform disjointed set operation, output only contains the final connected component of target object;The application avoids distributed lock predicament, realizes the unity of efficiency and correctness, significantly improves calculation real-time performance.

Description

Technical Field

[0001] This invention relates to the fields of graph theory and distributed technology, and in particular to a HIDE disjoint-setup data processing method based on a distributed architecture. Background Technology

[0002] In the digital economy era, the accelerated integration of fintech and financial markets has not only changed transaction models and improved market efficiency, but also made potential illegal and irregular activities increasingly concealed and complex, posing new challenges to market supervision. As a core component of market supervision, the monitoring of suspected related accounts focuses on identifying potential connections between accounts from massive amounts of multi-dimensional data, including account registration information, behavioral characteristics, funding sources, and transaction terminals. During the monitoring process, any accounts with directly or indirectly consistent information (such as the same contact information, devices, or funding channels) are grouped into the same related group. This grouping mathematically corresponds to the disjoint set problem. Its core business rule is that accounts in the same group must be related, while accounts in different groups are not related. However, actual business involves frequent and iterative exploration through repeated screening. The topological graph of accounts and relationships faced in a single monitoring session is extremely large, with millions of nodes (accounts) and billions of edges (relationships). Traditional technologies struggle to efficiently complete calculations at this scale. More importantly, excessively long computation times severely delay regulatory response, rendering risk monitoring ineffective. Therefore, how to complete the calculation of disjoint sets of ultra-large-scale topological graphs within a limited time has become a core technical bottleneck that urgently needs to be solved in the field of financial account supervision.

[0003] Currently, the industry standard algorithm for solving the disjoint set problem is the disjoint-set data structure (Union-Find) algorithm. Its core lies in establishing and dynamically maintaining a directed acyclic graph index. Through two operations, Find (finding the root node) and Union (merging sets), related nodes are gradually merged into sets represented by the same root node. According to existing theoretical and practical research, the lower bound of the computational complexity of the disjoint-set algorithm is no less than O(n). This means that in a single-process environment, the computation time is insufficient to meet the real-time requirements of business operations when dealing with graphs with hundreds of millions of edges. Therefore, distributing computational tasks has become an inevitable direction for technological evolution.

[0004] Existing distributed disjoint-set data structure schemes mainly follow two classic distributed locking strategies, but both have serious drawbacks: The optimistic locking strategy involves worker nodes writing directly to the DAG index without checking for conflicts. Since DAG indexes are inherently unsafe across multiple processes, this strategy leads to inconsistent root node attribution, unreliable calculation results, and an inability to meet the stringent accuracy requirements of business applications.

[0005] The pessimistic locking strategy requires all index update operations to be processed serially through the central master node to ensure the consistency of the DAG index. While this guarantees the correctness of the results, it concentrates the majority of the computational pressure and communication overhead on the master node, making it the system bottleneck. This approach is essentially a "pseudo-distributed" or load balancing solution, and its performance cannot be effectively improved by simply adding more hardware resources; it also has poor scalability.

[0006] In summary, traditional solutions treat the entire topology graph as a homogeneous whole, failing to address the inherent characteristics of financial linked account topologies through distributed design. This leads to the inherent dilemma of distributed locks. In computations of disjoint sets in ultra-large-scale graphs, either correctness (optimistic locking) is sacrificed for speed, or parallel efficiency is sacrificed (pessimistic locking) to ensure correctness. This results in a contradiction between computational efficiency and accuracy, and limited distributed scalability, making it difficult to meet the urgent needs of fields such as fintech regulation for real-time and accurate correlation analysis of massive relational data. Summary of the Invention

[0007] This invention provides a HIDE disjoint-setup data processing method based on a distributed architecture to overcome the above-mentioned technical problems.

[0008] A HIDE disjoint-set data processing method based on a distributed architecture, applied in a distributed cluster to query and simplify large-scale time series graph data, is characterized by comprising: S1: Set atomization conditions, collect raw relational data from the data source according to the atomization conditions, and obtain an atomized dataset; the atomization conditions include date fragments and non-target object entity types; S2: Perform disjoint set operations on the atomic dataset, extract the DAG index, and convert it into a DAG edge set; S3: Based on the scale attribute of non-target object entity types in the atomic dataset, the HashBucket algorithm is introduced to perform hash value bucketing on the DAG edge set that meets the bucketing condition according to the hash value of the non-target object node. The bucketing result is stored with the corresponding atomicity condition and the DAG edge set that does not meet the bucketing condition and the corresponding atomicity condition to form a pre-computation cache set. S4: Based on preset grouping conditions, calculate the target cache block set based on the master node in the distributed cluster and the pre-computed cache set to generate an atomic condition set; introduce a knapsack algorithm to divide the atomic condition set into n graph-scale balanced task groups according to the number of DAG edges corresponding to each target cache block, and assign them to n working nodes in the distributed cluster for processing. S5: For each working node, load all DAG edge sets corresponding to the assigned task group, perform local fusion and calculation on the DAG edge sets, generate a simplified DAG equivalent subgraph containing only the target object node, and send it to the master node; S6: The master node collects the DAG equivalent subgraphs sent by all worker nodes, connects all DAG equivalent subgraphs to form a connected DAG graph; performs the disjoint set operation on the connected DAG graph again, and outputs the final connected component result containing only the target object.

[0009] Furthermore, disjoint set operations are performed on the atomic dataset to extract the DAG indexes and convert them into DAG edge sets, including: For each of the atomic datasets, perform the following sub-steps: The Union-Find algorithm is executed on the atomic dataset to obtain the locally connected components of all nodes in the dataset. Traverse the locally connected components and generate a DAG structure for each connected component. According to a preset selection rule, a representative node of the connected component is taken as the root node and the remaining nodes are taken as child nodes, forming a set of directed edges from the child nodes to the root node, i.e., the DAG index. The set of directed edges of all connected components is aggregated to form a DAG edge set representing the overall connectivity of the atomized dataset.

[0010] Furthermore, based on the size attribute of non-target object entity types in the atomic dataset, it is determined whether the preset bucketing conditions are met, including: Obtain the historical data volume of the non-target object entity type node; if the historical data volume exceeds a preset threshold, it is determined that the bucketing condition is met; otherwise, it is determined that the bucketing condition is not met.

[0011] Furthermore, the HashBucket algorithm is introduced to bucket the DAG edge set according to the hash values ​​of non-target object nodes, and the results are stored along with the corresponding atomicity conditions, including: A preset number of buckets K is used to traverse each directed edge in the DAG edge set; Perform a hash calculation on the non-target node associated with the edge, and then take the modulo of K to obtain the hash value; The directed edge whose hash value is the same as the current hash bucket number is assigned to the corresponding hash bucket, generating K DAG edge sets; Each DAG edge set and its corresponding atomization condition are packaged and stored.

[0012] Furthermore, based on preset grouping conditions, the master node in the distributed cluster calculates the target cache block set according to the pre-computed cache set, generating an atomic condition set, including: The target time range is obtained by parsing the grouping conditions. Non-target object entity type collection ; The target time range is discretized by natural days to obtain a set of time dimensions; Perform a Cartesian product operation between the time dimension set and the non-target object entity type set to generate an atomic condition set.

[0013] Furthermore, a knapsack algorithm is introduced to divide the atomicity condition set into n graph-scale balanced task groups based on the number of DAG edges corresponding to each target cache block, including: For each atomization condition in the set of atomization conditions, query the estimated number of edges in the corresponding DAG edge set, and use it as the weight of the condition; Using the weights as input, the number of task groups n is the number of knapsacks, and the knapsack algorithm is executed with the goal of minimizing the difference between the total weights of each knapsack, i.e., the total weights of the task groups. Output n graph-scale balanced task groups, where each task group contains atomic conditions of the same type but on different dates; each subset is assigned as a computation task group to a worker node.

[0014] Furthermore, for each working node, all DAG edge sets corresponding to the assigned task group are loaded, and the DAG edge sets are locally fused and calculated to generate a simplified DAG equivalent subgraph containing only the target object nodes, which is then sent to the master node, including: Work nodes read the DAG edge sets corresponding to all atomization conditions within their task groups from storage in parallel; The read DAG edge sets are concatenated to form a local joint graph. ; The Union-Find algorithm is executed on the local union graph, based on the local union graph at each working node. Calculate the locally connected components to obtain the equivalent subgraph of the local DAG; From the equivalent local DAG subgraph, delete all nodes whose node type is not the target object and their associated edges to obtain a simplified DAG subgraph containing only the target object nodes. The simplified DAG subgraph is sent to the master node.

[0015] Furthermore, the master node collects the DAG equivalent subgraphs sent by all worker nodes, connects all DAG equivalent subgraphs to form a connected DAG graph; the disjoint set operation is then performed on the connected DAG graph again, outputting only the final connected component result of the target object, including: S61. Initialize the global simplified union graph structure and the global result queue; S62. The master node asynchronously receives the simplified DAG subgraphs returned by each working node and stores the edge set of each simplified DAG subgraph received into the global result queue. S63. The master node starts an incremental merge loop. When the global result queue is not empty and the number of processed results has not reached the total number of worker nodes, the following operations are performed: S631: Extract the edge set of the simplified DAG subgraph from the global result queue in sequence according to the time series; S632: Merge the extracted edge set with the current global simplified joint graph structure, and update the global simplified joint graph; S64. After the simplified DAG subgraph edge sets of all working nodes are merged, execute the Union-Find algorithm on the final global simplified joint graph to calculate the final connectivity between all target object nodes. S65. Output the final connectivity relationship in the form of a disjoint-set data structure, as the final connected component result.

[0016] Beneficial effects: This invention provides a HIDE disjoint-setup data processing method based on a distributed architecture, which has the following advantages: 1. This invention avoids the dilemma of distributed locks, achieving a balance between efficiency and correctness: Through two core paths—"pre-computation caching" and "hierarchical graph simplification"—this invention compresses and simplifies data before distributed computation. By introducing hash bucketing to pre-organize and cache the DAG edge set, this invention preprocesses massive amounts of raw relational data in the time dimension (outside of peak business periods), significantly reducing the real-time computation load in the response phase. By dividing the original topological graph G into several subgraphs, the equivalent DAG index of each subgraph is calculated in parallel on each worker node. In each equivalent sub-DAG, each node retains only one edge pointing to its root node, thus significantly compressing the edge set of the subgraph. This eliminates the need for complex edge dependency negotiation and synchronization between worker nodes, completely avoiding the optimistic or pessimistic locking mechanisms necessary to maintain consistency in traditional distributed disjoint-setup algorithms. Only the simplified, edge-free subgraphs need final aggregation computation, thereby improving parallel processing efficiency while ensuring that the computation result is mathematically equivalent to the global computation. 2. This invention significantly improves real-time computation performance: By partitioning tasks into graph-scale balanced components, this invention ensures load balancing among worker nodes, resulting in an approximately linear decrease in the overall system response time as the number of worker nodes increases. This meets the urgent needs of fields such as fintech regulation for real-time and near-real-time correlation analysis of massive amounts of data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a HIDE disjoint-setup data processing method based on a distributed architecture provided by the present invention; Figure 2 This is a diagram of the architecture of the HIDE disjoint-set data structure of this invention; Figure 3 This is a schematic diagram illustrating the verification results of the original disjoint sets. Figure 4 This is a schematic diagram illustrating the verification results of the disjoint set of the HIDE disjoint set in this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This embodiment provides a HIDE disjoint-setup data processing method based on a distributed architecture, such as... Figure 1 As shown, it includes: S1: Set atomization conditions, collect raw relational data from the data source according to the atomization conditions, and obtain an atomized dataset; the atomization conditions include date fragments and non-target object entity types; S2: Perform disjoint set operations on the atomic dataset, extract the DAG index, and convert it into a DAG edge set; S3: Based on the scale attribute of non-target object entity types in the atomic dataset, the HashBucket algorithm is introduced to perform hash value bucketing on the DAG edge set that meets the bucketing condition according to the hash value of the non-target object node. The bucketing result is stored with the corresponding atomicity condition and the DAG edge set that does not meet the bucketing condition and the corresponding atomicity condition to form a pre-computation cache set. S4: Based on preset grouping conditions, calculate the target cache block set based on the master node in the distributed cluster and the pre-computed cache set to generate an atomic condition set; introduce a knapsack algorithm to divide the atomic condition set into n graph-scale balanced task groups according to the number of DAG edges corresponding to each target cache block, and assign them to n working nodes in the distributed cluster for processing. S5: For each working node, load all DAG edge sets corresponding to the assigned task group, perform local fusion and calculation on the DAG edge sets, generate a simplified DAG equivalent subgraph containing only the target object node, and send it to the master node; S6: The master node collects the DAG equivalent subgraphs sent by all worker nodes, connects all DAG equivalent subgraphs to form a connected DAG graph; performs the disjoint set operation on the connected DAG graph again, and outputs the final connected component result containing only the target object.

[0021] Specifically, the panoramic view of the HIDE disjoint-setup is as follows: Figure 2 As shown, the left part is the cache part corresponding to the time-dimensional strategy, and the right part is the response part corresponding to the spatial-dimensional strategy.

[0022] For the time-dimensional strategy, preparations will be made for the spatial-dimensional caching. The principle is to bucket the edges in the topology graph based on non-TOI nodes. This ensures that edges with the same non-TOI node as one of their endpoints are assigned to the same bucket, forming an atomic condition set, and ensures that the intersection between any two buckets is only TOI nodes.

[0023] For spatial dimension strategies, the topology graph will be spatially distributed among the servers in the distributed cluster. The computation of all spatial dimension strategies will occur after the group partitioning conditions are given, also known as the response part.

[0024] Therefore, in the spatial dimension strategy, the worker nodes of the distributed cluster can arbitrarily select several atomic condition sets, then merge and calculate the equivalent subgraph of the DAG, and delete non-TOI nodes. This allows the computational pressure to be distributed among the worker nodes of the distributed cluster, greatly reducing the workload of the master node while ensuring the correctness of the final disjoint set result.

[0025] In a specific embodiment, atomication conditions are set, and raw relational data is collected from the data source according to the atomication conditions to obtain an atomic dataset; the atomication conditions include a date segment dimension and a non-target object entity type dimension as follows: The atomicity condition is a tuple. In this embodiment, the atomicity condition is set to <entity, date>. The entity refers to a non-target object entity type, representing entity types in the original business table other than the core analysis object, such as IP address, device number, location information, etc. This is the key to utilizing the "heterogeneous topology" feature, where "Type of Interest (TOI)" is the core analysis object (such as an account), and "Non-TOI" is the entity it refers to. Data is collected from the SQL database based on the atomicity condition; the query results are constructed into a temporary topological subgraph. In this subgraph, each customer ID and IP address is a node, and a record between them constitutes an edge connecting the two nodes. This subgraph naturally follows the "heterogeneous topology logic," meaning that edges only exist between customer nodes (TOI) and IP nodes (Non-TOI), and IP nodes are not directly connected to each other. Since business data typically originates from relational databases, its abstracted topology graph naturally follows "heterogeneous topology logic." For any topology graph abstracted from a relational database, nodes can be divided into two types: types of interest (TOI) abstracted from the primary key column and types of non-interest (non-TOI) abstracted from other columns. In the topology graph... In the middle, the set of TOI nodes is The set of non-TOI nodes is The set of all nodes is the union of these two subsets: .at the same time, .

[0026] TOI node Non-TOI nodes The topological differences between them are as follows: , ,in and yes The two endpoints, then and Not all nodes will be non-TOI nodes. In other words, there will be no edge between two non-TOI nodes.

[0027] Finally, only for the final disjoint sets We are interested in the connectivity between TOI nodes. That is, only TOI nodes. Must be included middle.

[0028] The aforementioned topological properties are called heterogeneous topological logic. A topological graph that satisfies homogeneous topological logic is called a heterogeneous topological graph. Since heterogeneous topological logic exists in production data, this embodiment will utilize this topological characteristic to further optimize the computational burden on the master node during the distributed disjoint-setup process. In this embodiment, "customer" is defined as a "Type of Interest (TOI)" node, while "IP address," "device number," etc., are defined as "Non-TOI" nodes. According to "heterogeneous topological logic," edges only exist between TOI nodes and non-TOI nodes, and will not appear between two non-TOI nodes. This data structure characteristic is the prerequisite and application basis for subsequent efficient optimization.

[0029] In a specific embodiment, the scheme for performing disjoint set operations on the atomic dataset, extracting the DAG index, and converting it into a DAG edge set is as follows: For each of the atomic datasets, perform the following sub-steps: The Union-Find algorithm is executed on the atomic dataset to obtain the locally connected components of all nodes in the dataset. Traverse the locally connected components and generate a DAG structure for each connected component. According to a preset selection rule, a representative node of the connected component is taken as the root node and the remaining nodes are taken as child nodes, forming a set of directed edges from the child nodes to the root node, i.e., the DAG index. Specifically, the root node selection rule in this embodiment can be determined according to a specific disjoint-set data structure algorithm. For example, the smaller or larger one can be selected, or the one with more nodes pointing to the root node can be selected. Therefore, the specific selection rule is not limited, and those skilled in the art can set it according to the actual situation. The set of directed edges of all connected components is aggregated to form a DAG edge set representing the overall connectivity of the atomized dataset.

[0030] DAG equivalent graph: for any topological graph ,in , , , , Represents a set of nodes. Represents an edge set. for The DAG index. Then there exists Lemma 1: Topological graph DAG equivalent graph Disjoint sets are equivalent.

[0031] This lemma offers two advantages: low computational cost and efficient distributed merging. If two disjoint-set data structure operations are performed, the second disjoint-set data structure can be represented by a DAG equivalent graph. We can directly compute disjoint sets. On the one hand, according to Lemma 1, it is equivalent to computing from the original topological graph. The resulting disjoint sets. On the other hand, since the computational complexity of the disjoint-set data structure is close to... Obviously, in most topological graphs middle, Usually greater than or even much greater than However, in middle, Usually equal to or even less than Then it only costs The time. Therefore, using The second disjoint set not only yields an equivalent disjoint set, but also takes significantly less time than the first disjoint set.

[0032] This scheme, based on Lemma 1, abstracts a new graph from the DAG index of a topological graph G (whose edge set consists of edges from each node to its root node), whose disjoint set is completely equivalent to the original graph G. Therefore, by extracting and storing the DAG index, significant data compression is achieved for the first time while preserving all connectivity information of the original graph, laying the foundation for caching and transmission.

[0033] In a specific embodiment, based on the size attribute of non-target object entity types in the atomic dataset, it is determined whether the preset bucketing conditions are met. If so, the HashBucket algorithm is introduced to perform hash value bucketing on the DAG edge set according to the hash value of the non-target object nodes, and the results are stored along with the corresponding atomicity conditions. If not, the DAG edge set and atomicity conditions are directly stored to form a pre-computed cache set. Obtain the historical data volume of the non-target object node entity type; if the historical data volume exceeds a preset threshold, it is determined that the bucketing condition is met; otherwise, it is determined that the bucketing condition is not met. Under the condition of satisfying bucketing, the HashBucket algorithm is introduced to perform hash value bucketing on the DAG edge set according to the hash value of non-target object nodes, and then store it with the corresponding atomicity conditions, including: A preset number of buckets K is used to traverse each directed edge in the DAG edge set; Perform a hash calculation on the non-target node associated with the edge, and then take the modulo of K to obtain the hash value; The directed edge whose hash value is the same as the current hash bucket number is assigned to the corresponding hash bucket, generating K DAG edge sets; Each DAG edge set and its corresponding atomization condition are packaged and stored.

[0034] In this scheme, nodes with huge amounts of data are bucketed to ensure that subsequent distributed computing meets the disjointness requirement.

[0035] In a specific embodiment, based on preset grouping conditions, the master node in the distributed cluster calculates the target cache block set according to the pre-computed cache set to generate an atomic condition set; the knapsack algorithm is introduced to divide the atomic condition set into n graph-balanced task groups according to the number of DAG edges corresponding to each target cache block, and the groups are then distributed to n worker nodes in the distributed cluster for processing. The target time range is obtained by parsing the grouping conditions. and the collection of target and non-target object entity types ; The target time range is discretized by natural days to obtain a set of time dimensions; Perform a Cartesian product operation between the time dimension set and the target / non-target object entity type set to generate an atomic condition set. The knapsack algorithm is introduced to divide the atomicity condition set into n graph-balanced task groups based on the number of edges in the DAG corresponding to each target cache block, including: For each atomization condition in the set of atomization conditions, query the estimated number of edges in the corresponding DAG edge set, and use it as the weight of the condition; Using the weights as input, the number of task groups N is the number of knapsacks, and the knapsack algorithm is executed with the goal of minimizing the difference between the total weights of each knapsack, i.e., the total weights of the task groups. Output n graph-scale balanced task groups, where each task group contains atomic conditions of the same type but on different dates; each subset is assigned as a computation task group to a worker node.

[0036] In a specific embodiment, for each working node, the following scheme is employed: loading all DAG edge sets corresponding to the assigned task group, performing local fusion and calculation on the DAG edge sets, generating a simplified DAG equivalent subgraph containing only the target object node, and sending it to the master node: Work nodes read the DAG edge sets corresponding to all atomization conditions within their task groups from storage in parallel; The read DAG edge sets are concatenated to form a local joint graph. ; The Union-Find algorithm is executed on the local union graph, based on the local union graph at each working node. Calculate the locally connected components to obtain the equivalent subgraph of the local DAG; From the equivalent local DAG subgraph, delete all nodes whose node type is not the target object and their associated edges to obtain a simplified DAG subgraph containing only the target object nodes. The simplified DAG subgraph is sent to the master node.

[0037] Remove all edges whose endpoints are non-target object (TOI) nodes to generate a simplified subgraph containing only TOI nodes. This step is based on Theorem 2: If and only if the original topology is isolated according to the principle of non-TOI node isolation, the original topology graph is... Split into subgraphs , Then from the DAG equivalent subgraph Delete non-TOI nodes As This will still result in disjoint sets being equivalent.

[0038] Specifically, , , .but disjoint sets Equivalent to disjoint sets Subgraph splitting based on non-TOI node isolation means: , yes The necessary and sufficient condition for this.

[0039] The value of Theorem 2 lies in its ability to maintain the correctness and equivalence of the final disjoint set result while discarding non-TOI nodes. In terms of algorithm optimization, optimization means less computation. It can effectively address the optimization bottleneck of HIDE disjoint-set data structures and significantly improve the upper bound of the algorithm.

[0040] Since the "non-TOI node isolation" condition has been ensured through hash bucketing during the caching phase, all non-TOI nodes and their associated edges can be safely deleted from the subgraphs assigned to worker nodes that satisfy this condition. According to Theorem 2, this operation will not affect the correctness of the final connected components between TOI nodes. This step is key to overcoming the performance bottleneck. It performs a second and more thorough compression of the data size (especially the number of edges) before distributed merging, making the amount of data that needs to be aggregated to the master node extremely small, thereby solving the computational pressure bottleneck of the master node in traditional distributed disjoint-set data structures.

[0041] In a specific embodiment, the master node collects the DAG equivalent subgraphs sent by all worker nodes, connects all DAG equivalent subgraphs to form a connected DAG graph, and then performs a disjoint set operation on the connected DAG graph to output only the final connected component result of the target object. S61. Initialize the global simplified union graph structure and the global result queue; S62. The master node asynchronously receives the simplified DAG subgraphs returned by each working node and stores the edge set of each simplified DAG subgraph received into the global result queue. S63. The master node starts an incremental merge loop. When the global result queue is not empty and the number of processed results has not reached the total number of worker nodes, the following operations are performed: S631: Extract the edge set of the simplified DAG subgraph from the global result queue in sequence according to the time series; S632: Merge the extracted edge set with the current global simplified joint graph structure, and update the global simplified joint graph; S64. After the simplified DAG subgraph edge sets of all working nodes are merged, execute the Union-Find algorithm on the final global simplified joint graph to calculate the final connectivity between all target object nodes. S65. Output the final connectivity relationship in the form of a disjoint-set data structure, as the final connected component result.

[0042] To verify the effectiveness of this invention, a comparison is made between the original disjoint-set data structure and the HIDE disjoint-set data structure: Generate topology graph ,in , , ,side Generates according to isomorphic topological logic.

[0043] Disjoint sets of G using the original disjoint-set data structure, such as... Figure 3 As shown, the disjoint sets obtained using the disjoint-set data structure of this application are as follows: Figure 4 As shown, the number of worker nodes, nw, is set to 2. Based on the disjoint set of the original disjoint-set data structure, the disjoint set contains 5 clusters: C0 = {1, 6, 8, 9, 11}, C1 = {3, 4}, C2 = {2, 5, 7}, C3 = {12}, and C4 = {10}. On the other hand, based on the disjoint set of the HIDE disjoint-set data structure, the disjoint set also contains 5 clusters: C0 = {1, 6, 8, 9, 11}, C1 = {3, 4}, C2 = {2, 5, 7}, C3 = {12}, and C4 = {10}.

[0044] For any topological graph, there exists only one unique set of disjoint sets. Therefore, the only thing that needs to be verified is whether the disjoint sets of the original disjoint-set data structure and the disjoint sets of the HIDE disjoint-set data structure are the same. Clearly, the disjoint sets of the topological graph G obtained by the original disjoint-set data structure and the HIDE disjoint-set data structure are the same. A brief verification has been completed.

[0045] Table 1 provides a brief comparison of response times between the original disjoint-set data structure and the HIDE disjoint-set data structure, with three original disjoint-set data structures tested as a control group. It is worth noting that running on a 3-node distributed cluster using GraphX's control strategy may be slower than the other two single-node methods. The table shows different numbers of worker nodes. The response time of the HIDE disjoint-setup is used to demonstrate the performance of the HIDE disjoint-setup.

[0046] Table 1:

[0047] As can be seen from the table, HIDE disjoint-set data structure will significantly reduce the response time of computations on disjoint sets.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A HIDE disjoint-setup data processing method based on a distributed architecture, applied in a distributed cluster to query and simplify large-scale time-series graph data, characterized in that... include: S1: Set the atomization conditions, collect the original relational data from the data source according to the atomization conditions, and obtain the atomized dataset; The atomization conditions include date fragments and non-target object entity types; S2: Perform disjoint set operations on the atomic dataset, extract the DAG index, and convert it into a DAG edge set; S3: Based on the scale attribute of non-target object entity types in the atomic dataset, the HashBucket algorithm is introduced to perform hash value bucketing on the DAG edge set that meets the bucketing condition according to the hash value of the non-target object node. The bucketing result is stored with the corresponding atomicity condition and the DAG edge set that does not meet the bucketing condition and the corresponding atomicity condition to form a pre-computation cache set. S4: Based on preset grouping conditions, calculate the target cache block set based on the master node in the distributed cluster and the pre-computed cache set to generate an atomic condition set; introduce a knapsack algorithm to divide the atomic condition set into n graph-scale balanced task groups according to the number of DAG edges corresponding to each target cache block, and assign them to n working nodes in the distributed cluster for processing. S5: For each working node, load all DAG edge sets corresponding to the assigned task group, perform local fusion and calculation on the DAG edge sets, generate a simplified DAG equivalent subgraph containing only the target object node, and send it to the master node; S6: The master node collects the DAG equivalent subgraphs sent by all worker nodes, connects all DAG equivalent subgraphs to form a connected DAG graph; performs the disjoint set operation on the connected DAG graph again, and outputs the final connected component result containing only the target object.

2. The HIDE disjoint-setup data processing method based on a distributed architecture according to claim 1, characterized in that, Perform disjoint set operations on the atomic dataset, extract the DAG indices, and convert them into DAG edge sets, including: For each of the atomic datasets, perform the following sub-steps: The Union-Find algorithm is executed on the atomic dataset to obtain the locally connected components of all nodes in the dataset. Traverse the locally connected components and generate a DAG structure for each connected component. According to a preset selection rule, a representative node of the connected component is taken as the root node and the remaining nodes are taken as child nodes, forming a set of directed edges from the child nodes to the root node, i.e., the DAG index. The set of directed edges of all connected components is aggregated to form a DAG edge set representing the overall connectivity of the atomized dataset.

3. The HIDE disjoint-setup data processing method based on a distributed architecture according to claim 1, characterized in that, Based on the size attribute of non-target object entity types in the atomic dataset, determine whether the preset bucketing conditions are met, including: Obtain the historical data volume of the non-target object entity type node; if the historical data volume exceeds a preset threshold, it is determined that the bucketing condition is met; otherwise, it is determined that the bucketing condition is not met.

4. The HIDE disjoint-setup data processing method based on a distributed architecture according to claim 1, characterized in that, The HashBucket algorithm is introduced to bucket the DAG edge set according to the hash values ​​of non-target object nodes. The results are then stored along with the corresponding atomicity conditions, including: A preset number of buckets K is used to traverse each directed edge in the DAG edge set; Perform a hash calculation on the non-target node associated with the edge, and then take the modulo of K to obtain the hash value; The directed edge whose hash value is the same as the current hash bucket number is assigned to the corresponding hash bucket, generating K DAG edge sets; Each DAG edge set and its corresponding atomization condition are packaged and stored.

5. The HIDE disjoint-setup data processing method based on a distributed architecture according to claim 1, characterized in that, Based on preset grouping conditions, the master node in the distributed cluster calculates the target cache block set according to the pre-computed cache set, generating an atomic condition set, including: The target time range is obtained by parsing the grouping conditions. Non-target object entity type collection ; The target time range is discretized by natural days to obtain a set of time dimensions; Perform a Cartesian product operation between the time dimension set and the non-target object entity type set to generate an atomic condition set.

6. The HIDE disjoint-setup data processing method based on a distributed architecture according to claim 1, characterized in that, The knapsack algorithm is introduced to divide the atomicity condition set into n graph-balanced task groups based on the number of edges in the DAG corresponding to each target cache block, including: For each atomization condition in the set of atomization conditions, query the estimated number of edges in the corresponding DAG edge set, and use it as the weight of the condition; Using the weights as input, the number of task groups n is the number of knapsacks, and the knapsack algorithm is executed with the goal of minimizing the difference between the total weights of each knapsack, i.e., the total weights of the task groups. Output n graph-scale balanced task groups, where each task group contains atomic conditions of the same type but on different dates; each subset is assigned as a computation task group to a worker node.

7. The HIDE disjoint-setup data processing method based on a distributed architecture according to claim 1, characterized in that, For each worker node, load all DAG edge sets corresponding to the assigned task group, perform local fusion and computation on the DAG edge sets, generate a simplified DAG equivalent subgraph containing only the target object nodes, and send it to the master node, including: Work nodes read the DAG edge sets corresponding to all atomization conditions within their task groups from storage in parallel; The read DAG edge sets are concatenated to form a local joint graph. ; The Union-Find algorithm is executed on the local union graph, based on the local union graph at each working node. Calculate the locally connected components to obtain the equivalent subgraph of the local DAG; From the equivalent local DAG subgraph, delete all nodes whose node type is not the target object and their associated edges to obtain a simplified DAG subgraph containing only the target object nodes. The simplified DAG subgraph is sent to the master node.

8. The HIDE disjoint-setup data processing method based on a distributed architecture according to claim 1, characterized in that, The master node collects the equivalent DAG subgraphs sent by all worker nodes, connects all the equivalent DAG subgraphs to form a connected DAG graph; the disjoint set operation is then performed on the connected DAG graph again, outputting only the final connected component results of the target object, including: S61. Initialize the global simplified union graph structure and the global result queue; S62. The master node asynchronously receives the simplified DAG subgraphs returned by each working node and stores the edge set of each simplified DAG subgraph received into the global result queue. S63. The master node starts an incremental merge loop. When the global result queue is not empty and the number of processed results has not reached the total number of worker nodes, the following operations are performed: S631: Extract the edge set of the simplified DAG subgraph from the global result queue in sequence according to the time series; S632: Merge the extracted edge set with the current global simplified joint graph structure, and update the global simplified joint graph; S64. After the simplified DAG subgraph edge sets of all working nodes are merged, execute the Union-Find algorithm on the final global simplified joint graph to calculate the final connectivity between all target object nodes. S65. Output the final connectivity relationship in the form of a disjoint-set data structure, as the final connected component result.

Images