A large-scale data analysis method based on DBSCAN algorithm

Abstract

The application discloses a large-scale data analysis method based on a DBSCAN algorithm, which comprises the following steps: step one, calculating an adjacency list through ring communication; step two, locally preliminarily determining a temporary cluster number of a core point; step three, alternately broadcasting a synchronous core point cluster number; and step four, updating a boundary point cluster number through reverse ring communication. The application can well solve the efficiency problem of the DBSCAN algorithm in large-scale data clustering analysis, ingeniously decomposes the calculation step dependence problem of the recursive calculation process of the DBSCAN algorithm, makes the algorithm applicable to a distributed execution model, makes the work load of different nodes in a cluster as balanced as possible, greatly reduces the communication amount in distributed calculation, and uses a non-blocking communication mode to make the calculation and communication overlap, reduces idle waiting time, fully utilizes the memory of a calculation node, and makes the massive data clustering analysis on the cluster become a reality.

Description

Technical Field

[0001] This invention relates to big data clustering algorithms and distributed computing technology, and in particular to a large-scale data analysis method based on the DBSCAN algorithm. Background Technology

[0002] With the continuous innovation of information technology, the need to extract useful information from massive amounts of data is becoming increasingly urgent. However, how to process massive amounts of data remains a significant challenge. Data mining techniques, which extract useful information from data, have become increasingly important with the advent of the big data era. Cluster analysis in the field of data mining is widely used in practice due to its versatility. In short, how to apply cluster analysis techniques to large-scale systems and massive amounts of data is one of the major challenges in big data processing.

[0003] The DBSCAN algorithm, proposed in 1996 by Ester Martin et al., is a density-based clustering method. It uses data objects whose neighbors (Eps) have at least a threshold number of data objects (MinPts) as core points. Each core point and its Eps neighborhood defines a cluster; multiple core points may belong to the same cluster. This clustering method can discover clusters of arbitrary shapes and effectively handle noisy points. The algorithm is simple in principle and highly efficient in practice, and is currently widely used in industrial production and scientific research.

[0004] The traditional serial DBSCAN algorithm arbitrarily selects a data object, calculates its distance to all other data objects, and determines whether it is a core point based on the number of objects in its neighborhood (Eps). If it is a core point, the objects in its neighborhood are marked with the same cluster ID (CID) as the core point, and this process is repeated recursively. If it is not a core point, it is marked as NOISE. When the data scale is very large, the memory of a single machine node cannot hold all the data, and the computational load is so large that it cannot be completed within a reasonable time. To address this scenario, this invention designs a distributed DBSCAN algorithm that can run on large-scale systems with tens of thousands of computing nodes. Some existing solutions implement distributed DBSCAN algorithms based on big data frameworks such as Spark, but these solutions suffer from poor performance due to insufficient memory utilization and redundant communication. Other implementations aim to partition the feature space of the data and then cluster it within each partition to reduce the computational load. However, as the dimensionality of data increases, the distribution of data often becomes extremely complex, and simple partitioning methods become very unstable in terms of effect and performance; complex partitioning methods also have the problem of excessive time consumption in the partitioning process, and many scenarios only require a one-time cluster analysis, which is not worth the time. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the DBSCAN clustering algorithm in the prior art, which has good accuracy but takes too long to execute in practical use, and to provide a large-scale data analysis method based on the DBSCAN algorithm.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] A large-scale data analysis method based on the DBSCAN algorithm includes the following steps:

[0008] Step 1: Calculate the adjacency list using ring communication;

[0009] Step 2: Preliminarily determine the temporary cluster number of the core point in a localized area;

[0010] Step 3: Broadcast the synchronized core point cluster number in turn;

[0011] Step 4: Update the boundary point cluster number using reverse ring communication.

[0012] Preferably, step one includes the following steps:

[0013] Step S1: After distributing the dataset evenly to the computing nodes in the cluster, calculate the distance between data objects within each computing node, and then organize them into an adjacency list of internal data objects and a table of the number of domain objects.

[0014] Step S2: Calculate the domain relationships of data across nodes using ring communication.

[0015] As a preferred option, the temporary cluster number of the core point is initially determined locally in step two. Specifically, a core point is arbitrarily selected, and then a cluster number is assigned to this core point as its own ID. Then, all core points in its Eps neighborhood are determined according to the adjacency list, and then cluster numbers are assigned. Then, this process is recursively executed on the core points in the Eps neighborhood.

[0016] Preferably, step four includes the following steps:

[0017] Step S41: Establish a local cluster number equivalence table, specifically in the form of CID. s →CID t An array representing the temporary cluster number (CID). s and CID t They belong to the same cluster number in the clustering algorithm results, and numerically they are CID. s ≤CID t ;

[0018] Step S42: This includes steps S421 and S422, which are performed simultaneously.

[0019] Step S421: Calculate node N i Center, towards the set of computation nodes The core of broadcasting itself, and then the receiver based on N. i The core point and its adjacency list determine the cluster number of the core point; step S422 is if... Then we will calculate node N. j The core point set is broadcast to the set {N} t For each node in the set |0≤t≤i}, calculate its cluster number and return the result to N. j Step S43: Unify the cluster number equivalence table;

[0020] Step S44: All nodes update the core node cluster number within the node according to the received cluster number equivalence table.

[0021] As a preferred option, step four involves updating the boundary point cluster number via reverse ring communication. Specifically, each node calculates the cluster number of the boundary points of other nodes based on the adjacency list and then returns it to the other nodes.

[0022] As a preferred option, step S43 specifically involves: first, node N... 2k+1 Send the cluster number equivalence table to N 2k The N sets are then integrated using a disjoint-set data structure; then N... 4k+2 Cluster number table unified to N 4k Once integration is complete, this process continues until the data converges on node 0. Node 0 then broadcasts the final cluster number equivalence table to all nodes.

[0023] Preferably, step S421 specifically involves scanning the core points among the receiving nodes, checking their adjacency lists, and if there are N... i The core node then begins updating the cluster number: if the cluster number (CID) of the core node is the original local temporary cluster number, then it is assigned a cluster number, and all local data objects corresponding to the original local temporary cluster number are updated with cluster numbers; if during the update process it is found that other nodes have already been assigned cluster numbers, this conflict indicates that the three belong to the same cluster, then these two cluster numbers are recorded in a temporary local cluster number equivalence table, and the recording method is to create a new CID. s →CID t right.

[0024] The beneficial effects of this invention are as follows: This invention effectively solves the efficiency problem of the DBSCAN algorithm in large-scale data clustering analysis. It cleverly decomposes the computational step dependency problem in the recursive computation process of the DBSCAN algorithm, making it applicable to a distributed execution model and balancing the workload of different nodes in the cluster as much as possible. It significantly reduces the amount of communication in distributed computing and utilizes non-blocking communication methods to overlap computation and communication, reducing idle waiting time. It fully utilizes the memory of computing nodes, making massive data clustering analysis on the cluster a reality. Attached Figure Description

[0025] Figure 1 This is a flowchart of the present invention;

[0026] Figure 2 This is a flowchart of the unified global cluster number phase;

[0027] Figure 3 This is a schematic diagram illustrating the data communication direction when determining the adjacency list and count list across nodes, using a cluster with a total of 5 nodes as an example.

[0028] Figure 4 This is a schematic diagram of the communication direction of the broadcast synchronization core point cluster number in stages 1 and 2, taking a cluster with a total of 5 nodes as an example.

[0029] Figure 5 This is a schematic diagram of the communication direction of the broadcast synchronization core point cluster number in stages 3 and 4, taking a cluster with a total of 5 nodes as an example.

[0030] Figure 6 This is a data flow graph of the binary tree communication process. Detailed Implementation

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings:

[0032] This embodiment discloses a distributed DBSCAN algorithm for large-scale computing clusters. The algorithm calculates the adjacency list through ring communication, broadcasts the core node cluster number in turn, and updates the boundary node cluster number through reverse ring communication. Finally, it realizes a distributed clustering algorithm that is mathematically equivalent to the original DBSCAN algorithm. It makes full use of the computer's physical memory and communication bandwidth, and balances the workload of each computing node, so that the clustering analysis of high-dimensional large-scale datasets can be completed in an effective time.

[0033] like Figure 1 As shown, the method in this embodiment specifically includes the following steps:

[0034] Step (1): Calculate the adjacency list using ring communication

[0035] Step S1: After distributing the dataset evenly to the computing nodes in the cluster, calculate the distance between data objects within each computing node, and then organize them into an adjacency list of internal data objects and a table of the number of domain objects.

[0036] The original dataset is in HDF5 file format, and the data row format is as follows:<ID,RawData> Here, ID is the data index and RawData is the raw data object. The cluster processes are started via an MPI initialization script, and the data is evenly distributed to the memory of each node using the MPI-based HDF5 data sending interface.

[0037] Each node receives a data matrix P. i ={p i1 p i2 , ..., p im After that, the distance calculation formula dist(p) is used. is p it )=|p is | 2 +|p it | 2 -2p is ·p it To calculate the Euclidean distance, specifically, to calculate |p is | 2 Let i = 1, 2, ..., m, then calculate dis(p) is p it ), s = 1, 2, ..., m, s ≤ t ≤ n. The above calculations are implemented using matrix algorithms from the high-performance computing library Eigen.

[0038] Each computation node maintains an adjacency list corresponding to all internal records and a list of the number of data objects within the Eps neighborhood, namely NTable and NumTable. The adjacency list is an array of pointers pointing to an array containing the set of objects within the Eps neighborhood of the data objects. The neighbor count table is an array corresponding to the rows of the data matrix and is used in later steps to determine the core points. For two data objects, "within the Eps neighborhood" is mutual, but they are only recorded in the adjacency list row with the smaller index value. The NumTable positions corresponding to these two data objects are incremented by one.

[0039] Step S2: Calculate the domain relationships of data across nodes using ring communication.

[0040] The original serial DBSCAN clustering algorithm requires comparing the distances between all pairwise data objects, and then calculating the distance relationships between data objects across nodes. To efficiently utilize bandwidth and overlap computation and communication, a ring-shaped communication method and a non-blocking communication method are adopted. In the first transmission, data D0 is transmitted from computation node N0 to computation node N1, forming a ring in this direction. In the second transmission, computation node N0 sends data to computation node N2, while computation node N1 sends data to computation node N3. Accordingly, in the k-th transmission, data D... i From the calculation node N i Transmitted to computing node N (k+i)％N Each process is called a stage. Figure 3 This diagram illustrates the process in a cluster of 5 computing nodes. The left diagram shows the loop of the first communication: N0->N1->N2->N3->N4->N0, and the right diagram shows the loop of the second communication: N0->N2->N4->N1->N3->N0. After (N-1) / 2 communications, if N is an even number, a final round of peer-to-peer communication is required, where each peer node takes half of the data and sends it.

[0041] The computation node receives data while simultaneously calculating distances to update its internal NTable and NumTable. In the MPI programming model, to allow communication and computation to overlap, a fixed-granularity communication approach is used: the Isend send interface function sends M data objects, and the receiver uses the Irecv receive interface function to retrieve the data. Upon successful data retrieval, an asynchronous thread is notified to perform distance calculations and subsequent steps. The asynchronous thread utilizes a high-performance computation library to perform NTable and NumTable calculations in parallel, while simultaneously filling a temporary object count array SendNumTable within the corresponding sending node's Eps domain. When the calculation is complete, SendNumTable is returned to the sending node.

[0042] After receiving SendNumTable, the original sending node merges it into its own NumTable using vector addition. After all stages of the ring communication are completed, the NumTable of each computing node represents the number of objects in the Eps neighborhood of the data object. When it exceeds the threshold MinPts, it becomes a core node.

[0043] Finally, use a global barrier to synchronize all compute nodes before proceeding with the following steps.

[0044] Step (II): Preliminary determination of the temporary cluster number of the core point in a localized area.

[0045] The core points are determined based on the NumTable. Specifically, the data objects corresponding to the IDs that satisfy NumTable[ID]≥MinPts are marked as core points.

[0046] In the DBSCAN algorithm, the cluster number of a core point is only related to other core points. Locally, each node can assign a temporary cluster number to a core point based on the distance relationship between core points. Specifically, a core point is arbitrarily selected, and a cluster number is assigned to this core point as its own ID. Then, all core points in its Eps neighborhood are determined according to the adjacency list, and cluster numbers are assigned. This process is recursively executed for the core points in the Eps neighborhood.

[0047] Step (3): Broadcast the synchronization core point cluster number in turn.

[0048] Each is calculated using node N. i Broadcasts are executed sequentially, with i = 0, 1, ..., N-1 as the center. For the specific execution flow, please refer to... Figure 2 The flowchart is shown below, and the description will be elaborated based on the flowchart.

[0049] S41: Establish a local cluster number equivalence table. The specific form of the cluster number equivalence table is CID. s →CID t An array representing the temporary cluster number (CID) s and CID t They belong to the same cluster number in the clustering algorithm results, and numerically they are CID. s ≤CID t .

[0050] Step S42: This includes steps S421 and S422, which are performed simultaneously. Step S421: Calculate node N. i Center, towards the set of computation nodes The core of broadcasting is that the receiver determines the frequency based on N. i The cluster number of the core point is determined by the core point and its adjacency list. Specifically, this involves scanning the core points in the receiving nodes, checking their adjacency lists, and if there is an N... i The core node then begins updating the cluster number: if the cluster number (CID) of the core node is the original local temporary cluster number, then it is assigned a cluster number, and all local data objects corresponding to the original local temporary cluster number are updated with cluster numbers; if during the update process it is found that other nodes have already been assigned cluster numbers, this conflict indicates that the three belong to the same cluster, then these two cluster numbers are recorded in a temporary local cluster number equivalence table, and the recording method is to create a new CID. s →CID t Yes. (Reference) Figure 4 In the example of a 5-node cluster, the solid lines in the diagram represent the broadcast nodes and directions.

[0051] S422: If Then we will calculate node N. j The core point set is broadcast to the set {N} t For each node in the set |0≤t≤i}, calculate its cluster number and return the result to N. j This step is executed simultaneously with S42.

[0052] When calculating the cluster number, conflicts may occur as seen in S42; these should be handled as before. At node N... j When integrating the received results, if conflicts are encountered, they should be handled in the same way as before. (See reference) Figure 4 An example of a 5-node cluster, Figure 4 The dotted lines in the diagram represent the converging nodes and the direction of information transmission. Figure 5 There is no such communication in China.

[0053] S43: Uniform Cluster Number Equivalence Table

[0054] Since different temporary cluster numbers may belong to the same final cluster number when calculating cluster numbers, a cluster number equivalence table is used for the final integration. During integration, a binary tree communication method is used, and the communication direction can be referenced. Figure 6 The case with 8 nodes. First, node N. 2k+1 Send the cluster number equivalence table to N 2k The N sets are then integrated using a disjoint-set data structure; then N... 4k+2 Cluster number table unified to N 4k Once the integration is complete, continue in this manner.

[0055] Finally, the data is aggregated on node 0, and then node 0 broadcasts the final cluster number equivalence table to all nodes.

[0056] S44: All nodes update the core node cluster number within the node according to the received cluster number equivalence table.

[0057] (iv): Reverse ring communication updates boundary point cluster number

[0058] Each node calculates the cluster number of the boundary points of other nodes based on the adjacency list, and then returns it to the other nodes. Specifically, the implementation follows a circular communication direction opposite to step 2. During the k-th transmission process, node N is calculated... (k+i)％N First calculate node N i The middle boundary point cluster number is then transmitted to node N. i Node N i Upon receipt, update the cluster number of the boundary points. Note that some data objects may not be within the Eps neighborhood of any core point; these are considered noise and marked as NOISE by default.

[0059] Finally, all data objects were assigned a cluster number or a NOISE flag.

[0060] The advantages of this invention are that it parallelizes DBSCAN, consumes less memory resources, and has short computation and communication times, with the algorithm's performance scaling almost linearly.

[0061] This invention effectively solves the efficiency problem of the DBSCAN algorithm in large-scale data clustering analysis. It cleverly decomposes the computational dependency problem of the DBSCAN algorithm's recursive computation process, making it applicable to a distributed execution model. This balances the workload of different nodes in the cluster, significantly reducing communication volume in distributed computing. Furthermore, it utilizes non-blocking communication to overlap computation and communication, minimizing idle waiting time. By fully utilizing the memory of computing nodes, it enables clustering analysis of massive amounts of data on a cluster, making this a reality.

[0062] It should be noted that the above examples are merely one specific embodiment of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. In short, all variations that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should be considered within the scope of protection of this invention.

Claims

1. A large-scale data analysis method based on the DBSCAN algorithm, characterized in that, Includes the following steps: Step 1: Calculate the adjacency list using ring communication; Step 2: Preliminarily determine the temporary cluster number of the core point in a localized area; Step 3: Broadcast the synchronized core point cluster number in turn; Step 4: Update the boundary point cluster number using reverse ring communication; Step one includes the following steps: Step S1: After distributing the dataset evenly to the computing nodes in the cluster, calculate the distance between data objects within each computing node, and then organize them into an adjacency list of internal data objects and a table of the number of domain objects. Step S2: Calculate the adjacency relationship of data across nodes using ring communication; Step three includes the following steps: Step S31: Establish a local cluster number equivalence table. The specific form of the cluster number equivalence table is as follows: An array representing temporary cluster numbers. and They belong to the same cluster number in the clustering algorithm results, and numerically... ; Step S32: This includes steps S321 and S322, which are performed simultaneously. Step S321: Calculate the node Center, towards the set of computation nodes The core of broadcasting itself, and then the receiver based on... The core point and its adjacency list determine the cluster number of the core point; step S322 is if... Then the node will be calculated. Core point set broadcast to collection The cluster number is calculated from the nodes in the set, and the result is returned to... ; Step S33: Use a binary tree communication method to unify the cluster number equivalence table; Step S34: All nodes update the core node cluster number within the node according to the received cluster number equivalence table.

2. The large-scale data analysis method based on the DBSCAN algorithm according to claim 1, characterized in that, The preliminary determination of the temporary cluster number of the core point in step two is as follows: arbitrarily select a core point, assign a cluster number to this core point as its own ID, then determine all core points in its Eps neighborhood according to the adjacency list, and then assign cluster numbers to all core points; then recursively execute step two for all other core points in the Eps neighborhood.

3. The large-scale data analysis method based on the DBSCAN algorithm according to claim 1, characterized in that, Step four, updating the boundary point cluster number via reverse ring communication, involves each node calculating the boundary point cluster number of other nodes based on the adjacency list, and then returning it to the other nodes.

4. The large-scale data analysis method based on the DBSCAN algorithm according to claim 1, characterized in that, The specific step S33 is as follows: First, the node Send the cluster number equivalence table to The disjoint-set data structure is used to integrate the data; then... Cluster number table unified to Once integration is complete, this process continues until the data converges on node 0. Node 0 then broadcasts the final cluster number equivalence table to all nodes.

5. The large-scale data analysis method based on the DBSCAN algorithm according to claim 1, characterized in that, Step S321 specifically involves scanning the core points among the receiving nodes, checking their adjacency lists, and if any are found... The core node then begins updating its cluster number: if the cluster number (CID) of that core node is the original local temporary cluster number, then it is assigned a cluster number, and all local data objects corresponding to the original local temporary cluster number are updated with new cluster numbers; if during the update process it is found that other nodes have already been assigned cluster numbers, this conflict indicates that the three belong to the same cluster, then these two cluster numbers are recorded in a temporary local cluster number equivalence table, recorded by creating a new table. right.

Images