Mass data-oriented clustering method and system

Abstract

The invention provides a clustering method and system for mass data, and relates to the technical field of data.The method comprises the steps that a space is subjected to grid division, all data points are distributed into corresponding grids, and a sub-grid structure with the side length being half of the grid structure is created; constructing a grid index and retrieving neighbors of the grid; in the sub-grid structure, traversing the neighborhood of each data point and calculating the distance, and identifying the core points of the mass data; and according to a core point identification result of the mass data, merging core grids and distributing non-core points, and completing rapid clustering oriented to the mass data. According to the method, the problem that a traditional density-based clustering algorithm cannot process massive scale data sets (especially when the parameter MinPts is relatively large) is solved.

Description

Technical Field

[0001] The present invention belongs to the field of data technology, and in particular relates to a clustering method and system for massive data. Background Art

[0002] In the field of data processing and analysis, clustering plays a vital role as an unsupervised learning method. It can mine hidden structures and patterns from complex and large data sets without pre-defined labels, providing data scientists and researchers with a powerful tool to deeply understand the characteristics of data. Among them, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) has long been widely recognized and applied because of its unique density-based clustering mechanism, which can effectively identify clusters of arbitrary shapes and show excellent performance when processing noisy data.

[0003] However, with the rapid development of information technology, we are facing an unprecedented flood of data. In the era of big data, the amount of data to be processed often reaches 10^9 or even higher, which poses a severe challenge to the computational efficiency of traditional algorithms. Although the DBSCAN algorithm has many advantages in theory, its low efficiency caused by computational complexity has become increasingly prominent when facing such a large data set, becoming a major bottleneck restricting its widespread application. In order to meet this challenge, the research community has continuously explored and proposed a series of improved algorithms (for example, appr-DBSCAN, GriT-DBSCAN, etc.) to improve the efficiency of DBSCAN. These algorithms have optimized the execution speed of DBSCAN to a certain extent, but they are still unable to cope with the processing of ultra-large-scale data sets, especially when a large MinPts (minimum number of points) parameter needs to be set to ensure the quality of clustering, and the performance is significantly reduced.

[0004] Real-world data often follows the assumption of independent and identical distribution, which means that in a dynamically changing database environment, in order to maintain the consistency of clustering results and comparability across time points, the MinPts parameter in the DBSCAN algorithm needs to be adjusted accordingly according to the growth of data size to ensure the stability of clustering density. However, this adjustment often leads to a significant increase in the MinPts value, further exacerbating the performance burden of the algorithm when processing large-scale data. Under the existing improved algorithm framework, this performance problem caused by parameter adjustment is particularly prominent, making it difficult for them to effectively respond to the clustering needs of massive data, limiting the comprehensive application potential of the DBSCAN algorithm in the big data era. In the prior art, there are the following problems:

[0005] 1. Insufficient performance under massive data, making it difficult to complete calculations within a reasonable time:

[0006] When processing large-scale data sets, the existing DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm needs to calculate the distance between each point and all other points in the data set, as well as the subsequent density connection operation, which leads to extremely high computational complexity and significantly increased running time, making it difficult to meet the real-time or high-efficiency requirements in practical applications.

[0007] 2. When the value of the parameter MinPts is large, the performance is insufficient and it is difficult to complete the calculation within a reasonable time:

[0008] MinPts (minimum number of points) is a key parameter in the DBSCAN algorithm, which determines the minimum number of neighbor points required to form a dense area. When MinPts is set larger, the algorithm needs to check the neighbors of each point in a larger range, which further increases the computational burden and makes it difficult for the algorithm to complete in a reasonable time, especially when the data scale is already large. Summary of the invention

[0009] In view of the above-mentioned deficiencies in the prior art, the present invention provides a clustering method and system for massive data, which solves the problem that traditional density-based clustering algorithms cannot process massive data sets (especially when the parameter MinPts is large).

[0010] In order to achieve the above purpose, the technical solution adopted by the present invention is: a clustering method for massive data, comprising the following steps:

[0011] S1. Divide the space into grids, assign all data points to the corresponding grids, and create a sub-grid structure with a side length of half the grid structure;

[0012] S2, build sub-grid index and retrieve sub-grid neighbors;

[0013] S3. In the sub-grid structure, the core points of the massive data are identified by traversing the neighborhood of each data point and calculating the distance;

[0014] S4. Based on the core point identification results of the massive data, the core grids are merged and non-core points are allocated to complete the rapid clustering of the massive data.

[0015] Furthermore, the S1 is specifically:

[0016] Use the following formula to calculate the grid coordinates (c 1 ,c2 ,...,c i ):

[0017]

[0018] i∈(0,1,...,d-1)

[0019] Among them, c i represents the coordinates of the grid to which the data point p belongs in the i-th dimension, ε represents the neighborhood radius, and p i Represents the coordinates of data point p in the i-th dimension, mn i It represents the minimum coordinate value of all data points in the i-th dimension, and d represents the data dimension;

[0020] Based on the grid coordinates, the lowest binary bits of the d data dimensions are combined as the sub-grid coordinates of the data point p;

[0021] For grid coordinates, all data points are sorted by their grid coordinates in cardinality order, and corresponding grid structures are created in sequence, where the number of members in each subgrid of each grid is cached when obtaining the grid coordinates of data point p;

[0022] For the subgrid coordinates, d binary numbers form a d-bit binary integer, which represents the subgrid number of the data point p in the grid to which it belongs;

[0023] After processing all data points, count sorting is performed on the members according to the sub-grid numbers in each grid, and based on the sorting results, a sub-grid member count table is constructed to complete the establishment of the sub-grid structure, where the side length of the sub-grid structure is half of the grid structure.

[0024] Furthermore, the S3 is specifically:

[0025] S301, check whether the number of members in each sub-grid structure is at least parameter MinPts, if so, mark each member of the sub-grid whose members are at least parameter MinPts as a core point, otherwise, go to step S302;

[0026] S302, initializing a temporary counter, and setting the initial value of the temporary counter to the member count in the sub-grid structure;

[0027] S303, from the nearest to the farthest distance, process each adjacent subgrid of the current subgrid of each member p, wherein for each processed subgrid, from the nearest to the farthest distance, preferentially calculate the distance of the member of the subgrid closest to member p, and the members of the adjacent subgrid are directly determined as neighbors of member p without calculation, and their member counts are directly added to the neighbor counter, wherein member p is data point p;

[0028] S304. Based on the processing result of step S303, when the number of neighbors of member p is at least the parameter MinPts, the calculation is terminated and member p is marked as a core point, thereby completing the identification of the core point.

[0029] The present invention also provides a rapid clustering system for massive data, comprising:

[0030] The first processing module is used to divide the space into grids, assign all data points to corresponding grids, and create a sub-grid structure with a side length half of the grid structure;

[0031] A second processing module, for constructing a subgrid index and retrieving the subgrid's neighbors;

[0032] The third processing module is used to identify the core points of the massive data by traversing the neighborhood of each data point and calculating the distance in the sub-grid structure;

[0033] The fourth processing module is used to merge core grids and allocate non-core points according to the core point recognition results of the massive data, so as to complete the rapid clustering of the massive data.

[0034] Beneficial effects of the present invention:

[0035] (1) Optimizing the data structure establishment method: The present invention is based on the grid DBSCAN algorithm. Different from the grid construction method of the traditional algorithm, the present invention adopts the radix sorting method to achieve efficient grid division, thereby saving computational overhead. The algorithm implements an accurate DBSCAN algorithm that can be quickly executed under massive data conditions. The algorithm has low sensitivity to the increase of the parameter MinPts and can maintain efficient performance even when the parameter MinPts takes a large value; Optimizing distance calculation: The present invention introduces a secondary grid structure (sub-grid structure) based on the traditional grid structure, thereby saving a large amount of unnecessary distance calculations in the core point identification step and significantly reducing the computational complexity.

[0036] (2) Compared with the prior art, the present invention demonstrates significant advantages in core point identification and overall algorithm efficiency, especially when processing large-scale data sets and the parameter MinPts is large. This superiority is particularly prominent. Existing similar technologies, such as grid-based DBSCAN, use a primary grid structure, which may generate a large number of unnecessary distance calculations in the core point identification stage to determine whether the points satisfy the neighbor relationship. This design will lead to a surge in the amount of calculation when the MinPts value is high, making distance calculations the main bottleneck of the algorithm's running time. In contrast, the present invention cleverly introduces a secondary grid structure (sub-grid structure). This key technical solution not only inherits the advantages of the traditional grid structure in fast spatial division, but also further expands the application potential of the grid structure through a finer grid granularity.

[0037] (3) In the "grid construction" step, the present invention efficiently establishes a sub-grid structure. This new data structure provides a solid optimization foundation for the subsequent core point identification step. The innovation of the present invention is particularly obvious in the "core point identification" stage. First, by utilizing the secondary grid structure, the present invention greatly expands the scope of "directly determining as neighbors without calculation" in the traditional grid structure. This means that in many cases, the neighbor relationship between points can be directly determined by grid position without the need for time-consuming distance calculations. Secondly, the present invention adopts a heuristic search method to search sub-grids from near to far. This strategy effectively reduces the number of unnecessary distance calculations because once a sufficient number of neighbor points are found, the search can be terminated early, thereby avoiding unnecessary computational overhead.

[0038] (4) The subgrid structure and core point identification method of the present invention can significantly reduce the amount of calculation and improve the efficiency of the algorithm when facing massive data sets, especially when the MinPts parameter is large. Compared with the prior art, the present invention not only inherits the advantages of the traditional grid structure, but also achieves a major technical breakthrough through refined grid division and heuristic search strategy, producing good technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the grid and sub-grid structure of the present invention.

[0040] Figure 2 Schematic diagram of the heuristic search method for the core point identification process of the present invention.

[0041] Figure 3 The present invention is a flow chart of the method.

[0042] Figure 4 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0043] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0044] Before describing the present invention, the following parameters are explained:

[0045] P: dataset;

[0046] p,q: data points;

[0047] n: data size;

[0048] d: data dimension;

[0049] ε: DBSCAN basic parameters: neighborhood radius;

[0050] MinPts: DBSCAN basic parameters: minimum number of core points;

[0051] p i : The coordinates of the data point p in the i-th dimension;

[0052] c i : The coordinates of the grid to which the data point p belongs in the i-th dimension;

[0053] mn i : The minimum coordinate value of all data points in the i-th dimension.

[0054] Example 1

[0055] like Figure 3 As shown, the present invention provides a clustering method for massive data, and its implementation method is as follows:

[0056] S1. Divide the space into grids, assign all data points to the corresponding grids, and create a sub-grid structure with a side length of half the grid structure. The implementation method is as follows:

[0057] Use the following formula to calculate the grid coordinates (c 1 ,c 2 ,...,c i ):

[0058]

[0059] i∈(0,1,...,d-1)

[0060] Among them, ci represents the coordinates of the grid to which the data point p belongs in the i-th dimension, ε represents the neighborhood radius, and p i Represents the coordinates of data point p in the i-th dimension, mn i It represents the minimum coordinate value of all data points in the i-th dimension, and d represents the data dimension;

[0061] Based on the grid coordinates, the lowest binary bits of the d data dimensions are combined as the sub-grid coordinates of the data point p;

[0062] For grid coordinates, all data points are sorted by their grid coordinates in cardinality order, and corresponding grid structures are created in sequence, where the number of members in each subgrid of each grid is cached when obtaining the grid coordinates of data point p;

[0063] For the subgrid coordinates, d binary numbers form a d-bit binary integer, which represents the subgrid number of the data point p in the grid to which it belongs;

[0064] After processing all data points, count sorting is performed on the members according to the sub-grid numbers in each grid, and based on the sorting results, a sub-grid member count table is constructed to complete the establishment of the sub-grid structure, where the side length of the sub-grid structure is half of the grid structure.

[0065] In this embodiment, the space is divided into two parts with side length Then assign all vector points to the corresponding grids and create a subgrid structure with a side length of half the grid, such as Figure 1 The strategy for creating grids and subgrids is as follows:

[0066] First, the grid coordinates of the data point p in the i-th dimension are calculated. The lowest bit of the obtained grid coordinates represents its subgrid coordinates in the i-th dimension. By performing a simple right shift operation, the grid coordinates of the point data p can be obtained. Then, the grid coordinates and subgrid coordinates are processed separately; for the grid coordinates, all data points are sorted according to their grid coordinates, and then the corresponding grids are created in sequence. For the subgrid coordinates, d binary numbers (0 or 1) form a d-bit binary integer, which represents the subgrid sequence number of the data point p in the grid to which it belongs. When the grid coordinates of each data point are obtained, the number of members in each subgrid of each grid is also cached. After processing all data points, the members are counted and sorted according to the subgrid number in each grid. By establishing a grid member list and a subgrid member count table, efficient subgrid retrieval is achieved.

[0067] S2, build sub-grid index and retrieve sub-grid neighbors;

[0068] In this embodiment, the grid indexing step aims to efficiently query the neighboring grids of a grid and avoid searching the entire data set. This step finds the neighboring grids of all grids and caches them for use in subsequent steps. The grid indexing step of the present invention adopts an existing solution.

[0069] S3. In the sub-grid structure, the core points of the massive data are identified by traversing the neighborhood of each data point and calculating the distance. The implementation method is as follows:

[0070] S301, check whether the number of members in each sub-grid structure is at least parameter MinPts, if so, mark each member of the sub-grid whose members are at least parameter MinPts as a core point, otherwise, go to step S302;

[0071] S302, initializing a temporary counter, and setting the initial value of the temporary counter to the member count in the sub-grid structure;

[0072] S303, from the nearest to the farthest distance, process each adjacent subgrid of the current subgrid of each member p, wherein for each processed subgrid, from the nearest to the farthest distance, preferentially calculate the distance of the member of the subgrid closest to member p, and the members of the adjacent subgrid are directly determined as neighbors of member p without calculation, and their member counts are directly added to the neighbor counter, wherein member p is data point p;

[0073] S304. Based on the processing result of step S303, when the number of neighbors of member p is at least the parameter MinPts, the calculation is terminated and member p is marked as a core point, thereby completing the identification of the core point.

[0074] In this embodiment, the purpose of this step is to identify whether each point is a core point by traversing the neighborhood of each point and calculating the distance. A core point is defined as a point with at least MinPts points in its ε-neighborhood, such as Figure 2 shown.

[0075] In this embodiment, using the established sub-grid structure, the algorithm identifies core points as follows: first, check whether the number of members of each grid is at least the parameter MinPts, and mark the members of the grids where each member is at least the parameter MinPts as core points; then, for the grids that do not meet the above conditions, for each member p therein:

[0076] 1. Initialize a temporary counter and set its initial value to the member count of the grid;

[0077] 2. Process each neighboring grid of member p's current grid from the closest to the farthest distance. For each processed grid, prioritize the distances of members of subgrids that are closer to member p. Members of adjacent subgrids are directly determined as neighbors of member p, without calculation, and their member counts are directly added to the neighbor counter.

[0078] 3. When the number of neighbors of member p is at least the parameter MinPts, terminate the calculation and mark member p as a core point.

[0079] S4. Based on the core point identification results of the massive data, the core grids are merged and non-core points are allocated to complete the rapid clustering of the massive data.

[0080] In this embodiment, the purpose of this step is to merge adjacent grids with core points and assign non-core points to nearby clusters. This step of the present invention adopts an existing solution.

[0081] Through the above-mentioned setting, the subgrid structure and core point identification method of the present invention can significantly reduce the amount of calculation and improve the efficiency of the algorithm when facing a massive data set, especially when the parameter MinPts is large. Compared with the prior art, the present invention not only inherits the advantages of the traditional grid structure, but also achieves a major technical breakthrough through refined grid division and heuristic search strategy, and produces good technical effects.

[0082] Example 2

[0083] like Figure 4 As shown, the present invention provides a rapid clustering system for massive data, comprising:

[0084] The first processing module is used to divide the space into grids, assign all data points to corresponding grids, and create a sub-grid structure with a side length half of the grid structure;

[0085] A second processing module, for constructing a subgrid index and retrieving the subgrid's neighbors;

[0086] The third processing module is used to identify the core points of the massive data by traversing the neighborhood of each data point and calculating the distance in the sub-grid structure;

[0087] The fourth processing module is used to merge core grids and allocate non-core points according to the core point recognition results of the massive data, so as to complete the rapid clustering of the massive data.

[0088] like Figure 4 The fast clustering system provided in the illustrated embodiment can execute the technical solution shown in the fast clustering method in the above-mentioned method embodiment 1, and its implementation principle and beneficial effects are similar and will not be repeated here.

[0089] In this embodiment, the present application can divide the functional units according to the fast clustering method. For example, each function can be divided into each functional unit, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the present invention is schematic and is only a logical division. There may be other division methods in actual implementation.

[0090] In this embodiment, in order to realize the principles and beneficial effects of the fast clustering method, the fast clustering system includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the schematic units and algorithm steps described in the embodiments disclosed in the present invention, the present invention can be implemented in the form of hardware and / or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven manner depends on the specific application and design constraints of the technical solution. Different methods can be used for each specific application to implement the described function, but such implementation should not be considered to exceed the scope of this application.

Claims

1. A clustering method for massive data, characterized in that: The following steps are involved: S1. Divide the space into grids, assign all data points to the corresponding grids, and create a sub-grid structure with a side length of half the grid structure; S2, build sub-grid index and retrieve sub-grid neighbors; S3. In the sub-grid structure, the core points of the massive data are identified by traversing the neighborhood of each data point and calculating the distance; S4. Based on the core point identification results of the massive data, the core grids are merged and non-core points are allocated to complete the rapid clustering of the massive data.

2. The clustering method for massive data according to claim 1, characterized in that: The S1 is specifically: Use the following formula to calculate the grid coordinates (c1, c2, ..., c i ): i∈(0,1,...,d-1) Among them, c i represents the coordinates of the grid to which the data point p belongs in the i-th dimension, ε represents the neighborhood radius, and p i Represents the coordinates of data point p in the i-th dimension, mn i It represents the minimum coordinate value of all data points in the i-th dimension, and d represents the data dimension; Based on the grid coordinates, the lowest binary bits of the d data dimensions are combined as the sub-grid coordinates of the data point p; For grid coordinates, all data points are sorted by their grid coordinates in cardinality order, and corresponding grid structures are created in sequence, where the number of members in each subgrid of each grid is cached when obtaining the grid coordinates of data point p; For subgrid coordinates, d binary numbers form a d-bit binary integer, which represents the subgrid number of the data point p in the grid to which it belongs; After processing all data points, count sorting is performed on the members according to the sub-grid numbers in each grid, and based on the sorting results, a sub-grid member count table is constructed to complete the establishment of the sub-grid structure, where the side length of the sub-grid structure is half of the grid structure.

3. The clustering method for massive data according to claim 1, characterized in that: The S3 is specifically: S301, check whether the number of members in each sub-grid structure is at least parameter MinPts, if so, mark each member of the sub-grid whose members are at least parameter MinPts as a core point, otherwise, go to step S302; S302, initializing a temporary counter, and setting the initial value of the temporary counter to the member count in the sub-grid structure; S303, from the nearest to the farthest distance, process each adjacent subgrid of the current subgrid of each member p, wherein for each processed subgrid, from the nearest to the farthest distance, preferentially calculate the distance of the member of the subgrid closest to member p, and the members of the adjacent subgrid are directly determined as neighbors of member p without calculation, and their member counts are directly added to the neighbor counter, wherein member p is data point p; S304. Based on the processing result of step S303, when the number of neighbors of member p is at least the parameter MinPts, the calculation is terminated and member p is marked as a core point, thereby completing the identification of the core point.

4. A fast clustering system for massive data, characterized in that: include: The first processing module is used to divide the space into grids, assign all data points to corresponding grids, and create a sub-grid structure with a side length half of the grid structure; A second processing module, for constructing a subgrid index and retrieving the subgrid's neighbors; The third processing module is used to identify the core points of the massive data by traversing the neighborhood of each data point and calculating the distance in the sub-grid structure; The fourth processing module is used to merge core grids and allocate non-core points according to the core point recognition results of the massive data, so as to complete the rapid clustering of the massive data.

Images