Data dynamic partition storage method and system based on adaptive clustering

Abstract

The invention discloses a data dynamic partition storage method and system based on adaptive clustering, and relates to the field of data processing. The method comprises the following steps: S1, extracting multi-scale geometric features of a high-dimensional data set, calculating a local curvature and generating a curvature feature matrix; s2, constructing a feature distance matrix and a similar matrix based on the curvature feature matrix, and generating a low-dimensional embedding matrix; s3, executing self-optimization clustering according to the low-dimensional embedded matrix, determining a cluster number through singular value distribution, and generating an initial cluster; s4, calculating a stability factor of each cluster, and triggering cluster splitting or merging operation according to a threshold value to form updated cluster division; and S5, performing incremental processing on newly added data points, obtaining a new point local curvature through local curvature gradient correction, mapping the new point local curvature to a low-dimensional space, dynamically deciding affiliation based on a cluster radius, and updating partitions. Through multi-scale feature extraction, self-optimization clustering and incremental updating mechanisms, high-dimensional data clustering precision, robustness and calculation efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, specifically to a dynamic partitioning and storage method and system for data based on adaptive clustering. Background Technology

[0002] With the rapid increase in data scale and dimensionality, traditional data storage and clustering methods are struggling to meet the requirements of real-time performance and efficiency. Especially for high-dimensional datasets, effectively extracting and processing the geometric features of the data during clustering, and avoiding the curse of dimensionality while ensuring clustering accuracy, have become major research challenges. Existing clustering methods largely rely on static data partitioning, making it difficult to adapt to dynamic data changes, and they often suffer from high computational complexity when processing large-scale data.

[0003] To address these issues, dynamic partitioning storage based on adaptive clustering has gradually become an effective solution. This method not only adaptively adjusts the clustering structure according to different scales of the data, but also performs incremental updates as new data arrives, ensuring the real-time performance and accuracy of clustering. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a data dynamic partitioning storage method and system based on adaptive clustering to solve the above-mentioned technical problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a dynamic data partitioning and storage method based on adaptive clustering, comprising:

[0006] S1: Extract multi-scale geometric features from high-dimensional datasets by constructing a multi-scale adjacency matrix and calculating the local curvature of each data point at each scale to generate a curvature feature matrix;

[0007] S2: Based on the curvature feature matrix, construct the feature distance matrix and similarity matrix to generate a low-dimensional embedding matrix;

[0008] S3: Perform self-optimizing clustering based on the low-dimensional embedding matrix, determine the number of clusters using the singular value distribution of the self-optimizing index matrix of angular similarity and adaptive distance similarity, and generate the initial cluster partition;

[0009] S4: Calculate the stability factor of each cluster, and trigger cluster splitting or merging operations based on the adaptively updated splitting and merging thresholds to form the updated cluster partitioning;

[0010] S5: Incremental processing is performed on newly added data points. The local curvature of the new point is obtained by gradient correction of the local curvature of the neighboring points. After mapping to a low-dimensional space, the association relationship is dynamically determined based on the cluster radius, and the partition structure is updated.

[0011] The present invention is further configured such that S1 includes:

[0012] Based on the distance distribution characteristics of high-dimensional datasets, an initial scale parameter and scale growth factor are preset to construct a scale set with exponentially increasing scale.

[0013] For each scale, an adjacency matrix is ​​constructed. If the Euclidean distance between two data points is less than or equal to the current scale value, an adjacency relationship is established. The adjacency weight is calculated using an exponential decay function of the ratio of the square of the Euclidean distance to the square of the current scale value.

[0014] Based on adjacency weights, calculate the weighted average distance of each data point at each scale;

[0015] Calculate the local curvature of the data points at each scale based on the adjacency weight, weighted average distance, and Euclidean distance between each data point and its neighbors at the current scale.

[0016] By integrating the local curvature of data points at all scales, a curvature feature matrix that integrates multi-scale geometric features is constructed.

[0017] The present invention is further configured such that S2 includes:

[0018] Calculate the scale-weighted feature distance between any two data points in the curvature feature matrix, and construct a feature distance matrix. The scale-weighted feature distance is obtained by summing the absolute difference of the local curvature of the two data points at each scale after scale attenuation weighting.

[0019] The similarity is calculated by performing a negative exponential transformation based on the ratio of scale-weighted feature distance matrix elements to local adaptive bandwidth parameters, and a similarity matrix is ​​constructed.

[0020] Construct the degree matrix and Laplacian matrix using similarity matrices, solve the generalized eigenvalue problem of the Laplacian matrix relative to the degree matrix, and construct a low-dimensional embedding matrix by selecting the eigenvectors corresponding to the first q smallest non-zero eigenvalues.

[0021] The local adaptive bandwidth parameter is determined by calculating the average distance of a preset number of nearest neighbors for each data point in the feature distance matrix.

[0022] The present invention is further configured such that S3 includes:

[0023] Based on the feature vectors of all data points in the low-dimensional embedding matrix, a self-optimizing index matrix is ​​constructed by calculating the product of the angle cosine similarity between two feature vectors and the distance exponential decay function based on the local distance adaptive parameter.

[0024] Singular value decomposition is performed on the self-optimization index matrix to extract the descending sequence of singular values;

[0025] The number of clusters, c, is determined by calculating the maximum difference between adjacent singular values;

[0026] Using the first c columns of the left singular vector matrix generated by singular value decomposition as input features, a clustering algorithm is executed to generate the initial cluster partition;

[0027] The local distance adaptive parameter corresponds to each feature vector itself and is determined by the average Euclidean distance between it and a preset number of nearest neighbor feature vectors in the low-dimensional embedding matrix.

[0028] The present invention is further configured such that S4 includes:

[0029] For each cluster in the current cluster partitioning, the stability factor is obtained by calculating the sum of the squared Euclidean distances from all data points in the cluster to the cluster center and the logarithmic weighted product of the neighborhood dispersion of the corresponding points, and then dividing by the cluster size.

[0030] When the cluster size is greater than the preset minimum cluster threshold and the stability factor is greater than the preset splitting threshold, perform a binary k-means splitting operation;

[0031] For any two cluster combinations, if the mean of the distances between all point pairs calculated by the Gaussian kernel function is greater than the preset merging threshold, then the cluster merging operation is performed.

[0032] The splitting threshold is updated using the median value of all cluster stability factors, and the merging threshold is updated using the median value of the mean of the Gaussian kernel functions among all non-identical clusters.

[0033] The present invention is further configured such that S5 includes:

[0034] For each new data point, calculate its Euclidean distance to historical data points to determine the nearest neighbor historical data point;

[0035] The local curvature of the new point is generated based on the local curvature and the linear offset correction value of the curvature gradient direction of the nearest neighbor historical data point.

[0036] The local curvature of newly added data points is projected onto the low-dimensional embedding matrix using a mapping matrix from historical local curvature to low-dimensional embedding.

[0037] Calculate the Euclidean distance from the low-dimensional embedding matrix of the newly added data point to the center of each cluster. If the minimum distance is less than or equal to the radius threshold of the corresponding cluster, it is assigned to the nearest cluster; otherwise, it is stored independently as a new cluster.

[0038] The radius threshold is determined by the maximum distance from all points within the corresponding cluster to the cluster center.

[0039] The present invention is further configured such that the linear offset correction value of the curvature gradient direction is obtained by projection operation in Euclidean space.

[0040] The present invention is further configured such that when a new data point is stored independently as a new cluster, the cluster radius threshold is initialized to the Euclidean distance from the new point to the nearest cluster center.

[0041] The present invention is further configured such that the method also includes automatically identifying and processing noise points in the data to optimize the accuracy of the clustering results.

[0042] The present invention also provides a data dynamic partitioning storage system based on adaptive clustering, the system comprising:

[0043] Multi-scale feature extraction module: used to extract multi-scale geometric features of high-dimensional datasets. It generates a curvature feature matrix by constructing a multi-scale adjacency matrix and calculating the local curvature of each data point at each scale.

[0044] Feature dimensionality reduction module: used to construct feature distance matrix and similarity matrix based on curvature feature matrix, and generate low-dimensional embedding matrix;

[0045] Self-optimizing clustering module: Used to perform self-optimizing clustering based on low-dimensional embedding matrix, determine the number of clusters by using the singular value distribution of the self-optimizing index matrix of angular similarity and adaptive distance similarity, and generate initial cluster partitioning;

[0046] Cluster Evolution Module: Used to calculate the stability factor of each cluster, and trigger cluster splitting or merging operations based on the adaptively updated splitting and merging thresholds to form the updated cluster partitions;

[0047] Incremental update module: Used to perform incremental processing on newly added data points. It obtains the local curvature of the new point by gradient correction of the local curvature of the neighboring points, maps it to a low-dimensional space, and then dynamically decides the affiliation relationship based on the cluster radius to update the partition structure.

[0048] This invention provides a data dynamic partitioning and storage method and system based on adaptive clustering. The method involves: S1: Extracting multi-scale geometric features from a high-dimensional dataset, constructing a multi-scale adjacency matrix and calculating the local curvature of each data point at each scale to generate a curvature feature matrix; S2: Based on the curvature feature matrix, constructing a feature distance matrix and a similarity matrix to generate a low-dimensional embedding matrix; S3: Performing self-optimizing clustering based on the low-dimensional embedding matrix, determining the number of clusters using the singular value distribution of the self-optimizing index matrix of angular similarity and adaptive distance similarity, and generating an initial cluster partition; S4: Calculating the stability factor of each cluster, triggering cluster splitting or merging operations based on adaptively updated splitting and merging thresholds to form an updated cluster partition; S5: Incrementally processing newly added data points, obtaining the local curvature of new points through gradient correction of the local curvature of neighboring points, mapping it to a low-dimensional space, and dynamically deciding the affiliation relationship based on the cluster radius to update the partitioning structure. The beneficial effects include:

[0049] 1. By extracting multi-scale geometric features, calculating local curvature, and using low-dimensional embedding matrices, the latent structure of data points in high-dimensional data can be accurately identified, thereby improving the accuracy and robustness of clustering. Especially when dealing with complex datasets, it can reduce the problem of poor performance of traditional clustering methods in high-dimensional data.

[0050] 2. Through self-optimizing clustering and incremental update mechanisms, this invention can automatically adjust the clustering structure according to changes in data, ensuring the real-time and adaptability of cluster partitioning. The incremental processing method for new data points avoids the computational pressure of traditional batch processing methods, improving the system's response speed and data update efficiency.

[0051] 3. By calculating the stability factor of the cluster and combining it with splitting and merging strategies, the cluster partitioning can be automatically adjusted according to the evolution of the dataset, thereby maintaining the dynamic optimization of the partitioning results. This mechanism avoids data distortion and computational redundancy caused by static cluster partitioning.

[0052] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0054] Figure 1 A flowchart illustrating a dynamic data partitioning and storage method based on adaptive clustering is shown as an exemplary embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram illustrating the structure of a dynamic data partitioning storage system based on adaptive clustering, as an exemplary embodiment of the present invention. Detailed Implementation

[0056] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0057] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0058] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0059] Example 1

[0060] A dynamic data partitioning and storage method based on adaptive clustering, such as Figure 1 As shown, it includes:

[0061] S1: Extract multi-scale geometric features from high-dimensional datasets by constructing a multi-scale adjacency matrix and calculating the local curvature of each data point at each scale to generate a curvature feature matrix;

[0062] S2: Based on the curvature feature matrix, construct the feature distance matrix and similarity matrix to generate a low-dimensional embedding matrix;

[0063] S3: Perform self-optimizing clustering based on the low-dimensional embedding matrix, determine the number of clusters using the singular value distribution of the self-optimizing index matrix of angular similarity and adaptive distance similarity, and generate the initial cluster partition;

[0064] S4: Calculate the stability factor of each cluster, and trigger cluster splitting or merging operations based on the adaptively updated splitting and merging thresholds to form the updated cluster partitioning;

[0065] S5: Incremental processing is performed on newly added data points. The local curvature of the new point is obtained by gradient correction of the local curvature of the neighboring points. After mapping to a low-dimensional space, the association relationship is dynamically determined based on the cluster radius, and the partition structure is updated.

[0066] The present invention is further configured such that S1 includes:

[0067] Based on the distance distribution characteristics of high-dimensional datasets, an initial scale parameter and scale growth factor are preset to construct a scale set with exponentially increasing scale.

[0068] For each scale, an adjacency matrix is ​​constructed. If the Euclidean distance between two data points is less than or equal to the current scale value, an adjacency relationship is established. The adjacency weight is calculated using an exponential decay function of the ratio of the square of the Euclidean distance to the square of the current scale value.

[0069] Based on adjacency weights, calculate the weighted average distance of each data point at each scale;

[0070] Calculate the local curvature of the data points at each scale based on the adjacency weight, weighted average distance, and Euclidean distance between each data point and its neighbors at the current scale.

[0071] By integrating the local curvature of data points across all scales, a curvature feature matrix fusing multi-scale geometric features is constructed. Specifically, in this embodiment, the high-dimensional dataset is set as follows: Where D is a high-dimensional dataset, x i For the i-th data point, d high For the original feature dimensions of the data, For d high Let be a real space, i be the index of a data point, and n be the number of data points. Based on the distance distribution characteristics of the high-dimensional dataset, an initial scale parameter and a scale growth factor are preset, and the scale set S is formed by exponentially increasing the scale set. This scale set is used to characterize the local structure of the data in different neighborhoods. The specific form of the scale set is: S = {s} m =R·α m-1 |m=1,2,…,M}, where S is the scale set, s m Let R be the current scale value, α be the scale parameter, α be the scale growth factor, M be the number of scales, and m be the scale index. The scale parameter R can be preset according to the distance distribution characteristics of the high-dimensional dataset. For example, the Euclidean distance between all data points in the high-dimensional dataset can be calculated, and the maximum distance can be used as a reference value for the initial scale parameter to determine a suitable scale setting. α is used to control the exponential expansion of the scale, and its value range is [1, 10]. At each scale s... m Next, construct an adjacency matrix for all n data points. For any two data points x... i With x j If the Euclidean distance between the two satisfies: ∥x i -x j ∥≤s m , where s m If the current scale value is used as the threshold for the neighborhood range, then the data point x in the adjacency matrix is... i With x j Establish adjacency relationships and calculate the corresponding adjacency weights. The calculation logic for adjacency weights is as follows: in, The adjacency weight represents the number of elements in the adjacency matrix at scale s. m Next data point xi With x j Similarity measure; ∥x i -x j ∥ represents data point x i With x j The Euclidean distance between the two points; through this exponential decay function, a local neighborhood topology is established at a specific scale, enhancing the influence between nearby data points and reducing the interference of distant data points; if the Euclidean distance between two points does not satisfy the above conditions, i.e., ∥x i -x j ∥>s m The corresponding adjacency weight is: For each data point x i The Euclidean distances of points in the neighborhood are weighted and summed according to the adjacency weights to obtain the distance of the point at scale s. m The weighted average distance is calculated as follows: in, For data points at scale s m The weighted average distance between each data point and its neighbors is used to calculate the distance at scale s. Combining the adjacency weight, weighted average distance, and Euclidean distance between each data point and its neighbors, the distance at scale s is calculated for each data point. m The local curvature is used to describe the degree of curvature of the data point in the local geometric space. The calculation logic of the local curvature is as follows: in, For data point x i At scale s m The local curvature under the data point reflects the degree of curvature of the geometric structure in the neighborhood of the data point. To be at scale s m Next data point x i The neighborhood set of x, that is, all x i The Euclidean distance is less than or equal to s m The set of data points; the local curvature of the data points at different scales. Integrate the data to construct the curvature feature matrix K∈R n×M Where K is the curvature feature matrix, used to represent the local curvature information of each data point at different scales, and each row of the curvature feature matrix corresponds to the local curvature of a data point at all scales; R n×M Let K be the dimension of the curvature feature matrix, n be the number of data points, and M be the number of scales; the elements in the curvature feature matrix are represented as follows: K i,m Let x be the element in the i-th row and m-th column of the curvature feature matrix, defined as the data point x. i At scale s mThe local curvature of the data is analyzed. Through the above steps, this embodiment realizes the extraction of multi-scale geometric features from high-dimensional data. Compared with the traditional single-scale neighborhood method, multi-scale curvature can comprehensively reflect the local bending and structural features of the data, and has stronger robustness in dealing with high-dimensional noise and redundant dimensions.

[0072] The present invention is further configured such that S2 includes:

[0073] Calculate the scale-weighted feature distance between any two data points in the curvature feature matrix, and construct a feature distance matrix. The scale-weighted feature distance is obtained by summing the absolute difference of the local curvature of the two data points at each scale after scale attenuation weighting.

[0074] The similarity is calculated by performing a negative exponential transformation based on the ratio of scale-weighted feature distance matrix elements to local adaptive bandwidth parameters, and a similarity matrix is ​​constructed.

[0075] Construct the degree matrix and Laplacian matrix using similarity matrices, solve the generalized eigenvalue problem of the Laplacian matrix relative to the degree matrix, and construct a low-dimensional embedding matrix by selecting the eigenvectors corresponding to the first q smallest non-zero eigenvalues.

[0076] The local adaptive bandwidth parameter is determined by calculating the average distance of a preset number of nearest neighbors for each data point in the feature distance matrix; specifically, in this example, for the curvature feature matrix K∈R n×M Any two data points x i and x j The absolute difference of local curvature at each scale is calculated, and the values ​​are accumulated by weighting according to scale attenuation to obtain the scale-weighted feature distance. The calculation logic of the scale-weighted feature distance is as follows: Where, Δ ij For data point x i With x j Scale-weighted feature distance; K i,m and K j,m Data points x i and x j At scale s m The local curvature is denoted by τ, which is the scale decay coefficient used to emphasize the stability of the structure at large scales, and its value ranges from [1,5]. The exponential decay function exp(-m / τ) is used to suppress the influence of small-scale noise on the distance while maintaining the dominant role of the large-scale structure. Based on the scale-weighted feature distance of all data point pairs, a feature distance matrix is ​​constructed, and the form of the feature distance matrix is: Δ∈R n×n This is used to quantify the difference between any two data points in the curvature feature matrix; to account for density differences around the data points, a local adaptive bandwidth parameter is introduced, calculated by the data point x. iThe local adaptive bandwidth parameter is obtained by averaging the distances to its k nearest neighbors in the feature distance matrix Δ. The calculation logic is as follows: σ i For data point x i The local adaptive bandwidth parameter is used to measure the bandwidth of data point x. i The density of the surrounding neighborhood; k is the preset number of nearest neighbors; N k (Δ i (x) represents the data point. i The set of k nearest neighbors; by introducing σ i The scale-weighted feature distance can adaptively adjust across different density regions, thus ensuring the comparability of distance metrics; based on the scale-weighted feature distance Δ in the distance feature matrix... i j and local adaptive bandwidth parameter σ i The similarity between data points is calculated using the following logic: Among them, W ij For data point x i and x j similarity, W ij A larger value indicates that the two data points are more similar; σ j For data point x j The local adaptive bandwidth parameter; exp(·) is a negative exponential function used to map the scale-weighted feature distance to the [0,1] interval; the above similarity calculation formula converts the scale-weighted feature distance into similarity through the local adaptive bandwidth parameter, making the similarity of data points in different density regions comparable; based on the similarity between all data point pairs, a similarity matrix is ​​constructed, and the form of the similarity matrix is: W∈R n×n The similarity matrix W is used to represent the similarity of data points in a multi-scale feature space. The elements of the similarity matrix W... ij The larger the value, the higher the value of the data point x. i and x j The more similar the matrix, the better; construct the degree matrix D∈R based on the similarity matrix W. n ×n The calculation logic for the diagonal elements in the degree matrix is ​​as follows: Among them, D ii Let x be the i-th diagonal element of the degree matrix D, representing the data point x. i The sum of similarities to all other data points is used to measure the similarity of data point x. i The overall connection weights; the degree matrix D is used to statistically analyze data points x. i Connection strength with the entire dataset; construct the Laplacian matrix L∈R based on the obtained degree matrix D and similarity matrix W. n×nThe construction logic of the Laplacian matrix is ​​as follows: L = DW, where L is the Laplacian matrix, used to reflect the local and global geometric relationships between data points. For the Laplacian matrix L and the degree matrix D, the generalized feature problem is solved: Lv = λDv, where v is the eigenvector, used to reflect the overall topological relationship and local neighborhood structure between data points; λ is the eigenvalue, corresponding to the quantization index of the eigenvector v. To achieve low-dimensional embedding representation, the eigenvectors corresponding to the q smallest non-zero eigenvalues ​​are selected from the solved eigenvectors to form the low-dimensional embedding matrix V∈R. n×q Each row of the low-dimensional embedding matrix V is v i Corresponding data point x i Feature vectors in a low-dimensional space; this low-dimensional embedding means that while preserving the high-dimensional geometric structure of the data, dimensional redundancy is reduced, thereby improving the efficiency and accuracy of subsequent clustering.

[0077] The present invention is further configured such that S3 includes:

[0078] Based on the feature vectors of all data points in the low-dimensional embedding matrix, a self-optimizing index matrix is ​​constructed by calculating the product of the angle cosine similarity between two feature vectors and the distance exponential decay function based on the local distance adaptive parameter.

[0079] Singular value decomposition is performed on the self-optimization index matrix to extract the descending sequence of singular values;

[0080] The number of clusters, c, is determined by calculating the maximum difference between adjacent singular values;

[0081] Using the first c columns of the left singular vector matrix generated by singular value decomposition as input features, a clustering algorithm is executed to generate the initial cluster partition;

[0082] The local distance adaptive parameter corresponds to each feature vector itself and is determined by the average Euclidean distance between it and a predetermined number of nearest neighbor feature vectors in the low-dimensional embedding matrix; specifically, the self-optimization index matrix Z∈R n×n This is used to fuse angular similarity between feature vectors and distance similarity based on local distance adaptive parameters. The elements of the self-optimizing index matrix consist of two parts: the cosine similarity between feature vectors and the distance exponential decay function defined by the Euclidean distance between feature vectors and the local distance adaptive parameters. The calculation logic of the elements of the self-optimizing index matrix is ​​as follows: Among them, Z ij For each element in the self-optimizing index matrix, v represents the eigenvector. i and v j Similarity between them; γ i and γ j These are local distance adaptive parameters, used to reflect the feature vector v. iand v j Local density characteristics; γ i The calculation logic is as follows: Where, j∈N k (v i ) is related to the feature vector v i f nearest neighbor sets; Used to calculate the eigenvector v i The average distance γ between it and its f nearest neighbors j The calculation logic is the same, and will not be elaborated here; For the feature vector v i and v j The cosine similarity between two feature vectors is calculated by normalizing the inner product of the two feature vectors. The larger the inner product, the more similar the two feature vectors are. <v i ,v j > is the feature vector v i and v j The inner product, ∥v i ∥ and ∥v j ∥ are the feature vectors v i and v j The modulus length; The function is an exponential decay function, representing distance similarity; to automatically determine the number of clusters in the data, the self-optimizing index matrix Z∈R is used. n×n Perform singular value decomposition: Z = UΣU T We obtain the singular value matrix Σ and the left singular vector matrix U, where Σ is a diagonal matrix containing all singular values. The singular values ​​reflect the importance of each dimension of the data structure, and Σ has the form: Σ=diag(σ1≥σ2≥…≥σ n U is the corresponding left singular vector matrix, representing the distribution of data in each singular value direction; by analyzing the singular value sequence {σ1,σ2,…,σ...} n Sort the data and calculate the difference between adjacent singular values ​​to determine the clustering structure. The calculation logic for the difference between adjacent singular values ​​is: δ r =σ r -σ r+1 , where δ r σ is the difference between the r-th singular value and its next singular value r+1. The larger the difference, the greater the change in the intrinsic structure of the data. The point of change corresponding to the largest difference is the cluster number; r Let be the r-th singular value in the singular value decomposition; r is the index in the singular value sequence, r = 1, 2, ..., n-1, representing the gap between the first singular value and the second-to-last singular value; the cluster number is the maximum value among the differences between adjacent singular values, i.e.: Where c is the number of clusters; using the first c columns of the left singular vector matrix U as feature input, the k-means clustering algorithm is executed to generate the initial cluster partition: C(0)={C1,C2,...,C c The method described above automatically determines the number of clusters, avoiding the subjectivity of manually setting the number of clusters required in traditional clustering methods. At the same time, by combining the weighted parameters of angular similarity and distance adaptive parameters, the accuracy of clustering is improved, which is especially suitable for processing datasets with complex geometric structures, thereby enhancing the clustering accuracy.

[0083] The present invention is further configured such that S4 includes:

[0084] For each cluster in the current cluster partitioning, the stability factor is obtained by calculating the sum of the squared Euclidean distances from all data points in the cluster to the cluster center and the logarithmic weighted product of the neighborhood dispersion of the corresponding points, and then dividing by the cluster size.

[0085] When the cluster size is greater than the preset minimum cluster threshold and the stability factor is greater than the preset splitting threshold, perform a binary k-means splitting operation;

[0086] For any two cluster combinations, if the mean of the distances between all point pairs calculated by the Gaussian kernel function is greater than the preset merging threshold, then the cluster merging operation is performed.

[0087] The splitting threshold is updated using the median value of all cluster stability factors, and the merging threshold is updated using the median value of the mean of the Gaussian kernel functions among all non-identical clusters. Specifically, the cluster stability factor is used to evaluate the stability of a cluster. It is calculated by summing the squared Euclidean distances from the feature vectors of all data points within the cluster to the cluster center and the logarithmic weighted product of the neighborhood dispersion, and then dividing by the cluster size. This stability factor combines information from both intra-cluster compactness and neighborhood dispersion. Specifically, the higher the intra-cluster compactness, i.e., the smaller the distance between the feature vectors of the data points and the cluster center, the larger the cluster stability factor. Neighborhood dispersion is measured by calculating the similarity between the feature vectors of data points within the cluster. When the distance between adjacent points is large, the intra-cluster dispersion increases, thus affecting the cluster stability. For each cluster C... h Calculate its stability factor. The calculation logic for the stability factor is as follows: Where, η h For cluster C h The stability factor is used to measure the combined compactness and dispersion of the feature vectors of data points within a cluster; m h For cluster C h The center of cluster C is represented by the center of cluster C. h The mean of the feature vectors of all data points in the dataset, i.e.: |C h |For cluster C hThe number of feature vectors of data points in the middle; ∥v i -m h ∥ 2 v is the feature vector of data points within the cluster. i With cluster center m h The square of the Euclidean distance between the data points represents the density of the data points to the cluster center; the smaller the value, the closer the data points are to the cluster center. η represents the neighborhood dispersion, used to assess the degree of dispersion within a cluster. Based on the stability factor and the cluster size, clusters can split or merge. A cluster can split or merge when its stability factor is greater than a preset split threshold and its size is greater than a preset minimum cluster threshold. h >θ split And |C h |>μ min Perform a binary k-means operation to further subdivide the cluster into two subclusters, where θ split To set the preset splitting threshold, μ min The preset minimum cluster threshold is used; for any two clusters C a and C b If the mean distance between the feature vectors of all its data point pairs, calculated using the Gaussian kernel function, is greater than a preset merging threshold, that is: Then perform a cluster merge operation to merge the two clusters, where |C a |and|C b |representing cluster C a and C b The number of feature vectors of data points; β is the parameter of the Gaussian kernel function, used to control the distance decay effect, and its value ranges from [0.1,1]. θ is a Gaussian kernel function used to convert the Euclidean distance between the feature vectors of two data points into a similarity value; merge The preset merging threshold is θ; the splitting threshold is θ. split The stability factor is updated using the median value of the stability factor for all clusters, i.e.: The updated split threshold; the merge threshold θ merge The update is performed using the median value of the mean of the Gaussian kernels among all non-identical clusters, i.e.: The updated merging threshold is used; through stability factor and adaptive threshold updates, the cluster partitioning process does not require human intervention, and automatically splits and merges according to data characteristics, improving the adaptability and accuracy of clustering results.

[0088] The present invention is further configured such that S5 includes:

[0089] For each new data point, calculate its Euclidean distance to historical data points to determine the nearest neighbor historical data point;

[0090] The local curvature of the new point is generated based on the local curvature and the linear offset correction value of the curvature gradient direction of the nearest neighbor historical data point.

[0091] The local curvature of newly added data points is projected onto the low-dimensional embedding matrix using a mapping matrix from historical local curvature to low-dimensional embedding.

[0092] Calculate the Euclidean distance from the low-dimensional embedding matrix of the newly added data point to the center of each cluster. If the minimum distance is less than or equal to the radius threshold of the corresponding cluster, it is assigned to the nearest cluster; otherwise, it is stored independently as a new cluster.

[0093] The radius threshold is determined by the maximum distance from all points within the corresponding cluster to the cluster center; specifically, let the newly added data point set be: Among them, D new For the newly added data point set, For the j-th newly added data point, where j is the index of the data point and p is the number of newly added data points; for each newly added data point By calculating the Euclidean distance between the new data point and historical data points, the historical data point with the smallest Euclidean distance to the new data point is selected as its nearest neighbor historical data point. The local curvature of newly added data points is based on the local curvature of the nearest neighbor historical data points. The calculation logic for the local curvature of newly added data points is as follows: The curvature is corrected by the linear offset correction value in the curvature gradient direction. in, For new data points At scale s m Local curvature below; For new data points At scale s m The correction value is obtained by using the nearest neighbor historical data points. The local curvature gradient is calculated. The calculation logic is as follows: in, For nearest neighbor historical data points At scale s m The gradient of the local curvature represents the rate at which the curvature changes with the location of the data point; The difference in feature vectors between a newly added data point and its nearest neighbor historical data point; Let P be the unit vector between the new data point and its nearest neighbor historical data point. Through the gradient correction method described above, the local curvature of the new data point is incrementally updated, ensuring a smooth transition of the new data point to the overall curvature structure. The local curvature of the new data point is projected onto the low-dimensional embedding mapping matrix P of the historical data points to obtain its low-dimensional embedding representation, i.e., the feature vector. The mapping matrix P is obtained from the curvature feature matrix and low-dimensional embedding matrix of the historical data points in step S2. A linear mapping relationship is established between the curvature feature matrix and the low-dimensional embedding matrix, and the mapping matrix P is solved by the least squares method. The feature vector of the newly added data points is calculated. to each cluster center m h Euclidean distance The calculation logic is as follows: If the minimum Euclidean distance of the newly added data points is less than or equal to the cluster radius threshold, that is: If a new data point is added, it will be assigned to that cluster; otherwise, it will be stored as a new, independent cluster. Where ρ h The radius threshold is determined by calculating the maximum distance from all points within the cluster to the cluster center, ρ. h The calculation logic is as follows: This radius threshold helps determine whether newly added data points conform to the density and shape of the current cluster, avoiding over- or erroneous cluster partitioning. Through the incremental update described above, the final output is a low-dimensional embedding matrix containing the newly added data points, and a cluster set updated based on the cluster affiliation decision. This set may include new clusters or existing clusters that have been adjusted. By adopting an incremental update method, the data structure can be updated smoothly, avoiding the recalculation of all data points, thereby improving computational efficiency, especially when processing dynamic data. At the same time, cluster partitioning achieves adaptability through computational stability and dynamic adjustment of the cluster radius threshold, overcoming the limitations of fixed cluster numbers and thresholds in traditional methods. In addition, newly added data points can be naturally integrated into the existing cluster structure, avoiding the high computational cost of re-clustering every time new data is added.

[0094] The present invention is further configured such that the linear offset correction value of the curvature gradient direction is obtained through projection operation in Euclidean space; specifically, through projection operation in Euclidean space, the linear offset correction value of the curvature gradient is calculated based on the distance between the new data point and the historical data point; the curvature feature of the new data point is corrected by the spatial difference between the curvature gradient of the historical data point and the new point, thereby ensuring the consistency of the data points in the low-dimensional space; this process improves the accuracy of incremental updates, ensures geometric consistency in the clustering process, and enhances the dynamic adaptability of the method, enabling the features of the new data point to be adaptively adjusted to cope with changes in the data structure.

[0095] The present invention is further configured such that when a new data point is stored independently as a new cluster, the radius threshold of the cluster is initialized to the Euclidean distance from the new data point to the nearest cluster center. Specifically, when a new data point cannot belong to an existing cluster, the new data point will be stored independently as a new cluster. In this process, to ensure the rationality of the new cluster, the radius threshold of the cluster is initialized to the Euclidean distance from the new data point to the nearest cluster center. This initialization process helps to set the boundaries of the new cluster, prevents the excessive expansion of the new cluster, and provides a valid reference for subsequent cluster merging or splitting operations. Through this setting, the generation of new clusters can ensure effective isolation between clusters while maintaining data structure consistency.

[0096] The present invention further includes the method of automatically identifying and processing noise points in the data to optimize the accuracy of clustering results. Specifically, a noise point identification mechanism is introduced to optimize clustering accuracy by automatically identifying and processing noise points in the data. During the clustering process, if some data points deviate significantly from the distance of most cluster members in the low-dimensional space, or cannot be assigned to any cluster, the system will automatically identify these points as noise and remove them. This process effectively reduces the interference of noise points on the clustering results, improves the accuracy and robustness of clustering, and ensures the efficiency of the final clustering results and the consistency of the inherent structure of the dataset.

[0097] Example 2

[0098] Please see Figure 2 This exemplary data dynamic partitioning storage system based on adaptive clustering includes:

[0099] Multi-scale feature extraction module: used to extract multi-scale geometric features of high-dimensional datasets. It generates a curvature feature matrix by constructing a multi-scale adjacency matrix and calculating the local curvature of each data point at each scale.

[0100] Feature dimensionality reduction module: used to construct feature distance matrix and similarity matrix based on curvature feature matrix, and generate low-dimensional embedding matrix;

[0101] Self-optimizing clustering module: Used to perform self-optimizing clustering based on low-dimensional embedding matrix, determine the number of clusters by using the singular value distribution of the self-optimizing index matrix of angular similarity and adaptive distance similarity, and generate initial cluster partitioning;

[0102] Cluster Evolution Module: Used to calculate the stability factor of each cluster, and trigger cluster splitting or merging operations based on the adaptively updated splitting and merging thresholds to form the updated cluster partitions;

[0103] Incremental update module: Used to perform incremental processing on newly added data points. It obtains the local curvature of the new point by gradient correction of the local curvature of the neighboring points, maps it to a low-dimensional space, and then dynamically decides the affiliation relationship based on the cluster radius to update the partition structure.

[0104] It should be noted that the adaptive clustering-based dynamic data partitioning storage system and the adaptive clustering-based dynamic data partitioning storage method provided in the above embodiments belong to the same concept. The specific methods by which each module and unit performs its operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the adaptive clustering-based dynamic data partitioning storage system provided in the above embodiments can be configured to have different functional modules perform the functions as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A dynamic data partitioning and storage method based on adaptive clustering, characterized in that, include: S1: Extract multi-scale geometric features from high-dimensional datasets by constructing a multi-scale adjacency matrix and calculating the local curvature of each data point at each scale to generate a curvature feature matrix; S2: Based on the curvature feature matrix, construct the feature distance matrix and similarity matrix to generate a low-dimensional embedding matrix; S3: Perform self-optimizing clustering based on the low-dimensional embedding matrix, determine the number of clusters using the singular value distribution of the self-optimizing index matrix of angular similarity and adaptive distance similarity, and generate the initial cluster partition; S4: Calculate the stability factor of each cluster, and trigger cluster splitting or merging operations based on the adaptively updated splitting and merging thresholds to form the updated cluster partitioning; S5: Incremental processing is performed on newly added data points. The local curvature of the new point is obtained by gradient correction of the local curvature of the neighboring points. After mapping to a low-dimensional space, the association relationship is dynamically determined based on the cluster radius, and the partition structure is updated.

2. The data dynamic partitioning and storage method based on adaptive clustering according to claim 1, characterized in that, S1 includes: Based on the distance distribution characteristics of high-dimensional datasets, an initial scale parameter and scale growth factor are preset to construct a scale set with exponentially increasing scale. For each scale, an adjacency matrix is ​​constructed. If the Euclidean distance between two data points is less than or equal to the current scale value, an adjacency relationship is established. The adjacency weight is calculated using an exponential decay function of the ratio of the square of the Euclidean distance to the square of the current scale value. Based on adjacency weights, calculate the weighted average distance of each data point at each scale; Calculate the local curvature of the data points at each scale based on the adjacency weight, weighted average distance, and Euclidean distance between each data point and its neighbors at the current scale. By integrating the local curvature of data points at all scales, a curvature feature matrix that integrates multi-scale geometric features is constructed.

3. The data dynamic partitioning and storage method based on adaptive clustering according to claim 1, characterized in that, S2 includes: Calculate the scale-weighted feature distance between any two data points in the curvature feature matrix, and construct a feature distance matrix. The scale-weighted feature distance is obtained by summing the absolute difference of the local curvature of the two data points at each scale after scale attenuation weighting. The similarity is calculated by performing a negative exponential transformation based on the ratio of scale-weighted feature distance matrix elements to local adaptive bandwidth parameters, and a similarity matrix is ​​constructed. Construct the degree matrix and Laplacian matrix using similarity matrices, solve the generalized eigenvalue problem of the Laplacian matrix relative to the degree matrix, and construct a low-dimensional embedding matrix by selecting the eigenvectors corresponding to the first q smallest non-zero eigenvalues. The local adaptive bandwidth parameter is determined by calculating the average distance of a preset number of nearest neighbors for each data point in the feature distance matrix.

4. The data dynamic partitioning and storage method based on adaptive clustering according to claim 1, characterized in that, S3 includes: Based on the feature vectors of all data points in the low-dimensional embedding matrix, a self-optimizing index matrix is ​​constructed by calculating the product of the angle cosine similarity between two feature vectors and the distance exponential decay function based on the local distance adaptive parameter. Singular value decomposition is performed on the self-optimization index matrix to extract the descending sequence of singular values; The number of clusters, c, is determined by calculating the maximum difference between adjacent singular values; Using the first c columns of the left singular vector matrix generated by singular value decomposition as input features, a clustering algorithm is executed to generate the initial cluster partition; The local distance adaptive parameter corresponds to each feature vector itself and is determined by the average Euclidean distance between it and a preset number of nearest neighbor feature vectors in the low-dimensional embedding matrix.

5. The data dynamic partitioning and storage method based on adaptive clustering according to claim 1, characterized in that, S4 includes: For each cluster in the current cluster partitioning, the stability factor is obtained by calculating the sum of the squared Euclidean distances from all data points in the cluster to the cluster center and the logarithmic weighted product of the neighborhood dispersion of the corresponding points, and then dividing by the cluster size. When the cluster size is greater than the preset minimum cluster threshold and the stability factor is greater than the preset splitting threshold, perform a binary k-means splitting operation; For any two cluster combinations, if the mean of the distances between all point pairs calculated by the Gaussian kernel function is greater than the preset merging threshold, then the cluster merging operation is performed. The splitting threshold is updated using the median value of all cluster stability factors, and the merging threshold is updated using the median value of the mean of the Gaussian kernel functions among all non-identical clusters.

6. The data dynamic partitioning and storage method based on adaptive clustering according to claim 1, characterized in that, S5 includes: For each new data point, calculate its Euclidean distance to historical data points to determine the nearest neighbor historical data point; The local curvature of the new point is generated based on the local curvature and the linear offset correction value of the curvature gradient direction of the nearest neighbor historical data point. The local curvature of newly added data points is projected onto the low-dimensional embedding matrix using a mapping matrix from historical local curvature to low-dimensional embedding. Calculate the Euclidean distance from the low-dimensional embedding matrix of the newly added data point to the center of each cluster. If the minimum distance is less than or equal to the radius threshold of the corresponding cluster, it is assigned to the nearest cluster; otherwise, it is stored independently as a new cluster. The radius threshold is determined by the maximum distance from all points within the corresponding cluster to the cluster center.

7. A data dynamic partitioning and storage method based on adaptive clustering according to claim 6, characterized in that, The linear offset correction value of the curvature gradient direction is obtained through projection operation in Euclidean space.

8. A data dynamic partitioning and storage method based on adaptive clustering according to claim 6, characterized in that, When a new data point is stored independently as a new cluster, the initial cluster radius threshold is the Euclidean distance from the new point to the nearest cluster center.

9. A data dynamic partitioning and storage method based on adaptive clustering according to claim 1, characterized in that, The method also includes automatically identifying and processing noise points in the data to optimize the accuracy of clustering results.

10. A data dynamic partitioning storage system based on adaptive clustering, used to implement the data dynamic partitioning storage method based on adaptive clustering as described in any one of claims 1-9, characterized in that, include: Multi-scale feature extraction module: used to extract multi-scale geometric features of high-dimensional datasets. It generates a curvature feature matrix by constructing a multi-scale adjacency matrix and calculating the local curvature of each data point at each scale. Feature dimensionality reduction module: used to construct feature distance matrix and similarity matrix based on curvature feature matrix, and generate low-dimensional embedding matrix; Self-optimizing clustering module: Used to perform self-optimizing clustering based on low-dimensional embedding matrix, determine the number of clusters by using the singular value distribution of the self-optimizing index matrix of angular similarity and adaptive distance similarity, and generate initial cluster partitioning; Cluster Evolution Module: Used to calculate the stability factor of each cluster, and trigger cluster splitting or merging operations based on the adaptively updated splitting and merging thresholds to form the updated cluster partitioning; Incremental update module: Used to perform incremental processing on newly added data points. It obtains the local curvature of the new point by gradient correction of the local curvature of the neighboring points, maps it to a low-dimensional space, and then dynamically decides the affiliation relationship based on the cluster radius to update the partition structure.

Images