Fair clustering optimization method for large-scale data

Abstract

The application discloses a fair clustering optimization method for large-scale data, which is applied to the technical field of data processing and comprises the following steps: pre-processing an obtained data set, marking data of sensitive attributes in the data set, and dividing the pre-processed data set into a plurality of subsets; distributing the plurality of subsets to a plurality of nodes, locally clustering the plurality of nodes, and obtaining local clustering results; wherein the clustering process comprises the following steps: iteratively evaluating the local clustering results according to statistical characteristics of the data set and a clustering quality evaluation index, and obtaining an optimal clustering number; monitoring the distribution of the data of the sensitive attributes in the local clustering results by using a fairness constraint mechanism, and judging whether the local clustering results meet preset fairness constraint conditions; and iteratively optimizing the local clustering results, combining and evaluating the optimized local clustering results of the nodes based on a central server, and generating global clustering results.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a fair clustering optimization method for large-scale data. Background Technology

[0002] In the context of "big data," designing effective clustering algorithms to handle large-scale datasets has become a significant challenge in data mining and machine learning. Fair clustering optimization algorithms for large-scale data aim to improve computational efficiency while ensuring fairness among different groups. Existing research shows that traditional clustering methods such as K-means, hierarchical clustering, and spectral clustering are generally effective for small datasets, but their computational complexity increases rapidly and their efficiency decreases significantly when faced with massive datasets. Furthermore, because large-scale datasets contain groups with different characteristics and attributes, existing clustering methods may exhibit bias towards certain groups, leading to insufficient fairness in clustering results across different groups. Fair clustering optimization algorithms for large-scale data need to strike a balance between processing efficiency and fairness, while designing a reasonable set of parameters, such as the number of clusters and the distribution of attributes within clusters, to maximize the accuracy and performance of data analysis while ensuring the fairness of clustering results.

[0003] To overcome these shortcomings, this application proposes a fair clustering optimization method for large-scale data, which enables rapid processing of large-scale data and ensures the fairness of the clustering results and their rationality in practical applications. Summary of the Invention

[0004] The purpose of this application is to provide a fair clustering optimization method for large-scale data, aiming to address the limitations of traditional methods in large-scale data processing.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] This application provides a fair clustering optimization method for large-scale data, including:

[0007] The acquired dataset is preprocessed, data with sensitive attributes in the dataset are labeled, and the preprocessed dataset is divided into several subsets;

[0008] The subsets are assigned to several nodes, and local clustering is performed on these nodes to obtain local clustering results. The clustering process includes: iteratively evaluating the local clustering results based on the statistical characteristics of the dataset and clustering quality evaluation indicators to obtain the optimal number of clusters; using a fairness constraint mechanism to monitor the distribution of the sensitive attribute data in the local clustering results and determine whether the local clustering results meet preset fairness constraints; if not, the sensitive attribute data is redistributed by adjusting the weights.

[0009] The local clustering results are optimized and iterated. Based on the central server, the optimized local clustering results of each node are combined and evaluated to generate the global clustering results.

[0010] Furthermore, the step of iteratively evaluating the local clustering results based on the statistical characteristics of the dataset and clustering quality evaluation indicators to obtain the optimal number of clusters specifically includes the following steps:

[0011] Based on a preset range of cluster numbers, the number of clusters is adjusted during iterations, and the local clustering results are evaluated; when the number of clusters makes the clustering quality index optimal, then that number is determined to be the optimal number of clusters.

[0012] Furthermore, the step of monitoring the distribution of the sensitive attribute data in the local clustering results using a fairness constraint mechanism, determining whether the local clustering results meet the preset fairness constraint conditions, and reallocating the sensitive attribute data by adjusting the weights if not, specifically includes the following steps:

[0013] The dataset is labeled with data representing sensitive attributes, and different groups of attributes are labeled with different colors; that is, dataset X, and different color sets Col = {col1, col2, ..., col...} l The function col: X → Col is used to assign a color to each data point, corresponding to the group it belongs to.

[0014] Divide the dataset X into k clusters X1, X2, ..., X... k , so that in each cluster X i In the data, the proportion of groups corresponding to different colors satisfies the preset fairness constraints.

[0015] For any subset and any color h ∈Col, define the set col h (X0)={j∈Xcol(j=col h} represents the color attribute as col in X0. h A set; using This represents the proportion of the h-th class in set X';

[0016] For the pre-defined l-dimensional upper and lower bound vectors and After clustering, the proportion of class h in each cluster is within the preset upper and lower bounds:

[0017]

[0018] Furthermore, the step of optimizing and iterating the local clustering results, and generating the global clustering results based on the optimized local clustering results of each node by combining and evaluating them through the central server, specifically includes the following steps:

[0019] Cluster centers were selected based on the statistical characteristics of the dataset for preliminary clustering.

[0020] The location of the cluster center is updated by calculating the distance between each data point in the dataset and the cluster center; the update rule is to select the location with the smallest sum of distances from each data point to the cluster center as the new cluster center.

[0021] When the change in cluster centers is lower than a preset threshold, the iteration process ends, and the globally optimal clustering result is obtained.

[0022] Furthermore, the central server also includes:

[0023] The local clustering results are transmitted to the central server via the network.

[0024] The central server evaluates and integrates the local clustering results of each node based on the location of the cluster center and the distribution of data points among various clusters; by adjusting the cluster boundaries, the local clustering results among different nodes are made the same.

[0025] This application provides an apparatus comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a fair clustering optimization method for large-scale data; the processor is configured to execute the program instructions stored in the memory to implement fair clustering optimization for large-scale data.

[0026] This application provides a storage medium storing processor-executable program instructions for executing a fair clustering optimization method for large-scale data.

[0027] This application provides a fair clustering optimization method for large-scale data, which has the following beneficial effects:

[0028] (1) Through the distributed computing architecture, the efficiency of clustering computing has been significantly improved, enabling it to adapt to the processing needs of large-scale datasets, providing a clear technical direction for the optimization of clustering algorithms, and improving the performance of clustering algorithms when processing big data.

[0029] (2) In view of the shortcomings of existing clustering algorithms in fairness processing, this application introduces a fairness constraint mechanism to ensure the balanced distribution of data with different sensitive attributes and avoid the data bias problem in traditional clustering algorithms. Through the fairness constraint mechanism, the bias in the clustering process is effectively reduced, making the clustering results more applicable and fair in various application scenarios.

[0030] (3) This application proposes an adaptive cluster number determination step, which solves the problem that traditional clustering algorithms usually require manual determination of the number of clusters, which is cumbersome in parameter tuning; through data analysis and quality assessment, the optimal number of clusters is automatically determined, avoiding errors caused by human intervention;

[0031] (4) The dynamic cluster center update mechanism is adopted to make the clustering results more reasonable and interpretable in large-scale data, especially in scenarios with high data heterogeneity. This helps to better understand the internal structure of the clustering results and provides more interpretable results for subsequent analysis. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the fair clustering optimization method for large-scale data according to Embodiment 1 of this application;

[0033] Figure 2 This is a schematic diagram of hierarchical clustering in Embodiment 1 of this application;

[0034] Figure 3 This is a schematic diagram of the dynamic selection of cluster centers in Embodiment 1 of this application;

[0035] Figure 4 This is a schematic diagram of the device structure in Embodiment 3 of this application;

[0036] Figure 5 This is a schematic diagram of the storage medium structure of Embodiment 4 of this application. Detailed Implementation

[0037] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0038] The following analysis, based on relevant technologies, examines existing solutions.

[0039] Current clustering algorithms have been studied in various fields of image processing, natural language processing, and computer vision. Application CN112132153B discloses a tomato fruit recognition method and system based on clustering and morphological processing, belonging to the field of image processing and recognition technology. This method acquires and preprocesses the image to be recognized, generates a color difference image, performs image segmentation based on the K-means clustering algorithm, and combines morphological processing to remove image noise. Finally, it extracts circular edges through edge contours to identify the tomato fruit. Application CN111414479B discloses a label extraction method based on short text clustering, belonging to the field of natural language processing technology. This method includes extracting useful words from short texts and calculating the similarity between short texts using word2vec. Assuming each short text is a cluster center, the short texts are clustered according to similarity, and after two pruning and merging operations, the cluster centers of each cluster are finally extracted as labels. This method generates labels with complete semantics and representative of the category while performing short text clustering. Application number CN109086720B discloses a face clustering method, belonging to the field of computer vision and image processing technology. This method acquires a set of face images to be clustered, constructs a face relationship graph, and connects unconnected nodes based on face feature similarity. When unconnected nodes still exist, a face clustering algorithm based on adjacency relation is used for connection clustering to obtain a clustered node set. For isolated nodes, connection clustering is performed based on their similarity to the clustered node set. Finally, the clustered node sets are merged globally to obtain the clustering result of the face image set.

[0040] While current research extensively explores various algorithms for large-scale data clustering, few studies comprehensively analyze how to ensure fairness of results while improving computational efficiency from a macro-level, big data processing perspective. Large-scale data processing involves coordinating and balancing different group attributes, often accompanied by excessive computational resource consumption and result bias. As datasets continue to expand, the challenge of accurately solving clustering problems arises. The heterogeneity and complexity of large-scale data necessitate a deeper investigation into the trade-off between computational efficiency and fairness brought about by different clustering schemes, aiming to maximize the rationality and fairness of clustering results. This problem requires comprehensive consideration of numerous factors, including but not limited to data scale, differences in group characteristics, allocation of computational resources, and the robustness of algorithm design, to establish more efficient and fair clustering models, providing support and guidance for optimization methods that balance fairness and computational efficiency. Therefore, designing an optimization method that can automatically adjust the number of clusters and handle large-scale data is crucial. The purpose of this invention is to maximize computational efficiency while ensuring clustering fairness, overcoming the limitations of traditional methods in large-scale data processing.

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0042] Example 1

[0043] Please see Figure 1 This is a flowchart illustrating the fair clustering optimization method for large-scale data according to Embodiment 1 of this application; the steps include:

[0044] S1: Preprocess the acquired dataset, label the data with sensitive attributes in the dataset, and divide the preprocessed dataset into several subsets.

[0045] In this embodiment, to address the computational complexity issue in large-scale data processing, this application employs a distributed computing architecture. The large-scale dataset is divided into several subsets, and these subsets are distributed across multiple computing nodes using distributed computing platforms such as Hadoop and Spark. Each node independently performs local clustering processing.

[0046] It is understandable that a distributed computing architecture can significantly reduce the load on individual computing nodes, accelerate the clustering process of large-scale data through parallel processing, and greatly improve computing efficiency. At the same time, the reasonable partitioning of data ensures the smooth progress of the clustering process and effectively reduces communication overhead.

[0047] S2: Assign the subsets to several nodes, perform local clustering on the nodes, and obtain local clustering results; wherein, the clustering process includes: iteratively evaluating the local clustering results based on the statistical characteristics of the dataset and clustering quality evaluation indicators to obtain the optimal number of clusters; using a fairness constraint mechanism to monitor the distribution of the sensitive attribute data in the local clustering results, and determining whether the local clustering results meet the preset fairness constraint conditions; if not, reallocating the sensitive attribute data by adjusting the weights.

[0048] In this embodiment, the selection and updating of cluster centers is one of the key factors affecting the accuracy of clustering results. This application proposes a dynamic cluster center updating mechanism, which updates the position of cluster centers in real time based on the clustering feedback of each round during the clustering process, making the clustering results more accurate and stable. The specific implementation steps are as follows:

[0049] In the initial stage, initial cluster centers are randomly selected based on the statistical characteristics of the data for the first round of clustering. Afterwards, the positions of the cluster centers are updated by calculating the distance between each data point and the cluster center. The update rule is to select the location with the smallest sum of distances from all data points to the cluster center as the new cluster center.

[0050] Dynamic updating is an iterative process that continuously adjusts the cluster centers in each iteration, gradually improving the accuracy of the clustering results. The clustering process stops only when the change in cluster centers falls below a set threshold. This dynamic cluster center update mechanism effectively avoids the sensitivity to initial centers found in traditional clustering algorithms, while ensuring that the clustering results are continuously optimized in multiple iterations, gradually approaching the optimal solution.

[0051] Furthermore, traditional clustering algorithms typically require a pre-defined number of clusters, but for large-scale and complex datasets, determining the exact number of clusters beforehand is often difficult. To address this issue, this application proposes an adaptive method for determining the number of clusters. The specific implementation steps are as follows:

[0052] During clustering, the statistical characteristics of the dataset and clustering quality evaluation metrics, such as silhouette coefficient and inter-cluster distance, are used to assess the current clustering quality. Initially, a rough range for the number of clusters is given, and the number of clusters is adjusted in subsequent iterations to ensure the accuracy and reasonableness of the clustering results. The clustering effect is continuously evaluated through multiple iterations. When a certain number of clusters achieves the optimal clustering quality metric, this number is automatically determined as the optimal number of clusters. This adaptive method avoids human intervention, improves the adaptability and flexibility of the algorithm, and performs particularly well when dealing with datasets of different sizes and complexities. The adaptive clustering number determination method ensures the reasonableness and stability of the clustering results, avoids over- or under-clustering, and improves the adaptability of the clustering algorithm in large-scale data scenarios.

[0053] Furthermore, when processing data containing sensitive attributes, such as race, gender, and age, traditional clustering methods may lead to clustering results biased towards a particular group, resulting in data bias. To address this issue, this application introduces a fairness constraint mechanism to ensure that data with different sensitive attributes are treated fairly in the clustering results. The specific steps are as follows:

[0054] In step S1, the acquired dataset is preprocessed, and data with sensitive attributes are labeled. Groups of different attributes are labeled with different colors. That is, for a given dataset X, the different color sets Col = {col1, col2, ..., col...} l The function `col` (X → Col) is used to assign a color to each data point, corresponding to its group. The dataset X is divided into k clusters: X1, X2, ..., X...k So that in each cluster X i In this model, the proportions of different color groups satisfy the fairness constraint, meaning no group is overrepresented, and no group is overly neglected. For any subset... and any color h ∈Col, define the set col h (x0)={j∈X:col(j=col h} represents the color attribute as col in X0. h A collection. Using This represents the proportion of the h-th class in set X'. For a predefined l-dimensional upper and lower bound vector... After clustering, the proportion of class h in each cluster is within the preset upper and lower bounds:

[0055]

[0056] Clustering results that satisfy the above constraints are called fair clustering. During each clustering process, the distribution of different sensitive attributes (such as race or gender) within the clusters is monitored. If a group with a certain sensitive attribute is over-concentrated or neglected in a particular cluster, the weights in the clustering algorithm are adjusted to redistribute data points, making the data distribution of different sensitive attributes more balanced. By monitoring the distribution of different sensitive attributes in real time, the fairness constraint mechanism ensures the fairness of data processing, avoids imbalance in clustering results within a specific group, and thus improves the representativeness of the clustering results and the impartiality in practical applications. This fairness constraint mechanism is particularly suitable for highly heterogeneous datasets, effectively reducing bias in data processing and enhancing the social and biological value of cluster analysis.

[0057] S3: Optimize and iterate the local clustering results, and generate global clustering results based on the optimized local clustering results of each node combined and evaluated by the central server.

[0058] In this embodiment, cluster centers are selected based on the statistical characteristics of the dataset for preliminary clustering. The positions of the cluster centers are updated by calculating the distances between the data points in the dataset and the cluster centers; the update rule is to select the position with the smallest sum of distances from all data points to the cluster center as the new cluster center. When the change in the number of cluster centers is lower than a preset threshold, the iteration process ends, and the globally optimal clustering result is obtained.

[0059] The central server also includes: transmitting local clustering results to the central server via the network; the central server evaluating and integrating the local clustering results of each node based on the location of the cluster centers and the distribution of data points among different clusters. By adjusting the cluster boundaries, the local clustering results among different nodes are made identical. Through this integration process, not only can large-scale datasets be processed efficiently, but the global consistency and accuracy of the clustering results can also be ensured.

[0060] In summary, Embodiment 1 of this application utilizes distributed computing architecture, fairness constraint mechanisms, adaptive cluster number determination methods, and dynamic cluster center update mechanisms to achieve rapid processing of large-scale data while ensuring the fairness of clustering results and their rationality in practical applications. Experimental simulations and verification with real datasets demonstrate its efficiency and fairness in processing large-scale datasets. This invention can improve the fairness and biological rationality of clustering results while maintaining computational efficiency, particularly significantly reducing clustering bias when processing datasets with sensitive attributes.

[0061] Example 2

[0062] Please see Figure 4 This is a schematic diagram of the device structure of Embodiment 2 of this application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0063] The memory 52 stores program instructions for implementing the fair clustering optimization method for large-scale data described above.

[0064] Processor 51 is used to execute program instructions stored in memory 52 to implement a fair clustering optimization for large-scale data.

[0065] The processor 51 can also be referred to as a CPU (Central Processing Unit).

[0066] Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0067] Example 3

[0068] Please see Figure 5This is a schematic diagram of the storage medium in Embodiment 3 of this application. The storage medium in this embodiment stores a program file 61 capable of implementing all the above methods. This program file 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or devices such as computers, servers, mobile phones, and tablets.

[0069] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0070] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0071] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

[0072] Of course, the present invention may have many other embodiments. Based on this embodiment, other embodiments obtained by those skilled in the art without any creative effort are all within the scope of protection of the present invention.

Claims

1. A fair clustering optimization method for large-scale data, characterized in that, include: The acquired dataset is preprocessed, data with sensitive attributes in the dataset are labeled, and the preprocessed dataset is divided into several subsets; The subsets are assigned to several nodes, and local clustering is performed on these nodes to obtain local clustering results. The clustering process includes: iteratively evaluating the local clustering results based on the statistical characteristics of the dataset and clustering quality evaluation indicators to obtain the optimal number of clusters; using a fairness constraint mechanism to monitor the distribution of the sensitive attribute data in the local clustering results and determine whether the local clustering results meet preset fairness constraints; if not, the sensitive attribute data is redistributed by adjusting the weights. The local clustering results are optimized and iterated. Based on the central server, the optimized local clustering results of each node are combined and evaluated to generate a global clustering result. The step of monitoring the distribution of the sensitive attribute data in the local clustering results using a fairness constraint mechanism, determining whether the local clustering results meet the preset fairness constraint conditions, and reallocating the sensitive attribute data by adjusting the weights if not meeting the constraints, specifically includes the following steps: The dataset is labeled with data representing sensitive attributes, and groups of different attributes are labeled with different colors; that is, dataset X consists of different color sets. Using functions Assign a color to each data point, corresponding to the group it belongs to; Divide the dataset X into k clusters. , so that in each cluster In the data, the proportion of groups corresponding to different colors satisfies the preset fairness constraints. For any subset and any color Define a set In order to be in The color attribute is A set; using Indicates in set The proportion of the h-th group in the middle; For the preset Upper and lower bound vectors and After clustering, the proportion of class h in each cluster is within the preset upper and lower bounds: 。 2. The fair clustering optimization method for large-scale data according to claim 1, characterized in that, The step of iteratively evaluating the local clustering results based on the statistical characteristics of the dataset and clustering quality evaluation indicators to obtain the optimal number of clusters specifically includes the following steps: Based on a preset range of cluster numbers, the number of clusters is adjusted during iterations to evaluate the local clustering results; when the number of clusters makes the clustering quality evaluation index optimal, then that number is determined to be the optimal number of clusters.

3. The fair clustering optimization method for large-scale data according to claim 1, characterized in that, The step of optimizing and iterating the local clustering results, and generating the global clustering results based on the optimized local clustering results of each node by combining and evaluating them through the central server, specifically includes the following steps: Cluster centers were selected based on the statistical characteristics of the dataset for preliminary clustering. The location of the cluster center is updated by calculating the distance between each data point in the dataset and the cluster center; the update rule is to select the location with the smallest sum of distances from each data point to the cluster center as the new cluster center. When the change in cluster centers is lower than a preset threshold, the iteration process ends, and the global clustering result is obtained.

4. The fair clustering optimization method for large-scale data according to claim 1, characterized in that, The central server also includes: The local clustering results are transmitted to the central server via the network. The central server evaluates and integrates the local clustering results of each node based on the location of the cluster center and the distribution of data points among various clusters; by adjusting the cluster boundaries, the local clustering results among different nodes are made the same.

5. A device, characterized in that, The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing the fair clustering optimization method for large-scale data as described in any one of claims 1-4; the processor is used to execute the program instructions stored in the memory to implement fair clustering optimization for large-scale data.

6. A storage medium, characterized in that, The system stores processor-executable program instructions for performing the fair clustering optimization method for large-scale data as described in any one of claims 1-4.

Images