Hierarchical clustering incremental data efficient merging method
By using the point values determined by Re.n and Re.s in the hierarchical clustering incremental data merging to add data and add real and false data, the problem of inefficiency of traditional hierarchical clustering algorithms is solved, efficient merging and privacy protection are achieved, and sensitive information is prevented.
Patent Information
- Application Number
- CN202510702775.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-08-15
AI Technical Summary
Traditional hierarchical clustering algorithms are inefficient when processing dynamic incremental data, especially in IoT perception and real-time financial analysis, computing resources consume huge amounts, and there is a risk of sensitive information leakage. It is difficult for existing privacy protection technologies to effectively avoid cluster centers being imitated.
By adding data to the merged data group, using Re.n and Re.s to determine the starting point value and reproduction point value, the data content is disturbed by the mutual mixing of real and false data, avoiding the leakage of key elements, and achieving efficient data merging and privacy protection.
It improves the efficiency of hierarchical clustering incremental data merging, reduces computing resource consumption, and effectively protects data privacy through data disruption technology to prevent sensitive information leakage.
Smart Images
Figure CN120492957A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data merging, and in particular is a method for efficiently merging incremental data of hierarchical clustering. Background Art
[0002] With the advent of the big data era, hierarchical clustering, as a classic unsupervised learning method, has been widely used in data mining, pattern recognition and other fields.
[0003] It can effectively reveal the inherent hierarchical relationship of data by gradually merging similar data points by constructing a tree structure. However, traditional hierarchical clustering algorithms face efficiency bottlenecks when processing dynamic incremental data, especially in scenarios with continuous data inflow, such as IoT perception and real-time financial analysis. Frequent global reconstruction leads to huge consumption of computing resources, and efficient incremental merging strategies are urgently needed to optimize performance.
[0004] At the same time, data privacy protection has become a core challenge hindering the implementation of data processing technologies. Hierarchical clustering involves the aggregation and feature extraction of large amounts of data. Directly exposing raw data or intermediate clustering results, such as cluster centers and similarity matrices, can lead to the risk of sensitive information leakage. Relevant privacy protection technologies exist, but when it comes to clustering content, preventing cluster centers from being imitated and providing a simple and effective way to protect data is a challenge. This article proposes a solution based on this. Summary of the Invention
[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; To this end, the present invention proposes a hierarchical clustering incremental data efficient merging method, comprising the following steps: Add data to the data cluster after hierarchical clustering and MNL constraint, determine Re.n and Re.s according to the number of data in the data cluster and the memory size occupied, multiply Re.n and Re.s or divide the larger number by the smaller number to determine the selected value, mark the smallest non-zero number in the selected value as the starting point value, and mark the largest number as the recurrence point value; Get the cluster number sequence according to the distance from the data point of the data in the data cluster to the origin in ascending order, add the first supplementary data after the sequence number corresponding to the starting point value, then set the interval value to two, add the second supplementary data after the first supplementary data with the interval value of data; after the interval value is increased by one, insert the third supplementary data after the data corresponding to the number of traversed interval values, and repeat until the interval value is equal to the recurrence point value; Reset the interval value to the starting point value, add supplementary data, then set the interval value to two, add supplementary data until the interval value is equal to the recurrence point value again, loop, traverse all the data in the data group, and complete the addition of supplementary data.
[0006] Furthermore, in Re.n and Re.s, if the value of the units digit of the former is greater than the value of the units digit of the latter, Re.n and Re.s are multiplied to determine the selected value.
[0007] Furthermore, in Re.n and Re.s, if the value in the units digit of the former is not greater than the value in the units digit of the latter, the larger value of Re.n and Re.s is divided by the smaller value to determine the selected value.
[0008] Furthermore, the added supplementary data must ensure that the horizontal and vertical coordinates of its data points are within the horizontal point value range and the vertical point value range respectively.
[0009] Furthermore, the horizontal point value range and the vertical point value range are the range from the minimum value to the maximum value of the horizontal and vertical coordinates corresponding to all data points in the corresponding data cluster.
[0010] Furthermore, the distance between the added supplementary data and the origin is between the distance between the origin and two data points before and after the location where the supplementary data needs to be added.
[0011] Furthermore, the supplementary data is randomly generated.
[0012] Furthermore, the supplementary data is generated according to the aggregation number determined by the total number that needs to be added, and the data corresponding to the number of aggregation number values are selected in sequence and divided into the number of parts corresponding to the aggregation number values. The first part is taken and fused to form the first supplementary data. The second supplementary data is obtained by removing the data in the selected data group and selecting the data corresponding to the aggregation number in sequence again. The same method is used to determine each supplementary data.
[0013] Furthermore, the supplementary data is generated according to the aggregation number determined by the total number that needs to be added, and the data corresponding to the number of aggregation number values are selected in sequence and divided into the number of shares corresponding to the aggregation number values. The first share of the first data, the second share of the second data, ..., the last share of the last data are taken and then merged to form the first supplementary data; the second supplementary data is determined by removing the data in the selected data group and selecting the data corresponding to the aggregation number in sequence again, and using the same method to determine each supplementary data accordingly.
[0014] Furthermore, Re.n is divided by the total number and rounded up to obtain the aggregation number.
[0015] Compared with the prior art, the present invention has the following beneficial effects: This application adds data to the data group after data merging, determines Re.n and Re.s according to the number of data in the data group and the memory size occupied, and determines the selected value by multiplying Re.n and Re.s or dividing the larger number by the smaller number. The smallest non-zero number in the selected value is marked as the starting point value, and the largest number is marked as the recurrence point value; The starting point value and the recurrence point value can be used to add data to the processed merged data, and the specific content of the merged data can be disrupted by mixing true and false data, thereby avoiding the leakage caused by the decryption of key elements, thereby improving the better processing of efficient data merging; this application is simple, effective, and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 The present invention provides a flow chart of the method. DETAILED DESCRIPTION
[0017] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0018] See also Figure 1 ,This application provides a hierarchical clustering incremental data efficient merging method; As embodiment 1 of the present application, the method specifically includes the following steps: Add data to the merged data clusters. Merging here means performing hierarchical clustering and MNL constraints on the data to obtain several data clusters. Determine Re.n and Re.s based on the number of data in the data group and the memory size occupied. Multiply Re.n and Re.s or divide the larger number by the smaller number to determine the selected value. Mark the smallest non-zero number in the selected value as the starting point value, and mark the largest number as the recurrence point value. The cluster number sequence is obtained according to the distance between the data points of the data in the data cluster and the origin from small to large, and then the first supplementary data is added after the sequence number corresponding to the starting point value, and then the interval value is set to two, and the second supplementary data is added after the first supplementary data with an interval corresponding to the interval value; after the interval value is increased by one, the third supplementary data is inserted after the data corresponding to the number of traversed interval values, and repeated until the interval value is equal to the recurrence point value, and the interval value is set to be equal to the starting point value again, and the supplementary data is added, and the interval value is set to be equal to two again, and the supplementary data is added until the interval value is equal to the recurrence point value again, and the cycle is repeated until all the data in the data cluster are traversed and the supplementary data is added.
[0019] Example 2: This example is implemented on the basis of Example 1, except that The specific method is: Step 1: Data preprocessing: Standardize or generalize the data to reduce the granularity of sensitive attributes. Standardization can be done by using cosine normalization, while generalization can be done by replacing numerical data with intervals. Generate MNL constraint pairs: Calculate the bidirectional nearest neighbor relationship between data points and select sample pairs that are mutually nearest neighbors as constraint conditions.
[0020] Step 2: Perform hierarchical clustering and MNL constraint fusion: Initialize each data point as an independent cluster.
[0021] In each merging layer, only clusters that satisfy the MNL constraint are allowed to be merged (for example, the merger of cluster A and cluster B requires that all points in A and all points in B are nearest neighbors to each other).
[0022] Combine must-connect (ML) and must-connect (CL) constraints: for example, ML constraints force certain pairs of samples to be merged, and CL constraints prohibit incompatible clusters from being merged.
[0023] In the merged data group, Gaussian noise with a mean of 0 is added to the numerical data with the help of Gaussian mechanism; Laplace noise is added to the count data with the help of Laplace mechanism; Of course, this embodiment also provides a different data insertion method to achieve data privacy protection, specifically: Step 3: Perform data insertion encryption on the merged data group, where the data group is also called a data cluster; the specific method is: After merging, several clusters are obtained. A data cluster contains several data, each data corresponds to a data point, and the data point is used to indicate the location of the corresponding data. This process has completed the hierarchical clustering of the data and obtained several data clusters. Then, select a data group and obtain the number of data in the data group Re.n and the size of all data in the data group Re.s. The data size refers to the memory size occupied by the data; Then Re.n and Re.s are processed as follows: Get the values of the units digits of both, and mark the values of the units digits of Re.n and Re.s as number one and number two respectively; When the number one is greater than the number two, multiply Re.n and Re.s to get a value, which is marked as the selected value; Get the largest number among the selected values and mark it as the recurrence point value, then get the smallest non-zero number among the selected values and mark it as the starting point value; If the number one is not greater than the number two, then divide the larger of Re.n and Re.s by the smaller number. If the resulting value contains a decimal, it will be automatically rounded to five decimal places, and the last digit will be rounded up. The resulting value is marked as the selected value. The starting point value and the recurrence point value are determined in the same way. Specific examples of the above methods are: When there are 2581 data in the data group, that is, Re.n is 2581, if the corresponding Re.s is 382, then the number one is 1 and the number two is 2, so the number two is larger. At this time, 2581 / 382=6.75654 is needed; according to the above description, take five decimal places; that is, the selected value is 6.75654, then the recurrence point value is 7, and the starting point value is 4; If Re.n is 2585 and the corresponding Re.s is 383, then number one is 1 and number two is 2, so number two is larger. In this case, we need to convert 2581 / 382=6.74935. According to the above description, we take five decimal places, that is, we choose the value 6.74935. At this time, the recurrence point value is 9 and the starting point value is 3. After arranging the data in the data cluster according to the order of the corresponding original data points, the order here refers to the distance of the corresponding data points from the origin; Then sort them in ascending order of distance from the origin to get the cluster number sequence, obtain the data corresponding to the starting point value order in the cluster number sequence, and add the corresponding supplementary data after it. The specific adding method is: First, obtain all the data points, then obtain their minimum and maximum values on the horizontal and vertical coordinates respectively, and mark the ranges from the two minimum values to the maximum value as the horizontal point value range and the vertical point value range respectively; Then, on the basis of ensuring that the horizontal and vertical coordinate values are respectively within the horizontal point value range and the vertical point value range, the distance between the corresponding data point and the origin is between the distance between the two data points before and after the location where the supplementary data needs to be added and the origin; Randomly generate data points according to the above requirements; Then the supplementary data is placed at the position of the corresponding data point; the supplementary data here is randomly generated; After adding the first supplementary data, add the second supplementary data after the two data after the first supplementary data in the order of the group number sequence. The addition method is the same. Then add the third supplementary data after the three data after the second supplementary data. Repeat this rule until the number of supplementary data reaches the number of recurrence points. Then, add the first supplementary data according to the number of starting point values corresponding to the interval, and then add it again according to the above method, and repeat until all the data in the cluster number sequence are traversed; if the supplementary data to be added has passed all the numbers in the cluster number sequence that need to be started again, the process will stop and no additional data will be added.
[0024] In the specific implementation process, it is assumed that there are 8 data from D1, D2, D3, D4, D5, D6, ...., D5; at the same time, the recurrence point value is 7 and the starting point value is 4; First, add the first supplementary data between D4 and D5, then add the second supplementary data between D6 and D7, then add the third supplementary data between D9 and D10, then insert the fourth data between D13 and D14, .... continue until the seventh supplementary data is added between D31 and D32, then insert the eighth supplementary data between D35 and D36, then insert the eighth supplementary data between D37 and D38, and so on; Of course, as the third embodiment of the present application, this embodiment is implemented on the basis of the first embodiment, except that the supplementary data is generated in the following manner: Get the total number of supplementary data to be added, then divide Re.n by the total number, round the value to get the integer and mark it as the aggregate number; Starting from the first data of the data group, continuously select data of the corresponding number of aggregations, then divide each selected data into the corresponding number of aggregations, combine the first part of each data to form a new added data, thus obtaining the first added data, and then sequentially select data of the corresponding number of aggregations, perform the same process, obtain the second added data, and so on, obtain all the added data; As the fourth embodiment of the present application, this embodiment is implemented on the basis of the third embodiment, except that the combination method after each selected data is evenly divided into the corresponding aggregated number of parts is different in this embodiment. The combination method in this embodiment is: Combine the first copy of the first data, the second copy of the second data, ..., and the last copy of the last data to form a new added data, thereby obtaining the first added data; the rest of the methods are the same; Of course, based on the above embodiments, if this application adopts the method of embodiments two to four to add data, it is necessary to record the Re.n and Re.s of the original data, and also store a method of adding and deleting data in the opposite way of the above method, which is used to delete the added data, and this method is stored in an external device, that is, a USB flash drive, which is a peripheral that needs to be authenticated before it can be used, so that the data is not easily leaked.
[0025] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.
Claims
1. A hierarchical clustering incremental data efficient merging method, characterized by: The steps include: Add data to the merged data group, determine Re.n and Re.s according to the number of data in the data group and the memory size occupied, multiply Re.n and Re.s or divide the larger number by the smaller number to determine the selected value, mark the smallest non-zero number in the selected value as the starting point value, and mark the largest number as the recurrence point value; Get the cluster number sequence according to the distance from the data point of the data in the data cluster to the origin in ascending order, add the first supplementary data after the sequence number corresponding to the starting point value, then set the interval value to two, add the second supplementary data after the first supplementary data with the interval value of data; after the interval value is increased by one, insert the third supplementary data after the data corresponding to the number of traversed interval values, and repeat until the interval value is equal to the recurrence point value; Reset the interval value to the starting point value, add supplementary data, then set the interval value to two, add supplementary data until the interval value is equal to the recurrence point value again, loop, traverse all the data in the data group, and complete the addition of supplementary data.
2. The hierarchical clustering incremental data efficient merging method according to claim 1, characterized in that: In Re.n and Re.s, if the value in the units digit of the former is greater than the value in the units digit of the latter, then multiply Re.n and Re.s to determine the selected value.
3. The hierarchical clustering incremental data efficient merging method according to claim 1, characterized in that: In Re.n and Re.s, if the value in the units digit of the former is not greater than the value in the units digit of the latter, divide the larger value of Re.n and Re.s by the smaller value to determine the selected value.
4. The hierarchical clustering incremental data efficient merging method according to claim 1, characterized in that: The added supplementary data must ensure that the horizontal and vertical coordinates of its data points are within the horizontal point value range and the vertical point value range respectively.
5. The hierarchical clustering incremental data efficient merging method according to claim 1, characterized in that: The horizontal point value range and the vertical point value range are the minimum to maximum value ranges of the horizontal and vertical coordinates corresponding to all data points in the corresponding data group.
6. The hierarchical clustering incremental data efficient merging method according to claim 1, characterized in that: The distance between the added supplementary data and the origin is between the distance between the two data points before and after the location where the supplementary data needs to be added and the origin.
7. The hierarchical clustering incremental data efficient merging method according to claim 1, characterized in that: Supplementary data are randomly generated.
8. The hierarchical clustering incremental data efficient merging method according to claim 1, characterized in that: The supplementary data is generated according to the aggregation number determined by the total number that needs to be added. The data corresponding to the number of aggregation number values are selected in sequence and divided into the number of parts corresponding to the aggregation number values. The first part is taken and fused to form the first supplementary data. The second supplementary data is generated by removing the data in the selected data group and selecting the data corresponding to the aggregation number in sequence again. The same method is used to determine each supplementary data.
9. The hierarchical clustering incremental data efficient merging method according to claim 8, characterized in that: The supplementary data is generated according to the aggregation number determined by the total number of data to be added. The data corresponding to the number of aggregation number values are selected in sequence and divided into the number of shares corresponding to the aggregation number values. The first share of the first data, the second share of the second data, ..., the last share of the last data are taken and then merged to form the first supplementary data; the second supplementary data is determined by removing the data in the selected data group and selecting the data corresponding to the aggregation number in sequence again. The same method is used to determine each supplementary data in this way.
10. The hierarchical clustering incremental data efficient merging method according to claim 1, characterized in that: Divide Re.n by the total number and round it up to get the aggregate number.